* [PATCH 01/12] powerpc: Disable HFSCR:TM if TM not supported
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Otherwise KVM guests might mess with it even when told not
to causing bad thing interrupts in the host
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/kernel/setup_64.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index 9cfaa8b..b372b23 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -236,6 +236,16 @@ static void cpu_ready_for_interrupts(void)
mtspr(SPRN_LPCR, lpcr | LPCR_AIL_3);
}
+ /*
+ * Fixup HFSCR:TM based on CPU features. The bit is set by our
+ * early asm init because at that point we haven't updated our
+ * CPU features from firmware and device-tree. Here we have,
+ * so let's do it
+ */
+ if (early_cpu_has_feature(CPU_FTR_HVMODE) &&
+ !early_cpu_has_feature(CPU_FTR_TM_COMP))
+ mtspr(SPRN_HFSCR, mfspr(SPRN_HFSCR) & ~HFSCR_TM);
+
/* Set IR and DR in PACA MSR */
get_paca()->kernel_msr = MSR_KERNEL;
}
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 01/12] powerpc: Disable HFSCR:TM if TM not supported
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Otherwise KVM guests might mess with it even when told not
to causing bad thing interrupts in the host
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/kernel/setup_64.c | 10 ++++++++++
1 file changed, 10 insertions(+)
diff --git a/arch/powerpc/kernel/setup_64.c b/arch/powerpc/kernel/setup_64.c
index 9cfaa8b..b372b23 100644
--- a/arch/powerpc/kernel/setup_64.c
+++ b/arch/powerpc/kernel/setup_64.c
@@ -236,6 +236,16 @@ static void cpu_ready_for_interrupts(void)
mtspr(SPRN_LPCR, lpcr | LPCR_AIL_3);
}
+ /*
+ * Fixup HFSCR:TM based on CPU features. The bit is set by our
+ * early asm init because at that point we haven't updated our
+ * CPU features from firmware and device-tree. Here we have,
+ * so let's do it
+ */
+ if (early_cpu_has_feature(CPU_FTR_HVMODE) &&
+ !early_cpu_has_feature(CPU_FTR_TM_COMP))
+ mtspr(SPRN_HFSCR, mfspr(SPRN_HFSCR) & ~HFSCR_TM);
+
/* Set IR and DR in PACA MSR */
get_paca()->kernel_msr = MSR_KERNEL;
}
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 02/12] powerpc: Sync opal-api.h
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/opal-api.h | 302 ++++++++++++++++++++-----
arch/powerpc/include/asm/opal.h | 36 +++
arch/powerpc/platforms/powernv/opal-wrappers.S | 15 ++
3 files changed, 302 insertions(+), 51 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index a0aa285..a6053a6 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -40,6 +40,16 @@
#define OPAL_I2C_ARBT_LOST -22
#define OPAL_I2C_NACK_RCVD -23
#define OPAL_I2C_STOP_ERR -24
+#define OPAL_XSCOM_BUSY OPAL_BUSY
+#define OPAL_XSCOM_CHIPLET_OFF OPAL_WRONG_STATE
+#define OPAL_XSCOM_PARTIAL_GOOD -25
+#define OPAL_XSCOM_ADDR_ERROR -26
+#define OPAL_XSCOM_CLOCK_ERROR -27
+#define OPAL_XSCOM_PARITY_ERROR -28
+#define OPAL_XSCOM_TIMEOUT -29
+#define OPAL_XSCOM_CTR_OFFLINED -30
+#define OPAL_XIVE_PROVISIONING -31
+#define OPAL_XIVE_FREE_ACTIVE -32
/* API Tokens (in r0) */
#define OPAL_INVALID_CALL -1
@@ -168,7 +178,24 @@
#define OPAL_INT_SET_MFRR 125
#define OPAL_PCI_TCE_KILL 126
#define OPAL_NMMU_SET_PTCR 127
-#define OPAL_LAST 127
+#define OPAL_XIVE_RESET 128
+#define OPAL_XIVE_GET_IRQ_INFO 129
+#define OPAL_XIVE_GET_IRQ_CONFIG 130
+#define OPAL_XIVE_SET_IRQ_CONFIG 131
+#define OPAL_XIVE_GET_QUEUE_INFO 132
+#define OPAL_XIVE_SET_QUEUE_INFO 133
+#define OPAL_XIVE_DONATE_PAGE 134
+#define OPAL_XIVE_ALLOCATE_VP_BLOCK 135
+#define OPAL_XIVE_FREE_VP_BLOCK 136
+#define OPAL_XIVE_GET_VP_INFO 137
+#define OPAL_XIVE_SET_VP_INFO 138
+#define OPAL_XIVE_ALLOCATE_IRQ 139
+#define OPAL_XIVE_FREE_IRQ 140
+#define OPAL_XIVE_SYNC 141
+#define OPAL_XIVE_DUMP 142
+#define OPAL_XIVE_RESERVED3 143
+#define OPAL_XIVE_RESERVED4 144
+#define OPAL_LAST 144
/* Device tree flags */
@@ -176,14 +203,34 @@
* Flags set in power-mgmt nodes in device tree describing
* idle states that are supported in the platform.
*/
-
-#define OPAL_PM_TIMEBASE_STOP 0x00000002
-#define OPAL_PM_LOSE_HYP_CONTEXT 0x00002000
+#define OPAL_PM_DEC_STOP 0x00000001 /* Decrementer would stop */
+#define OPAL_PM_TIMEBASE_STOP 0x00000002 /* Needs timebase restore */
+#define OPAL_PM_LOSE_USER_CONTEXT 0x00001000 /* Restore GPRs like nap */
+#define OPAL_PM_LOSE_HYP_CONTEXT 0x00002000 /* Restore hypervisor
+ resource from PACA pointer */
#define OPAL_PM_LOSE_FULL_CONTEXT 0x00004000
#define OPAL_PM_NAP_ENABLED 0x00010000
#define OPAL_PM_SLEEP_ENABLED 0x00020000
#define OPAL_PM_WINKLE_ENABLED 0x00040000
#define OPAL_PM_SLEEP_ENABLED_ER1 0x00080000 /* with workaround */
+#define OPAL_USE_PMICR 0x00800000 /* Use SPR PMICR instruction */
+
+#define OPAL_PM_FASTSLEEP_PMICR 0x0000002000000000UL
+#define OPAL_PM_DEEPSLEEP_PMICR 0x0000003000000000UL
+#define OPAL_PM_SLEEP_PMICR_MASK 0x0000003000000000UL
+
+#define OPAL_PM_FASTWINKLE_PMICR 0x0000000000200000UL
+#define OPAL_PM_DEEPWINKLE_PMICR 0x0000000000300000UL
+#define OPAL_PM_WINKLE_PMICR_MASK 0x0000000000300000UL
+
+
+/*
+ * Flags for stop states. Use 2 bits to distinguish between
+ * deep and fast states. Deep states result in full context
+ * loss thereby requiring slw to partially restore state
+ * whereas fast state can function without the presence of
+ * slw.
+ */
#define OPAL_PM_STOP_INST_FAST 0x00100000
#define OPAL_PM_STOP_INST_DEEP 0x00200000
@@ -197,6 +244,10 @@
#ifndef __ASSEMBLY__
/* Other enums */
+enum OpalVendorApiTokens {
+ OPAL_START_VENDOR_API_RANGE = 1000, OPAL_END_VENDOR_API_RANGE = 1999
+};
+
enum OpalFreezeState {
OPAL_EEH_STOPPED_NOT_FROZEN = 0,
OPAL_EEH_STOPPED_MMIO_FREEZE = 1,
@@ -264,12 +315,36 @@ enum OpalErrinjectFunc {
OPAL_ERR_INJECT_FUNC_IOA_DMA_WR_TARGET = 19,
};
+enum OpalShpcAction {
+ OPAL_SHPC_GET_LINK_STATE = 0,
+ OPAL_SHPC_GET_SLOT_STATE = 1
+};
+
+enum OpalShpcLinkState {
+ OPAL_SHPC_LINK_DOWN = 0,
+ OPAL_SHPC_LINK_UP_x1 = 1,
+ OPAL_SHPC_LINK_UP_x2 = 2,
+ OPAL_SHPC_LINK_UP_x4 = 4,
+ OPAL_SHPC_LINK_UP_x8 = 8,
+ OPAL_SHPC_LINK_UP_x16 = 16,
+ OPAL_SHPC_LINK_UP_x32 = 32
+};
+
enum OpalMmioWindowType {
OPAL_M32_WINDOW_TYPE = 1,
OPAL_M64_WINDOW_TYPE = 2,
OPAL_IO_WINDOW_TYPE = 3
};
+enum OpalShpcSlotState {
+ OPAL_SHPC_DEV_NOT_PRESENT = 0,
+ OPAL_SHPC_DEV_PRESENT = 1
+};
+enum OpalShpcPowerState {
+ OPAL_SHPC_POWER_OFF = 0,
+ OPAL_SHPC_POWER_ON = 1
+};
+
enum OpalExceptionHandler {
OPAL_MACHINE_CHECK_HANDLER = 1,
OPAL_HYPERVISOR_MAINTENANCE_HANDLER = 2,
@@ -361,6 +436,11 @@ enum OpalPciResetState {
OPAL_ASSERT_RESET = 1
};
+enum OpalPciMaskAction {
+ OPAL_UNMASK_ERROR_TYPE = 0,
+ OPAL_MASK_ERROR_TYPE = 1
+};
+
enum OpalPciSlotPresence {
OPAL_PCI_SLOT_EMPTY = 0,
OPAL_PCI_SLOT_PRESENT = 1
@@ -385,6 +465,18 @@ enum OpalSlotLedState {
OPAL_SLOT_LED_STATE_ON = 1 /* LED is ON */
};
+enum OpalEpowStatus {
+ OPAL_EPOW_NONE = 0,
+ OPAL_EPOW_UPS = 1,
+ OPAL_EPOW_OVER_AMBIENT_TEMP = 2,
+ OPAL_EPOW_OVER_INTERNAL_TEMP = 3
+};
+
+enum OpalCheckTokenStatus {
+ OPAL_TOKEN_ABSENT = 0,
+ OPAL_TOKEN_PRESENT = 1
+};
+
/*
* Address cycle types for LPC accesses. These also correspond
* to the content of the first cell of the "reg" property for
@@ -434,6 +526,46 @@ struct opal_ipmi_msg {
uint8_t data[];
};
+/*
+ * EPOW status sharing (OPAL and the host)
+ *
+ * The host will pass on OPAL, a buffer of length OPAL_SYSEPOW_MAX
+ * with individual elements being 16 bits wide to fetch the system
+ * wide EPOW status. Each element in the buffer will contain the
+ * EPOW status in it's bit representation for a particular EPOW sub
+ * class as defined here. So multiple detailed EPOW status bits
+ * specific for any sub class can be represented in a single buffer
+ * element as it's bit representation.
+ */
+
+/* System EPOW type */
+enum OpalSysEpow {
+ OPAL_SYSEPOW_POWER = 0, /* Power EPOW */
+ OPAL_SYSEPOW_TEMP = 1, /* Temperature EPOW */
+ OPAL_SYSEPOW_COOLING = 2, /* Cooling EPOW */
+ OPAL_SYSEPOW_MAX = 3, /* Max EPOW categories */
+};
+
+/* Power EPOW */
+enum OpalSysPower {
+ OPAL_SYSPOWER_UPS = 0x0001, /* System on UPS power */
+ OPAL_SYSPOWER_CHNG = 0x0002, /* System power configuration change */
+ OPAL_SYSPOWER_FAIL = 0x0004, /* System impending power failure */
+ OPAL_SYSPOWER_INCL = 0x0008, /* System incomplete power */
+};
+
+/* Temperature EPOW */
+enum OpalSysTemp {
+ OPAL_SYSTEMP_AMB = 0x0001, /* System over ambient temperature */
+ OPAL_SYSTEMP_INT = 0x0002, /* System over internal temperature */
+ OPAL_SYSTEMP_HMD = 0x0004, /* System over ambient humidity */
+};
+
+/* Cooling EPOW */
+enum OpalSysCooling {
+ OPAL_SYSCOOL_INSF = 0x0001, /* System insufficient cooling */
+};
+
/* FSP memory errors handling */
enum OpalMemErr_Version {
OpalMemErr_V1 = 1,
@@ -456,6 +588,11 @@ enum OpalMemErr_DynErrType {
OPAL_MEM_DYNAMIC_DEALLOC = 0,
};
+/* OpalMemoryErrorData->flags */
+#define OPAL_MEM_CORRECTED_ERROR 0x0001
+#define OPAL_MEM_THRESHOLD_EXCEEDED 0x0002
+#define OPAL_MEM_ACK_REQUIRED 0x8000
+
struct OpalMemoryErrorData {
enum OpalMemErr_Version version:8; /* 0x00 */
enum OpalMemErrType type:8; /* 0x01 */
@@ -518,6 +655,7 @@ enum OpalHMI_XstopType {
CHECKSTOP_TYPE_UNKNOWN = 0,
CHECKSTOP_TYPE_CORE = 1,
CHECKSTOP_TYPE_NX = 2,
+ CHECKSTOP_TYPE_NPU = 3
};
enum OpalHMI_CoreXstopReason {
@@ -577,10 +715,10 @@ struct OpalHMIEvent {
struct {
uint8_t xstop_type; /* enum OpalHMI_XstopType */
uint8_t reserved_1[3];
- __be32 xstop_reason;
+ __be32 xstop_reason;
union {
- __be32 pir; /* for CHECKSTOP_TYPE_CORE */
- __be32 chip_id; /* for CHECKSTOP_TYPE_NX */
+ __be32 pir; /* for CHECKSTOP_TYPE_CORE */
+ __be32 chip_id; /* for CHECKSTOP_TYPE_NX */
} u;
} xstop_error;
} u;
@@ -645,7 +783,8 @@ enum {
enum {
OPAL_PHB_ERROR_DATA_TYPE_P7IOC = 1,
- OPAL_PHB_ERROR_DATA_TYPE_PHB3 = 2
+ OPAL_PHB_ERROR_DATA_TYPE_PHB3 = 2,
+ OPAL_PHB_ERROR_DATA_TYPE_PHB4 = 3
};
enum {
@@ -780,6 +919,11 @@ struct OpalIoPhb3ErrorData {
__be64 pestB[OPAL_PHB3_NUM_PEST_REGS];
};
+struct OpalIoPhb4ErrorData {
+ struct OpalIoPhbErrorCommon common;
+ // FIXME add phb4 specific stuff
+};
+
enum {
OPAL_REINIT_CPUS_HILE_BE = (1 << 0),
OPAL_REINIT_CPUS_HILE_LE = (1 << 1),
@@ -797,6 +941,7 @@ enum opal_prd_msg_type {
OPAL_PRD_MSG_TYPE_ATTN_ACK, /* HBRT --> OPAL */
OPAL_PRD_MSG_TYPE_OCC_ERROR, /* HBRT <-- OPAL */
OPAL_PRD_MSG_TYPE_OCC_RESET, /* HBRT <-- OPAL */
+ OPAL_PRD_MSG_TYPE_OCC_RESET_NOTIFY, /* HBRT --> OPAL */
};
struct opal_prd_msg_header {
@@ -805,13 +950,50 @@ struct opal_prd_msg_header {
__be16 size;
};
-struct opal_prd_msg;
-
-#define OCC_RESET 0
-#define OCC_LOAD 1
-#define OCC_THROTTLE 2
-#define OCC_MAX_THROTTLE_STATUS 5
+struct opal_prd_msg {
+ struct opal_prd_msg_header hdr;
+ __be32 token;
+ union {
+ struct {
+ __be64 version;
+ __be64 ipoll;
+ } init;
+ struct {
+ __be64 proc;
+ __be64 ipoll_status;
+ __be64 ipoll_mask;
+ } attn;
+ struct {
+ __be64 proc;
+ __be64 ipoll_ack;
+ } attn_ack;
+ struct {
+ __be64 chip;
+ } occ_error;
+ struct {
+ __be64 chip;
+ } occ_reset;
+ };
+};
+#define OCC_RESET 0
+#define OCC_LOAD 1
+#define OCC_THROTTLE 2
+#define OCC_MAX_THROTTLE_STATUS 5
+/*
+ * struct opal_occ_msg:
+ * type: OCC_RESET, OCC_LOAD, OCC_THROTTLE
+ * chip: chip id
+ * throttle status: indicates the reason why OCC may have limited
+ * the max Pstate of the chip.
+ * 0x00 = No throttle
+ * 0x01 = Power Cap
+ * 0x02 = Processor Over Temperature
+ * 0x03 = Power Supply Failure (currently not used)
+ * 0x04 = Over current (currently not used)
+ * 0x05 = OCC Reset (not reliable as some failures will not allow for
+ * OCC to update throttle status)
+ */
struct opal_occ_msg {
__be64 type;
__be64 chip;
@@ -857,6 +1039,10 @@ enum {
OPAL_PHB_CAPI_MODE_DMA = 4,
};
+/* CAPI feature flags (in device-tree) */
+#define OPAL_PHB_CAPI_FLAG_SNOOP_CONTROL 0x00000001
+#define OPAL_PHB_CAPI_FLAG_REVERT_TO_PCIE 0x00000002
+
/* OPAL I2C request */
struct opal_i2c_request {
uint8_t type;
@@ -875,59 +1061,73 @@ struct opal_i2c_request {
__be64 buffer_ra; /* Buffer real address */
};
-/*
- * EPOW status sharing (OPAL and the host)
- *
- * The host will pass on OPAL, a buffer of length OPAL_SYSEPOW_MAX
- * with individual elements being 16 bits wide to fetch the system
- * wide EPOW status. Each element in the buffer will contain the
- * EPOW status in it's bit representation for a particular EPOW sub
- * class as defined here. So multiple detailed EPOW status bits
- * specific for any sub class can be represented in a single buffer
- * element as it's bit representation.
+/* Argument to OPAL_CEC_REBOOT2() */
+enum {
+ OPAL_REBOOT_NORMAL = 0,
+ OPAL_REBOOT_PLATFORM_ERROR,
+};
+
+/* Argument to OPAL_PCI_TCE_KILL */
+enum {
+ OPAL_PCI_TCE_KILL_PAGES,
+ OPAL_PCI_TCE_KILL_PE,
+ OPAL_PCI_TCE_KILL_ALL,
+};
+
+/* The xive operation mode indicates the active "API" and
+ * corresponds to the "mode" parameter of the opal_xive_reset()
+ * call
*/
+enum {
+ OPAL_XIVE_MODE_EMU = 0,
+ OPAL_XIVE_MODE_EXPL = 1,
+};
-/* System EPOW type */
-enum OpalSysEpow {
- OPAL_SYSEPOW_POWER = 0, /* Power EPOW */
- OPAL_SYSEPOW_TEMP = 1, /* Temperature EPOW */
- OPAL_SYSEPOW_COOLING = 2, /* Cooling EPOW */
- OPAL_SYSEPOW_MAX = 3, /* Max EPOW categories */
+/* Flags for OPAL_XIVE_GET_IRQ_INFO */
+enum {
+ OPAL_XIVE_IRQ_TRIGGER_PAGE = 0x00000001,
+ OPAL_XIVE_IRQ_STORE_EOI = 0x00000002,
+ OPAL_XIVE_IRQ_LSI = 0x00000004,
+ OPAL_XIVE_IRQ_SHIFT_BUG = 0x00000008,
+ OPAL_XIVE_IRQ_MASK_VIA_FW = 0x00000010,
+ OPAL_XIVE_IRQ_EOI_VIA_FW = 0x00000020,
};
-/* Power EPOW */
-enum OpalSysPower {
- OPAL_SYSPOWER_UPS = 0x0001, /* System on UPS power */
- OPAL_SYSPOWER_CHNG = 0x0002, /* System power config change */
- OPAL_SYSPOWER_FAIL = 0x0004, /* System impending power failure */
- OPAL_SYSPOWER_INCL = 0x0008, /* System incomplete power */
+/* Flags for OPAL_XIVE_GET/SET_QUEUE_INFO */
+enum {
+ OPAL_XIVE_EQ_ENABLED = 0x00000001,
+ OPAL_XIVE_EQ_ALWAYS_NOTIFY = 0x00000002,
+ OPAL_XIVE_EQ_ESCALATE = 0x00000004,
};
-/* Temperature EPOW */
-enum OpalSysTemp {
- OPAL_SYSTEMP_AMB = 0x0001, /* System over ambient temperature */
- OPAL_SYSTEMP_INT = 0x0002, /* System over internal temperature */
- OPAL_SYSTEMP_HMD = 0x0004, /* System over ambient humidity */
+/* Flags for OPAL_XIVE_GET/SET_VP_INFO */
+enum {
+ OPAL_XIVE_VP_ENABLED = 0x00000001,
};
-/* Cooling EPOW */
-enum OpalSysCooling {
- OPAL_SYSCOOL_INSF = 0x0001, /* System insufficient cooling */
+/* "Any chip" replacement for chip ID for allocation functions */
+enum {
+ OPAL_XIVE_ANY_CHIP = 0xffffffff,
};
-/* Argument to OPAL_CEC_REBOOT2() */
+/* Xive sync options */
enum {
- OPAL_REBOOT_NORMAL = 0,
- OPAL_REBOOT_PLATFORM_ERROR = 1,
+ /* This bits are cumulative, arg is a girq */
+ XIVE_SYNC_EAS = 0x00000001, /* Sync irq source */
+ XIVE_SYNC_QUEUE = 0x00000002, /* Sync irq target */
};
-/* Argument to OPAL_PCI_TCE_KILL */
+/* Dump options */
enum {
- OPAL_PCI_TCE_KILL_PAGES,
- OPAL_PCI_TCE_KILL_PE,
- OPAL_PCI_TCE_KILL_ALL,
+ XIVE_DUMP_TM_HYP = 0,
+ XIVE_DUMP_TM_POOL = 1,
+ XIVE_DUMP_TM_OS = 2,
+ XIVE_DUMP_TM_USER = 3,
+ XIVE_DUMP_VP = 4,
+ XIVE_DUMP_EMU_STATE = 5,
};
#endif /* __ASSEMBLY__ */
#endif /* __OPAL_API_H */
+
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 1ff03a6..cb7d607 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -226,6 +226,42 @@ int64_t opal_pci_tce_kill(uint64_t phb_id, uint32_t kill_type,
uint32_t pe_num, uint32_t tce_size,
uint64_t dma_addr, uint32_t npages);
int64_t opal_nmmu_set_ptcr(uint64_t chip_id, uint64_t ptcr);
+int64_t opal_xive_reset(uint64_t version);
+int64_t opal_xive_get_irq_info(uint32_t girq,
+ __be64 *out_flags,
+ __be64 *out_eoi_page,
+ __be64 *out_trig_page,
+ __be32 *out_esb_shift,
+ __be32 *out_src_chip);
+int64_t opal_xive_get_irq_config(uint32_t girq, __be64 *out_vp,
+ uint8_t *out_prio, __be32 *out_lirq);
+int64_t opal_xive_set_irq_config(uint32_t girq, uint64_t vp, uint8_t prio,
+ uint32_t lirq);
+int64_t opal_xive_get_queue_info(uint64_t vp, uint32_t prio,
+ __be64 *out_qpage,
+ __be64 *out_qsize,
+ __be64 *out_qeoi_page,
+ __be32 *out_escalate_irq,
+ __be64 *out_qflags);
+int64_t opal_xive_set_queue_info(uint64_t vp, uint32_t prio,
+ uint64_t qpage,
+ uint64_t qsize,
+ uint64_t qflags);
+int64_t opal_xive_donate_page(uint32_t chip_id, uint64_t addr);
+int64_t opal_xive_alloc_vp_block(uint32_t alloc_order);
+int64_t opal_xive_free_vp_block(uint64_t vp);
+int64_t opal_xive_get_vp_info(uint64_t vp,
+ __be64 *out_flags,
+ __be64 *out_cam_value,
+ __be64 *out_report_cl_pair,
+ __be32 *out_chip_id);
+int64_t opal_xive_set_vp_info(uint64_t vp,
+ uint64_t flags,
+ uint64_t report_cl_pair);
+int64_t opal_xive_allocate_irq(uint32_t chip_id);
+int64_t opal_xive_free_irq(uint32_t girq);
+int64_t opal_xive_sync(uint32_t type, uint32_t id);
+int64_t opal_xive_dump(uint32_t type, uint32_t id);
/* Internal functions */
extern int early_init_dt_scan_opal(unsigned long node, const char *uname,
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index da8a0f7..085605a 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -301,3 +301,18 @@ OPAL_CALL(opal_int_eoi, OPAL_INT_EOI);
OPAL_CALL(opal_int_set_mfrr, OPAL_INT_SET_MFRR);
OPAL_CALL(opal_pci_tce_kill, OPAL_PCI_TCE_KILL);
OPAL_CALL(opal_nmmu_set_ptcr, OPAL_NMMU_SET_PTCR);
+OPAL_CALL(opal_xive_reset, OPAL_XIVE_RESET);
+OPAL_CALL(opal_xive_get_irq_info, OPAL_XIVE_GET_IRQ_INFO);
+OPAL_CALL(opal_xive_get_irq_config, OPAL_XIVE_GET_IRQ_CONFIG);
+OPAL_CALL(opal_xive_set_irq_config, OPAL_XIVE_SET_IRQ_CONFIG);
+OPAL_CALL(opal_xive_get_queue_info, OPAL_XIVE_GET_QUEUE_INFO);
+OPAL_CALL(opal_xive_set_queue_info, OPAL_XIVE_SET_QUEUE_INFO);
+OPAL_CALL(opal_xive_donate_page, OPAL_XIVE_DONATE_PAGE);
+OPAL_CALL(opal_xive_alloc_vp_block, OPAL_XIVE_ALLOCATE_VP_BLOCK);
+OPAL_CALL(opal_xive_free_vp_block, OPAL_XIVE_FREE_VP_BLOCK);
+OPAL_CALL(opal_xive_allocate_irq, OPAL_XIVE_ALLOCATE_IRQ);
+OPAL_CALL(opal_xive_free_irq, OPAL_XIVE_FREE_IRQ);
+OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
+OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
+OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
+OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 02/12] powerpc: Sync opal-api.h
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/opal-api.h | 302 ++++++++++++++++++++-----
arch/powerpc/include/asm/opal.h | 36 +++
arch/powerpc/platforms/powernv/opal-wrappers.S | 15 ++
3 files changed, 302 insertions(+), 51 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index a0aa285..a6053a6 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -40,6 +40,16 @@
#define OPAL_I2C_ARBT_LOST -22
#define OPAL_I2C_NACK_RCVD -23
#define OPAL_I2C_STOP_ERR -24
+#define OPAL_XSCOM_BUSY OPAL_BUSY
+#define OPAL_XSCOM_CHIPLET_OFF OPAL_WRONG_STATE
+#define OPAL_XSCOM_PARTIAL_GOOD -25
+#define OPAL_XSCOM_ADDR_ERROR -26
+#define OPAL_XSCOM_CLOCK_ERROR -27
+#define OPAL_XSCOM_PARITY_ERROR -28
+#define OPAL_XSCOM_TIMEOUT -29
+#define OPAL_XSCOM_CTR_OFFLINED -30
+#define OPAL_XIVE_PROVISIONING -31
+#define OPAL_XIVE_FREE_ACTIVE -32
/* API Tokens (in r0) */
#define OPAL_INVALID_CALL -1
@@ -168,7 +178,24 @@
#define OPAL_INT_SET_MFRR 125
#define OPAL_PCI_TCE_KILL 126
#define OPAL_NMMU_SET_PTCR 127
-#define OPAL_LAST 127
+#define OPAL_XIVE_RESET 128
+#define OPAL_XIVE_GET_IRQ_INFO 129
+#define OPAL_XIVE_GET_IRQ_CONFIG 130
+#define OPAL_XIVE_SET_IRQ_CONFIG 131
+#define OPAL_XIVE_GET_QUEUE_INFO 132
+#define OPAL_XIVE_SET_QUEUE_INFO 133
+#define OPAL_XIVE_DONATE_PAGE 134
+#define OPAL_XIVE_ALLOCATE_VP_BLOCK 135
+#define OPAL_XIVE_FREE_VP_BLOCK 136
+#define OPAL_XIVE_GET_VP_INFO 137
+#define OPAL_XIVE_SET_VP_INFO 138
+#define OPAL_XIVE_ALLOCATE_IRQ 139
+#define OPAL_XIVE_FREE_IRQ 140
+#define OPAL_XIVE_SYNC 141
+#define OPAL_XIVE_DUMP 142
+#define OPAL_XIVE_RESERVED3 143
+#define OPAL_XIVE_RESERVED4 144
+#define OPAL_LAST 144
/* Device tree flags */
@@ -176,14 +203,34 @@
* Flags set in power-mgmt nodes in device tree describing
* idle states that are supported in the platform.
*/
-
-#define OPAL_PM_TIMEBASE_STOP 0x00000002
-#define OPAL_PM_LOSE_HYP_CONTEXT 0x00002000
+#define OPAL_PM_DEC_STOP 0x00000001 /* Decrementer would stop */
+#define OPAL_PM_TIMEBASE_STOP 0x00000002 /* Needs timebase restore */
+#define OPAL_PM_LOSE_USER_CONTEXT 0x00001000 /* Restore GPRs like nap */
+#define OPAL_PM_LOSE_HYP_CONTEXT 0x00002000 /* Restore hypervisor
+ resource from PACA pointer */
#define OPAL_PM_LOSE_FULL_CONTEXT 0x00004000
#define OPAL_PM_NAP_ENABLED 0x00010000
#define OPAL_PM_SLEEP_ENABLED 0x00020000
#define OPAL_PM_WINKLE_ENABLED 0x00040000
#define OPAL_PM_SLEEP_ENABLED_ER1 0x00080000 /* with workaround */
+#define OPAL_USE_PMICR 0x00800000 /* Use SPR PMICR instruction */
+
+#define OPAL_PM_FASTSLEEP_PMICR 0x0000002000000000UL
+#define OPAL_PM_DEEPSLEEP_PMICR 0x0000003000000000UL
+#define OPAL_PM_SLEEP_PMICR_MASK 0x0000003000000000UL
+
+#define OPAL_PM_FASTWINKLE_PMICR 0x0000000000200000UL
+#define OPAL_PM_DEEPWINKLE_PMICR 0x0000000000300000UL
+#define OPAL_PM_WINKLE_PMICR_MASK 0x0000000000300000UL
+
+
+/*
+ * Flags for stop states. Use 2 bits to distinguish between
+ * deep and fast states. Deep states result in full context
+ * loss thereby requiring slw to partially restore state
+ * whereas fast state can function without the presence of
+ * slw.
+ */
#define OPAL_PM_STOP_INST_FAST 0x00100000
#define OPAL_PM_STOP_INST_DEEP 0x00200000
@@ -197,6 +244,10 @@
#ifndef __ASSEMBLY__
/* Other enums */
+enum OpalVendorApiTokens {
+ OPAL_START_VENDOR_API_RANGE = 1000, OPAL_END_VENDOR_API_RANGE = 1999
+};
+
enum OpalFreezeState {
OPAL_EEH_STOPPED_NOT_FROZEN = 0,
OPAL_EEH_STOPPED_MMIO_FREEZE = 1,
@@ -264,12 +315,36 @@ enum OpalErrinjectFunc {
OPAL_ERR_INJECT_FUNC_IOA_DMA_WR_TARGET = 19,
};
+enum OpalShpcAction {
+ OPAL_SHPC_GET_LINK_STATE = 0,
+ OPAL_SHPC_GET_SLOT_STATE = 1
+};
+
+enum OpalShpcLinkState {
+ OPAL_SHPC_LINK_DOWN = 0,
+ OPAL_SHPC_LINK_UP_x1 = 1,
+ OPAL_SHPC_LINK_UP_x2 = 2,
+ OPAL_SHPC_LINK_UP_x4 = 4,
+ OPAL_SHPC_LINK_UP_x8 = 8,
+ OPAL_SHPC_LINK_UP_x16 = 16,
+ OPAL_SHPC_LINK_UP_x32 = 32
+};
+
enum OpalMmioWindowType {
OPAL_M32_WINDOW_TYPE = 1,
OPAL_M64_WINDOW_TYPE = 2,
OPAL_IO_WINDOW_TYPE = 3
};
+enum OpalShpcSlotState {
+ OPAL_SHPC_DEV_NOT_PRESENT = 0,
+ OPAL_SHPC_DEV_PRESENT = 1
+};
+enum OpalShpcPowerState {
+ OPAL_SHPC_POWER_OFF = 0,
+ OPAL_SHPC_POWER_ON = 1
+};
+
enum OpalExceptionHandler {
OPAL_MACHINE_CHECK_HANDLER = 1,
OPAL_HYPERVISOR_MAINTENANCE_HANDLER = 2,
@@ -361,6 +436,11 @@ enum OpalPciResetState {
OPAL_ASSERT_RESET = 1
};
+enum OpalPciMaskAction {
+ OPAL_UNMASK_ERROR_TYPE = 0,
+ OPAL_MASK_ERROR_TYPE = 1
+};
+
enum OpalPciSlotPresence {
OPAL_PCI_SLOT_EMPTY = 0,
OPAL_PCI_SLOT_PRESENT = 1
@@ -385,6 +465,18 @@ enum OpalSlotLedState {
OPAL_SLOT_LED_STATE_ON = 1 /* LED is ON */
};
+enum OpalEpowStatus {
+ OPAL_EPOW_NONE = 0,
+ OPAL_EPOW_UPS = 1,
+ OPAL_EPOW_OVER_AMBIENT_TEMP = 2,
+ OPAL_EPOW_OVER_INTERNAL_TEMP = 3
+};
+
+enum OpalCheckTokenStatus {
+ OPAL_TOKEN_ABSENT = 0,
+ OPAL_TOKEN_PRESENT = 1
+};
+
/*
* Address cycle types for LPC accesses. These also correspond
* to the content of the first cell of the "reg" property for
@@ -434,6 +526,46 @@ struct opal_ipmi_msg {
uint8_t data[];
};
+/*
+ * EPOW status sharing (OPAL and the host)
+ *
+ * The host will pass on OPAL, a buffer of length OPAL_SYSEPOW_MAX
+ * with individual elements being 16 bits wide to fetch the system
+ * wide EPOW status. Each element in the buffer will contain the
+ * EPOW status in it's bit representation for a particular EPOW sub
+ * class as defined here. So multiple detailed EPOW status bits
+ * specific for any sub class can be represented in a single buffer
+ * element as it's bit representation.
+ */
+
+/* System EPOW type */
+enum OpalSysEpow {
+ OPAL_SYSEPOW_POWER = 0, /* Power EPOW */
+ OPAL_SYSEPOW_TEMP = 1, /* Temperature EPOW */
+ OPAL_SYSEPOW_COOLING = 2, /* Cooling EPOW */
+ OPAL_SYSEPOW_MAX = 3, /* Max EPOW categories */
+};
+
+/* Power EPOW */
+enum OpalSysPower {
+ OPAL_SYSPOWER_UPS = 0x0001, /* System on UPS power */
+ OPAL_SYSPOWER_CHNG = 0x0002, /* System power configuration change */
+ OPAL_SYSPOWER_FAIL = 0x0004, /* System impending power failure */
+ OPAL_SYSPOWER_INCL = 0x0008, /* System incomplete power */
+};
+
+/* Temperature EPOW */
+enum OpalSysTemp {
+ OPAL_SYSTEMP_AMB = 0x0001, /* System over ambient temperature */
+ OPAL_SYSTEMP_INT = 0x0002, /* System over internal temperature */
+ OPAL_SYSTEMP_HMD = 0x0004, /* System over ambient humidity */
+};
+
+/* Cooling EPOW */
+enum OpalSysCooling {
+ OPAL_SYSCOOL_INSF = 0x0001, /* System insufficient cooling */
+};
+
/* FSP memory errors handling */
enum OpalMemErr_Version {
OpalMemErr_V1 = 1,
@@ -456,6 +588,11 @@ enum OpalMemErr_DynErrType {
OPAL_MEM_DYNAMIC_DEALLOC = 0,
};
+/* OpalMemoryErrorData->flags */
+#define OPAL_MEM_CORRECTED_ERROR 0x0001
+#define OPAL_MEM_THRESHOLD_EXCEEDED 0x0002
+#define OPAL_MEM_ACK_REQUIRED 0x8000
+
struct OpalMemoryErrorData {
enum OpalMemErr_Version version:8; /* 0x00 */
enum OpalMemErrType type:8; /* 0x01 */
@@ -518,6 +655,7 @@ enum OpalHMI_XstopType {
CHECKSTOP_TYPE_UNKNOWN = 0,
CHECKSTOP_TYPE_CORE = 1,
CHECKSTOP_TYPE_NX = 2,
+ CHECKSTOP_TYPE_NPU = 3
};
enum OpalHMI_CoreXstopReason {
@@ -577,10 +715,10 @@ struct OpalHMIEvent {
struct {
uint8_t xstop_type; /* enum OpalHMI_XstopType */
uint8_t reserved_1[3];
- __be32 xstop_reason;
+ __be32 xstop_reason;
union {
- __be32 pir; /* for CHECKSTOP_TYPE_CORE */
- __be32 chip_id; /* for CHECKSTOP_TYPE_NX */
+ __be32 pir; /* for CHECKSTOP_TYPE_CORE */
+ __be32 chip_id; /* for CHECKSTOP_TYPE_NX */
} u;
} xstop_error;
} u;
@@ -645,7 +783,8 @@ enum {
enum {
OPAL_PHB_ERROR_DATA_TYPE_P7IOC = 1,
- OPAL_PHB_ERROR_DATA_TYPE_PHB3 = 2
+ OPAL_PHB_ERROR_DATA_TYPE_PHB3 = 2,
+ OPAL_PHB_ERROR_DATA_TYPE_PHB4 = 3
};
enum {
@@ -780,6 +919,11 @@ struct OpalIoPhb3ErrorData {
__be64 pestB[OPAL_PHB3_NUM_PEST_REGS];
};
+struct OpalIoPhb4ErrorData {
+ struct OpalIoPhbErrorCommon common;
+ // FIXME add phb4 specific stuff
+};
+
enum {
OPAL_REINIT_CPUS_HILE_BE = (1 << 0),
OPAL_REINIT_CPUS_HILE_LE = (1 << 1),
@@ -797,6 +941,7 @@ enum opal_prd_msg_type {
OPAL_PRD_MSG_TYPE_ATTN_ACK, /* HBRT --> OPAL */
OPAL_PRD_MSG_TYPE_OCC_ERROR, /* HBRT <-- OPAL */
OPAL_PRD_MSG_TYPE_OCC_RESET, /* HBRT <-- OPAL */
+ OPAL_PRD_MSG_TYPE_OCC_RESET_NOTIFY, /* HBRT --> OPAL */
};
struct opal_prd_msg_header {
@@ -805,13 +950,50 @@ struct opal_prd_msg_header {
__be16 size;
};
-struct opal_prd_msg;
-
-#define OCC_RESET 0
-#define OCC_LOAD 1
-#define OCC_THROTTLE 2
-#define OCC_MAX_THROTTLE_STATUS 5
+struct opal_prd_msg {
+ struct opal_prd_msg_header hdr;
+ __be32 token;
+ union {
+ struct {
+ __be64 version;
+ __be64 ipoll;
+ } init;
+ struct {
+ __be64 proc;
+ __be64 ipoll_status;
+ __be64 ipoll_mask;
+ } attn;
+ struct {
+ __be64 proc;
+ __be64 ipoll_ack;
+ } attn_ack;
+ struct {
+ __be64 chip;
+ } occ_error;
+ struct {
+ __be64 chip;
+ } occ_reset;
+ };
+};
+#define OCC_RESET 0
+#define OCC_LOAD 1
+#define OCC_THROTTLE 2
+#define OCC_MAX_THROTTLE_STATUS 5
+/*
+ * struct opal_occ_msg:
+ * type: OCC_RESET, OCC_LOAD, OCC_THROTTLE
+ * chip: chip id
+ * throttle status: indicates the reason why OCC may have limited
+ * the max Pstate of the chip.
+ * 0x00 = No throttle
+ * 0x01 = Power Cap
+ * 0x02 = Processor Over Temperature
+ * 0x03 = Power Supply Failure (currently not used)
+ * 0x04 = Over current (currently not used)
+ * 0x05 = OCC Reset (not reliable as some failures will not allow for
+ * OCC to update throttle status)
+ */
struct opal_occ_msg {
__be64 type;
__be64 chip;
@@ -857,6 +1039,10 @@ enum {
OPAL_PHB_CAPI_MODE_DMA = 4,
};
+/* CAPI feature flags (in device-tree) */
+#define OPAL_PHB_CAPI_FLAG_SNOOP_CONTROL 0x00000001
+#define OPAL_PHB_CAPI_FLAG_REVERT_TO_PCIE 0x00000002
+
/* OPAL I2C request */
struct opal_i2c_request {
uint8_t type;
@@ -875,59 +1061,73 @@ struct opal_i2c_request {
__be64 buffer_ra; /* Buffer real address */
};
-/*
- * EPOW status sharing (OPAL and the host)
- *
- * The host will pass on OPAL, a buffer of length OPAL_SYSEPOW_MAX
- * with individual elements being 16 bits wide to fetch the system
- * wide EPOW status. Each element in the buffer will contain the
- * EPOW status in it's bit representation for a particular EPOW sub
- * class as defined here. So multiple detailed EPOW status bits
- * specific for any sub class can be represented in a single buffer
- * element as it's bit representation.
+/* Argument to OPAL_CEC_REBOOT2() */
+enum {
+ OPAL_REBOOT_NORMAL = 0,
+ OPAL_REBOOT_PLATFORM_ERROR,
+};
+
+/* Argument to OPAL_PCI_TCE_KILL */
+enum {
+ OPAL_PCI_TCE_KILL_PAGES,
+ OPAL_PCI_TCE_KILL_PE,
+ OPAL_PCI_TCE_KILL_ALL,
+};
+
+/* The xive operation mode indicates the active "API" and
+ * corresponds to the "mode" parameter of the opal_xive_reset()
+ * call
*/
+enum {
+ OPAL_XIVE_MODE_EMU = 0,
+ OPAL_XIVE_MODE_EXPL = 1,
+};
-/* System EPOW type */
-enum OpalSysEpow {
- OPAL_SYSEPOW_POWER = 0, /* Power EPOW */
- OPAL_SYSEPOW_TEMP = 1, /* Temperature EPOW */
- OPAL_SYSEPOW_COOLING = 2, /* Cooling EPOW */
- OPAL_SYSEPOW_MAX = 3, /* Max EPOW categories */
+/* Flags for OPAL_XIVE_GET_IRQ_INFO */
+enum {
+ OPAL_XIVE_IRQ_TRIGGER_PAGE = 0x00000001,
+ OPAL_XIVE_IRQ_STORE_EOI = 0x00000002,
+ OPAL_XIVE_IRQ_LSI = 0x00000004,
+ OPAL_XIVE_IRQ_SHIFT_BUG = 0x00000008,
+ OPAL_XIVE_IRQ_MASK_VIA_FW = 0x00000010,
+ OPAL_XIVE_IRQ_EOI_VIA_FW = 0x00000020,
};
-/* Power EPOW */
-enum OpalSysPower {
- OPAL_SYSPOWER_UPS = 0x0001, /* System on UPS power */
- OPAL_SYSPOWER_CHNG = 0x0002, /* System power config change */
- OPAL_SYSPOWER_FAIL = 0x0004, /* System impending power failure */
- OPAL_SYSPOWER_INCL = 0x0008, /* System incomplete power */
+/* Flags for OPAL_XIVE_GET/SET_QUEUE_INFO */
+enum {
+ OPAL_XIVE_EQ_ENABLED = 0x00000001,
+ OPAL_XIVE_EQ_ALWAYS_NOTIFY = 0x00000002,
+ OPAL_XIVE_EQ_ESCALATE = 0x00000004,
};
-/* Temperature EPOW */
-enum OpalSysTemp {
- OPAL_SYSTEMP_AMB = 0x0001, /* System over ambient temperature */
- OPAL_SYSTEMP_INT = 0x0002, /* System over internal temperature */
- OPAL_SYSTEMP_HMD = 0x0004, /* System over ambient humidity */
+/* Flags for OPAL_XIVE_GET/SET_VP_INFO */
+enum {
+ OPAL_XIVE_VP_ENABLED = 0x00000001,
};
-/* Cooling EPOW */
-enum OpalSysCooling {
- OPAL_SYSCOOL_INSF = 0x0001, /* System insufficient cooling */
+/* "Any chip" replacement for chip ID for allocation functions */
+enum {
+ OPAL_XIVE_ANY_CHIP = 0xffffffff,
};
-/* Argument to OPAL_CEC_REBOOT2() */
+/* Xive sync options */
enum {
- OPAL_REBOOT_NORMAL = 0,
- OPAL_REBOOT_PLATFORM_ERROR = 1,
+ /* This bits are cumulative, arg is a girq */
+ XIVE_SYNC_EAS = 0x00000001, /* Sync irq source */
+ XIVE_SYNC_QUEUE = 0x00000002, /* Sync irq target */
};
-/* Argument to OPAL_PCI_TCE_KILL */
+/* Dump options */
enum {
- OPAL_PCI_TCE_KILL_PAGES,
- OPAL_PCI_TCE_KILL_PE,
- OPAL_PCI_TCE_KILL_ALL,
+ XIVE_DUMP_TM_HYP = 0,
+ XIVE_DUMP_TM_POOL = 1,
+ XIVE_DUMP_TM_OS = 2,
+ XIVE_DUMP_TM_USER = 3,
+ XIVE_DUMP_VP = 4,
+ XIVE_DUMP_EMU_STATE = 5,
};
#endif /* __ASSEMBLY__ */
#endif /* __OPAL_API_H */
+
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index 1ff03a6..cb7d607 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -226,6 +226,42 @@ int64_t opal_pci_tce_kill(uint64_t phb_id, uint32_t kill_type,
uint32_t pe_num, uint32_t tce_size,
uint64_t dma_addr, uint32_t npages);
int64_t opal_nmmu_set_ptcr(uint64_t chip_id, uint64_t ptcr);
+int64_t opal_xive_reset(uint64_t version);
+int64_t opal_xive_get_irq_info(uint32_t girq,
+ __be64 *out_flags,
+ __be64 *out_eoi_page,
+ __be64 *out_trig_page,
+ __be32 *out_esb_shift,
+ __be32 *out_src_chip);
+int64_t opal_xive_get_irq_config(uint32_t girq, __be64 *out_vp,
+ uint8_t *out_prio, __be32 *out_lirq);
+int64_t opal_xive_set_irq_config(uint32_t girq, uint64_t vp, uint8_t prio,
+ uint32_t lirq);
+int64_t opal_xive_get_queue_info(uint64_t vp, uint32_t prio,
+ __be64 *out_qpage,
+ __be64 *out_qsize,
+ __be64 *out_qeoi_page,
+ __be32 *out_escalate_irq,
+ __be64 *out_qflags);
+int64_t opal_xive_set_queue_info(uint64_t vp, uint32_t prio,
+ uint64_t qpage,
+ uint64_t qsize,
+ uint64_t qflags);
+int64_t opal_xive_donate_page(uint32_t chip_id, uint64_t addr);
+int64_t opal_xive_alloc_vp_block(uint32_t alloc_order);
+int64_t opal_xive_free_vp_block(uint64_t vp);
+int64_t opal_xive_get_vp_info(uint64_t vp,
+ __be64 *out_flags,
+ __be64 *out_cam_value,
+ __be64 *out_report_cl_pair,
+ __be32 *out_chip_id);
+int64_t opal_xive_set_vp_info(uint64_t vp,
+ uint64_t flags,
+ uint64_t report_cl_pair);
+int64_t opal_xive_allocate_irq(uint32_t chip_id);
+int64_t opal_xive_free_irq(uint32_t girq);
+int64_t opal_xive_sync(uint32_t type, uint32_t id);
+int64_t opal_xive_dump(uint32_t type, uint32_t id);
/* Internal functions */
extern int early_init_dt_scan_opal(unsigned long node, const char *uname,
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index da8a0f7..085605a 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -301,3 +301,18 @@ OPAL_CALL(opal_int_eoi, OPAL_INT_EOI);
OPAL_CALL(opal_int_set_mfrr, OPAL_INT_SET_MFRR);
OPAL_CALL(opal_pci_tce_kill, OPAL_PCI_TCE_KILL);
OPAL_CALL(opal_nmmu_set_ptcr, OPAL_NMMU_SET_PTCR);
+OPAL_CALL(opal_xive_reset, OPAL_XIVE_RESET);
+OPAL_CALL(opal_xive_get_irq_info, OPAL_XIVE_GET_IRQ_INFO);
+OPAL_CALL(opal_xive_get_irq_config, OPAL_XIVE_GET_IRQ_CONFIG);
+OPAL_CALL(opal_xive_set_irq_config, OPAL_XIVE_SET_IRQ_CONFIG);
+OPAL_CALL(opal_xive_get_queue_info, OPAL_XIVE_GET_QUEUE_INFO);
+OPAL_CALL(opal_xive_set_queue_info, OPAL_XIVE_SET_QUEUE_INFO);
+OPAL_CALL(opal_xive_donate_page, OPAL_XIVE_DONATE_PAGE);
+OPAL_CALL(opal_xive_alloc_vp_block, OPAL_XIVE_ALLOCATE_VP_BLOCK);
+OPAL_CALL(opal_xive_free_vp_block, OPAL_XIVE_FREE_VP_BLOCK);
+OPAL_CALL(opal_xive_allocate_irq, OPAL_XIVE_ALLOCATE_IRQ);
+OPAL_CALL(opal_xive_free_irq, OPAL_XIVE_FREE_IRQ);
+OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
+OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
+OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
+OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 03/12] powerpc: Add more PPC bit conversion macros
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Add 32 and 8 bit variants
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/bitops.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/powerpc/include/asm/bitops.h b/arch/powerpc/include/asm/bitops.h
index bc5fdfd..33a24fd 100644
--- a/arch/powerpc/include/asm/bitops.h
+++ b/arch/powerpc/include/asm/bitops.h
@@ -55,6 +55,14 @@
#define PPC_BITEXTRACT(bits, ppc_bit, dst_bit) \
((((bits) >> PPC_BITLSHIFT(ppc_bit)) & 1) << (dst_bit))
+#define PPC_BITLSHIFT32(be) (32 - 1 - (be))
+#define PPC_BIT32(bit) (1UL << PPC_BITLSHIFT32(bit))
+#define PPC_BITMASK32(bs, be) ((PPC_BIT32(bs) - PPC_BIT32(be))|PPC_BIT32(bs))
+
+#define PPC_BITLSHIFT8(be) (8 - 1 - (be))
+#define PPC_BIT8(bit) (1UL << PPC_BITLSHIFT8(bit))
+#define PPC_BITMASK8(bs, be) ((PPC_BIT8(bs) - PPC_BIT8(be))|PPC_BIT8(bs))
+
#include <asm/barrier.h>
/* Macro for generating the ***_bits() functions */
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 03/12] powerpc: Add more PPC bit conversion macros
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Add 32 and 8 bit variants
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/bitops.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/arch/powerpc/include/asm/bitops.h b/arch/powerpc/include/asm/bitops.h
index bc5fdfd..33a24fd 100644
--- a/arch/powerpc/include/asm/bitops.h
+++ b/arch/powerpc/include/asm/bitops.h
@@ -55,6 +55,14 @@
#define PPC_BITEXTRACT(bits, ppc_bit, dst_bit) \
((((bits) >> PPC_BITLSHIFT(ppc_bit)) & 1) << (dst_bit))
+#define PPC_BITLSHIFT32(be) (32 - 1 - (be))
+#define PPC_BIT32(bit) (1UL << PPC_BITLSHIFT32(bit))
+#define PPC_BITMASK32(bs, be) ((PPC_BIT32(bs) - PPC_BIT32(be))|PPC_BIT32(bs))
+
+#define PPC_BITLSHIFT8(be) (8 - 1 - (be))
+#define PPC_BIT8(bit) (1UL << PPC_BITLSHIFT8(bit))
+#define PPC_BITMASK8(bs, be) ((PPC_BIT8(bs) - PPC_BIT8(be))|PPC_BIT8(bs))
+
#include <asm/barrier.h>
/* Macro for generating the ***_bits() functions */
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 04/12] powerpc: Add optional smp_ops->prepare_cpu SMP callback
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Some platforms (will) need to perform allocations before bringing
a new CPU online. Doing it from smp_ops->setup_cpu is the wrong
thing to do:
- It has no useful failure path (too late)
- Calling any allocator will enable interrupts prematurely
causing problems with large decrementer among others
Instead, add a new callback that is called from __cpu_up (so from
the context trying to online the new CPU) at a point where we
can safely allocate and handle failures.
This will be used by XIVE support.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/smp.h | 1 +
arch/powerpc/kernel/smp.c | 10 ++++++++++
2 files changed, 11 insertions(+)
diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
index 32db16d..2f8e36f 100644
--- a/arch/powerpc/include/asm/smp.h
+++ b/arch/powerpc/include/asm/smp.h
@@ -44,6 +44,7 @@ struct smp_ops_t {
#endif
void (*probe)(void);
int (*kick_cpu)(int nr);
+ int (*prepare_cpu)(int nr);
void (*setup_cpu)(int nr);
void (*bringup_done)(void);
void (*take_timebase)(void);
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 46f89e6..b12f5f0 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -521,6 +521,16 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle)
cpu_idle_thread_init(cpu, tidle);
+ /*
+ * The platform might need to allocate resources prior to bringing
+ * up the CPU
+ */
+ if (smp_ops->prepare_cpu) {
+ rc = smp_ops->prepare_cpu(cpu);
+ if (rc)
+ return rc;
+ }
+
/* Make sure callin-map entry is 0 (can be leftover a CPU
* hotplug
*/
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 04/12] powerpc: Add optional smp_ops->prepare_cpu SMP callback
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Some platforms (will) need to perform allocations before bringing
a new CPU online. Doing it from smp_ops->setup_cpu is the wrong
thing to do:
- It has no useful failure path (too late)
- Calling any allocator will enable interrupts prematurely
causing problems with large decrementer among others
Instead, add a new callback that is called from __cpu_up (so from
the context trying to online the new CPU) at a point where we
can safely allocate and handle failures.
This will be used by XIVE support.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/smp.h | 1 +
arch/powerpc/kernel/smp.c | 10 ++++++++++
2 files changed, 11 insertions(+)
diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
index 32db16d..2f8e36f 100644
--- a/arch/powerpc/include/asm/smp.h
+++ b/arch/powerpc/include/asm/smp.h
@@ -44,6 +44,7 @@ struct smp_ops_t {
#endif
void (*probe)(void);
int (*kick_cpu)(int nr);
+ int (*prepare_cpu)(int nr);
void (*setup_cpu)(int nr);
void (*bringup_done)(void);
void (*take_timebase)(void);
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index 46f89e6..b12f5f0 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -521,6 +521,16 @@ int __cpu_up(unsigned int cpu, struct task_struct *tidle)
cpu_idle_thread_init(cpu, tidle);
+ /*
+ * The platform might need to allocate resources prior to bringing
+ * up the CPU
+ */
+ if (smp_ops->prepare_cpu) {
+ rc = smp_ops->prepare_cpu(cpu);
+ if (rc)
+ return rc;
+ }
+
/* Make sure callin-map entry is 0 (can be leftover a CPU
* hotplug
*/
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 05/12] powerpc/smp: Remove migrate_irq() custom implementation
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Some powerpc platforms use this to move IRQs away from a CPU
being unplugged. This function has several bugs such as not
taking the right locks or failing to NULL check pointers.
There's a new generic function doing exactly the same thing
without all the bugs, so let's use it instead.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/Kconfig | 1 +
arch/powerpc/include/asm/smp.h | 1 -
arch/powerpc/kernel/irq.c | 40 ----------------------------------------
arch/powerpc/kernel/smp.c | 9 ++++++++-
4 files changed, 9 insertions(+), 42 deletions(-)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 97a8bc8..29fb0e6 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -431,6 +431,7 @@ config HOTPLUG_CPU
bool "Support for enabling/disabling CPUs"
depends on SMP && (PPC_PSERIES || \
PPC_PMAC || PPC_POWERNV || FSL_SOC_BOOKE)
+ select GENERIC_IRQ_MIGRATION
---help---
Say Y here to be able to disable and re-enable individual
CPUs at runtime on SMP machines.
diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
index 2f8e36f..63fa780 100644
--- a/arch/powerpc/include/asm/smp.h
+++ b/arch/powerpc/include/asm/smp.h
@@ -62,7 +62,6 @@ extern void smp_generic_take_timebase(void);
DECLARE_PER_CPU(unsigned int, cpu_pvr);
#ifdef CONFIG_HOTPLUG_CPU
-extern void migrate_irqs(void);
int generic_cpu_disable(void);
void generic_cpu_die(unsigned int cpu);
void generic_set_cpu_dead(unsigned int cpu);
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index a018f5c..8ee7b44 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -442,46 +442,6 @@ u64 arch_irq_stat_cpu(unsigned int cpu)
return sum;
}
-#ifdef CONFIG_HOTPLUG_CPU
-void migrate_irqs(void)
-{
- struct irq_desc *desc;
- unsigned int irq;
- static int warned;
- cpumask_var_t mask;
- const struct cpumask *map = cpu_online_mask;
-
- alloc_cpumask_var(&mask, GFP_KERNEL);
-
- for_each_irq_desc(irq, desc) {
- struct irq_data *data;
- struct irq_chip *chip;
-
- data = irq_desc_get_irq_data(desc);
- if (irqd_is_per_cpu(data))
- continue;
-
- chip = irq_data_get_irq_chip(data);
-
- cpumask_and(mask, irq_data_get_affinity_mask(data), map);
- if (cpumask_any(mask) >= nr_cpu_ids) {
- pr_warn("Breaking affinity for irq %i\n", irq);
- cpumask_copy(mask, map);
- }
- if (chip->irq_set_affinity)
- chip->irq_set_affinity(data, mask, true);
- else if (desc->action && !(warned++))
- pr_err("Cannot set affinity for irq %i\n", irq);
- }
-
- free_cpumask_var(mask);
-
- local_irq_enable();
- mdelay(1);
- local_irq_disable();
-}
-#endif
-
static inline void check_stack_overflow(void)
{
#ifdef CONFIG_DEBUG_STACKOVERFLOW
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index b12f5f0..6e61cdb 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -439,7 +439,14 @@ int generic_cpu_disable(void)
#ifdef CONFIG_PPC64
vdso_data->processorCount--;
#endif
- migrate_irqs();
+ /* Update affinity of all IRQs previously aimed at this CPU */
+ irq_migrate_all_off_this_cpu();
+
+ /* Give the CPU time to drain in-flight ones */
+ local_irq_enable();
+ mdelay(1);
+ local_irq_disable();
+
return 0;
}
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 05/12] powerpc/smp: Remove migrate_irq() custom implementation
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
Some powerpc platforms use this to move IRQs away from a CPU
being unplugged. This function has several bugs such as not
taking the right locks or failing to NULL check pointers.
There's a new generic function doing exactly the same thing
without all the bugs, so let's use it instead.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/Kconfig | 1 +
arch/powerpc/include/asm/smp.h | 1 -
arch/powerpc/kernel/irq.c | 40 ----------------------------------------
arch/powerpc/kernel/smp.c | 9 ++++++++-
4 files changed, 9 insertions(+), 42 deletions(-)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index 97a8bc8..29fb0e6 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -431,6 +431,7 @@ config HOTPLUG_CPU
bool "Support for enabling/disabling CPUs"
depends on SMP && (PPC_PSERIES || \
PPC_PMAC || PPC_POWERNV || FSL_SOC_BOOKE)
+ select GENERIC_IRQ_MIGRATION
---help---
Say Y here to be able to disable and re-enable individual
CPUs at runtime on SMP machines.
diff --git a/arch/powerpc/include/asm/smp.h b/arch/powerpc/include/asm/smp.h
index 2f8e36f..63fa780 100644
--- a/arch/powerpc/include/asm/smp.h
+++ b/arch/powerpc/include/asm/smp.h
@@ -62,7 +62,6 @@ extern void smp_generic_take_timebase(void);
DECLARE_PER_CPU(unsigned int, cpu_pvr);
#ifdef CONFIG_HOTPLUG_CPU
-extern void migrate_irqs(void);
int generic_cpu_disable(void);
void generic_cpu_die(unsigned int cpu);
void generic_set_cpu_dead(unsigned int cpu);
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index a018f5c..8ee7b44 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -442,46 +442,6 @@ u64 arch_irq_stat_cpu(unsigned int cpu)
return sum;
}
-#ifdef CONFIG_HOTPLUG_CPU
-void migrate_irqs(void)
-{
- struct irq_desc *desc;
- unsigned int irq;
- static int warned;
- cpumask_var_t mask;
- const struct cpumask *map = cpu_online_mask;
-
- alloc_cpumask_var(&mask, GFP_KERNEL);
-
- for_each_irq_desc(irq, desc) {
- struct irq_data *data;
- struct irq_chip *chip;
-
- data = irq_desc_get_irq_data(desc);
- if (irqd_is_per_cpu(data))
- continue;
-
- chip = irq_data_get_irq_chip(data);
-
- cpumask_and(mask, irq_data_get_affinity_mask(data), map);
- if (cpumask_any(mask) >= nr_cpu_ids) {
- pr_warn("Breaking affinity for irq %i\n", irq);
- cpumask_copy(mask, map);
- }
- if (chip->irq_set_affinity)
- chip->irq_set_affinity(data, mask, true);
- else if (desc->action && !(warned++))
- pr_err("Cannot set affinity for irq %i\n", irq);
- }
-
- free_cpumask_var(mask);
-
- local_irq_enable();
- mdelay(1);
- local_irq_disable();
-}
-#endif
-
static inline void check_stack_overflow(void)
{
#ifdef CONFIG_DEBUG_STACKOVERFLOW
diff --git a/arch/powerpc/kernel/smp.c b/arch/powerpc/kernel/smp.c
index b12f5f0..6e61cdb 100644
--- a/arch/powerpc/kernel/smp.c
+++ b/arch/powerpc/kernel/smp.c
@@ -439,7 +439,14 @@ int generic_cpu_disable(void)
#ifdef CONFIG_PPC64
vdso_data->processorCount--;
#endif
- migrate_irqs();
+ /* Update affinity of all IRQs previously aimed at this CPU */
+ irq_migrate_all_off_this_cpu();
+
+ /* Give the CPU time to drain in-flight ones */
+ local_irq_enable();
+ mdelay(1);
+ local_irq_disable();
+
return 0;
}
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
The XIVE interrupt controller is the new interrupt controller
found in POWER9. It supports advanced virtualization capabilities
among other things.
Currently we use a set of firmware calls that simulate the old
"XICS" interrupt controller but this is fairly inefficient.
This adds the framework for using XIVE along with a native
backend which OPAL for configuration. Later, a backend allowing
the use in a KVM or PowerVM guest will also be provided.
This disables some fast path for interrupts in KVM when XIVE is
enabled as these rely on the firmware emulation code which is no
longer available when the XIVE is used natively by Linux.
A latter patch will make KVM also directly exploit the XIVE, thus
recovering the lost performance (and more).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/xive.h | 116 +++
arch/powerpc/include/asm/xmon.h | 2 +
arch/powerpc/platforms/powernv/Kconfig | 2 +
arch/powerpc/platforms/powernv/setup.c | 15 +-
arch/powerpc/platforms/powernv/smp.c | 39 +-
arch/powerpc/sysdev/Kconfig | 1 +
arch/powerpc/sysdev/Makefile | 1 +
arch/powerpc/sysdev/xive/Kconfig | 7 +
arch/powerpc/sysdev/xive/Makefile | 4 +
arch/powerpc/sysdev/xive/common.c | 1175 ++++++++++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 604 +++++++++++++++
arch/powerpc/sysdev/xive/xive-internal.h | 51 ++
arch/powerpc/sysdev/xive/xive-regs.h | 88 +++
arch/powerpc/xmon/xmon.c | 93 ++-
14 files changed, 2186 insertions(+), 12 deletions(-)
create mode 100644 arch/powerpc/include/asm/xive.h
create mode 100644 arch/powerpc/sysdev/xive/Kconfig
create mode 100644 arch/powerpc/sysdev/xive/Makefile
create mode 100644 arch/powerpc/sysdev/xive/common.c
create mode 100644 arch/powerpc/sysdev/xive/native.c
create mode 100644 arch/powerpc/sysdev/xive/xive-internal.h
create mode 100644 arch/powerpc/sysdev/xive/xive-regs.h
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
new file mode 100644
index 0000000..b1604b73
--- /dev/null
+++ b/arch/powerpc/include/asm/xive.h
@@ -0,0 +1,116 @@
+#ifndef _ASM_POWERPC_XIVE_H
+#define _ASM_POWERPC_XIVE_H
+
+#define XIVE_INVALID_VP 0xffffffff
+
+#ifdef CONFIG_PPC_XIVE
+
+extern void __iomem *xive_tm_area;
+extern u32 xive_tm_offset;
+
+/*
+ * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
+ * have it stored in the xive_cpu structure. We also cache
+ * for normal interrupts the current target CPU.
+ */
+struct xive_irq_data {
+ /* Setup by backend */
+ u64 flags;
+#define XIVE_IRQ_FLAG_STORE_EOI 0x01
+#define XIVE_IRQ_FLAG_LSI 0x02
+#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
+#define XIVE_IRQ_FLAG_MASK_FW 0x08
+#define XIVE_IRQ_FLAG_EOI_FW 0x10
+ u64 eoi_page;
+ void __iomem *eoi_mmio;
+ u64 trig_page;
+ void __iomem *trig_mmio;
+ u32 esb_shift;
+ int src_chip;
+
+ /* Setup/used by frontend */
+ int target;
+ bool saved_p;
+};
+#define XIVE_INVALID_CHIP_ID -1
+
+/* A queue tracking structure in a CPU */
+struct xive_q {
+ __be32 *qpage;
+ u32 msk;
+ u32 idx;
+ u32 toggle;
+ u64 eoi_phys;
+ void __iomem *eoi_mmio;
+ u32 esc_irq;
+ atomic_t count;
+ atomic_t pending_count;
+};
+
+/*
+ * "magic" ESB MMIO offsets
+ */
+#define XIVE_ESB_GET 0x800
+#define XIVE_ESB_SET_PQ_00 0xc00
+#define XIVE_ESB_SET_PQ_01 0xd00
+#define XIVE_ESB_SET_PQ_10 0xe00
+#define XIVE_ESB_SET_PQ_11 0xf00
+#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
+
+extern bool __xive_enabled;
+
+static inline bool xive_enabled(void) { return __xive_enabled; }
+
+extern bool xive_native_init(void);
+extern void xive_smp_probe(void);
+extern int xive_smp_prepare_cpu(unsigned int cpu);
+extern void xive_smp_setup_cpu(void);
+extern void xive_smp_disable_cpu(void);
+extern void xive_kexec_teardown_cpu(int secondary);
+extern void xive_shutdown(void);
+extern void xive_flush_interrupt(void);
+
+/* xmon hook */
+extern void xmon_xive_do_dump(int cpu);
+
+/* APIs used by KVM */
+extern u32 xive_native_default_eq_shift(void);
+extern u32 xive_native_alloc_vp_block(u32 max_vcpus);
+extern void xive_native_free_vp_block(u32 vp_base);
+extern int xive_native_populate_irq_data(u32 hw_irq,
+ struct xive_irq_data *data);
+extern void xive_cleanup_irq_data(struct xive_irq_data *xd);
+extern u32 xive_native_alloc_irq(void);
+extern void xive_native_free_irq(u32 irq);
+extern int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
+
+extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
+ __be32 *qpage, u32 order, bool can_escalate);
+extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
+
+extern bool __xive_irq_trigger(struct xive_irq_data *xd);
+extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
+extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
+
+extern bool is_xive_irq(struct irq_chip *chip);
+
+#else
+
+static inline bool xive_enabled(void) { return false; }
+
+static inline bool xive_native_init(void) { return false; }
+static inline void xive_smp_probe(void) { }
+extern inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
+static inline void xive_smp_setup_cpu(void) { }
+static inline void xive_smp_disable_cpu(void) { }
+static inline void xive_kexec_teardown_cpu(int secondary) { }
+static inline void xive_shutdown(void) { }
+static inline void xive_flush_interrupt(void) { }
+
+static inline u32 xive_native_alloc_vp_block(u32 max_vcpus)
+ { return XIVE_INVALID_VP; }
+static inline void xive_native_free_vp_block(u32 vp_base) { }
+
+#endif
+
+#endif /* _ASM_POWERPC_XIVE_H */
diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h
index 5eb8e59..eb42a0c 100644
--- a/arch/powerpc/include/asm/xmon.h
+++ b/arch/powerpc/include/asm/xmon.h
@@ -29,5 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { };
extern int cpus_are_in_xmon(void);
#endif
+extern void xmon_printf(const char *format, ...);
+
#endif /* __KERNEL __ */
#endif /* __ASM_POWERPC_XMON_H */
diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
index 3a07e4d..81ee2ed 100644
--- a/arch/powerpc/platforms/powernv/Kconfig
+++ b/arch/powerpc/platforms/powernv/Kconfig
@@ -4,6 +4,8 @@ config PPC_POWERNV
select PPC_NATIVE
select PPC_XICS
select PPC_ICP_NATIVE
+ select PPC_XIVE
+ select PPC_XIVE_NATIVE
select PPC_P7_NAP
select PCI
select PCI_MSI
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index d50c7d9..adceac9 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -32,6 +32,7 @@
#include <asm/machdep.h>
#include <asm/firmware.h>
#include <asm/xics.h>
+#include <asm/xive.h>
#include <asm/opal.h>
#include <asm/kexec.h>
#include <asm/smp.h>
@@ -76,7 +77,9 @@ static void __init pnv_init(void)
static void __init pnv_init_IRQ(void)
{
- xics_init();
+ /* Try using a XIVE if available, otherwise use a XICS */
+ if (!xive_native_init())
+ xics_init();
WARN_ON(!ppc_md.get_irq);
}
@@ -218,10 +221,12 @@ static void pnv_kexec_wait_secondaries_down(void)
static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
{
- xics_kexec_teardown_cpu(secondary);
+ if (xive_enabled())
+ xive_kexec_teardown_cpu(secondary);
+ else
+ xics_kexec_teardown_cpu(secondary);
/* On OPAL, we return all CPUs to firmware */
-
if (!firmware_has_feature(FW_FEATURE_OPAL))
return;
@@ -237,6 +242,10 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
/* Primary waits for the secondaries to have reached OPAL */
pnv_kexec_wait_secondaries_down();
+ /* Switch XIVE back to emulation mode */
+ if (xive_enabled())
+ xive_shutdown();
+
/*
* We might be running as little-endian - now that interrupts
* are disabled, reset the HILE bit to big-endian so we don't
diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 8b67e1e..f571955 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -29,6 +29,7 @@
#include <asm/vdso_datapage.h>
#include <asm/cputhreads.h>
#include <asm/xics.h>
+#include <asm/xive.h>
#include <asm/opal.h>
#include <asm/runlatch.h>
#include <asm/code-patching.h>
@@ -47,7 +48,9 @@
static void pnv_smp_setup_cpu(int cpu)
{
- if (cpu != boot_cpuid)
+ if (xive_enabled())
+ xive_smp_setup_cpu();
+ else if (cpu != boot_cpuid)
xics_setup_cpu();
#ifdef CONFIG_PPC_DOORBELL
@@ -132,7 +135,10 @@ static int pnv_smp_cpu_disable(void)
vdso_data->processorCount--;
if (cpu == boot_cpuid)
boot_cpuid = cpumask_any(cpu_online_mask);
- xics_migrate_irqs_away();
+ if (xive_enabled())
+ xive_smp_disable_cpu();
+ else
+ xics_migrate_irqs_away();
return 0;
}
@@ -213,9 +219,12 @@ static void pnv_smp_cpu_kill_self(void)
if (((srr1 & wmask) == SRR1_WAKEEE) ||
((srr1 & wmask) == SRR1_WAKEHVI) ||
(local_paca->irq_happened & PACA_IRQ_EE)) {
- if (cpu_has_feature(CPU_FTR_ARCH_300))
- icp_opal_flush_interrupt();
- else
+ if (cpu_has_feature(CPU_FTR_ARCH_300)) {
+ if (xive_enabled())
+ xive_flush_interrupt();
+ else
+ icp_opal_flush_interrupt();
+ } else
icp_native_flush_interrupt();
} else if ((srr1 & wmask) == SRR1_WAKEHDBELL) {
unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
@@ -252,10 +261,26 @@ static int pnv_cpu_bootable(unsigned int nr)
return smp_generic_cpu_bootable(nr);
}
+static int pnv_smp_prepare_cpu(int cpu)
+{
+ if (xive_enabled())
+ return xive_smp_prepare_cpu(cpu);
+ return 0;
+}
+
+static void __init pnv_smp_probe(void)
+{
+ if (xive_enabled())
+ xive_smp_probe();
+ else
+ xics_smp_probe();
+}
+
static struct smp_ops_t pnv_smp_ops = {
.message_pass = smp_muxed_ipi_message_pass,
- .cause_ipi = NULL, /* Filled at runtime by xics_smp_probe() */
- .probe = xics_smp_probe,
+ .cause_ipi = NULL, /* Filled at runtime by xi{cs,ve}_smp_probe() */
+ .probe = pnv_smp_probe,
+ .prepare_cpu = pnv_smp_prepare_cpu,
.kick_cpu = pnv_smp_kick_cpu,
.setup_cpu = pnv_smp_setup_cpu,
.cpu_bootable = pnv_cpu_bootable,
diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig
index 52dc165..caf882e 100644
--- a/arch/powerpc/sysdev/Kconfig
+++ b/arch/powerpc/sysdev/Kconfig
@@ -28,6 +28,7 @@ config PPC_MSI_BITMAP
default y if PPC_POWERNV
source "arch/powerpc/sysdev/xics/Kconfig"
+source "arch/powerpc/sysdev/xive/Kconfig"
config PPC_SCOM
bool
diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile
index a254824..c0ae11d 100644
--- a/arch/powerpc/sysdev/Makefile
+++ b/arch/powerpc/sysdev/Makefile
@@ -71,5 +71,6 @@ obj-$(CONFIG_PPC_EARLY_DEBUG_MEMCONS) += udbg_memcons.o
subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
obj-$(CONFIG_PPC_XICS) += xics/
+obj-$(CONFIG_PPC_XIVE) += xive/
obj-$(CONFIG_GE_FPGA) += ge/
diff --git a/arch/powerpc/sysdev/xive/Kconfig b/arch/powerpc/sysdev/xive/Kconfig
new file mode 100644
index 0000000..c8816c8
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/Kconfig
@@ -0,0 +1,7 @@
+config PPC_XIVE
+ def_bool n
+ select PPC_SMP_MUXED_IPI
+ select HARDIRQS_SW_RESEND
+
+config PPC_XIVE_NATIVE
+ def_bool n
diff --git a/arch/powerpc/sysdev/xive/Makefile b/arch/powerpc/sysdev/xive/Makefile
new file mode 100644
index 0000000..3fab303
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/Makefile
@@ -0,0 +1,4 @@
+subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
+
+obj-y += common.o
+obj-$(CONFIG_PPC_XIVE_NATIVE) += native.o
diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
new file mode 100644
index 0000000..96037e0
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/common.c
@@ -0,0 +1,1175 @@
+/*
+ * Copyright 2016,2017 IBM Corporation.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#include <linux/types.h>
+#include <linux/threads.h>
+#include <linux/kernel.h>
+#include <linux/irq.h>
+#include <linux/debugfs.h>
+#include <linux/smp.h>
+#include <linux/interrupt.h>
+#include <linux/seq_file.h>
+#include <linux/init.h>
+#include <linux/cpu.h>
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/msi.h>
+
+#include <asm/prom.h>
+#include <asm/io.h>
+#include <asm/smp.h>
+#include <asm/machdep.h>
+#include <asm/irq.h>
+#include <asm/errno.h>
+#include <asm/xive.h>
+#include <asm/xmon.h>
+
+#include "xive-regs.h"
+#include "xive-internal.h"
+
+#undef DEBUG_FLUSH
+#undef DEBUG_ALL
+
+#define DBG(fmt...) pr_devel("XIVE: " fmt)
+
+#ifdef DEBUG_ALL
+#define DBG_VERBOSE(fmt...) pr_devel("XIVE: " fmt)
+#else
+#define DBG_VERBOSE(fmt...) do { } while(0)
+#endif
+
+bool __xive_enabled;
+bool xive_cmdline_disabled;
+
+/* We use only one priority for now */
+static u8 xive_irq_priority;
+
+void __iomem *xive_tm_area;
+u32 xive_tm_offset;
+static const struct xive_ops *xive_ops;
+static struct irq_domain *xive_irq_domain;
+
+/* The IPIs all use the same logical irq number */
+static u32 xive_ipi_irq;
+
+/* Xive state for each CPU */
+static DEFINE_PER_CPU(struct xive_cpu *, xive_cpu);
+
+/*
+ * A "disabled" interrupt should never fire, to catch problems
+ * we set its logical number to this
+ */
+#define XIVE_BAD_IRQ 0x7fffffff
+#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
+
+/* An invalid CPU target */
+#define XIVE_INVALID_TARGET (-1)
+
+static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
+{
+ u32 cur;
+
+ if (!q->qpage)
+ return 0;
+ cur = be32_to_cpup(q->qpage + q->idx);
+ if ((cur >> 31) == q->toggle)
+ return 0;
+ if (!just_peek) {
+ q->idx = (q->idx + 1) & q->msk;
+ if (q->idx == 0)
+ q->toggle ^= 1;
+ }
+ return cur & 0x7fffffff;
+}
+
+static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
+{
+ u32 hirq = 0;
+ u8 prio;
+
+ /* Find highest pending priority */
+ while (xc->pending_prio != 0) {
+ struct xive_q *q;
+
+ prio = ffs(xc->pending_prio) - 1;
+ DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
+
+ /* Try to fetch */
+ hirq = xive_read_eq(&xc->queue[prio], prio, just_peek);
+
+ /* Found something ? That's it */
+ if (hirq)
+ break;
+
+ /* Clear pending bits */
+ xc->pending_prio &= ~(1 << prio);
+
+ /*
+ * Check if the queue count needs adjusting due to
+ * interrupts being moved away.
+ */
+ q = &xc->queue[prio];
+ if (atomic_read(&q->pending_count)) {
+ int p = atomic_xchg(&q->pending_count, 0);
+ if (p) {
+ WARN_ON(p > atomic_read(&q->count));
+ atomic_sub(p, &q->count);
+ }
+ }
+ }
+
+ /* If nothing was found, set CPPR to 0xff */
+ if (hirq == 0)
+ prio = 0xff;
+
+ /* Update HW CPPR to match if necessary */
+ if (prio != xc->cppr) {
+ DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n", prio);
+ xc->cppr = prio;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, prio);
+ }
+
+ return hirq;
+}
+
+#ifdef CONFIG_XMON
+static void xive_dump_eq(const char *name, struct xive_q *q)
+{
+ u32 i0, i1, idx;
+
+ if (!q->qpage)
+ return;
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
+ q->toggle, i0, i1);
+}
+
+void xmon_xive_do_dump(int cpu)
+{
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ struct xive_irq_data *xd;
+ uint64_t val, offset;
+
+ xmon_printf("XIVE state for CPU %d:\n", cpu);
+ xmon_printf(" pp=%02x cppr=%02x\n", xc->pending_prio, xc->cppr);
+ xive_dump_eq("IRQ", &xc->queue[xive_irq_priority]);
+ xd = &xc->ipi_data;
+ offset = 0x800;
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+ val = in_be64(xd->eoi_mmio + offset);
+ xmon_printf(" IPI state: %x:%c%c\n", xc->hw_ipi,
+ val & 2 ? 'P' : 'p',
+ val & 1 ? 'Q' : 'q');
+}
+#endif /* CONFIG_XMON */
+
+static void xive_update_pending_irqs(struct xive_cpu *xc)
+{
+ u8 he, cppr;
+ u16 ack;
+
+ /* Perform the acknowledge hypervisor to register cycle */
+ ack = be16_to_cpu(__raw_readw(xive_tm_area + TM_SPC_ACK_HV_REG));
+
+ /* Synchronize subsequent queue accesses */
+ mb();
+
+ DBG_VERBOSE("CPU %d get_irq, ack=%04x\n", smp_processor_id(), ack);
+
+ /* Check the HE field */
+ cppr = ack & 0xff;
+ he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8));
+ switch(he) {
+ case TM_QW3_NSR_HE_NONE:
+ break;
+ case TM_QW3_NSR_HE_PHYS:
+ if (cppr == 0xff)
+ return;
+ xc->pending_prio |= 1 << cppr;
+ if (cppr >= xc->cppr)
+ pr_err("XIVE: CPU %d odd ack CPPR, got %d at %d\n",
+ smp_processor_id(), cppr, xc->cppr);
+ xc->cppr = cppr;
+ break;
+ case TM_QW3_NSR_HE_POOL:
+ case TM_QW3_NSR_HE_LSI:
+ pr_err("XIVE: CPU %d got unexpected interrupt type HE=%d\n",
+ smp_processor_id(), he);
+ return;
+ }
+}
+
+static unsigned int xive_get_irq(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ u32 hirq;
+
+ /*
+ * This can be called either as a result of a HW interrupt or
+ * as a "replay" because EOI decided there was still something
+ * in one of the queues.
+ *
+ * First we perform an ACK cycle in order to update our mask
+ * of pending priorities. This will also have the effect of
+ * updating the CPPR to the most favored pending interrupts.
+ *
+ * In the future, if we have a way to differenciate a first
+ * entry (on HW interrupt) from a replay triggered by EOI,
+ * we could skip this on replays unless we soft-mask tells us
+ * that a new HW interrupt occurred.
+ */
+ xive_update_pending_irqs(xc);
+
+ DBG_VERBOSE("get_irq: pending=%02x\n", xc->pending_prio);
+
+ hirq = xive_scan_interrupts(xc, false);
+
+ DBG_VERBOSE("get_irq: got irq 0x%x, new pending=0x%02x\n",
+ hirq, xc->pending_prio);
+
+ /* Return pending interrupt if any */
+ if (hirq == XIVE_BAD_IRQ)
+ return 0;
+ return hirq;
+}
+
+
+static void xive_do_queue_eoi(struct xive_cpu *xc)
+{
+ if (xive_scan_interrupts(xc, true) != 0) {
+ DBG_VERBOSE("eoi: pending=0x%02x\n", xc->pending_prio);
+ force_external_irq_replay();
+ }
+}
+
+static u8 xive_poke_esb(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = in_be64(xd->eoi_mmio + offset);
+
+ return (u8)val;
+}
+
+static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd)
+{
+ /* If the XIVE supports the new "store EOI facility, use it */
+ if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
+ out_be64(xd->eoi_mmio, 0);
+ else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
+ if (WARN_ON_ONCE(!xive_ops->eoi))
+ return;
+ xive_ops->eoi(hw_irq);
+ } else {
+ uint8_t eoi_val;
+
+ /*
+ * Otherwise for EOI, we use the special MMIO that does
+ * a clear of both P and Q and returns the old Q.
+ *
+ * This allows us to then do a re-trigger if Q was set
+ * rather than synthetizing an interrupt in software
+ */
+ eoi_val = xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
+ DBG_VERBOSE("eoi_val=%x\n", offset, eoi_val);
+
+ if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
+ return;
+
+ /* Re-trigger */
+ if (xd->trig_mmio)
+ out_be64(xd->trig_mmio, 0);
+ }
+
+}
+
+static void xive_irq_eoi(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+
+ DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
+ d->irq, irqd_to_hwirq(d), xc->pending_prio);
+
+ if (!irqd_irq_disabled(d))
+ xive_do_source_eoi(irqd_to_hwirq(d), xd);
+
+ /*
+ * Clear saved_p to indicate that it's no longer occupying
+ * a queue slot on the target queue
+ */
+ xd->saved_p = false;
+
+ xive_do_queue_eoi(xc);
+}
+
+static void xive_do_source_set_mask(struct xive_irq_data *xd,
+ bool masked)
+{
+ if (masked)
+ xive_poke_esb(xd, XIVE_ESB_SET_PQ_01);
+ else
+ xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
+}
+
+static bool xive_try_pick_target(int cpu)
+{
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ struct xive_q *q = &xc->queue[xive_irq_priority];
+ int max;
+
+ /* Calculate max number of interrupts in that queue.
+ *
+ * We leave a gap of 1 just in case...
+ */
+ max = (q->msk + 1) - 1;
+ return !!atomic_add_unless(&q->count, 1, max);
+}
+
+static void xive_dec_target_count(int cpu)
+{
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ struct xive_q *q = &xc->queue[xive_irq_priority];
+
+ if (WARN_ON(cpu < 0))
+ return;
+
+ /*
+ * We increment the "pending count" which will be used
+ * to decrement the target queue count whenever it's next
+ * processed and found empty. This ensure that we don't
+ * decrement while we still have the interrupt there
+ * occupying a slot.
+ */
+ atomic_inc(&q->pending_count);
+}
+
+static int xive_find_target_in_mask(const struct cpumask *mask,
+ unsigned int fuzz)
+{
+ int cpu, first, num, i;
+
+ /* Pick up a starting point CPU in the mask based on fuzz */
+ num = cpumask_weight(mask);
+ first = (fuzz++) % num;
+
+ /* Locate it */
+ cpu = cpumask_first(mask);
+ for (i = 0; i < first; i++)
+ cpu = cpumask_next(cpu, mask);
+ first = cpu;
+
+ /*
+ * Now go through the entire mask until we find a valid
+ * target.
+ */
+ for (;;) {
+ /*
+ * We re-check online as the fallback case passes us
+ * an untested affinity mask
+ */
+ if (cpu_online(cpu) && xive_try_pick_target(cpu))
+ return cpu;
+ cpu = cpumask_next(cpu, mask);
+ if (cpu == first)
+ break;
+ }
+ return -1;
+}
+
+static int xive_pick_irq_target(struct irq_data *d,
+ const struct cpumask *affinity)
+{
+ static unsigned int fuzz;
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ cpumask_var_t mask;
+ int cpu = -1;
+
+ /*
+ * Pick a target CPU for an interrupt. This is done at
+ * startup or if the affinity is changed in a way that
+ * invalidates the current target.
+ */
+
+ /* If we have chip IDs, first we try to build a mask of
+ * CPUs matching ther CPU and find a target in there
+ */
+ if (xd->src_chip != XIVE_INVALID_CHIP_ID &&
+ zalloc_cpumask_var(&mask, GFP_ATOMIC)) {
+ /* Build a mask of matching chip IDs */
+ for_each_cpu_and(cpu, affinity, cpu_online_mask) {
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ if (xc->chip_id == xd->src_chip)
+ cpumask_set_cpu(cpu, mask);
+ }
+ /* Try to find a target */
+ if (!cpumask_empty(mask))
+ cpu = xive_find_target_in_mask(mask, fuzz++);
+ free_cpumask_var(mask);
+ if (cpu >= 0)
+ return cpu;
+ fuzz--;
+ }
+
+ /* No chip IDs, fallback to using the affinity mask */
+ return xive_find_target_in_mask(affinity, fuzz++);
+}
+
+static unsigned int xive_irq_startup(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ int target, rc;
+
+ DBG("xive_irq_startup: irq %d [0x%x] data @%p\n",
+ d->irq, hw_irq, d);
+
+#ifdef CONFIG_PCI_MSI
+ /*
+ * The generic MSI code returns with the interrupt disabled on the
+ * card, using the MSI mask bits. Firmware doesn't appear to unmask
+ * at that level, so we do it here by hand.
+ */
+ if (irq_data_get_msi_desc(d))
+ pci_msi_unmask_irq(d);
+#endif
+
+ /* Pick a target */
+ target = xive_pick_irq_target(d, irq_data_get_affinity_mask(d));
+ if (target == XIVE_INVALID_TARGET) {
+ /* Try again breaking affinity */
+ target = xive_pick_irq_target(d, cpu_online_mask);
+ if (target == XIVE_INVALID_TARGET)
+ return -ENXIO;
+ pr_warn("XIVE: irq %d started with broken affinity\n",
+ d->irq);
+ }
+ xd->target = target;
+
+ /*
+ * Configure the logical number to be the Linux IRQ number
+ * and set the target queue
+ */
+ rc = xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(target),
+ xive_irq_priority, d->irq);
+ if (rc)
+ return rc;
+
+ /* Unmask the ESB */
+ xive_do_source_set_mask(xd, false);
+
+ return 0;
+}
+
+static void xive_irq_shutdown(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+
+ DBG("xive_irq_shutdown: irq %d [0x%x] data @%p\n",
+ d->irq, hw_irq, d);
+
+ if (WARN_ON(xd->target == XIVE_INVALID_TARGET))
+ return;
+
+ /* Mask the interrupt at the source */
+ xive_do_source_set_mask(xd, true);
+
+ /* Mask the interrupt in HW in the IVT/EAS */
+ xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ 0xff, hw_irq);
+
+ xive_dec_target_count(xd->target);
+ xd->target = XIVE_INVALID_TARGET;
+}
+
+static void xive_irq_unmask(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ DBG("xive_irq_unmask: irq %d data @%p\n", d->irq, xd);
+
+ /*
+ * This is a workaround for PCI LSI problems on P9, for
+ * these, we call FW to set the mask. The problems might
+ * be fixed by P9 DD2.0, if that is the case, we will make
+ * this a DD1 workaround only
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ xive_irq_priority, d->irq);
+ return;
+ }
+
+ xive_do_source_set_mask(xd, false);
+}
+
+static void xive_irq_mask(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ DBG("xive_irq_mask: irq %d data @%p\n", d->irq, xd);
+
+ /*
+ * This is a workaround for PCI LSI problems on P9, for
+ * these, we call OPAL to set the mask. The problems might
+ * be fixed by P9 DD2.0, if that is the case, we will make
+ * this a DD1 workaround only
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ 0xff, d->irq);
+ return;
+ }
+
+ xive_do_source_set_mask(xd, true);
+}
+
+static int xive_irq_set_affinity(struct irq_data *d,
+ const struct cpumask *cpumask,
+ bool force)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ u32 target, old_target;
+ int rc = 0;
+
+ DBG("xive_irq_set_affinity: irq %d\n", d->irq);
+
+ /* Is this valid ? */
+ if (cpumask_any_and(cpumask, cpu_online_mask) >= nr_cpu_ids)
+ return -EINVAL;
+
+ /* If existing target is already in the new mask, and is
+ * online then do nothing.
+ */
+ if (cpu_online(xd->target) &&
+ cpumask_test_cpu(xd->target, cpumask))
+ return IRQ_SET_MASK_OK;
+
+ /* Pick a new target */
+ target = xive_pick_irq_target(d, cpumask);
+
+ /* No target found */
+ if (target == XIVE_INVALID_TARGET)
+ return -ENXIO;
+
+ old_target = xd->target;
+
+ /*
+ * Only configure the irq if it's not currently passed-through to
+ * a KVM guest
+ */
+ rc = xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(target),
+ xive_irq_priority, d->irq);
+ if (rc < 0) {
+ pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
+ return rc;
+ }
+
+ DBG(" target: 0x%x\n", target);
+ xd->target = target;
+
+ /* Give up previous target */
+ if (old_target != XIVE_INVALID_TARGET)
+ xive_dec_target_count(old_target);
+
+ return IRQ_SET_MASK_OK;
+}
+
+static int xive_irq_set_type(struct irq_data *d, unsigned int flow_type)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ /*
+ * We only support these. This has really no effect other than setting
+ * the corresponding descriptor bits mind you but those will in turn
+ * affect the resend function when re-enabling an edge interrupt.
+ *
+ * Set set the default to edge as explained in map().
+ */
+ if (flow_type == IRQ_TYPE_DEFAULT || flow_type == IRQ_TYPE_NONE)
+ flow_type = IRQ_TYPE_EDGE_RISING;
+
+ if (flow_type != IRQ_TYPE_EDGE_RISING &&
+ flow_type != IRQ_TYPE_LEVEL_LOW)
+ return -EINVAL;
+
+ irqd_set_trigger_type(d, flow_type);
+
+ /*
+ * Double check it matches what the FW thinks
+ *
+ * NOTE: We don't know yet if the PAPR interface will provide
+ * the LSI vs MSI information appart from the device-tree so
+ * this check might have to move into an optional backend call
+ * that is specific to the native backend
+ */
+ if ((flow_type == IRQ_TYPE_LEVEL_LOW) !=
+ !!(xd->flags & XIVE_IRQ_FLAG_LSI))
+ pr_warn("XIVE: Interrupt %d (HW 0x%x) type mismatch,"
+ " Linux says %s, FW says %s\n",
+ d->irq, (u32)irqd_to_hwirq(d),
+ (flow_type == IRQ_TYPE_LEVEL_LOW) ? "Level" : "Edge",
+ (xd->flags & XIVE_IRQ_FLAG_LSI) ? "Level" : "Edge");
+
+ return IRQ_SET_MASK_OK_NOCOPY;
+}
+
+static int xive_irq_retrigger(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ /* This should be only for MSIs */
+ if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
+ return 0;
+
+ /*
+ * To perform a retrigger, we first set the PQ bits to
+ * 11, then perform an EOI.
+ */
+ xive_poke_esb(xd, XIVE_ESB_SET_PQ_11);
+
+ /*
+ * Note: We pass "0" to the hw_irq argument in order to
+ * avoid calling into the backend EOI code which we don't
+ * want to do in the case of a re-trigger. Backends typically
+ * only do EOI for LSIs anyway.
+ */
+ xive_do_source_eoi(0, xd);
+
+ return 1;
+}
+
+static struct irq_chip xive_irq_chip = {
+ .name = "XIVE-IRQ",
+ .irq_startup = xive_irq_startup,
+ .irq_shutdown = xive_irq_shutdown,
+ .irq_eoi = xive_irq_eoi,
+ .irq_mask = xive_irq_mask,
+ .irq_unmask = xive_irq_unmask,
+ .irq_set_affinity = xive_irq_set_affinity,
+ .irq_set_type = xive_irq_set_type,
+ .irq_retrigger = xive_irq_retrigger,
+};
+
+bool is_xive_irq(struct irq_chip *chip)
+{
+ return chip == &xive_irq_chip;
+}
+
+void xive_cleanup_irq_data(struct xive_irq_data *xd)
+{
+ if (xd->eoi_mmio) {
+ iounmap(xd->eoi_mmio);
+ if (xd->eoi_mmio == xd->trig_mmio)
+ xd->trig_mmio = NULL;
+ xd->eoi_mmio = NULL;
+ }
+ if (xd->trig_mmio) {
+ iounmap(xd->trig_mmio);
+ xd->trig_mmio = NULL;
+ }
+}
+
+static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
+{
+ struct xive_irq_data *xd;
+ int rc;
+
+ xd = kzalloc(sizeof(struct xive_irq_data), GFP_KERNEL);
+ if (!xd)
+ return -ENOMEM;
+ rc = xive_ops->populate_irq_data(hw, xd);
+ if (rc) {
+ kfree(xd);
+ return rc;
+ }
+ xd->target = XIVE_INVALID_TARGET;
+ irq_set_handler_data(virq, xd);
+
+ return 0;
+}
+
+static void xive_irq_free_data(unsigned int virq)
+{
+ struct xive_irq_data *xd = irq_get_handler_data(virq);
+
+ if (!xd)
+ return;
+ irq_set_handler_data(virq, NULL);
+ xive_cleanup_irq_data(xd);
+ kfree(xd);
+}
+
+#ifdef CONFIG_SMP
+
+static void xive_cause_ipi(int cpu, unsigned long msg)
+{
+ struct xive_cpu *xc;
+ struct xive_irq_data *xd;
+
+ xc = per_cpu(xive_cpu, cpu);
+
+ DBG_VERBOSE("IPI msg#%ld CPU %d -> %d (HW IRQ 0x%x)\n",
+ msg, smp_processor_id(), cpu, xc->hw_ipi);
+
+ xd = &xc->ipi_data;
+ if (WARN_ON(!xd->trig_mmio))
+ return;
+ out_be64(xd->trig_mmio, 0);
+}
+
+static irqreturn_t xive_muxed_ipi_action(int irq, void *dev_id)
+{
+ return smp_ipi_demux();
+}
+
+static void xive_ipi_eoi(struct irq_data *d)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+
+ /* Handle possible race with unplug and drop stale IPIs */
+ if (!xc)
+ return;
+ xive_do_source_eoi(xc->hw_ipi, &xc->ipi_data);
+ xive_do_queue_eoi(xc);
+}
+
+static void xive_ipi_unmask(struct irq_data *d)
+{
+ /* Nothing to do, we never mask IPIs, but the callback
+ * must exist
+ */
+}
+
+static void xive_ipi_mask(struct irq_data *d)
+{
+ /* Nothing to do, we never mask IPIs, but the callback
+ * must exist
+ */
+}
+
+static struct irq_chip xive_ipi_chip = {
+ .name = "XIVE-IPI",
+ .irq_eoi = xive_ipi_eoi,
+ .irq_mask = xive_ipi_mask,
+ .irq_unmask = xive_ipi_unmask,
+};
+
+static void __init xive_request_ipi(void)
+{
+ unsigned int virq;
+
+ /* Initialize it */
+ virq = irq_create_mapping(xive_irq_domain, 0);
+ xive_ipi_irq = virq;
+
+ BUG_ON(request_irq(virq, xive_muxed_ipi_action,
+ IRQF_PERCPU | IRQF_NO_THREAD, "IPI", NULL));
+}
+
+static int xive_setup_cpu_ipi(unsigned int cpu)
+{
+ struct xive_cpu *xc;
+ int rc;
+
+ pr_debug("XIVE: Setting up IPI for CPU %d\n", cpu);
+
+ xc = per_cpu(xive_cpu, cpu);
+
+ /* Check if we are already setup */
+ if (xc->hw_ipi != 0)
+ return 0;
+
+ /* Grab an IPI from the backend, this will populate xc->hw_ipi */
+ if (xive_ops->get_ipi(cpu, xc))
+ return -EIO;
+
+ /* Populate the IRQ data in the xive_cpu structure and
+ * configure the HW / enable the IPIs
+ */
+ rc = xive_ops->populate_irq_data(xc->hw_ipi, &xc->ipi_data);
+ if (rc) {
+ pr_err("XIVE: Failed to populate IPI data on CPU %d\n", cpu);
+ return -EIO;
+ }
+ rc = xive_ops->configure_irq(xc->hw_ipi,
+ get_hard_smp_processor_id(cpu),
+ xive_irq_priority, xive_ipi_irq);
+ if (rc) {
+ pr_err("XIVE: Failed to map IPI CPU %d\n", cpu);
+ return -EIO;
+ }
+ DBG("XIVE: CPU %d HW IPI %x, virq %d, trig_mmio=%p\n", cpu,
+ xc->hw_ipi, xive_ipi_irq, xc->ipi_data.trig_mmio);
+
+ /* Unmask it */
+ xive_do_source_set_mask(&xc->ipi_data, false);
+
+ return 0;
+}
+
+static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
+{
+ /* Disable the IPI and free the IRQ data */
+
+ /* Already cleaned up ? */
+ if (xc->hw_ipi == 0)
+ return;
+
+ /* Mask the IPI */
+ xive_do_source_set_mask(&xc->ipi_data, true);
+
+ /*
+ * Note: We don't call xive_cleanup_irq_data() to free
+ * the mappings as this is called from an IPI on kexec
+ * which is not a safe environment to call iounmap()
+ */
+
+ /* Deconfigure/mask in the backend */
+ xive_ops->configure_irq(xc->hw_ipi, hard_smp_processor_id(),
+ 0xff, xive_ipi_irq);
+
+ /* Free the IPIs in the backend */
+ xive_ops->put_ipi(cpu, xc);
+}
+
+void __init xive_smp_probe(void)
+{
+ smp_ops->cause_ipi = xive_cause_ipi;
+
+ /* Register the IPI */
+ xive_request_ipi();
+
+ /* Allocate and setup IPI for the boot CPU */
+ xive_setup_cpu_ipi(smp_processor_id());
+}
+
+#endif /* CONFIG_SMP */
+
+static int xive_irq_domain_map(struct irq_domain *h, unsigned int virq,
+ irq_hw_number_t hw)
+{
+ int rc;
+
+ /*
+ * Mark interrupts as edge sensitive by default so that resend
+ * actually works. Will fix that up below if needed.
+ */
+ irq_clear_status_flags(virq, IRQ_LEVEL);
+
+ /* IPIs are special and come up with HW number 0 */
+ if (hw == 0) {
+ /*
+ * IPIs are marked per-cpu. We use separate HW interrupts under
+ * the hood but associated with the same "linux" interrupt
+ */
+ irq_set_chip_and_handler(virq, &xive_ipi_chip,
+ handle_percpu_irq);
+ return 0;
+ }
+
+ rc = xive_irq_alloc_data(virq, hw);
+ if (rc)
+ return rc;
+
+ irq_set_chip_and_handler(virq, &xive_irq_chip, handle_fasteoi_irq);
+
+ return 0;
+}
+
+static void xive_irq_domain_unmap(struct irq_domain *d, unsigned int virq)
+{
+ struct irq_data *data = irq_get_irq_data(virq);
+ unsigned int hw_irq;
+
+ if (!data)
+ return;
+ hw_irq = (unsigned int)irqd_to_hwirq(data);
+ if (hw_irq)
+ xive_irq_free_data(virq);
+}
+
+static int xive_irq_domain_xlate(struct irq_domain *h, struct device_node *ct,
+ const u32 *intspec, unsigned int intsize,
+ irq_hw_number_t *out_hwirq, unsigned int *out_flags)
+
+{
+ *out_hwirq = intspec[0];
+
+ /*
+ * If intsize is at least 2, we look for the type in the second cell,
+ * we assume the LSB indicates a level interrupt.
+ */
+ if (intsize > 1) {
+ if (intspec[1] & 1)
+ *out_flags = IRQ_TYPE_LEVEL_LOW;
+ else
+ *out_flags = IRQ_TYPE_EDGE_RISING;
+ } else
+ *out_flags = IRQ_TYPE_LEVEL_LOW;
+
+ return 0;
+}
+
+static int xive_irq_domain_match(struct irq_domain *h, struct device_node *node,
+ enum irq_domain_bus_token bus_token)
+{
+ return xive_ops->match(node);
+}
+
+static const struct irq_domain_ops xive_irq_domain_ops = {
+ .match = xive_irq_domain_match,
+ .map = xive_irq_domain_map,
+ .unmap = xive_irq_domain_unmap,
+ .xlate = xive_irq_domain_xlate,
+};
+
+static void __init xive_init_host(void)
+{
+ xive_irq_domain = irq_domain_add_nomap(NULL, XIVE_MAX_IRQ,
+ &xive_irq_domain_ops, NULL);
+ BUG_ON(xive_irq_domain == NULL);
+ irq_set_default_host(xive_irq_domain);
+}
+
+static void xive_cleanup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
+{
+ if (xc->queue[xive_irq_priority].qpage)
+ xive_ops->cleanup_queue(cpu, xc, xive_irq_priority);
+}
+
+static int xive_setup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
+{
+ int rc = 0;
+
+ /* We setup 1 queues for now with a 64k page */
+ if (!xc->queue[xive_irq_priority].qpage)
+ rc = xive_ops->setup_queue(cpu, xc, xive_irq_priority);
+
+ return rc;
+}
+
+static int xive_prepare_cpu(unsigned int cpu)
+{
+ struct xive_cpu *xc;
+
+ xc = per_cpu(xive_cpu, cpu);
+ if (!xc) {
+ struct device_node *np;
+
+ xc = kzalloc_node(sizeof(struct xive_cpu),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!xc)
+ return -ENOMEM;
+ np = of_get_cpu_node(cpu, NULL);
+ if (np)
+ xc->chip_id = of_get_ibm_chip_id(np);
+ of_node_put(np);
+
+ per_cpu(xive_cpu, cpu) = xc;
+ }
+
+ /* Setup EQs if not already */
+ return xive_setup_cpu_queues(cpu, xc);
+}
+
+static void xive_setup_cpu(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+
+ /* Debug: Dump the TM state */
+ DBG("CPU %d [HW 0x%02x] VT=%02x\n",
+ smp_processor_id(), hard_smp_processor_id(),
+ in_8(xive_tm_area + xive_tm_offset + TM_WORD2));
+
+ /* The backend might have additional things to do */
+ if (xive_ops->setup_cpu)
+ xive_ops->setup_cpu(smp_processor_id(), xc);
+
+ /* Set CPPR to 0xff to enable flow of interrupts */
+ xc->cppr = 0xff;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
+}
+
+#ifdef CONFIG_SMP
+void xive_smp_setup_cpu(void)
+{
+ DBG("XIVE: SMP setup CPU %d\n", smp_processor_id());
+
+ /* This will have already been done on the boot CPU */
+ if (smp_processor_id() != boot_cpuid)
+ xive_setup_cpu();
+
+}
+
+int xive_smp_prepare_cpu(unsigned int cpu)
+{
+ int rc;
+
+ /* Allocate per-CPU data and queues */
+ rc = xive_prepare_cpu(cpu);
+ if (rc)
+ return rc;
+
+ /* Allocate and setup IPI for the new CPU */
+ return xive_setup_cpu_ipi(cpu);
+}
+
+#ifdef CONFIG_HOTPLUG_CPU
+static void xive_flush_cpu_queue(unsigned int cpu, struct xive_cpu *xc)
+{
+ u32 irq;
+
+ /* We assume local irqs are disabled */
+ WARN_ON(!irqs_disabled());
+
+ /* Check what's already in the CPU queue */
+ while ((irq = xive_scan_interrupts(xc, false)) != 0) {
+ /*
+ * We need to re-route that interrupt to its new distination.
+ * First get and lock the descriptor
+ */
+ struct irq_desc *desc = irq_to_desc(irq);
+ struct irq_data *d = irq_desc_get_irq_data(desc);
+ struct xive_irq_data *xd;
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+
+ /*
+ * Ignore anything that isn't a XIVE irq and ignore
+ * IPIs, so can just be dropped.
+ */
+ if (d->domain != xive_irq_domain || hw_irq == 0)
+ continue;
+#ifdef DEBUG_FLUSH
+ pr_info("CPU %d: Got irq %d while offline, re-routing...\n",
+ cpu, irq);
+#endif
+ raw_spin_lock(&desc->lock);
+ xd = irq_desc_get_handler_data(desc);
+
+ /* For LSIs, we EOI, this will cause a resend if it's
+ * still asserted. Otherwise do an MSI retrigger
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_LSI)
+ xive_do_source_eoi(irqd_to_hwirq(d), xd);
+ else
+ xive_irq_retrigger(d);
+ raw_spin_unlock(&desc->lock);
+ }
+}
+
+void xive_smp_disable_cpu(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ unsigned int cpu = smp_processor_id();
+
+ /* Migrate interrupts away from the CPU */
+ irq_migrate_all_off_this_cpu();
+
+ /* Set CPPR to 0 to disable flow of interrupts */
+ xc->cppr = 0;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
+
+ /* Flush everything still in the queue */
+ xive_flush_cpu_queue(cpu, xc);
+
+ /* Re-enable CPPR */
+ xc->cppr = 0xff;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
+}
+
+void xive_flush_interrupt(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ unsigned int cpu = smp_processor_id();
+
+ /* Called if an interrupt occurs while the CPU is hot unplugged */
+ xive_flush_cpu_queue(cpu, xc);
+}
+
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#endif /* CONFIG_SMP */
+
+void xive_kexec_teardown_cpu(int secondary)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ unsigned int cpu = smp_processor_id();
+
+ /* Set CPPR to 0 to disable flow of interrupts */
+ xc->cppr = 0;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
+
+ /* Backend cleanup if any */
+ if (xive_ops->teardown_cpu)
+ xive_ops->teardown_cpu(cpu, xc);
+
+ /* Get rid of IPI */
+ xive_cleanup_cpu_ipi(cpu, xc);
+
+ /* Disable and free the queues */
+ xive_cleanup_cpu_queues(cpu, xc);
+}
+
+void xive_shutdown(void)
+{
+ xive_ops->shutdown();
+}
+
+bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
+ u8 max_prio)
+{
+ xive_tm_area = area;
+ xive_tm_offset = offset;
+ xive_ops = ops;
+ xive_irq_priority = max_prio;
+
+ ppc_md.get_irq = xive_get_irq;
+ __xive_enabled = true;
+
+ DBG("Initializing host..\n");
+ xive_init_host();
+
+ DBG("Initializing boot CPU..\n");
+
+ /* Allocate per-CPU data and queues */
+ xive_prepare_cpu(smp_processor_id());
+
+ /* Get ready for interrupts */
+ xive_setup_cpu();
+
+ pr_info("XIVE: Interrupt handling intialized with %s backend\n",
+ xive_ops->name);
+ pr_info("XIVE: Using priority %d for all interrupts\n", max_prio);
+
+ return true;
+}
+
+static int __init xive_off(char *arg)
+{
+ xive_cmdline_disabled = true;
+ return 0;
+}
+__setup("xive=off", xive_off);
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
new file mode 100644
index 0000000..26cc6bf
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -0,0 +1,604 @@
+/*
+ * Copyright 2016,2017 IBM Corporation.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#include <linux/types.h>
+#include <linux/irq.h>
+#include <linux/debugfs.h>
+#include <linux/smp.h>
+#include <linux/interrupt.h>
+#include <linux/seq_file.h>
+#include <linux/init.h>
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/cpumask.h>
+#include <linux/mm.h>
+
+#include <asm/prom.h>
+#include <asm/io.h>
+#include <asm/smp.h>
+#include <asm/irq.h>
+#include <asm/errno.h>
+#include <asm/xive.h>
+#include <asm/opal.h>
+
+#include "xive-regs.h"
+#include "xive-internal.h"
+
+#define DBG(fmt...) pr_devel("XIVE: " fmt)
+
+/* Enable this for using queue MMIO page for EOI. We don't currently
+ * use it as we always notify
+ */
+#undef USE_QUEUE_MMIO
+
+static u32 xive_provision_size;
+static u32 *xive_provision_chips;
+static u32 xive_provision_chip_count;
+static u32 xive_queue_shift;
+static u32 xive_pool_vps = XIVE_INVALID_VP;
+static struct kmem_cache *xive_provision_cache;
+
+int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
+{
+ __be64 flags, eoi_page, trig_page;
+ __be32 esb_shift, src_chip;
+ u64 opal_flags;
+ s64 rc;
+
+ memset(data, 0, sizeof(*data));
+
+ rc = opal_xive_get_irq_info(hw_irq, &flags, &eoi_page, &trig_page,
+ &esb_shift, &src_chip);
+ if (rc) {
+ pr_err("XIVE: opal_xive_get_irq_info(0x%x) returned %lld\n",
+ hw_irq, rc);
+ return -EINVAL;
+ }
+
+ opal_flags = be64_to_cpu(flags);
+ if (opal_flags & OPAL_XIVE_IRQ_STORE_EOI)
+ data->flags |= XIVE_IRQ_FLAG_STORE_EOI;
+ if (opal_flags & OPAL_XIVE_IRQ_LSI)
+ data->flags |= XIVE_IRQ_FLAG_LSI;
+ if (opal_flags & OPAL_XIVE_IRQ_SHIFT_BUG)
+ data->flags |= XIVE_IRQ_FLAG_SHIFT_BUG;
+ if (opal_flags & OPAL_XIVE_IRQ_MASK_VIA_FW)
+ data->flags |= XIVE_IRQ_FLAG_MASK_FW;
+ if (opal_flags & OPAL_XIVE_IRQ_EOI_VIA_FW)
+ data->flags |= XIVE_IRQ_FLAG_EOI_FW;
+ data->eoi_page = be64_to_cpu(eoi_page);
+ data->trig_page = be64_to_cpu(trig_page);
+ data->esb_shift = be32_to_cpu(esb_shift);
+ data->src_chip = be32_to_cpu(src_chip);
+
+ data->eoi_mmio = ioremap(data->eoi_page, 1u << data->esb_shift);
+ if (!data->eoi_mmio) {
+ pr_err("XIVE: Failed to map EOI page for irq 0x%x\n", hw_irq);
+ return -ENOMEM;
+ }
+
+ if (!data->trig_page)
+ return 0;
+ if (data->trig_page == data->eoi_page) {
+ data->trig_mmio = data->eoi_mmio;
+ return 0;
+ }
+
+ data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
+ if (!data->trig_mmio) {
+ pr_err("XIVE: Failed to map trigger page for irq 0x%x\n", hw_irq);
+ return -ENOMEM;
+ }
+ return 0;
+}
+
+int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
+{
+ s64 rc;
+
+ for (;;) {
+ rc = opal_xive_set_irq_config(hw_irq, target, prio, sw_irq);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ return rc == 0 ? 0 : -ENXIO;
+}
+
+/* This can be called multiple time to change a queue configuration */
+int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
+ __be32 *qpage, u32 order, bool can_escalate)
+{
+ s64 rc = 0;
+ __be64 qeoi_page_be;
+ __be32 esc_irq_be;
+ u64 flags, qpage_phys;
+
+ /* If there's an actual queue page, clean it */
+ if (order) {
+ BUG_ON(!qpage);
+ qpage_phys = __pa(qpage);
+ } else
+ qpage_phys = 0;
+
+ /* Initialize the rest of the fields */
+ q->msk = order ? ((1u << (order - 2)) - 1) : 0;
+ q->idx = 0;
+ q->toggle = 0;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, NULL, NULL,
+ &qeoi_page_be,
+ &esc_irq_be,
+ NULL);
+ if (rc) {
+ pr_err("XIVE: Error %lld getting queue info prio %d\n",
+ rc, prio);
+ rc = -EIO;
+ goto fail;
+ }
+ q->eoi_phys = be64_to_cpu(qeoi_page_be);
+
+#ifdef USE_QUEUE_MMIO
+ if (!q->eoi_mmio)
+ q->eoi_mmio = ioremap(q->eoi_phys, PAGE_SIZE);
+ if (!q->eoi_mmio) {
+ pr_err("XIVE: Failed to map queue MMIO prio %d CPU %d\n",
+ rc, prio, cpu);
+ rc = -ENOMEM;
+ goto fail;
+ }
+#endif /* USE_QUEUE_MMIO */
+
+ /* Default flags */
+ flags = OPAL_XIVE_EQ_ALWAYS_NOTIFY | OPAL_XIVE_EQ_ENABLED;
+
+ /* Escalation needed ? */
+ if (can_escalate) {
+ q->esc_irq = be32_to_cpu(esc_irq_be);
+ flags |= OPAL_XIVE_EQ_ESCALATE;
+ }
+
+ /* Configure and enable the queue in HW */
+ for (;;) {
+ rc = opal_xive_set_queue_info(vp_id, prio, qpage_phys, order, flags);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ if (rc) {
+ pr_err("XIVE: Error %lld setting queue for prio %d\n",
+ rc, prio);
+ rc = -EIO;
+ } else {
+ /*
+ * KVM code requires all of the above to be visible before
+ * q->qpage is set due to how it manages IPI EOIs
+ */
+ wmb();
+ q->qpage = qpage;
+ }
+ fail:
+ return rc;
+}
+
+static void __xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
+{
+ s64 rc;
+
+ /* Disable the queue in HW */
+ for (;;) {
+ rc = opal_xive_set_queue_info(vp_id, prio, 0, 0, 0);
+ break;
+ msleep(1);
+ }
+ if (rc)
+ pr_err("XIVE: Error %lld disabling queue for prio %d\n",
+ rc, prio);
+}
+
+void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
+{
+ __xive_native_disable_queue(vp_id, q, prio);
+
+ if (q->eoi_mmio)
+ iounmap(q->eoi_mmio);
+ q->eoi_mmio = NULL;
+}
+
+static int xive_native_setup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
+{
+ struct xive_q *q = &xc->queue[prio];
+ unsigned int alloc_order;
+ struct page *pages;
+ __be32 *qpage;
+
+ alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
+ (xive_queue_shift - PAGE_SHIFT) : 0;
+ pages = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL, alloc_order);
+ if (!pages)
+ return -ENOMEM;
+ qpage = (__be32 *)page_address(pages);
+ memset(qpage, 0, 1 << xive_queue_shift);
+ return xive_native_configure_queue(get_hard_smp_processor_id(cpu),
+ q, prio, qpage, xive_queue_shift, false);
+}
+
+static void xive_native_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
+{
+ struct xive_q *q = &xc->queue[prio];
+ unsigned int alloc_order;
+
+ /*
+ * We use the variant with no iounmap as this is called on exec
+ * from an IPI and iounmap isn't safe
+ */
+ __xive_native_disable_queue(get_hard_smp_processor_id(cpu), q, prio);
+ alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
+ (xive_queue_shift - PAGE_SHIFT) : 0;
+ free_pages((unsigned long)q->qpage, alloc_order);
+ q->qpage = NULL;
+}
+
+static bool xive_native_match(struct device_node *node)
+{
+ return of_device_is_compatible(node, "ibm,opal-xive-vc");
+}
+
+#ifdef CONFIG_SMP
+static int xive_native_get_ipi(unsigned int cpu, struct xive_cpu *xc)
+{
+ struct device_node *np;
+ unsigned int chip_id;
+ s64 irq;
+
+ /* Find the chip ID */
+ np = of_get_cpu_node(cpu, NULL);
+ if (np) {
+ if (of_property_read_u32(np, "ibm,chip-id", &chip_id) < 0)
+ chip_id = 0;
+ }
+
+ /* Allocate an IPI and populate info about it */
+ for (;;) {
+ irq = opal_xive_allocate_irq(chip_id);
+ if (irq == OPAL_BUSY) {
+ msleep(1);
+ continue;
+ }
+ if (irq < 0) {
+ pr_err("XIVE: Failed to allocate IPI on CPU %d\n",
+ cpu);
+ return -ENXIO;
+ }
+ xc->hw_ipi = irq;
+ break;
+ }
+ return 0;
+}
+
+u32 xive_native_alloc_irq(void)
+{
+ s64 rc;
+
+ for (;;) {
+ rc = opal_xive_allocate_irq(OPAL_XIVE_ANY_CHIP);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ if (rc < 0)
+ return 0;
+ return rc;
+}
+
+void xive_native_free_irq(u32 irq)
+{
+ for (;;) {
+ s64 rc = opal_xive_free_irq(irq);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+}
+
+static void xive_native_put_ipi(unsigned int cpu, struct xive_cpu *xc)
+{
+ s64 rc;
+
+ /* Free the IPI */
+ if (!xc->hw_ipi)
+ return;
+ for (;;) {
+ rc = opal_xive_free_irq(xc->hw_ipi);
+ if (rc == OPAL_BUSY) {
+ msleep(1);
+ continue;
+ }
+ xc->hw_ipi = 0;
+ break;
+ }
+}
+#endif /* CONFIG_SMP */
+
+static void xive_native_shutdown(void)
+{
+ /* Switch the XIVE to emulation mode */
+ opal_xive_reset(OPAL_XIVE_MODE_EMU);
+}
+
+static void xive_native_eoi(u32 hw_irq)
+{
+ /* Not normally used except if specific interrupts need
+ * a workaround on EOI
+ */
+ opal_int_eoi(hw_irq);
+}
+
+static void xive_native_setup_cpu(unsigned int cpu, struct xive_cpu *xc)
+{
+ s64 rc;
+ u32 vp;
+ __be64 vp_cam_be;
+ u64 vp_cam;
+
+ if (xive_pool_vps == XIVE_INVALID_VP)
+ return;
+
+ /* Enable the pool VP */
+ vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
+ pr_debug("XIVE: CPU %d setting up pool VP 0x%x\n", cpu, vp);
+ for (;;) {
+ rc = opal_xive_set_vp_info(vp, OPAL_XIVE_VP_ENABLED, 0);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ if (rc) {
+ pr_err("XIVE: Failed to enable pool VP on CPU %d\n", cpu);
+ return;
+ }
+
+ /* Grab it's CAM value */
+ rc = opal_xive_get_vp_info(vp, NULL, &vp_cam_be, NULL, NULL);
+ if (rc) {
+ pr_err("XIVE: Failed to get pool VP info CPU %d\n", cpu);
+ return;
+ }
+ vp_cam = be64_to_cpu(vp_cam_be);
+
+ pr_debug("XIVE: VP CAM = %llx\n", vp_cam);
+
+ /* Push it on the CPU (set LSMFB to 0xff to skip backlog scan) */
+ pr_debug("XIVE: (Old HW value: %08x)\n",
+ in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
+ out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD0, 0xff);
+ out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2,
+ TM_QW2W2_VP | vp_cam);
+ pr_debug("XIVE: (New HW value: %08x)\n",
+ in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
+}
+
+static void xive_native_teardown_cpu(unsigned int cpu, struct xive_cpu *xc)
+{
+ s64 rc;
+ u32 vp;
+
+ if (xive_pool_vps == XIVE_INVALID_VP)
+ return;
+
+ /* Pull the pool VP from the CPU */
+ in_be64(xive_tm_area + TM_SPC_PULL_POOL_CTX);
+
+ /* Disable it */
+ vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
+ for (;;) {
+ rc = opal_xive_set_vp_info(vp, 0, 0);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+}
+
+static void xive_native_sync_source(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_EAS, hw_irq);
+}
+
+static const struct xive_ops xive_native_ops = {
+ .populate_irq_data = xive_native_populate_irq_data,
+ .configure_irq = xive_native_configure_irq,
+ .setup_queue = xive_native_setup_queue,
+ .cleanup_queue = xive_native_cleanup_queue,
+ .match = xive_native_match,
+ .shutdown = xive_native_shutdown,
+ .eoi = xive_native_eoi,
+ .setup_cpu = xive_native_setup_cpu,
+ .teardown_cpu = xive_native_teardown_cpu,
+ .sync_source = xive_native_sync_source,
+#ifdef CONFIG_SMP
+ .get_ipi = xive_native_get_ipi,
+ .put_ipi = xive_native_put_ipi,
+#endif /* CONFIG_SMP */
+ .name = "native",
+};
+
+static bool xive_parse_provisioning(struct device_node *np)
+{
+ int rc;
+
+ if (of_property_read_u32(np, "ibm,xive-provision-page-size",
+ &xive_provision_size) < 0)
+ return true;
+ rc = of_property_count_elems_of_size(np, "ibm,xive-provision-chips", 4);
+ if (rc < 0) {
+ pr_err("XIVE: Error %d getting provision chips array\n", rc);
+ return false;
+ }
+ xive_provision_chip_count = rc;
+ if (rc == 0)
+ return true;
+
+ xive_provision_chips = kzalloc(4 * xive_provision_chip_count,
+ GFP_KERNEL);
+ BUG_ON(!xive_provision_chips);
+
+ rc = of_property_read_u32_array(np, "ibm,xive-provision-chips",
+ xive_provision_chips,
+ xive_provision_chip_count);
+ if (rc < 0) {
+ pr_err("XIVE: Error %d reading provision chips array\n", rc);
+ return false;
+ }
+
+ xive_provision_cache = kmem_cache_create("xive-provision",
+ xive_provision_size,
+ xive_provision_size,
+ 0, NULL);
+ if (!xive_provision_cache) {
+ pr_err("XIVE: Failed to allocate provision cache\n");
+ return false;
+ }
+ return true;
+}
+
+u32 xive_native_default_eq_shift(void)
+{
+ return xive_queue_shift;
+}
+
+bool xive_native_init(void)
+{
+ struct device_node *np;
+ struct resource r;
+ void __iomem *tm_area;
+ struct property *prop;
+ u8 max_prio = 7;
+ const __be32 *p;
+ u32 val;
+ s64 rc;
+
+ if (xive_cmdline_disabled)
+ return false;
+
+ DBG("xive_native_init()\n");
+ np = of_find_compatible_node(NULL, NULL, "ibm,opal-xive-pe");
+ if (!np) {
+ DBG("not found !\n");
+ return false;
+ }
+ DBG("Found %s\n", np->full_name);
+
+ /* Resource 1 is HV window */
+ if (of_address_to_resource(np, 1, &r)) {
+ pr_err("XIVE: Failed to get TM area resource\n");
+ return false;
+ }
+ tm_area = ioremap(r.start, resource_size(&r));
+ if (!tm_area) {
+ pr_err("XIVE: Failed to map TM area\n");
+ return false;
+ }
+
+ /* Read number of priorities */
+ if (of_property_read_u32(np, "ibm,xive-#priorities", &val) == 0)
+ max_prio = val - 1;
+
+ /* Iterate the EQ sizes and pick one */
+ of_property_for_each_u32(np, "ibm,xive-eq-sizes", prop, p, val) {
+ xive_queue_shift = val;
+ if (val == PAGE_SHIFT)
+ break;
+ }
+
+ /* Grab size of provisionning pages */
+ xive_parse_provisioning(np);
+
+ /* Switch the XIVE to exploitation mode */
+ rc = opal_xive_reset(OPAL_XIVE_MODE_EXPL);
+ if (rc) {
+ pr_err("XIVE: Switch to exploitation mode failed"
+ " with error %lld\n", rc);
+ return false;
+ }
+
+ /* Initialize XIVE core with our backend */
+ if (!xive_core_init(&xive_native_ops, tm_area, TM_QW3_HV_PHYS,
+ max_prio)) {
+ opal_xive_reset(OPAL_XIVE_MODE_EMU);
+ return false;
+ }
+ pr_info("XIVE: Using %dkB queues\n", 1 << (xive_queue_shift - 10));
+ return true;
+}
+
+static bool xive_native_provision_pages(void)
+{
+ u32 i;
+ void *p;
+
+ for (i = 0; i < xive_provision_chip_count; i++) {
+ u32 chip = xive_provision_chips[i];
+
+ /* XXX TODO: Try to make the allocation local to the node where
+ * the chip reside
+ */
+ p = kmem_cache_alloc(xive_provision_cache, GFP_KERNEL);
+ if (!p) {
+ pr_err("XIVE: Failed to allocate provisioning page\n");
+ return false;
+ }
+ opal_xive_donate_page(chip, __pa(p));
+ }
+ return true;
+}
+
+u32 xive_native_alloc_vp_block(u32 max_vcpus)
+{
+ s64 rc;
+ u32 order;
+
+ order = fls(max_vcpus) - 1;
+ pr_info("XIVE: VP block alloc, for max VCPUs %d"
+ " use order %d\n", max_vcpus, order);
+ for (;;) {
+ rc = opal_xive_alloc_vp_block(order);
+ switch (rc) {
+ case OPAL_BUSY:
+ msleep(1);
+ break;
+ case OPAL_XIVE_PROVISIONING:
+ if (!xive_native_provision_pages())
+ return XIVE_INVALID_VP;
+ break;
+ default:
+ if (rc < 0) {
+ pr_err("XIVE: OPAL failed to allocate VCPUs"
+ " order %d, err %lld\n",
+ order, rc);
+ return XIVE_INVALID_VP;
+ }
+ return rc;
+ }
+ }
+}
+EXPORT_SYMBOL_GPL(xive_native_alloc_vp_block);
+
+void xive_native_free_vp_block(u32 vp_base)
+{
+ s64 rc;
+
+ if (vp_base == XIVE_INVALID_VP)
+ return;
+
+ rc = opal_xive_free_vp_block(vp_base);
+ if (rc < 0)
+ pr_warn("XIVE: OPAL error %lld freeing VP block\n", rc);
+}
+EXPORT_SYMBOL_GPL(xive_native_free_vp_block);
diff --git a/arch/powerpc/sysdev/xive/xive-internal.h b/arch/powerpc/sysdev/xive/xive-internal.h
new file mode 100644
index 0000000..e736fc5
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/xive-internal.h
@@ -0,0 +1,51 @@
+#ifndef __XIVE_INTERNAL_H
+#define __XIVE_INTERNAL_H
+
+/* Each CPU carry one of these with various per-CPU state */
+struct xive_cpu {
+#ifdef CONFIG_SMP
+ /* HW irq number and data of IPI */
+ u32 hw_ipi;
+ struct xive_irq_data ipi_data;
+#endif /* CONFIG_SMP */
+
+ int chip_id;
+
+ /* Queue datas. Only one is populated */
+#define XIVE_MAX_QUEUES 8
+ struct xive_q queue[XIVE_MAX_QUEUES];
+
+ /* Pending mask. Each bit corresponds to a priority that
+ * potentially has pending interrupts
+ */
+ u8 pending_prio;
+
+ /* Cache of HW CPPR */
+ u8 cppr;
+};
+
+/* Backend ops */
+struct xive_ops {
+ int (*populate_irq_data)(u32 hw_irq, struct xive_irq_data *data);
+ int (*configure_irq)(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
+ int (*setup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
+ void (*cleanup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
+ void (*setup_cpu)(unsigned int cpu, struct xive_cpu *xc);
+ void (*teardown_cpu)(unsigned int cpu, struct xive_cpu *xc);
+ bool (*match)(struct device_node *np);
+ void (*shutdown)(void);
+ void (*eoi)(u32 hw_irq);
+ void (*sync_source)(u32 hw_irq);
+#ifdef CONFIG_SMP
+ int (*get_ipi)(unsigned int cpu, struct xive_cpu *xc);
+ void (*put_ipi)(unsigned int cpu, struct xive_cpu *xc);
+#endif
+ const char *name;
+};
+
+bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
+ u8 max_prio);
+
+extern bool xive_cmdline_disabled;
+
+#endif /* __XIVE_INTERNAL_H */
diff --git a/arch/powerpc/sysdev/xive/xive-regs.h b/arch/powerpc/sysdev/xive/xive-regs.h
new file mode 100644
index 0000000..f1edb23
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/xive-regs.h
@@ -0,0 +1,88 @@
+#ifndef __XIVE_REGS_H__
+#define __XIVE_REGS_H__
+
+/*
+ * TM registers
+ */
+
+/* TM register offsets */
+#define TM_QW0_USER 0x000 /* All rings */
+#define TM_QW1_OS 0x010 /* Ring 0..2 */
+#define TM_QW2_HV_POOL 0x020 /* Ring 0..1 */
+#define TM_QW3_HV_PHYS 0x030 /* Ring 0..1 */
+
+/* Byte offsets inside a QW QW0 QW1 QW2 QW3 */
+#define TM_NSR 0x0 /* + + - + */
+#define TM_CPPR 0x1 /* - + - + */
+#define TM_IPB 0x2 /* - + + + */
+#define TM_LSMFB 0x3 /* - + + + */
+#define TM_ACK_CNT 0x4 /* - + - - */
+#define TM_INC 0x5 /* - + - + */
+#define TM_AGE 0x6 /* - + - + */
+#define TM_PIPR 0x7 /* - + - + */
+
+#define TM_WORD0 0x0
+#define TM_WORD1 0x4
+
+/* QW word 2 contains the valid bit at the top and other fields
+ * depending on the QW
+ */
+#define TM_WORD2 0x8
+#define TM_QW0W2_VU PPC_BIT32(0)
+#define TM_QW0W2_LOGIC_SERV PPC_BITMASK32(1,31) // XX 2,31 ?
+#define TM_QW1W2_VO PPC_BIT32(0)
+#define TM_QW1W2_OS_CAM PPC_BITMASK32(8,31)
+#define TM_QW2W2_VP PPC_BIT32(0)
+#define TM_QW2W2_POOL_CAM PPC_BITMASK32(8,31)
+#define TM_QW3W2_VT PPC_BIT32(0)
+#define TM_QW3W2_LP PPC_BIT32(6)
+#define TM_QW3W2_LE PPC_BIT32(7)
+#define TM_QW3W2_T PPC_BIT32(31)
+
+/* In addition to normal loads to "peek" and writes (only when invalid)
+ * using 4 and 8 bytes accesses, the above registers support these
+ * "special" byte operations:
+ *
+ * - Byte load from QW0[NSR] - User level NSR (EBB)
+ * - Byte store to QW0[NSR] - User level NSR (EBB)
+ * - Byte load/store to QW1[CPPR] and QW3[CPPR] - CPPR access
+ * - Byte load from QW3[TM_WORD2] - Read VT||00000||LP||LE on thrd 0
+ * otherwise VT||0000000
+ * - Byte store to QW3[TM_WORD2] - Set VT bit (and LP/LE if present)
+ *
+ * Then we have all these "special" CI ops at these offset that trigger
+ * all sorts of side effects:
+ */
+#define TM_SPC_ACK_EBB 0x800 /* Load8 ack EBB to reg*/
+#define TM_SPC_ACK_OS_REG 0x810 /* Load16 ack OS irq to reg */
+#define TM_SPC_PUSH_USR_CTX 0x808 /* Store32 Push/Validate user context */
+#define TM_SPC_PULL_USR_CTX 0x808 /* Load32 Pull/Invalidate user context */
+#define TM_SPC_SET_OS_PENDING 0x812 /* Store8 Set OS irq pending bit */
+#define TM_SPC_PULL_OS_CTX 0x818 /* Load32/Load64 Pull/Invalidate OS context to reg */
+#define TM_SPC_PULL_POOL_CTX 0x828 /* Load32/Load64 Pull/Invalidate Pool context to reg*/
+#define TM_SPC_ACK_HV_REG 0x830 /* Load16 ack HV irq to reg */
+#define TM_SPC_PULL_USR_CTX_OL 0xc08 /* Store8 Pull/Inval usr ctx to odd line */
+#define TM_SPC_ACK_OS_EL 0xc10 /* Store8 ack OS irq to even line */
+#define TM_SPC_ACK_HV_POOL_EL 0xc20 /* Store8 ack HV evt pool to even line */
+#define TM_SPC_ACK_HV_EL 0xc30 /* Store8 ack HV irq to even line */
+/* XXX more... */
+
+/* NSR fields for the various QW ack types */
+#define TM_QW0_NSR_EB PPC_BIT8(0)
+#define TM_QW1_NSR_EO PPC_BIT8(0)
+#define TM_QW3_NSR_HE PPC_BITMASK8(0,1)
+#define TM_QW3_NSR_HE_NONE 0
+#define TM_QW3_NSR_HE_POOL 1
+#define TM_QW3_NSR_HE_PHYS 2
+#define TM_QW3_NSR_HE_LSI 3
+#define TM_QW3_NSR_I PPC_BIT8(2)
+#define TM_QW3_NSR_GRP_LVL PPC_BIT8(3,7)
+
+/* Utilities to manipulate these (originaly from OPAL) */
+#define MASK_TO_LSH(m) (__builtin_ffsl(m) - 1)
+#define GETFIELD(m, v) (((v) & (m)) >> MASK_TO_LSH(m))
+#define SETFIELD(m, v, val) \
+ (((v) & ~(m)) | ((((typeof(v))(val)) << MASK_TO_LSH(m)) & (m)))
+
+
+#endif /* __XIVE_H__ */
diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index 16321ad..c71e919 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -48,7 +48,7 @@
#include <asm/reg.h>
#include <asm/debug.h>
#include <asm/hw_breakpoint.h>
-
+#include <asm/xive.h>
#include <asm/opal.h>
#include <asm/firmware.h>
@@ -232,7 +232,13 @@ Commands:\n\
"\
dr dump stream of raw bytes\n\
dt dump the tracing buffers (uses printk)\n\
- e print exception information\n\
+"
+#ifdef CONFIG_PPC_POWERNV
+" dx# dump xive on CPU\n\
+ dxi# dump xive irq state\n\
+ dxa dump xive on all CPUs\n"
+#endif
+" e print exception information\n\
f flush cache\n\
la lookup symbol+offset of specified address\n\
ls lookup address of specified symbol\n\
@@ -2338,6 +2344,81 @@ static void dump_pacas(void)
}
#endif
+#ifdef CONFIG_PPC_POWERNV
+static void dump_one_xive(int cpu)
+{
+ unsigned int hwid = get_hard_smp_processor_id(cpu);
+
+ opal_xive_dump(XIVE_DUMP_TM_HYP, hwid);
+ opal_xive_dump(XIVE_DUMP_TM_POOL, hwid);
+ opal_xive_dump(XIVE_DUMP_TM_OS, hwid);
+ opal_xive_dump(XIVE_DUMP_TM_USER, hwid);
+ opal_xive_dump(XIVE_DUMP_VP, hwid);
+ opal_xive_dump(XIVE_DUMP_EMU_STATE, hwid);
+
+ if (setjmp(bus_error_jmp) != 0) {
+ catch_memory_errors = 0;
+ printf("*** Error dumping xive on cpu %d\n", cpu);
+ return;
+ }
+
+ catch_memory_errors = 1;
+ sync();
+ xmon_xive_do_dump(cpu);
+ sync();
+ __delay(200);
+ catch_memory_errors = 0;
+}
+
+static void dump_all_xives(void)
+{
+ int cpu;
+
+ if (num_possible_cpus() == 0) {
+ printf("No possible cpus, use 'dx #' to dump individual cpus\n");
+ return;
+ }
+
+ for_each_possible_cpu(cpu)
+ dump_one_xive(cpu);
+}
+
+static void dump_one_xive_irq(uint32_t num)
+{
+ int64_t rc;
+ __be64 vp;
+ uint8_t prio;
+ __be32 lirq;
+
+ rc = opal_xive_get_irq_config(num, &vp, &prio, &lirq);
+ xmon_printf("IRQ 0x%x config: vp=0x%llx prio=%d lirq=0x%x (rc=%lld)\n",
+ num, be64_to_cpu(vp), prio, be32_to_cpu(lirq), rc);
+}
+
+static void dump_xives(void)
+{
+ unsigned long num;
+ int c;
+
+ c = inchar();
+ if (c == 'a') {
+ dump_all_xives();
+ return;
+ } else if (c == 'i') {
+ if (scanhex(&num))
+ dump_one_xive_irq(num);
+ return;
+ }
+
+ termch = c; /* Put c back, it wasn't 'a' */
+
+ if (scanhex(&num))
+ dump_one_xive(num);
+ else
+ dump_one_xive(xmon_owner);
+}
+#endif /* CONFIG_PPC_POWERNV */
+
static void dump_by_size(unsigned long addr, long count, int size)
{
unsigned char temp[16];
@@ -2386,6 +2467,14 @@ dump(void)
return;
}
#endif
+#ifdef CONFIG_PPC_POWERNV
+ if (c == 'x') {
+ xmon_start_pagination();
+ dump_xives();
+ xmon_end_pagination();
+ return;
+ }
+#endif
if (c == '\n')
termch = c;
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
The XIVE interrupt controller is the new interrupt controller
found in POWER9. It supports advanced virtualization capabilities
among other things.
Currently we use a set of firmware calls that simulate the old
"XICS" interrupt controller but this is fairly inefficient.
This adds the framework for using XIVE along with a native
backend which OPAL for configuration. Later, a backend allowing
the use in a KVM or PowerVM guest will also be provided.
This disables some fast path for interrupts in KVM when XIVE is
enabled as these rely on the firmware emulation code which is no
longer available when the XIVE is used natively by Linux.
A latter patch will make KVM also directly exploit the XIVE, thus
recovering the lost performance (and more).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/xive.h | 116 +++
arch/powerpc/include/asm/xmon.h | 2 +
arch/powerpc/platforms/powernv/Kconfig | 2 +
arch/powerpc/platforms/powernv/setup.c | 15 +-
arch/powerpc/platforms/powernv/smp.c | 39 +-
arch/powerpc/sysdev/Kconfig | 1 +
arch/powerpc/sysdev/Makefile | 1 +
arch/powerpc/sysdev/xive/Kconfig | 7 +
arch/powerpc/sysdev/xive/Makefile | 4 +
arch/powerpc/sysdev/xive/common.c | 1175 ++++++++++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 604 +++++++++++++++
arch/powerpc/sysdev/xive/xive-internal.h | 51 ++
arch/powerpc/sysdev/xive/xive-regs.h | 88 +++
arch/powerpc/xmon/xmon.c | 93 ++-
14 files changed, 2186 insertions(+), 12 deletions(-)
create mode 100644 arch/powerpc/include/asm/xive.h
create mode 100644 arch/powerpc/sysdev/xive/Kconfig
create mode 100644 arch/powerpc/sysdev/xive/Makefile
create mode 100644 arch/powerpc/sysdev/xive/common.c
create mode 100644 arch/powerpc/sysdev/xive/native.c
create mode 100644 arch/powerpc/sysdev/xive/xive-internal.h
create mode 100644 arch/powerpc/sysdev/xive/xive-regs.h
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
new file mode 100644
index 0000000..b1604b73
--- /dev/null
+++ b/arch/powerpc/include/asm/xive.h
@@ -0,0 +1,116 @@
+#ifndef _ASM_POWERPC_XIVE_H
+#define _ASM_POWERPC_XIVE_H
+
+#define XIVE_INVALID_VP 0xffffffff
+
+#ifdef CONFIG_PPC_XIVE
+
+extern void __iomem *xive_tm_area;
+extern u32 xive_tm_offset;
+
+/*
+ * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
+ * have it stored in the xive_cpu structure. We also cache
+ * for normal interrupts the current target CPU.
+ */
+struct xive_irq_data {
+ /* Setup by backend */
+ u64 flags;
+#define XIVE_IRQ_FLAG_STORE_EOI 0x01
+#define XIVE_IRQ_FLAG_LSI 0x02
+#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
+#define XIVE_IRQ_FLAG_MASK_FW 0x08
+#define XIVE_IRQ_FLAG_EOI_FW 0x10
+ u64 eoi_page;
+ void __iomem *eoi_mmio;
+ u64 trig_page;
+ void __iomem *trig_mmio;
+ u32 esb_shift;
+ int src_chip;
+
+ /* Setup/used by frontend */
+ int target;
+ bool saved_p;
+};
+#define XIVE_INVALID_CHIP_ID -1
+
+/* A queue tracking structure in a CPU */
+struct xive_q {
+ __be32 *qpage;
+ u32 msk;
+ u32 idx;
+ u32 toggle;
+ u64 eoi_phys;
+ void __iomem *eoi_mmio;
+ u32 esc_irq;
+ atomic_t count;
+ atomic_t pending_count;
+};
+
+/*
+ * "magic" ESB MMIO offsets
+ */
+#define XIVE_ESB_GET 0x800
+#define XIVE_ESB_SET_PQ_00 0xc00
+#define XIVE_ESB_SET_PQ_01 0xd00
+#define XIVE_ESB_SET_PQ_10 0xe00
+#define XIVE_ESB_SET_PQ_11 0xf00
+#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
+
+extern bool __xive_enabled;
+
+static inline bool xive_enabled(void) { return __xive_enabled; }
+
+extern bool xive_native_init(void);
+extern void xive_smp_probe(void);
+extern int xive_smp_prepare_cpu(unsigned int cpu);
+extern void xive_smp_setup_cpu(void);
+extern void xive_smp_disable_cpu(void);
+extern void xive_kexec_teardown_cpu(int secondary);
+extern void xive_shutdown(void);
+extern void xive_flush_interrupt(void);
+
+/* xmon hook */
+extern void xmon_xive_do_dump(int cpu);
+
+/* APIs used by KVM */
+extern u32 xive_native_default_eq_shift(void);
+extern u32 xive_native_alloc_vp_block(u32 max_vcpus);
+extern void xive_native_free_vp_block(u32 vp_base);
+extern int xive_native_populate_irq_data(u32 hw_irq,
+ struct xive_irq_data *data);
+extern void xive_cleanup_irq_data(struct xive_irq_data *xd);
+extern u32 xive_native_alloc_irq(void);
+extern void xive_native_free_irq(u32 irq);
+extern int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
+
+extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
+ __be32 *qpage, u32 order, bool can_escalate);
+extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
+
+extern bool __xive_irq_trigger(struct xive_irq_data *xd);
+extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
+extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
+
+extern bool is_xive_irq(struct irq_chip *chip);
+
+#else
+
+static inline bool xive_enabled(void) { return false; }
+
+static inline bool xive_native_init(void) { return false; }
+static inline void xive_smp_probe(void) { }
+extern inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
+static inline void xive_smp_setup_cpu(void) { }
+static inline void xive_smp_disable_cpu(void) { }
+static inline void xive_kexec_teardown_cpu(int secondary) { }
+static inline void xive_shutdown(void) { }
+static inline void xive_flush_interrupt(void) { }
+
+static inline u32 xive_native_alloc_vp_block(u32 max_vcpus)
+ { return XIVE_INVALID_VP; }
+static inline void xive_native_free_vp_block(u32 vp_base) { }
+
+#endif
+
+#endif /* _ASM_POWERPC_XIVE_H */
diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h
index 5eb8e59..eb42a0c 100644
--- a/arch/powerpc/include/asm/xmon.h
+++ b/arch/powerpc/include/asm/xmon.h
@@ -29,5 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { };
extern int cpus_are_in_xmon(void);
#endif
+extern void xmon_printf(const char *format, ...);
+
#endif /* __KERNEL __ */
#endif /* __ASM_POWERPC_XMON_H */
diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
index 3a07e4d..81ee2ed 100644
--- a/arch/powerpc/platforms/powernv/Kconfig
+++ b/arch/powerpc/platforms/powernv/Kconfig
@@ -4,6 +4,8 @@ config PPC_POWERNV
select PPC_NATIVE
select PPC_XICS
select PPC_ICP_NATIVE
+ select PPC_XIVE
+ select PPC_XIVE_NATIVE
select PPC_P7_NAP
select PCI
select PCI_MSI
diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
index d50c7d9..adceac9 100644
--- a/arch/powerpc/platforms/powernv/setup.c
+++ b/arch/powerpc/platforms/powernv/setup.c
@@ -32,6 +32,7 @@
#include <asm/machdep.h>
#include <asm/firmware.h>
#include <asm/xics.h>
+#include <asm/xive.h>
#include <asm/opal.h>
#include <asm/kexec.h>
#include <asm/smp.h>
@@ -76,7 +77,9 @@ static void __init pnv_init(void)
static void __init pnv_init_IRQ(void)
{
- xics_init();
+ /* Try using a XIVE if available, otherwise use a XICS */
+ if (!xive_native_init())
+ xics_init();
WARN_ON(!ppc_md.get_irq);
}
@@ -218,10 +221,12 @@ static void pnv_kexec_wait_secondaries_down(void)
static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
{
- xics_kexec_teardown_cpu(secondary);
+ if (xive_enabled())
+ xive_kexec_teardown_cpu(secondary);
+ else
+ xics_kexec_teardown_cpu(secondary);
/* On OPAL, we return all CPUs to firmware */
-
if (!firmware_has_feature(FW_FEATURE_OPAL))
return;
@@ -237,6 +242,10 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
/* Primary waits for the secondaries to have reached OPAL */
pnv_kexec_wait_secondaries_down();
+ /* Switch XIVE back to emulation mode */
+ if (xive_enabled())
+ xive_shutdown();
+
/*
* We might be running as little-endian - now that interrupts
* are disabled, reset the HILE bit to big-endian so we don't
diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 8b67e1e..f571955 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -29,6 +29,7 @@
#include <asm/vdso_datapage.h>
#include <asm/cputhreads.h>
#include <asm/xics.h>
+#include <asm/xive.h>
#include <asm/opal.h>
#include <asm/runlatch.h>
#include <asm/code-patching.h>
@@ -47,7 +48,9 @@
static void pnv_smp_setup_cpu(int cpu)
{
- if (cpu != boot_cpuid)
+ if (xive_enabled())
+ xive_smp_setup_cpu();
+ else if (cpu != boot_cpuid)
xics_setup_cpu();
#ifdef CONFIG_PPC_DOORBELL
@@ -132,7 +135,10 @@ static int pnv_smp_cpu_disable(void)
vdso_data->processorCount--;
if (cpu = boot_cpuid)
boot_cpuid = cpumask_any(cpu_online_mask);
- xics_migrate_irqs_away();
+ if (xive_enabled())
+ xive_smp_disable_cpu();
+ else
+ xics_migrate_irqs_away();
return 0;
}
@@ -213,9 +219,12 @@ static void pnv_smp_cpu_kill_self(void)
if (((srr1 & wmask) = SRR1_WAKEEE) ||
((srr1 & wmask) = SRR1_WAKEHVI) ||
(local_paca->irq_happened & PACA_IRQ_EE)) {
- if (cpu_has_feature(CPU_FTR_ARCH_300))
- icp_opal_flush_interrupt();
- else
+ if (cpu_has_feature(CPU_FTR_ARCH_300)) {
+ if (xive_enabled())
+ xive_flush_interrupt();
+ else
+ icp_opal_flush_interrupt();
+ } else
icp_native_flush_interrupt();
} else if ((srr1 & wmask) = SRR1_WAKEHDBELL) {
unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
@@ -252,10 +261,26 @@ static int pnv_cpu_bootable(unsigned int nr)
return smp_generic_cpu_bootable(nr);
}
+static int pnv_smp_prepare_cpu(int cpu)
+{
+ if (xive_enabled())
+ return xive_smp_prepare_cpu(cpu);
+ return 0;
+}
+
+static void __init pnv_smp_probe(void)
+{
+ if (xive_enabled())
+ xive_smp_probe();
+ else
+ xics_smp_probe();
+}
+
static struct smp_ops_t pnv_smp_ops = {
.message_pass = smp_muxed_ipi_message_pass,
- .cause_ipi = NULL, /* Filled at runtime by xics_smp_probe() */
- .probe = xics_smp_probe,
+ .cause_ipi = NULL, /* Filled at runtime by xi{cs,ve}_smp_probe() */
+ .probe = pnv_smp_probe,
+ .prepare_cpu = pnv_smp_prepare_cpu,
.kick_cpu = pnv_smp_kick_cpu,
.setup_cpu = pnv_smp_setup_cpu,
.cpu_bootable = pnv_cpu_bootable,
diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig
index 52dc165..caf882e 100644
--- a/arch/powerpc/sysdev/Kconfig
+++ b/arch/powerpc/sysdev/Kconfig
@@ -28,6 +28,7 @@ config PPC_MSI_BITMAP
default y if PPC_POWERNV
source "arch/powerpc/sysdev/xics/Kconfig"
+source "arch/powerpc/sysdev/xive/Kconfig"
config PPC_SCOM
bool
diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile
index a254824..c0ae11d 100644
--- a/arch/powerpc/sysdev/Makefile
+++ b/arch/powerpc/sysdev/Makefile
@@ -71,5 +71,6 @@ obj-$(CONFIG_PPC_EARLY_DEBUG_MEMCONS) += udbg_memcons.o
subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
obj-$(CONFIG_PPC_XICS) += xics/
+obj-$(CONFIG_PPC_XIVE) += xive/
obj-$(CONFIG_GE_FPGA) += ge/
diff --git a/arch/powerpc/sysdev/xive/Kconfig b/arch/powerpc/sysdev/xive/Kconfig
new file mode 100644
index 0000000..c8816c8
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/Kconfig
@@ -0,0 +1,7 @@
+config PPC_XIVE
+ def_bool n
+ select PPC_SMP_MUXED_IPI
+ select HARDIRQS_SW_RESEND
+
+config PPC_XIVE_NATIVE
+ def_bool n
diff --git a/arch/powerpc/sysdev/xive/Makefile b/arch/powerpc/sysdev/xive/Makefile
new file mode 100644
index 0000000..3fab303
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/Makefile
@@ -0,0 +1,4 @@
+subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
+
+obj-y += common.o
+obj-$(CONFIG_PPC_XIVE_NATIVE) += native.o
diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
new file mode 100644
index 0000000..96037e0
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/common.c
@@ -0,0 +1,1175 @@
+/*
+ * Copyright 2016,2017 IBM Corporation.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#include <linux/types.h>
+#include <linux/threads.h>
+#include <linux/kernel.h>
+#include <linux/irq.h>
+#include <linux/debugfs.h>
+#include <linux/smp.h>
+#include <linux/interrupt.h>
+#include <linux/seq_file.h>
+#include <linux/init.h>
+#include <linux/cpu.h>
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/msi.h>
+
+#include <asm/prom.h>
+#include <asm/io.h>
+#include <asm/smp.h>
+#include <asm/machdep.h>
+#include <asm/irq.h>
+#include <asm/errno.h>
+#include <asm/xive.h>
+#include <asm/xmon.h>
+
+#include "xive-regs.h"
+#include "xive-internal.h"
+
+#undef DEBUG_FLUSH
+#undef DEBUG_ALL
+
+#define DBG(fmt...) pr_devel("XIVE: " fmt)
+
+#ifdef DEBUG_ALL
+#define DBG_VERBOSE(fmt...) pr_devel("XIVE: " fmt)
+#else
+#define DBG_VERBOSE(fmt...) do { } while(0)
+#endif
+
+bool __xive_enabled;
+bool xive_cmdline_disabled;
+
+/* We use only one priority for now */
+static u8 xive_irq_priority;
+
+void __iomem *xive_tm_area;
+u32 xive_tm_offset;
+static const struct xive_ops *xive_ops;
+static struct irq_domain *xive_irq_domain;
+
+/* The IPIs all use the same logical irq number */
+static u32 xive_ipi_irq;
+
+/* Xive state for each CPU */
+static DEFINE_PER_CPU(struct xive_cpu *, xive_cpu);
+
+/*
+ * A "disabled" interrupt should never fire, to catch problems
+ * we set its logical number to this
+ */
+#define XIVE_BAD_IRQ 0x7fffffff
+#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
+
+/* An invalid CPU target */
+#define XIVE_INVALID_TARGET (-1)
+
+static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
+{
+ u32 cur;
+
+ if (!q->qpage)
+ return 0;
+ cur = be32_to_cpup(q->qpage + q->idx);
+ if ((cur >> 31) = q->toggle)
+ return 0;
+ if (!just_peek) {
+ q->idx = (q->idx + 1) & q->msk;
+ if (q->idx = 0)
+ q->toggle ^= 1;
+ }
+ return cur & 0x7fffffff;
+}
+
+static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
+{
+ u32 hirq = 0;
+ u8 prio;
+
+ /* Find highest pending priority */
+ while (xc->pending_prio != 0) {
+ struct xive_q *q;
+
+ prio = ffs(xc->pending_prio) - 1;
+ DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
+
+ /* Try to fetch */
+ hirq = xive_read_eq(&xc->queue[prio], prio, just_peek);
+
+ /* Found something ? That's it */
+ if (hirq)
+ break;
+
+ /* Clear pending bits */
+ xc->pending_prio &= ~(1 << prio);
+
+ /*
+ * Check if the queue count needs adjusting due to
+ * interrupts being moved away.
+ */
+ q = &xc->queue[prio];
+ if (atomic_read(&q->pending_count)) {
+ int p = atomic_xchg(&q->pending_count, 0);
+ if (p) {
+ WARN_ON(p > atomic_read(&q->count));
+ atomic_sub(p, &q->count);
+ }
+ }
+ }
+
+ /* If nothing was found, set CPPR to 0xff */
+ if (hirq = 0)
+ prio = 0xff;
+
+ /* Update HW CPPR to match if necessary */
+ if (prio != xc->cppr) {
+ DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n", prio);
+ xc->cppr = prio;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, prio);
+ }
+
+ return hirq;
+}
+
+#ifdef CONFIG_XMON
+static void xive_dump_eq(const char *name, struct xive_q *q)
+{
+ u32 i0, i1, idx;
+
+ if (!q->qpage)
+ return;
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
+ q->toggle, i0, i1);
+}
+
+void xmon_xive_do_dump(int cpu)
+{
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ struct xive_irq_data *xd;
+ uint64_t val, offset;
+
+ xmon_printf("XIVE state for CPU %d:\n", cpu);
+ xmon_printf(" pp=%02x cppr=%02x\n", xc->pending_prio, xc->cppr);
+ xive_dump_eq("IRQ", &xc->queue[xive_irq_priority]);
+ xd = &xc->ipi_data;
+ offset = 0x800;
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+ val = in_be64(xd->eoi_mmio + offset);
+ xmon_printf(" IPI state: %x:%c%c\n", xc->hw_ipi,
+ val & 2 ? 'P' : 'p',
+ val & 1 ? 'Q' : 'q');
+}
+#endif /* CONFIG_XMON */
+
+static void xive_update_pending_irqs(struct xive_cpu *xc)
+{
+ u8 he, cppr;
+ u16 ack;
+
+ /* Perform the acknowledge hypervisor to register cycle */
+ ack = be16_to_cpu(__raw_readw(xive_tm_area + TM_SPC_ACK_HV_REG));
+
+ /* Synchronize subsequent queue accesses */
+ mb();
+
+ DBG_VERBOSE("CPU %d get_irq, ack=%04x\n", smp_processor_id(), ack);
+
+ /* Check the HE field */
+ cppr = ack & 0xff;
+ he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8));
+ switch(he) {
+ case TM_QW3_NSR_HE_NONE:
+ break;
+ case TM_QW3_NSR_HE_PHYS:
+ if (cppr = 0xff)
+ return;
+ xc->pending_prio |= 1 << cppr;
+ if (cppr >= xc->cppr)
+ pr_err("XIVE: CPU %d odd ack CPPR, got %d at %d\n",
+ smp_processor_id(), cppr, xc->cppr);
+ xc->cppr = cppr;
+ break;
+ case TM_QW3_NSR_HE_POOL:
+ case TM_QW3_NSR_HE_LSI:
+ pr_err("XIVE: CPU %d got unexpected interrupt type HE=%d\n",
+ smp_processor_id(), he);
+ return;
+ }
+}
+
+static unsigned int xive_get_irq(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ u32 hirq;
+
+ /*
+ * This can be called either as a result of a HW interrupt or
+ * as a "replay" because EOI decided there was still something
+ * in one of the queues.
+ *
+ * First we perform an ACK cycle in order to update our mask
+ * of pending priorities. This will also have the effect of
+ * updating the CPPR to the most favored pending interrupts.
+ *
+ * In the future, if we have a way to differenciate a first
+ * entry (on HW interrupt) from a replay triggered by EOI,
+ * we could skip this on replays unless we soft-mask tells us
+ * that a new HW interrupt occurred.
+ */
+ xive_update_pending_irqs(xc);
+
+ DBG_VERBOSE("get_irq: pending=%02x\n", xc->pending_prio);
+
+ hirq = xive_scan_interrupts(xc, false);
+
+ DBG_VERBOSE("get_irq: got irq 0x%x, new pending=0x%02x\n",
+ hirq, xc->pending_prio);
+
+ /* Return pending interrupt if any */
+ if (hirq = XIVE_BAD_IRQ)
+ return 0;
+ return hirq;
+}
+
+
+static void xive_do_queue_eoi(struct xive_cpu *xc)
+{
+ if (xive_scan_interrupts(xc, true) != 0) {
+ DBG_VERBOSE("eoi: pending=0x%02x\n", xc->pending_prio);
+ force_external_irq_replay();
+ }
+}
+
+static u8 xive_poke_esb(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = in_be64(xd->eoi_mmio + offset);
+
+ return (u8)val;
+}
+
+static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd)
+{
+ /* If the XIVE supports the new "store EOI facility, use it */
+ if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
+ out_be64(xd->eoi_mmio, 0);
+ else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
+ if (WARN_ON_ONCE(!xive_ops->eoi))
+ return;
+ xive_ops->eoi(hw_irq);
+ } else {
+ uint8_t eoi_val;
+
+ /*
+ * Otherwise for EOI, we use the special MMIO that does
+ * a clear of both P and Q and returns the old Q.
+ *
+ * This allows us to then do a re-trigger if Q was set
+ * rather than synthetizing an interrupt in software
+ */
+ eoi_val = xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
+ DBG_VERBOSE("eoi_val=%x\n", offset, eoi_val);
+
+ if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
+ return;
+
+ /* Re-trigger */
+ if (xd->trig_mmio)
+ out_be64(xd->trig_mmio, 0);
+ }
+
+}
+
+static void xive_irq_eoi(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+
+ DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
+ d->irq, irqd_to_hwirq(d), xc->pending_prio);
+
+ if (!irqd_irq_disabled(d))
+ xive_do_source_eoi(irqd_to_hwirq(d), xd);
+
+ /*
+ * Clear saved_p to indicate that it's no longer occupying
+ * a queue slot on the target queue
+ */
+ xd->saved_p = false;
+
+ xive_do_queue_eoi(xc);
+}
+
+static void xive_do_source_set_mask(struct xive_irq_data *xd,
+ bool masked)
+{
+ if (masked)
+ xive_poke_esb(xd, XIVE_ESB_SET_PQ_01);
+ else
+ xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
+}
+
+static bool xive_try_pick_target(int cpu)
+{
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ struct xive_q *q = &xc->queue[xive_irq_priority];
+ int max;
+
+ /* Calculate max number of interrupts in that queue.
+ *
+ * We leave a gap of 1 just in case...
+ */
+ max = (q->msk + 1) - 1;
+ return !!atomic_add_unless(&q->count, 1, max);
+}
+
+static void xive_dec_target_count(int cpu)
+{
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ struct xive_q *q = &xc->queue[xive_irq_priority];
+
+ if (WARN_ON(cpu < 0))
+ return;
+
+ /*
+ * We increment the "pending count" which will be used
+ * to decrement the target queue count whenever it's next
+ * processed and found empty. This ensure that we don't
+ * decrement while we still have the interrupt there
+ * occupying a slot.
+ */
+ atomic_inc(&q->pending_count);
+}
+
+static int xive_find_target_in_mask(const struct cpumask *mask,
+ unsigned int fuzz)
+{
+ int cpu, first, num, i;
+
+ /* Pick up a starting point CPU in the mask based on fuzz */
+ num = cpumask_weight(mask);
+ first = (fuzz++) % num;
+
+ /* Locate it */
+ cpu = cpumask_first(mask);
+ for (i = 0; i < first; i++)
+ cpu = cpumask_next(cpu, mask);
+ first = cpu;
+
+ /*
+ * Now go through the entire mask until we find a valid
+ * target.
+ */
+ for (;;) {
+ /*
+ * We re-check online as the fallback case passes us
+ * an untested affinity mask
+ */
+ if (cpu_online(cpu) && xive_try_pick_target(cpu))
+ return cpu;
+ cpu = cpumask_next(cpu, mask);
+ if (cpu = first)
+ break;
+ }
+ return -1;
+}
+
+static int xive_pick_irq_target(struct irq_data *d,
+ const struct cpumask *affinity)
+{
+ static unsigned int fuzz;
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ cpumask_var_t mask;
+ int cpu = -1;
+
+ /*
+ * Pick a target CPU for an interrupt. This is done at
+ * startup or if the affinity is changed in a way that
+ * invalidates the current target.
+ */
+
+ /* If we have chip IDs, first we try to build a mask of
+ * CPUs matching ther CPU and find a target in there
+ */
+ if (xd->src_chip != XIVE_INVALID_CHIP_ID &&
+ zalloc_cpumask_var(&mask, GFP_ATOMIC)) {
+ /* Build a mask of matching chip IDs */
+ for_each_cpu_and(cpu, affinity, cpu_online_mask) {
+ struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
+ if (xc->chip_id = xd->src_chip)
+ cpumask_set_cpu(cpu, mask);
+ }
+ /* Try to find a target */
+ if (!cpumask_empty(mask))
+ cpu = xive_find_target_in_mask(mask, fuzz++);
+ free_cpumask_var(mask);
+ if (cpu >= 0)
+ return cpu;
+ fuzz--;
+ }
+
+ /* No chip IDs, fallback to using the affinity mask */
+ return xive_find_target_in_mask(affinity, fuzz++);
+}
+
+static unsigned int xive_irq_startup(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ int target, rc;
+
+ DBG("xive_irq_startup: irq %d [0x%x] data @%p\n",
+ d->irq, hw_irq, d);
+
+#ifdef CONFIG_PCI_MSI
+ /*
+ * The generic MSI code returns with the interrupt disabled on the
+ * card, using the MSI mask bits. Firmware doesn't appear to unmask
+ * at that level, so we do it here by hand.
+ */
+ if (irq_data_get_msi_desc(d))
+ pci_msi_unmask_irq(d);
+#endif
+
+ /* Pick a target */
+ target = xive_pick_irq_target(d, irq_data_get_affinity_mask(d));
+ if (target = XIVE_INVALID_TARGET) {
+ /* Try again breaking affinity */
+ target = xive_pick_irq_target(d, cpu_online_mask);
+ if (target = XIVE_INVALID_TARGET)
+ return -ENXIO;
+ pr_warn("XIVE: irq %d started with broken affinity\n",
+ d->irq);
+ }
+ xd->target = target;
+
+ /*
+ * Configure the logical number to be the Linux IRQ number
+ * and set the target queue
+ */
+ rc = xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(target),
+ xive_irq_priority, d->irq);
+ if (rc)
+ return rc;
+
+ /* Unmask the ESB */
+ xive_do_source_set_mask(xd, false);
+
+ return 0;
+}
+
+static void xive_irq_shutdown(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+
+ DBG("xive_irq_shutdown: irq %d [0x%x] data @%p\n",
+ d->irq, hw_irq, d);
+
+ if (WARN_ON(xd->target = XIVE_INVALID_TARGET))
+ return;
+
+ /* Mask the interrupt at the source */
+ xive_do_source_set_mask(xd, true);
+
+ /* Mask the interrupt in HW in the IVT/EAS */
+ xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ 0xff, hw_irq);
+
+ xive_dec_target_count(xd->target);
+ xd->target = XIVE_INVALID_TARGET;
+}
+
+static void xive_irq_unmask(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ DBG("xive_irq_unmask: irq %d data @%p\n", d->irq, xd);
+
+ /*
+ * This is a workaround for PCI LSI problems on P9, for
+ * these, we call FW to set the mask. The problems might
+ * be fixed by P9 DD2.0, if that is the case, we will make
+ * this a DD1 workaround only
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ xive_irq_priority, d->irq);
+ return;
+ }
+
+ xive_do_source_set_mask(xd, false);
+}
+
+static void xive_irq_mask(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ DBG("xive_irq_mask: irq %d data @%p\n", d->irq, xd);
+
+ /*
+ * This is a workaround for PCI LSI problems on P9, for
+ * these, we call OPAL to set the mask. The problems might
+ * be fixed by P9 DD2.0, if that is the case, we will make
+ * this a DD1 workaround only
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ 0xff, d->irq);
+ return;
+ }
+
+ xive_do_source_set_mask(xd, true);
+}
+
+static int xive_irq_set_affinity(struct irq_data *d,
+ const struct cpumask *cpumask,
+ bool force)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ u32 target, old_target;
+ int rc = 0;
+
+ DBG("xive_irq_set_affinity: irq %d\n", d->irq);
+
+ /* Is this valid ? */
+ if (cpumask_any_and(cpumask, cpu_online_mask) >= nr_cpu_ids)
+ return -EINVAL;
+
+ /* If existing target is already in the new mask, and is
+ * online then do nothing.
+ */
+ if (cpu_online(xd->target) &&
+ cpumask_test_cpu(xd->target, cpumask))
+ return IRQ_SET_MASK_OK;
+
+ /* Pick a new target */
+ target = xive_pick_irq_target(d, cpumask);
+
+ /* No target found */
+ if (target = XIVE_INVALID_TARGET)
+ return -ENXIO;
+
+ old_target = xd->target;
+
+ /*
+ * Only configure the irq if it's not currently passed-through to
+ * a KVM guest
+ */
+ rc = xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(target),
+ xive_irq_priority, d->irq);
+ if (rc < 0) {
+ pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
+ return rc;
+ }
+
+ DBG(" target: 0x%x\n", target);
+ xd->target = target;
+
+ /* Give up previous target */
+ if (old_target != XIVE_INVALID_TARGET)
+ xive_dec_target_count(old_target);
+
+ return IRQ_SET_MASK_OK;
+}
+
+static int xive_irq_set_type(struct irq_data *d, unsigned int flow_type)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ /*
+ * We only support these. This has really no effect other than setting
+ * the corresponding descriptor bits mind you but those will in turn
+ * affect the resend function when re-enabling an edge interrupt.
+ *
+ * Set set the default to edge as explained in map().
+ */
+ if (flow_type = IRQ_TYPE_DEFAULT || flow_type = IRQ_TYPE_NONE)
+ flow_type = IRQ_TYPE_EDGE_RISING;
+
+ if (flow_type != IRQ_TYPE_EDGE_RISING &&
+ flow_type != IRQ_TYPE_LEVEL_LOW)
+ return -EINVAL;
+
+ irqd_set_trigger_type(d, flow_type);
+
+ /*
+ * Double check it matches what the FW thinks
+ *
+ * NOTE: We don't know yet if the PAPR interface will provide
+ * the LSI vs MSI information appart from the device-tree so
+ * this check might have to move into an optional backend call
+ * that is specific to the native backend
+ */
+ if ((flow_type = IRQ_TYPE_LEVEL_LOW) !+ !!(xd->flags & XIVE_IRQ_FLAG_LSI))
+ pr_warn("XIVE: Interrupt %d (HW 0x%x) type mismatch,"
+ " Linux says %s, FW says %s\n",
+ d->irq, (u32)irqd_to_hwirq(d),
+ (flow_type = IRQ_TYPE_LEVEL_LOW) ? "Level" : "Edge",
+ (xd->flags & XIVE_IRQ_FLAG_LSI) ? "Level" : "Edge");
+
+ return IRQ_SET_MASK_OK_NOCOPY;
+}
+
+static int xive_irq_retrigger(struct irq_data *d)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+
+ /* This should be only for MSIs */
+ if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
+ return 0;
+
+ /*
+ * To perform a retrigger, we first set the PQ bits to
+ * 11, then perform an EOI.
+ */
+ xive_poke_esb(xd, XIVE_ESB_SET_PQ_11);
+
+ /*
+ * Note: We pass "0" to the hw_irq argument in order to
+ * avoid calling into the backend EOI code which we don't
+ * want to do in the case of a re-trigger. Backends typically
+ * only do EOI for LSIs anyway.
+ */
+ xive_do_source_eoi(0, xd);
+
+ return 1;
+}
+
+static struct irq_chip xive_irq_chip = {
+ .name = "XIVE-IRQ",
+ .irq_startup = xive_irq_startup,
+ .irq_shutdown = xive_irq_shutdown,
+ .irq_eoi = xive_irq_eoi,
+ .irq_mask = xive_irq_mask,
+ .irq_unmask = xive_irq_unmask,
+ .irq_set_affinity = xive_irq_set_affinity,
+ .irq_set_type = xive_irq_set_type,
+ .irq_retrigger = xive_irq_retrigger,
+};
+
+bool is_xive_irq(struct irq_chip *chip)
+{
+ return chip = &xive_irq_chip;
+}
+
+void xive_cleanup_irq_data(struct xive_irq_data *xd)
+{
+ if (xd->eoi_mmio) {
+ iounmap(xd->eoi_mmio);
+ if (xd->eoi_mmio = xd->trig_mmio)
+ xd->trig_mmio = NULL;
+ xd->eoi_mmio = NULL;
+ }
+ if (xd->trig_mmio) {
+ iounmap(xd->trig_mmio);
+ xd->trig_mmio = NULL;
+ }
+}
+
+static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
+{
+ struct xive_irq_data *xd;
+ int rc;
+
+ xd = kzalloc(sizeof(struct xive_irq_data), GFP_KERNEL);
+ if (!xd)
+ return -ENOMEM;
+ rc = xive_ops->populate_irq_data(hw, xd);
+ if (rc) {
+ kfree(xd);
+ return rc;
+ }
+ xd->target = XIVE_INVALID_TARGET;
+ irq_set_handler_data(virq, xd);
+
+ return 0;
+}
+
+static void xive_irq_free_data(unsigned int virq)
+{
+ struct xive_irq_data *xd = irq_get_handler_data(virq);
+
+ if (!xd)
+ return;
+ irq_set_handler_data(virq, NULL);
+ xive_cleanup_irq_data(xd);
+ kfree(xd);
+}
+
+#ifdef CONFIG_SMP
+
+static void xive_cause_ipi(int cpu, unsigned long msg)
+{
+ struct xive_cpu *xc;
+ struct xive_irq_data *xd;
+
+ xc = per_cpu(xive_cpu, cpu);
+
+ DBG_VERBOSE("IPI msg#%ld CPU %d -> %d (HW IRQ 0x%x)\n",
+ msg, smp_processor_id(), cpu, xc->hw_ipi);
+
+ xd = &xc->ipi_data;
+ if (WARN_ON(!xd->trig_mmio))
+ return;
+ out_be64(xd->trig_mmio, 0);
+}
+
+static irqreturn_t xive_muxed_ipi_action(int irq, void *dev_id)
+{
+ return smp_ipi_demux();
+}
+
+static void xive_ipi_eoi(struct irq_data *d)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+
+ /* Handle possible race with unplug and drop stale IPIs */
+ if (!xc)
+ return;
+ xive_do_source_eoi(xc->hw_ipi, &xc->ipi_data);
+ xive_do_queue_eoi(xc);
+}
+
+static void xive_ipi_unmask(struct irq_data *d)
+{
+ /* Nothing to do, we never mask IPIs, but the callback
+ * must exist
+ */
+}
+
+static void xive_ipi_mask(struct irq_data *d)
+{
+ /* Nothing to do, we never mask IPIs, but the callback
+ * must exist
+ */
+}
+
+static struct irq_chip xive_ipi_chip = {
+ .name = "XIVE-IPI",
+ .irq_eoi = xive_ipi_eoi,
+ .irq_mask = xive_ipi_mask,
+ .irq_unmask = xive_ipi_unmask,
+};
+
+static void __init xive_request_ipi(void)
+{
+ unsigned int virq;
+
+ /* Initialize it */
+ virq = irq_create_mapping(xive_irq_domain, 0);
+ xive_ipi_irq = virq;
+
+ BUG_ON(request_irq(virq, xive_muxed_ipi_action,
+ IRQF_PERCPU | IRQF_NO_THREAD, "IPI", NULL));
+}
+
+static int xive_setup_cpu_ipi(unsigned int cpu)
+{
+ struct xive_cpu *xc;
+ int rc;
+
+ pr_debug("XIVE: Setting up IPI for CPU %d\n", cpu);
+
+ xc = per_cpu(xive_cpu, cpu);
+
+ /* Check if we are already setup */
+ if (xc->hw_ipi != 0)
+ return 0;
+
+ /* Grab an IPI from the backend, this will populate xc->hw_ipi */
+ if (xive_ops->get_ipi(cpu, xc))
+ return -EIO;
+
+ /* Populate the IRQ data in the xive_cpu structure and
+ * configure the HW / enable the IPIs
+ */
+ rc = xive_ops->populate_irq_data(xc->hw_ipi, &xc->ipi_data);
+ if (rc) {
+ pr_err("XIVE: Failed to populate IPI data on CPU %d\n", cpu);
+ return -EIO;
+ }
+ rc = xive_ops->configure_irq(xc->hw_ipi,
+ get_hard_smp_processor_id(cpu),
+ xive_irq_priority, xive_ipi_irq);
+ if (rc) {
+ pr_err("XIVE: Failed to map IPI CPU %d\n", cpu);
+ return -EIO;
+ }
+ DBG("XIVE: CPU %d HW IPI %x, virq %d, trig_mmio=%p\n", cpu,
+ xc->hw_ipi, xive_ipi_irq, xc->ipi_data.trig_mmio);
+
+ /* Unmask it */
+ xive_do_source_set_mask(&xc->ipi_data, false);
+
+ return 0;
+}
+
+static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
+{
+ /* Disable the IPI and free the IRQ data */
+
+ /* Already cleaned up ? */
+ if (xc->hw_ipi = 0)
+ return;
+
+ /* Mask the IPI */
+ xive_do_source_set_mask(&xc->ipi_data, true);
+
+ /*
+ * Note: We don't call xive_cleanup_irq_data() to free
+ * the mappings as this is called from an IPI on kexec
+ * which is not a safe environment to call iounmap()
+ */
+
+ /* Deconfigure/mask in the backend */
+ xive_ops->configure_irq(xc->hw_ipi, hard_smp_processor_id(),
+ 0xff, xive_ipi_irq);
+
+ /* Free the IPIs in the backend */
+ xive_ops->put_ipi(cpu, xc);
+}
+
+void __init xive_smp_probe(void)
+{
+ smp_ops->cause_ipi = xive_cause_ipi;
+
+ /* Register the IPI */
+ xive_request_ipi();
+
+ /* Allocate and setup IPI for the boot CPU */
+ xive_setup_cpu_ipi(smp_processor_id());
+}
+
+#endif /* CONFIG_SMP */
+
+static int xive_irq_domain_map(struct irq_domain *h, unsigned int virq,
+ irq_hw_number_t hw)
+{
+ int rc;
+
+ /*
+ * Mark interrupts as edge sensitive by default so that resend
+ * actually works. Will fix that up below if needed.
+ */
+ irq_clear_status_flags(virq, IRQ_LEVEL);
+
+ /* IPIs are special and come up with HW number 0 */
+ if (hw = 0) {
+ /*
+ * IPIs are marked per-cpu. We use separate HW interrupts under
+ * the hood but associated with the same "linux" interrupt
+ */
+ irq_set_chip_and_handler(virq, &xive_ipi_chip,
+ handle_percpu_irq);
+ return 0;
+ }
+
+ rc = xive_irq_alloc_data(virq, hw);
+ if (rc)
+ return rc;
+
+ irq_set_chip_and_handler(virq, &xive_irq_chip, handle_fasteoi_irq);
+
+ return 0;
+}
+
+static void xive_irq_domain_unmap(struct irq_domain *d, unsigned int virq)
+{
+ struct irq_data *data = irq_get_irq_data(virq);
+ unsigned int hw_irq;
+
+ if (!data)
+ return;
+ hw_irq = (unsigned int)irqd_to_hwirq(data);
+ if (hw_irq)
+ xive_irq_free_data(virq);
+}
+
+static int xive_irq_domain_xlate(struct irq_domain *h, struct device_node *ct,
+ const u32 *intspec, unsigned int intsize,
+ irq_hw_number_t *out_hwirq, unsigned int *out_flags)
+
+{
+ *out_hwirq = intspec[0];
+
+ /*
+ * If intsize is at least 2, we look for the type in the second cell,
+ * we assume the LSB indicates a level interrupt.
+ */
+ if (intsize > 1) {
+ if (intspec[1] & 1)
+ *out_flags = IRQ_TYPE_LEVEL_LOW;
+ else
+ *out_flags = IRQ_TYPE_EDGE_RISING;
+ } else
+ *out_flags = IRQ_TYPE_LEVEL_LOW;
+
+ return 0;
+}
+
+static int xive_irq_domain_match(struct irq_domain *h, struct device_node *node,
+ enum irq_domain_bus_token bus_token)
+{
+ return xive_ops->match(node);
+}
+
+static const struct irq_domain_ops xive_irq_domain_ops = {
+ .match = xive_irq_domain_match,
+ .map = xive_irq_domain_map,
+ .unmap = xive_irq_domain_unmap,
+ .xlate = xive_irq_domain_xlate,
+};
+
+static void __init xive_init_host(void)
+{
+ xive_irq_domain = irq_domain_add_nomap(NULL, XIVE_MAX_IRQ,
+ &xive_irq_domain_ops, NULL);
+ BUG_ON(xive_irq_domain = NULL);
+ irq_set_default_host(xive_irq_domain);
+}
+
+static void xive_cleanup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
+{
+ if (xc->queue[xive_irq_priority].qpage)
+ xive_ops->cleanup_queue(cpu, xc, xive_irq_priority);
+}
+
+static int xive_setup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
+{
+ int rc = 0;
+
+ /* We setup 1 queues for now with a 64k page */
+ if (!xc->queue[xive_irq_priority].qpage)
+ rc = xive_ops->setup_queue(cpu, xc, xive_irq_priority);
+
+ return rc;
+}
+
+static int xive_prepare_cpu(unsigned int cpu)
+{
+ struct xive_cpu *xc;
+
+ xc = per_cpu(xive_cpu, cpu);
+ if (!xc) {
+ struct device_node *np;
+
+ xc = kzalloc_node(sizeof(struct xive_cpu),
+ GFP_KERNEL, cpu_to_node(cpu));
+ if (!xc)
+ return -ENOMEM;
+ np = of_get_cpu_node(cpu, NULL);
+ if (np)
+ xc->chip_id = of_get_ibm_chip_id(np);
+ of_node_put(np);
+
+ per_cpu(xive_cpu, cpu) = xc;
+ }
+
+ /* Setup EQs if not already */
+ return xive_setup_cpu_queues(cpu, xc);
+}
+
+static void xive_setup_cpu(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+
+ /* Debug: Dump the TM state */
+ DBG("CPU %d [HW 0x%02x] VT=%02x\n",
+ smp_processor_id(), hard_smp_processor_id(),
+ in_8(xive_tm_area + xive_tm_offset + TM_WORD2));
+
+ /* The backend might have additional things to do */
+ if (xive_ops->setup_cpu)
+ xive_ops->setup_cpu(smp_processor_id(), xc);
+
+ /* Set CPPR to 0xff to enable flow of interrupts */
+ xc->cppr = 0xff;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
+}
+
+#ifdef CONFIG_SMP
+void xive_smp_setup_cpu(void)
+{
+ DBG("XIVE: SMP setup CPU %d\n", smp_processor_id());
+
+ /* This will have already been done on the boot CPU */
+ if (smp_processor_id() != boot_cpuid)
+ xive_setup_cpu();
+
+}
+
+int xive_smp_prepare_cpu(unsigned int cpu)
+{
+ int rc;
+
+ /* Allocate per-CPU data and queues */
+ rc = xive_prepare_cpu(cpu);
+ if (rc)
+ return rc;
+
+ /* Allocate and setup IPI for the new CPU */
+ return xive_setup_cpu_ipi(cpu);
+}
+
+#ifdef CONFIG_HOTPLUG_CPU
+static void xive_flush_cpu_queue(unsigned int cpu, struct xive_cpu *xc)
+{
+ u32 irq;
+
+ /* We assume local irqs are disabled */
+ WARN_ON(!irqs_disabled());
+
+ /* Check what's already in the CPU queue */
+ while ((irq = xive_scan_interrupts(xc, false)) != 0) {
+ /*
+ * We need to re-route that interrupt to its new distination.
+ * First get and lock the descriptor
+ */
+ struct irq_desc *desc = irq_to_desc(irq);
+ struct irq_data *d = irq_desc_get_irq_data(desc);
+ struct xive_irq_data *xd;
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+
+ /*
+ * Ignore anything that isn't a XIVE irq and ignore
+ * IPIs, so can just be dropped.
+ */
+ if (d->domain != xive_irq_domain || hw_irq = 0)
+ continue;
+#ifdef DEBUG_FLUSH
+ pr_info("CPU %d: Got irq %d while offline, re-routing...\n",
+ cpu, irq);
+#endif
+ raw_spin_lock(&desc->lock);
+ xd = irq_desc_get_handler_data(desc);
+
+ /* For LSIs, we EOI, this will cause a resend if it's
+ * still asserted. Otherwise do an MSI retrigger
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_LSI)
+ xive_do_source_eoi(irqd_to_hwirq(d), xd);
+ else
+ xive_irq_retrigger(d);
+ raw_spin_unlock(&desc->lock);
+ }
+}
+
+void xive_smp_disable_cpu(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ unsigned int cpu = smp_processor_id();
+
+ /* Migrate interrupts away from the CPU */
+ irq_migrate_all_off_this_cpu();
+
+ /* Set CPPR to 0 to disable flow of interrupts */
+ xc->cppr = 0;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
+
+ /* Flush everything still in the queue */
+ xive_flush_cpu_queue(cpu, xc);
+
+ /* Re-enable CPPR */
+ xc->cppr = 0xff;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
+}
+
+void xive_flush_interrupt(void)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ unsigned int cpu = smp_processor_id();
+
+ /* Called if an interrupt occurs while the CPU is hot unplugged */
+ xive_flush_cpu_queue(cpu, xc);
+}
+
+#endif /* CONFIG_HOTPLUG_CPU */
+
+#endif /* CONFIG_SMP */
+
+void xive_kexec_teardown_cpu(int secondary)
+{
+ struct xive_cpu *xc = __this_cpu_read(xive_cpu);
+ unsigned int cpu = smp_processor_id();
+
+ /* Set CPPR to 0 to disable flow of interrupts */
+ xc->cppr = 0;
+ out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
+
+ /* Backend cleanup if any */
+ if (xive_ops->teardown_cpu)
+ xive_ops->teardown_cpu(cpu, xc);
+
+ /* Get rid of IPI */
+ xive_cleanup_cpu_ipi(cpu, xc);
+
+ /* Disable and free the queues */
+ xive_cleanup_cpu_queues(cpu, xc);
+}
+
+void xive_shutdown(void)
+{
+ xive_ops->shutdown();
+}
+
+bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
+ u8 max_prio)
+{
+ xive_tm_area = area;
+ xive_tm_offset = offset;
+ xive_ops = ops;
+ xive_irq_priority = max_prio;
+
+ ppc_md.get_irq = xive_get_irq;
+ __xive_enabled = true;
+
+ DBG("Initializing host..\n");
+ xive_init_host();
+
+ DBG("Initializing boot CPU..\n");
+
+ /* Allocate per-CPU data and queues */
+ xive_prepare_cpu(smp_processor_id());
+
+ /* Get ready for interrupts */
+ xive_setup_cpu();
+
+ pr_info("XIVE: Interrupt handling intialized with %s backend\n",
+ xive_ops->name);
+ pr_info("XIVE: Using priority %d for all interrupts\n", max_prio);
+
+ return true;
+}
+
+static int __init xive_off(char *arg)
+{
+ xive_cmdline_disabled = true;
+ return 0;
+}
+__setup("xive=off", xive_off);
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
new file mode 100644
index 0000000..26cc6bf
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -0,0 +1,604 @@
+/*
+ * Copyright 2016,2017 IBM Corporation.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * as published by the Free Software Foundation; either version
+ * 2 of the License, or (at your option) any later version.
+ */
+#include <linux/types.h>
+#include <linux/irq.h>
+#include <linux/debugfs.h>
+#include <linux/smp.h>
+#include <linux/interrupt.h>
+#include <linux/seq_file.h>
+#include <linux/init.h>
+#include <linux/of.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/cpumask.h>
+#include <linux/mm.h>
+
+#include <asm/prom.h>
+#include <asm/io.h>
+#include <asm/smp.h>
+#include <asm/irq.h>
+#include <asm/errno.h>
+#include <asm/xive.h>
+#include <asm/opal.h>
+
+#include "xive-regs.h"
+#include "xive-internal.h"
+
+#define DBG(fmt...) pr_devel("XIVE: " fmt)
+
+/* Enable this for using queue MMIO page for EOI. We don't currently
+ * use it as we always notify
+ */
+#undef USE_QUEUE_MMIO
+
+static u32 xive_provision_size;
+static u32 *xive_provision_chips;
+static u32 xive_provision_chip_count;
+static u32 xive_queue_shift;
+static u32 xive_pool_vps = XIVE_INVALID_VP;
+static struct kmem_cache *xive_provision_cache;
+
+int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
+{
+ __be64 flags, eoi_page, trig_page;
+ __be32 esb_shift, src_chip;
+ u64 opal_flags;
+ s64 rc;
+
+ memset(data, 0, sizeof(*data));
+
+ rc = opal_xive_get_irq_info(hw_irq, &flags, &eoi_page, &trig_page,
+ &esb_shift, &src_chip);
+ if (rc) {
+ pr_err("XIVE: opal_xive_get_irq_info(0x%x) returned %lld\n",
+ hw_irq, rc);
+ return -EINVAL;
+ }
+
+ opal_flags = be64_to_cpu(flags);
+ if (opal_flags & OPAL_XIVE_IRQ_STORE_EOI)
+ data->flags |= XIVE_IRQ_FLAG_STORE_EOI;
+ if (opal_flags & OPAL_XIVE_IRQ_LSI)
+ data->flags |= XIVE_IRQ_FLAG_LSI;
+ if (opal_flags & OPAL_XIVE_IRQ_SHIFT_BUG)
+ data->flags |= XIVE_IRQ_FLAG_SHIFT_BUG;
+ if (opal_flags & OPAL_XIVE_IRQ_MASK_VIA_FW)
+ data->flags |= XIVE_IRQ_FLAG_MASK_FW;
+ if (opal_flags & OPAL_XIVE_IRQ_EOI_VIA_FW)
+ data->flags |= XIVE_IRQ_FLAG_EOI_FW;
+ data->eoi_page = be64_to_cpu(eoi_page);
+ data->trig_page = be64_to_cpu(trig_page);
+ data->esb_shift = be32_to_cpu(esb_shift);
+ data->src_chip = be32_to_cpu(src_chip);
+
+ data->eoi_mmio = ioremap(data->eoi_page, 1u << data->esb_shift);
+ if (!data->eoi_mmio) {
+ pr_err("XIVE: Failed to map EOI page for irq 0x%x\n", hw_irq);
+ return -ENOMEM;
+ }
+
+ if (!data->trig_page)
+ return 0;
+ if (data->trig_page = data->eoi_page) {
+ data->trig_mmio = data->eoi_mmio;
+ return 0;
+ }
+
+ data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
+ if (!data->trig_mmio) {
+ pr_err("XIVE: Failed to map trigger page for irq 0x%x\n", hw_irq);
+ return -ENOMEM;
+ }
+ return 0;
+}
+
+int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
+{
+ s64 rc;
+
+ for (;;) {
+ rc = opal_xive_set_irq_config(hw_irq, target, prio, sw_irq);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ return rc = 0 ? 0 : -ENXIO;
+}
+
+/* This can be called multiple time to change a queue configuration */
+int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
+ __be32 *qpage, u32 order, bool can_escalate)
+{
+ s64 rc = 0;
+ __be64 qeoi_page_be;
+ __be32 esc_irq_be;
+ u64 flags, qpage_phys;
+
+ /* If there's an actual queue page, clean it */
+ if (order) {
+ BUG_ON(!qpage);
+ qpage_phys = __pa(qpage);
+ } else
+ qpage_phys = 0;
+
+ /* Initialize the rest of the fields */
+ q->msk = order ? ((1u << (order - 2)) - 1) : 0;
+ q->idx = 0;
+ q->toggle = 0;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, NULL, NULL,
+ &qeoi_page_be,
+ &esc_irq_be,
+ NULL);
+ if (rc) {
+ pr_err("XIVE: Error %lld getting queue info prio %d\n",
+ rc, prio);
+ rc = -EIO;
+ goto fail;
+ }
+ q->eoi_phys = be64_to_cpu(qeoi_page_be);
+
+#ifdef USE_QUEUE_MMIO
+ if (!q->eoi_mmio)
+ q->eoi_mmio = ioremap(q->eoi_phys, PAGE_SIZE);
+ if (!q->eoi_mmio) {
+ pr_err("XIVE: Failed to map queue MMIO prio %d CPU %d\n",
+ rc, prio, cpu);
+ rc = -ENOMEM;
+ goto fail;
+ }
+#endif /* USE_QUEUE_MMIO */
+
+ /* Default flags */
+ flags = OPAL_XIVE_EQ_ALWAYS_NOTIFY | OPAL_XIVE_EQ_ENABLED;
+
+ /* Escalation needed ? */
+ if (can_escalate) {
+ q->esc_irq = be32_to_cpu(esc_irq_be);
+ flags |= OPAL_XIVE_EQ_ESCALATE;
+ }
+
+ /* Configure and enable the queue in HW */
+ for (;;) {
+ rc = opal_xive_set_queue_info(vp_id, prio, qpage_phys, order, flags);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ if (rc) {
+ pr_err("XIVE: Error %lld setting queue for prio %d\n",
+ rc, prio);
+ rc = -EIO;
+ } else {
+ /*
+ * KVM code requires all of the above to be visible before
+ * q->qpage is set due to how it manages IPI EOIs
+ */
+ wmb();
+ q->qpage = qpage;
+ }
+ fail:
+ return rc;
+}
+
+static void __xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
+{
+ s64 rc;
+
+ /* Disable the queue in HW */
+ for (;;) {
+ rc = opal_xive_set_queue_info(vp_id, prio, 0, 0, 0);
+ break;
+ msleep(1);
+ }
+ if (rc)
+ pr_err("XIVE: Error %lld disabling queue for prio %d\n",
+ rc, prio);
+}
+
+void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
+{
+ __xive_native_disable_queue(vp_id, q, prio);
+
+ if (q->eoi_mmio)
+ iounmap(q->eoi_mmio);
+ q->eoi_mmio = NULL;
+}
+
+static int xive_native_setup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
+{
+ struct xive_q *q = &xc->queue[prio];
+ unsigned int alloc_order;
+ struct page *pages;
+ __be32 *qpage;
+
+ alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
+ (xive_queue_shift - PAGE_SHIFT) : 0;
+ pages = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL, alloc_order);
+ if (!pages)
+ return -ENOMEM;
+ qpage = (__be32 *)page_address(pages);
+ memset(qpage, 0, 1 << xive_queue_shift);
+ return xive_native_configure_queue(get_hard_smp_processor_id(cpu),
+ q, prio, qpage, xive_queue_shift, false);
+}
+
+static void xive_native_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
+{
+ struct xive_q *q = &xc->queue[prio];
+ unsigned int alloc_order;
+
+ /*
+ * We use the variant with no iounmap as this is called on exec
+ * from an IPI and iounmap isn't safe
+ */
+ __xive_native_disable_queue(get_hard_smp_processor_id(cpu), q, prio);
+ alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
+ (xive_queue_shift - PAGE_SHIFT) : 0;
+ free_pages((unsigned long)q->qpage, alloc_order);
+ q->qpage = NULL;
+}
+
+static bool xive_native_match(struct device_node *node)
+{
+ return of_device_is_compatible(node, "ibm,opal-xive-vc");
+}
+
+#ifdef CONFIG_SMP
+static int xive_native_get_ipi(unsigned int cpu, struct xive_cpu *xc)
+{
+ struct device_node *np;
+ unsigned int chip_id;
+ s64 irq;
+
+ /* Find the chip ID */
+ np = of_get_cpu_node(cpu, NULL);
+ if (np) {
+ if (of_property_read_u32(np, "ibm,chip-id", &chip_id) < 0)
+ chip_id = 0;
+ }
+
+ /* Allocate an IPI and populate info about it */
+ for (;;) {
+ irq = opal_xive_allocate_irq(chip_id);
+ if (irq = OPAL_BUSY) {
+ msleep(1);
+ continue;
+ }
+ if (irq < 0) {
+ pr_err("XIVE: Failed to allocate IPI on CPU %d\n",
+ cpu);
+ return -ENXIO;
+ }
+ xc->hw_ipi = irq;
+ break;
+ }
+ return 0;
+}
+
+u32 xive_native_alloc_irq(void)
+{
+ s64 rc;
+
+ for (;;) {
+ rc = opal_xive_allocate_irq(OPAL_XIVE_ANY_CHIP);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ if (rc < 0)
+ return 0;
+ return rc;
+}
+
+void xive_native_free_irq(u32 irq)
+{
+ for (;;) {
+ s64 rc = opal_xive_free_irq(irq);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+}
+
+static void xive_native_put_ipi(unsigned int cpu, struct xive_cpu *xc)
+{
+ s64 rc;
+
+ /* Free the IPI */
+ if (!xc->hw_ipi)
+ return;
+ for (;;) {
+ rc = opal_xive_free_irq(xc->hw_ipi);
+ if (rc = OPAL_BUSY) {
+ msleep(1);
+ continue;
+ }
+ xc->hw_ipi = 0;
+ break;
+ }
+}
+#endif /* CONFIG_SMP */
+
+static void xive_native_shutdown(void)
+{
+ /* Switch the XIVE to emulation mode */
+ opal_xive_reset(OPAL_XIVE_MODE_EMU);
+}
+
+static void xive_native_eoi(u32 hw_irq)
+{
+ /* Not normally used except if specific interrupts need
+ * a workaround on EOI
+ */
+ opal_int_eoi(hw_irq);
+}
+
+static void xive_native_setup_cpu(unsigned int cpu, struct xive_cpu *xc)
+{
+ s64 rc;
+ u32 vp;
+ __be64 vp_cam_be;
+ u64 vp_cam;
+
+ if (xive_pool_vps = XIVE_INVALID_VP)
+ return;
+
+ /* Enable the pool VP */
+ vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
+ pr_debug("XIVE: CPU %d setting up pool VP 0x%x\n", cpu, vp);
+ for (;;) {
+ rc = opal_xive_set_vp_info(vp, OPAL_XIVE_VP_ENABLED, 0);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ if (rc) {
+ pr_err("XIVE: Failed to enable pool VP on CPU %d\n", cpu);
+ return;
+ }
+
+ /* Grab it's CAM value */
+ rc = opal_xive_get_vp_info(vp, NULL, &vp_cam_be, NULL, NULL);
+ if (rc) {
+ pr_err("XIVE: Failed to get pool VP info CPU %d\n", cpu);
+ return;
+ }
+ vp_cam = be64_to_cpu(vp_cam_be);
+
+ pr_debug("XIVE: VP CAM = %llx\n", vp_cam);
+
+ /* Push it on the CPU (set LSMFB to 0xff to skip backlog scan) */
+ pr_debug("XIVE: (Old HW value: %08x)\n",
+ in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
+ out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD0, 0xff);
+ out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2,
+ TM_QW2W2_VP | vp_cam);
+ pr_debug("XIVE: (New HW value: %08x)\n",
+ in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
+}
+
+static void xive_native_teardown_cpu(unsigned int cpu, struct xive_cpu *xc)
+{
+ s64 rc;
+ u32 vp;
+
+ if (xive_pool_vps = XIVE_INVALID_VP)
+ return;
+
+ /* Pull the pool VP from the CPU */
+ in_be64(xive_tm_area + TM_SPC_PULL_POOL_CTX);
+
+ /* Disable it */
+ vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
+ for (;;) {
+ rc = opal_xive_set_vp_info(vp, 0, 0);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+}
+
+static void xive_native_sync_source(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_EAS, hw_irq);
+}
+
+static const struct xive_ops xive_native_ops = {
+ .populate_irq_data = xive_native_populate_irq_data,
+ .configure_irq = xive_native_configure_irq,
+ .setup_queue = xive_native_setup_queue,
+ .cleanup_queue = xive_native_cleanup_queue,
+ .match = xive_native_match,
+ .shutdown = xive_native_shutdown,
+ .eoi = xive_native_eoi,
+ .setup_cpu = xive_native_setup_cpu,
+ .teardown_cpu = xive_native_teardown_cpu,
+ .sync_source = xive_native_sync_source,
+#ifdef CONFIG_SMP
+ .get_ipi = xive_native_get_ipi,
+ .put_ipi = xive_native_put_ipi,
+#endif /* CONFIG_SMP */
+ .name = "native",
+};
+
+static bool xive_parse_provisioning(struct device_node *np)
+{
+ int rc;
+
+ if (of_property_read_u32(np, "ibm,xive-provision-page-size",
+ &xive_provision_size) < 0)
+ return true;
+ rc = of_property_count_elems_of_size(np, "ibm,xive-provision-chips", 4);
+ if (rc < 0) {
+ pr_err("XIVE: Error %d getting provision chips array\n", rc);
+ return false;
+ }
+ xive_provision_chip_count = rc;
+ if (rc = 0)
+ return true;
+
+ xive_provision_chips = kzalloc(4 * xive_provision_chip_count,
+ GFP_KERNEL);
+ BUG_ON(!xive_provision_chips);
+
+ rc = of_property_read_u32_array(np, "ibm,xive-provision-chips",
+ xive_provision_chips,
+ xive_provision_chip_count);
+ if (rc < 0) {
+ pr_err("XIVE: Error %d reading provision chips array\n", rc);
+ return false;
+ }
+
+ xive_provision_cache = kmem_cache_create("xive-provision",
+ xive_provision_size,
+ xive_provision_size,
+ 0, NULL);
+ if (!xive_provision_cache) {
+ pr_err("XIVE: Failed to allocate provision cache\n");
+ return false;
+ }
+ return true;
+}
+
+u32 xive_native_default_eq_shift(void)
+{
+ return xive_queue_shift;
+}
+
+bool xive_native_init(void)
+{
+ struct device_node *np;
+ struct resource r;
+ void __iomem *tm_area;
+ struct property *prop;
+ u8 max_prio = 7;
+ const __be32 *p;
+ u32 val;
+ s64 rc;
+
+ if (xive_cmdline_disabled)
+ return false;
+
+ DBG("xive_native_init()\n");
+ np = of_find_compatible_node(NULL, NULL, "ibm,opal-xive-pe");
+ if (!np) {
+ DBG("not found !\n");
+ return false;
+ }
+ DBG("Found %s\n", np->full_name);
+
+ /* Resource 1 is HV window */
+ if (of_address_to_resource(np, 1, &r)) {
+ pr_err("XIVE: Failed to get TM area resource\n");
+ return false;
+ }
+ tm_area = ioremap(r.start, resource_size(&r));
+ if (!tm_area) {
+ pr_err("XIVE: Failed to map TM area\n");
+ return false;
+ }
+
+ /* Read number of priorities */
+ if (of_property_read_u32(np, "ibm,xive-#priorities", &val) = 0)
+ max_prio = val - 1;
+
+ /* Iterate the EQ sizes and pick one */
+ of_property_for_each_u32(np, "ibm,xive-eq-sizes", prop, p, val) {
+ xive_queue_shift = val;
+ if (val = PAGE_SHIFT)
+ break;
+ }
+
+ /* Grab size of provisionning pages */
+ xive_parse_provisioning(np);
+
+ /* Switch the XIVE to exploitation mode */
+ rc = opal_xive_reset(OPAL_XIVE_MODE_EXPL);
+ if (rc) {
+ pr_err("XIVE: Switch to exploitation mode failed"
+ " with error %lld\n", rc);
+ return false;
+ }
+
+ /* Initialize XIVE core with our backend */
+ if (!xive_core_init(&xive_native_ops, tm_area, TM_QW3_HV_PHYS,
+ max_prio)) {
+ opal_xive_reset(OPAL_XIVE_MODE_EMU);
+ return false;
+ }
+ pr_info("XIVE: Using %dkB queues\n", 1 << (xive_queue_shift - 10));
+ return true;
+}
+
+static bool xive_native_provision_pages(void)
+{
+ u32 i;
+ void *p;
+
+ for (i = 0; i < xive_provision_chip_count; i++) {
+ u32 chip = xive_provision_chips[i];
+
+ /* XXX TODO: Try to make the allocation local to the node where
+ * the chip reside
+ */
+ p = kmem_cache_alloc(xive_provision_cache, GFP_KERNEL);
+ if (!p) {
+ pr_err("XIVE: Failed to allocate provisioning page\n");
+ return false;
+ }
+ opal_xive_donate_page(chip, __pa(p));
+ }
+ return true;
+}
+
+u32 xive_native_alloc_vp_block(u32 max_vcpus)
+{
+ s64 rc;
+ u32 order;
+
+ order = fls(max_vcpus) - 1;
+ pr_info("XIVE: VP block alloc, for max VCPUs %d"
+ " use order %d\n", max_vcpus, order);
+ for (;;) {
+ rc = opal_xive_alloc_vp_block(order);
+ switch (rc) {
+ case OPAL_BUSY:
+ msleep(1);
+ break;
+ case OPAL_XIVE_PROVISIONING:
+ if (!xive_native_provision_pages())
+ return XIVE_INVALID_VP;
+ break;
+ default:
+ if (rc < 0) {
+ pr_err("XIVE: OPAL failed to allocate VCPUs"
+ " order %d, err %lld\n",
+ order, rc);
+ return XIVE_INVALID_VP;
+ }
+ return rc;
+ }
+ }
+}
+EXPORT_SYMBOL_GPL(xive_native_alloc_vp_block);
+
+void xive_native_free_vp_block(u32 vp_base)
+{
+ s64 rc;
+
+ if (vp_base = XIVE_INVALID_VP)
+ return;
+
+ rc = opal_xive_free_vp_block(vp_base);
+ if (rc < 0)
+ pr_warn("XIVE: OPAL error %lld freeing VP block\n", rc);
+}
+EXPORT_SYMBOL_GPL(xive_native_free_vp_block);
diff --git a/arch/powerpc/sysdev/xive/xive-internal.h b/arch/powerpc/sysdev/xive/xive-internal.h
new file mode 100644
index 0000000..e736fc5
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/xive-internal.h
@@ -0,0 +1,51 @@
+#ifndef __XIVE_INTERNAL_H
+#define __XIVE_INTERNAL_H
+
+/* Each CPU carry one of these with various per-CPU state */
+struct xive_cpu {
+#ifdef CONFIG_SMP
+ /* HW irq number and data of IPI */
+ u32 hw_ipi;
+ struct xive_irq_data ipi_data;
+#endif /* CONFIG_SMP */
+
+ int chip_id;
+
+ /* Queue datas. Only one is populated */
+#define XIVE_MAX_QUEUES 8
+ struct xive_q queue[XIVE_MAX_QUEUES];
+
+ /* Pending mask. Each bit corresponds to a priority that
+ * potentially has pending interrupts
+ */
+ u8 pending_prio;
+
+ /* Cache of HW CPPR */
+ u8 cppr;
+};
+
+/* Backend ops */
+struct xive_ops {
+ int (*populate_irq_data)(u32 hw_irq, struct xive_irq_data *data);
+ int (*configure_irq)(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
+ int (*setup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
+ void (*cleanup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
+ void (*setup_cpu)(unsigned int cpu, struct xive_cpu *xc);
+ void (*teardown_cpu)(unsigned int cpu, struct xive_cpu *xc);
+ bool (*match)(struct device_node *np);
+ void (*shutdown)(void);
+ void (*eoi)(u32 hw_irq);
+ void (*sync_source)(u32 hw_irq);
+#ifdef CONFIG_SMP
+ int (*get_ipi)(unsigned int cpu, struct xive_cpu *xc);
+ void (*put_ipi)(unsigned int cpu, struct xive_cpu *xc);
+#endif
+ const char *name;
+};
+
+bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
+ u8 max_prio);
+
+extern bool xive_cmdline_disabled;
+
+#endif /* __XIVE_INTERNAL_H */
diff --git a/arch/powerpc/sysdev/xive/xive-regs.h b/arch/powerpc/sysdev/xive/xive-regs.h
new file mode 100644
index 0000000..f1edb23
--- /dev/null
+++ b/arch/powerpc/sysdev/xive/xive-regs.h
@@ -0,0 +1,88 @@
+#ifndef __XIVE_REGS_H__
+#define __XIVE_REGS_H__
+
+/*
+ * TM registers
+ */
+
+/* TM register offsets */
+#define TM_QW0_USER 0x000 /* All rings */
+#define TM_QW1_OS 0x010 /* Ring 0..2 */
+#define TM_QW2_HV_POOL 0x020 /* Ring 0..1 */
+#define TM_QW3_HV_PHYS 0x030 /* Ring 0..1 */
+
+/* Byte offsets inside a QW QW0 QW1 QW2 QW3 */
+#define TM_NSR 0x0 /* + + - + */
+#define TM_CPPR 0x1 /* - + - + */
+#define TM_IPB 0x2 /* - + + + */
+#define TM_LSMFB 0x3 /* - + + + */
+#define TM_ACK_CNT 0x4 /* - + - - */
+#define TM_INC 0x5 /* - + - + */
+#define TM_AGE 0x6 /* - + - + */
+#define TM_PIPR 0x7 /* - + - + */
+
+#define TM_WORD0 0x0
+#define TM_WORD1 0x4
+
+/* QW word 2 contains the valid bit at the top and other fields
+ * depending on the QW
+ */
+#define TM_WORD2 0x8
+#define TM_QW0W2_VU PPC_BIT32(0)
+#define TM_QW0W2_LOGIC_SERV PPC_BITMASK32(1,31) // XX 2,31 ?
+#define TM_QW1W2_VO PPC_BIT32(0)
+#define TM_QW1W2_OS_CAM PPC_BITMASK32(8,31)
+#define TM_QW2W2_VP PPC_BIT32(0)
+#define TM_QW2W2_POOL_CAM PPC_BITMASK32(8,31)
+#define TM_QW3W2_VT PPC_BIT32(0)
+#define TM_QW3W2_LP PPC_BIT32(6)
+#define TM_QW3W2_LE PPC_BIT32(7)
+#define TM_QW3W2_T PPC_BIT32(31)
+
+/* In addition to normal loads to "peek" and writes (only when invalid)
+ * using 4 and 8 bytes accesses, the above registers support these
+ * "special" byte operations:
+ *
+ * - Byte load from QW0[NSR] - User level NSR (EBB)
+ * - Byte store to QW0[NSR] - User level NSR (EBB)
+ * - Byte load/store to QW1[CPPR] and QW3[CPPR] - CPPR access
+ * - Byte load from QW3[TM_WORD2] - Read VT||00000||LP||LE on thrd 0
+ * otherwise VT||0000000
+ * - Byte store to QW3[TM_WORD2] - Set VT bit (and LP/LE if present)
+ *
+ * Then we have all these "special" CI ops at these offset that trigger
+ * all sorts of side effects:
+ */
+#define TM_SPC_ACK_EBB 0x800 /* Load8 ack EBB to reg*/
+#define TM_SPC_ACK_OS_REG 0x810 /* Load16 ack OS irq to reg */
+#define TM_SPC_PUSH_USR_CTX 0x808 /* Store32 Push/Validate user context */
+#define TM_SPC_PULL_USR_CTX 0x808 /* Load32 Pull/Invalidate user context */
+#define TM_SPC_SET_OS_PENDING 0x812 /* Store8 Set OS irq pending bit */
+#define TM_SPC_PULL_OS_CTX 0x818 /* Load32/Load64 Pull/Invalidate OS context to reg */
+#define TM_SPC_PULL_POOL_CTX 0x828 /* Load32/Load64 Pull/Invalidate Pool context to reg*/
+#define TM_SPC_ACK_HV_REG 0x830 /* Load16 ack HV irq to reg */
+#define TM_SPC_PULL_USR_CTX_OL 0xc08 /* Store8 Pull/Inval usr ctx to odd line */
+#define TM_SPC_ACK_OS_EL 0xc10 /* Store8 ack OS irq to even line */
+#define TM_SPC_ACK_HV_POOL_EL 0xc20 /* Store8 ack HV evt pool to even line */
+#define TM_SPC_ACK_HV_EL 0xc30 /* Store8 ack HV irq to even line */
+/* XXX more... */
+
+/* NSR fields for the various QW ack types */
+#define TM_QW0_NSR_EB PPC_BIT8(0)
+#define TM_QW1_NSR_EO PPC_BIT8(0)
+#define TM_QW3_NSR_HE PPC_BITMASK8(0,1)
+#define TM_QW3_NSR_HE_NONE 0
+#define TM_QW3_NSR_HE_POOL 1
+#define TM_QW3_NSR_HE_PHYS 2
+#define TM_QW3_NSR_HE_LSI 3
+#define TM_QW3_NSR_I PPC_BIT8(2)
+#define TM_QW3_NSR_GRP_LVL PPC_BIT8(3,7)
+
+/* Utilities to manipulate these (originaly from OPAL) */
+#define MASK_TO_LSH(m) (__builtin_ffsl(m) - 1)
+#define GETFIELD(m, v) (((v) & (m)) >> MASK_TO_LSH(m))
+#define SETFIELD(m, v, val) \
+ (((v) & ~(m)) | ((((typeof(v))(val)) << MASK_TO_LSH(m)) & (m)))
+
+
+#endif /* __XIVE_H__ */
diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
index 16321ad..c71e919 100644
--- a/arch/powerpc/xmon/xmon.c
+++ b/arch/powerpc/xmon/xmon.c
@@ -48,7 +48,7 @@
#include <asm/reg.h>
#include <asm/debug.h>
#include <asm/hw_breakpoint.h>
-
+#include <asm/xive.h>
#include <asm/opal.h>
#include <asm/firmware.h>
@@ -232,7 +232,13 @@ Commands:\n\
"\
dr dump stream of raw bytes\n\
dt dump the tracing buffers (uses printk)\n\
- e print exception information\n\
+"
+#ifdef CONFIG_PPC_POWERNV
+" dx# dump xive on CPU\n\
+ dxi# dump xive irq state\n\
+ dxa dump xive on all CPUs\n"
+#endif
+" e print exception information\n\
f flush cache\n\
la lookup symbol+offset of specified address\n\
ls lookup address of specified symbol\n\
@@ -2338,6 +2344,81 @@ static void dump_pacas(void)
}
#endif
+#ifdef CONFIG_PPC_POWERNV
+static void dump_one_xive(int cpu)
+{
+ unsigned int hwid = get_hard_smp_processor_id(cpu);
+
+ opal_xive_dump(XIVE_DUMP_TM_HYP, hwid);
+ opal_xive_dump(XIVE_DUMP_TM_POOL, hwid);
+ opal_xive_dump(XIVE_DUMP_TM_OS, hwid);
+ opal_xive_dump(XIVE_DUMP_TM_USER, hwid);
+ opal_xive_dump(XIVE_DUMP_VP, hwid);
+ opal_xive_dump(XIVE_DUMP_EMU_STATE, hwid);
+
+ if (setjmp(bus_error_jmp) != 0) {
+ catch_memory_errors = 0;
+ printf("*** Error dumping xive on cpu %d\n", cpu);
+ return;
+ }
+
+ catch_memory_errors = 1;
+ sync();
+ xmon_xive_do_dump(cpu);
+ sync();
+ __delay(200);
+ catch_memory_errors = 0;
+}
+
+static void dump_all_xives(void)
+{
+ int cpu;
+
+ if (num_possible_cpus() = 0) {
+ printf("No possible cpus, use 'dx #' to dump individual cpus\n");
+ return;
+ }
+
+ for_each_possible_cpu(cpu)
+ dump_one_xive(cpu);
+}
+
+static void dump_one_xive_irq(uint32_t num)
+{
+ int64_t rc;
+ __be64 vp;
+ uint8_t prio;
+ __be32 lirq;
+
+ rc = opal_xive_get_irq_config(num, &vp, &prio, &lirq);
+ xmon_printf("IRQ 0x%x config: vp=0x%llx prio=%d lirq=0x%x (rc=%lld)\n",
+ num, be64_to_cpu(vp), prio, be32_to_cpu(lirq), rc);
+}
+
+static void dump_xives(void)
+{
+ unsigned long num;
+ int c;
+
+ c = inchar();
+ if (c = 'a') {
+ dump_all_xives();
+ return;
+ } else if (c = 'i') {
+ if (scanhex(&num))
+ dump_one_xive_irq(num);
+ return;
+ }
+
+ termch = c; /* Put c back, it wasn't 'a' */
+
+ if (scanhex(&num))
+ dump_one_xive(num);
+ else
+ dump_one_xive(xmon_owner);
+}
+#endif /* CONFIG_PPC_POWERNV */
+
static void dump_by_size(unsigned long addr, long count, int size)
{
unsigned char temp[16];
@@ -2386,6 +2467,14 @@ dump(void)
return;
}
#endif
+#ifdef CONFIG_PPC_POWERNV
+ if (c = 'x') {
+ xmon_start_pagination();
+ dump_xives();
+ xmon_end_pagination();
+ return;
+ }
+#endif
if (c = '\n')
termch = c;
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 07/12] powerpc/kvm: Massage order of #include
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
We traditionally have linux/ before asm/
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/kvm/book3s.c | 8 ++++----
arch/powerpc/kvm/book3s_hv.c | 18 +++++++++---------
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index b6b5c18..aedacef 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -20,6 +20,10 @@
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/miscdevice.h>
+#include <linux/gfp.h>
+#include <linux/sched.h>
+#include <linux/vmalloc.h>
+#include <linux/highmem.h>
#include <asm/reg.h>
#include <asm/cputable.h>
@@ -31,10 +35,6 @@
#include <asm/kvm_book3s.h>
#include <asm/mmu_context.h>
#include <asm/page.h>
-#include <linux/gfp.h>
-#include <linux/sched.h>
-#include <linux/vmalloc.h>
-#include <linux/highmem.h>
#include "book3s.h"
#include "trace.h"
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 1ec86d9..fadb75a 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -35,6 +35,15 @@
#include <linux/srcu.h>
#include <linux/miscdevice.h>
#include <linux/debugfs.h>
+#include <linux/gfp.h>
+#include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/hugetlb.h>
+#include <linux/kvm_irqfd.h>
+#include <linux/irqbypass.h>
+#include <linux/module.h>
+#include <linux/compiler.h>
+#include <linux/of.h>
#include <asm/reg.h>
#include <asm/cputable.h>
@@ -58,15 +67,6 @@
#include <asm/mmu.h>
#include <asm/opal.h>
#include <asm/xics.h>
-#include <linux/gfp.h>
-#include <linux/vmalloc.h>
-#include <linux/highmem.h>
-#include <linux/hugetlb.h>
-#include <linux/kvm_irqfd.h>
-#include <linux/irqbypass.h>
-#include <linux/module.h>
-#include <linux/compiler.h>
-#include <linux/of.h>
#include "book3s.h"
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 07/12] powerpc/kvm: Massage order of #include
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
We traditionally have linux/ before asm/
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/kvm/book3s.c | 8 ++++----
arch/powerpc/kvm/book3s_hv.c | 18 +++++++++---------
2 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index b6b5c18..aedacef 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -20,6 +20,10 @@
#include <linux/slab.h>
#include <linux/module.h>
#include <linux/miscdevice.h>
+#include <linux/gfp.h>
+#include <linux/sched.h>
+#include <linux/vmalloc.h>
+#include <linux/highmem.h>
#include <asm/reg.h>
#include <asm/cputable.h>
@@ -31,10 +35,6 @@
#include <asm/kvm_book3s.h>
#include <asm/mmu_context.h>
#include <asm/page.h>
-#include <linux/gfp.h>
-#include <linux/sched.h>
-#include <linux/vmalloc.h>
-#include <linux/highmem.h>
#include "book3s.h"
#include "trace.h"
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index 1ec86d9..fadb75a 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -35,6 +35,15 @@
#include <linux/srcu.h>
#include <linux/miscdevice.h>
#include <linux/debugfs.h>
+#include <linux/gfp.h>
+#include <linux/vmalloc.h>
+#include <linux/highmem.h>
+#include <linux/hugetlb.h>
+#include <linux/kvm_irqfd.h>
+#include <linux/irqbypass.h>
+#include <linux/module.h>
+#include <linux/compiler.h>
+#include <linux/of.h>
#include <asm/reg.h>
#include <asm/cputable.h>
@@ -58,15 +67,6 @@
#include <asm/mmu.h>
#include <asm/opal.h>
#include <asm/xics.h>
-#include <linux/gfp.h>
-#include <linux/vmalloc.h>
-#include <linux/highmem.h>
-#include <linux/hugetlb.h>
-#include <linux/kvm_irqfd.h>
-#include <linux/irqbypass.h>
-#include <linux/module.h>
-#include <linux/compiler.h>
-#include <linux/of.h>
#include "book3s.h"
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 08/12] powerpc/kvm: Make kvmppc_xics_create_icp static
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
It's only used within the same file it's defined
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/kvm_ppc.h | 4 ----
arch/powerpc/kvm/book3s_xics.c | 2 +-
2 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index dd11c4c..bfef1ae 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -478,7 +478,6 @@ extern void kvmppc_free_host_rm_ops(void);
extern void kvmppc_free_pimap(struct kvm *kvm);
extern int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall);
extern void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu);
-extern int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu, unsigned long server);
extern int kvm_vm_ioctl_xics_irq(struct kvm *kvm, struct kvm_irq_level *args);
extern int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd);
extern u64 kvmppc_xics_get_icp(struct kvm_vcpu *vcpu);
@@ -507,9 +506,6 @@ static inline int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall)
static inline int kvmppc_xics_enabled(struct kvm_vcpu *vcpu)
{ return 0; }
static inline void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu) { }
-static inline int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu,
- unsigned long server)
- { return -EINVAL; }
static inline int kvm_vm_ioctl_xics_irq(struct kvm *kvm,
struct kvm_irq_level *args)
{ return -ENOTTY; }
diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
index e48803e..ef4fd52 100644
--- a/arch/powerpc/kvm/book3s_xics.c
+++ b/arch/powerpc/kvm/book3s_xics.c
@@ -1084,7 +1084,7 @@ static struct kvmppc_ics *kvmppc_xics_create_ics(struct kvm *kvm,
return xics->ics[icsid];
}
-int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu, unsigned long server_num)
+static int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu, unsigned long server_num)
{
struct kvmppc_icp *icp;
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 08/12] powerpc/kvm: Make kvmppc_xics_create_icp static
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
It's only used within the same file it's defined
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/kvm_ppc.h | 4 ----
arch/powerpc/kvm/book3s_xics.c | 2 +-
2 files changed, 1 insertion(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index dd11c4c..bfef1ae 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -478,7 +478,6 @@ extern void kvmppc_free_host_rm_ops(void);
extern void kvmppc_free_pimap(struct kvm *kvm);
extern int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall);
extern void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu);
-extern int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu, unsigned long server);
extern int kvm_vm_ioctl_xics_irq(struct kvm *kvm, struct kvm_irq_level *args);
extern int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd);
extern u64 kvmppc_xics_get_icp(struct kvm_vcpu *vcpu);
@@ -507,9 +506,6 @@ static inline int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall)
static inline int kvmppc_xics_enabled(struct kvm_vcpu *vcpu)
{ return 0; }
static inline void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu) { }
-static inline int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu,
- unsigned long server)
- { return -EINVAL; }
static inline int kvm_vm_ioctl_xics_irq(struct kvm *kvm,
struct kvm_irq_level *args)
{ return -ENOTTY; }
diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
index e48803e..ef4fd52 100644
--- a/arch/powerpc/kvm/book3s_xics.c
+++ b/arch/powerpc/kvm/book3s_xics.c
@@ -1084,7 +1084,7 @@ static struct kvmppc_ics *kvmppc_xics_create_ics(struct kvm *kvm,
return xics->ics[icsid];
}
-int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu, unsigned long server_num)
+static int kvmppc_xics_create_icp(struct kvm_vcpu *vcpu, unsigned long server_num)
{
struct kvmppc_icp *icp;
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 09/12] powerpc/kvm: Remove obsolete kvm_vm_ioctl_xics_irq declaration
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
The function doesn't exist anymore
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/kvm_ppc.h | 4 ----
1 file changed, 4 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index bfef1ae..0c41865 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -478,7 +478,6 @@ extern void kvmppc_free_host_rm_ops(void);
extern void kvmppc_free_pimap(struct kvm *kvm);
extern int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall);
extern void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu);
-extern int kvm_vm_ioctl_xics_irq(struct kvm *kvm, struct kvm_irq_level *args);
extern int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd);
extern u64 kvmppc_xics_get_icp(struct kvm_vcpu *vcpu);
extern int kvmppc_xics_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
@@ -506,9 +505,6 @@ static inline int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall)
static inline int kvmppc_xics_enabled(struct kvm_vcpu *vcpu)
{ return 0; }
static inline void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu) { }
-static inline int kvm_vm_ioctl_xics_irq(struct kvm *kvm,
- struct kvm_irq_level *args)
- { return -ENOTTY; }
static inline int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd)
{ return 0; }
#endif
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 09/12] powerpc/kvm: Remove obsolete kvm_vm_ioctl_xics_irq declaration
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
The function doesn't exist anymore
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/kvm_ppc.h | 4 ----
1 file changed, 4 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index bfef1ae..0c41865 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -478,7 +478,6 @@ extern void kvmppc_free_host_rm_ops(void);
extern void kvmppc_free_pimap(struct kvm *kvm);
extern int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall);
extern void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu);
-extern int kvm_vm_ioctl_xics_irq(struct kvm *kvm, struct kvm_irq_level *args);
extern int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd);
extern u64 kvmppc_xics_get_icp(struct kvm_vcpu *vcpu);
extern int kvmppc_xics_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
@@ -506,9 +505,6 @@ static inline int kvmppc_xics_rm_complete(struct kvm_vcpu *vcpu, u32 hcall)
static inline int kvmppc_xics_enabled(struct kvm_vcpu *vcpu)
{ return 0; }
static inline void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu) { }
-static inline int kvm_vm_ioctl_xics_irq(struct kvm *kvm,
- struct kvm_irq_level *args)
- { return -ENOTTY; }
static inline int kvmppc_xics_hcall(struct kvm_vcpu *vcpu, u32 cmd)
{ return 0; }
#endif
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 10/12] powerpc: Consolidate variants of real-mode MMIOs
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
We have all sort of variants of MMIO accessors for the real mode
instructions. This creates a clean set of accessors based on
Linux normal naming conventions, replacing all occurrences of
the old ones in the tree.
I have purposefully removed the "out/in" variants in favor of
only including __raw variants. Any code using these is already
pretty much hand tuned to operate in a very specific environment.
I've fixed up the 2 users (only one of them actually needed
a barrier in the first place).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/io.h | 98 ++++++++++++++++---------------
arch/powerpc/include/asm/kvm_book3s_asm.h | 2 +-
arch/powerpc/include/asm/kvm_ppc.h | 2 +-
arch/powerpc/kvm/book3s_hv_builtin.c | 21 +++----
arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +-
arch/powerpc/platforms/powernv/rng.c | 2 +-
arch/powerpc/sysdev/xics/icp-native.c | 8 +--
7 files changed, 68 insertions(+), 69 deletions(-)
diff --git a/arch/powerpc/include/asm/io.h b/arch/powerpc/include/asm/io.h
index 5ed2924..45c136a 100644
--- a/arch/powerpc/include/asm/io.h
+++ b/arch/powerpc/include/asm/io.h
@@ -192,24 +192,8 @@ DEF_MMIO_OUT_D(out_le32, 32, stw);
#endif /* __BIG_ENDIAN */
-/*
- * Cache inhibitied accessors for use in real mode, you don't want to use these
- * unless you know what you're doing.
- *
- * NB. These use the cpu byte ordering.
- */
-DEF_MMIO_OUT_X(out_rm8, 8, stbcix);
-DEF_MMIO_OUT_X(out_rm16, 16, sthcix);
-DEF_MMIO_OUT_X(out_rm32, 32, stwcix);
-DEF_MMIO_IN_X(in_rm8, 8, lbzcix);
-DEF_MMIO_IN_X(in_rm16, 16, lhzcix);
-DEF_MMIO_IN_X(in_rm32, 32, lwzcix);
-
#ifdef __powerpc64__
-DEF_MMIO_OUT_X(out_rm64, 64, stdcix);
-DEF_MMIO_IN_X(in_rm64, 64, ldcix);
-
#ifdef __BIG_ENDIAN__
DEF_MMIO_OUT_D(out_be64, 64, std);
DEF_MMIO_IN_D(in_be64, 64, ld);
@@ -242,35 +226,6 @@ static inline void out_be64(volatile u64 __iomem *addr, u64 val)
#endif
#endif /* __powerpc64__ */
-
-/*
- * Simple Cache inhibited accessors
- * Unlike the DEF_MMIO_* macros, these don't include any h/w memory
- * barriers, callers need to manage memory barriers on their own.
- * These can only be used in hypervisor real mode.
- */
-
-static inline u32 _lwzcix(unsigned long addr)
-{
- u32 ret;
-
- __asm__ __volatile__("lwzcix %0,0, %1"
- : "=r" (ret) : "r" (addr) : "memory");
- return ret;
-}
-
-static inline void _stbcix(u64 addr, u8 val)
-{
- __asm__ __volatile__("stbcix %0,0,%1"
- : : "r" (val), "r" (addr) : "memory");
-}
-
-static inline void _stwcix(u64 addr, u32 val)
-{
- __asm__ __volatile__("stwcix %0,0,%1"
- : : "r" (val), "r" (addr) : "memory");
-}
-
/*
* Low level IO stream instructions are defined out of line for now
*/
@@ -417,15 +372,64 @@ static inline void __raw_writeq(unsigned long v, volatile void __iomem *addr)
}
/*
- * Real mode version of the above. stdcix is only supposed to be used
- * in hypervisor real mode as per the architecture spec.
+ * Real mode versions of the above. Those instructions are only supposed
+ * to be used in hypervisor real mode as per the architecture spec.
*/
+static inline void __raw_rm_writeb(u8 val, volatile void __iomem *paddr)
+{
+ __asm__ __volatile__("stbcix %0,0,%1"
+ : : "r" (val), "r" (paddr) : "memory");
+}
+
+static inline void __raw_rm_writew(u16 val, volatile void __iomem *paddr)
+{
+ __asm__ __volatile__("sthcix %0,0,%1"
+ : : "r" (val), "r" (paddr) : "memory");
+}
+
+static inline void __raw_rm_writel(u32 val, volatile void __iomem *paddr)
+{
+ __asm__ __volatile__("stwcix %0,0,%1"
+ : : "r" (val), "r" (paddr) : "memory");
+}
+
static inline void __raw_rm_writeq(u64 val, volatile void __iomem *paddr)
{
__asm__ __volatile__("stdcix %0,0,%1"
: : "r" (val), "r" (paddr) : "memory");
}
+static inline u8 __raw_rm_readb(volatile void __iomem *paddr)
+{
+ u8 ret;
+ __asm__ __volatile__("lbzcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
+
+static inline u16 __raw_rm_readw(volatile void __iomem *paddr)
+{
+ u16 ret;
+ __asm__ __volatile__("lhzcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
+
+static inline u32 __raw_rm_readl(volatile void __iomem *paddr)
+{
+ u32 ret;
+ __asm__ __volatile__("lwzcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
+
+static inline u64 __raw_rm_readq(volatile void __iomem *paddr)
+{
+ u64 ret;
+ __asm__ __volatile__("ldcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
#endif /* __powerpc64__ */
/*
diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
index d318d43..0593d94 100644
--- a/arch/powerpc/include/asm/kvm_book3s_asm.h
+++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
@@ -110,7 +110,7 @@ struct kvmppc_host_state {
u8 ptid;
struct kvm_vcpu *kvm_vcpu;
struct kvmppc_vcore *kvm_vcore;
- unsigned long xics_phys;
+ void __iomem *xics_phys;
u32 saved_xirr;
u64 dabr;
u64 host_mmcr[7]; /* MMCR 0,1,A, SIAR, SDAR, MMCR2, SIER */
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 0c41865..c387799 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -409,7 +409,7 @@ struct openpic;
extern void kvm_cma_reserve(void) __init;
static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)
{
- paca[cpu].kvm_hstate.xics_phys = addr;
+ paca[cpu].kvm_hstate.xics_phys = (void __iomem *)addr;
}
static inline u32 kvmppc_get_xics_latch(void)
diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c
index 4d6c64b..d48f9b6 100644
--- a/arch/powerpc/kvm/book3s_hv_builtin.c
+++ b/arch/powerpc/kvm/book3s_hv_builtin.c
@@ -193,12 +193,6 @@ long kvmppc_h_random(struct kvm_vcpu *vcpu)
return H_HARDWARE;
}
-static inline void rm_writeb(unsigned long paddr, u8 val)
-{
- __asm__ __volatile__("stbcix %0,0,%1"
- : : "r" (val), "r" (paddr) : "memory");
-}
-
/*
* Send an interrupt or message to another CPU.
* The caller needs to include any barrier needed to order writes
@@ -206,7 +200,7 @@ static inline void rm_writeb(unsigned long paddr, u8 val)
*/
void kvmhv_rm_send_ipi(int cpu)
{
- unsigned long xics_phys;
+ void __iomem *xics_phys;
unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
/* On POWER9 we can use msgsnd for any destination cpu. */
@@ -227,7 +221,7 @@ void kvmhv_rm_send_ipi(int cpu)
/* Else poke the target with an IPI */
xics_phys = paca[cpu].kvm_hstate.xics_phys;
if (xics_phys)
- rm_writeb(xics_phys + XICS_MFRR, IPI_PRIORITY);
+ __raw_rm_writeb(IPI_PRIORITY, xics_phys + XICS_MFRR);
else
opal_int_set_mfrr(get_hard_smp_processor_id(cpu), IPI_PRIORITY);
}
@@ -397,7 +391,7 @@ long kvmppc_read_intr(void)
static long kvmppc_read_one_intr(bool *again)
{
- unsigned long xics_phys;
+ void __iomem *xics_phys;
u32 h_xirr;
__be32 xirr;
u32 xisr;
@@ -415,7 +409,7 @@ static long kvmppc_read_one_intr(bool *again)
if (!xics_phys)
rc = opal_int_get_xirr(&xirr, false);
else
- xirr = _lwzcix(xics_phys + XICS_XIRR);
+ xirr = __raw_rm_readl(xics_phys + XICS_XIRR);
if (rc < 0)
return 1;
@@ -445,8 +439,8 @@ static long kvmppc_read_one_intr(bool *again)
if (xisr == XICS_IPI) {
rc = 0;
if (xics_phys) {
- _stbcix(xics_phys + XICS_MFRR, 0xff);
- _stwcix(xics_phys + XICS_XIRR, xirr);
+ __raw_rm_writeb(0xff, xics_phys + XICS_MFRR);
+ __raw_rm_writel(xirr, xics_phys + XICS_XIRR);
} else {
opal_int_set_mfrr(hard_smp_processor_id(), 0xff);
rc = opal_int_eoi(h_xirr);
@@ -471,7 +465,8 @@ static long kvmppc_read_one_intr(bool *again)
* we need to resend that IPI, bummer
*/
if (xics_phys)
- _stbcix(xics_phys + XICS_MFRR, IPI_PRIORITY);
+ __raw_rm_writeb(IPI_PRIORITY,
+ xics_phys + XICS_MFRR);
else
opal_int_set_mfrr(hard_smp_processor_id(),
IPI_PRIORITY);
diff --git a/arch/powerpc/kvm/book3s_hv_rm_xics.c b/arch/powerpc/kvm/book3s_hv_rm_xics.c
index e78542d..3a1a463 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_xics.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_xics.c
@@ -766,7 +766,7 @@ unsigned long eoi_rc;
static void icp_eoi(struct irq_chip *c, u32 hwirq, __be32 xirr, bool *again)
{
- unsigned long xics_phys;
+ void __iomem *xics_phys;
int64_t rc;
rc = pnv_opal_pci_msi_eoi(c, hwirq);
@@ -779,7 +779,7 @@ static void icp_eoi(struct irq_chip *c, u32 hwirq, __be32 xirr, bool *again)
/* EOI it */
xics_phys = local_paca->kvm_hstate.xics_phys;
if (xics_phys) {
- _stwcix(xics_phys + XICS_XIRR, xirr);
+ __raw_rm_writel(xirr, xics_phys + XICS_XIRR);
} else {
rc = opal_int_eoi(be32_to_cpu(xirr));
*again = rc > 0;
diff --git a/arch/powerpc/platforms/powernv/rng.c b/arch/powerpc/platforms/powernv/rng.c
index 5dcbdea..1a9d843 100644
--- a/arch/powerpc/platforms/powernv/rng.c
+++ b/arch/powerpc/platforms/powernv/rng.c
@@ -62,7 +62,7 @@ int powernv_get_random_real_mode(unsigned long *v)
rng = raw_cpu_read(powernv_rng);
- *v = rng_whiten(rng, in_rm64(rng->regs_real));
+ *v = rng_whiten(rng, __raw_rm_readq(rng->regs_real));
return 1;
}
diff --git a/arch/powerpc/sysdev/xics/icp-native.c b/arch/powerpc/sysdev/xics/icp-native.c
index 8a6a043..f0f3f47 100644
--- a/arch/powerpc/sysdev/xics/icp-native.c
+++ b/arch/powerpc/sysdev/xics/icp-native.c
@@ -168,15 +168,15 @@ void icp_native_cause_ipi_rm(int cpu)
* Need the physical address of the XICS to be
* previously saved in kvm_hstate in the paca.
*/
- unsigned long xics_phys;
+ void __iomem *xics_phys;
/*
* Just like the cause_ipi functions, it is required to
- * include a full barrier (out8 includes a sync) before
- * causing the IPI.
+ * include a full barrier before causing the IPI.
*/
xics_phys = paca[cpu].kvm_hstate.xics_phys;
- out_rm8((u8 *)(xics_phys + XICS_MFRR), IPI_PRIORITY);
+ mb();
+ __raw_rm_writeb(IPI_PRIORITY, xics_phys + XICS_MFRR);
}
#endif
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 10/12] powerpc: Consolidate variants of real-mode MMIOs
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
We have all sort of variants of MMIO accessors for the real mode
instructions. This creates a clean set of accessors based on
Linux normal naming conventions, replacing all occurrences of
the old ones in the tree.
I have purposefully removed the "out/in" variants in favor of
only including __raw variants. Any code using these is already
pretty much hand tuned to operate in a very specific environment.
I've fixed up the 2 users (only one of them actually needed
a barrier in the first place).
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/io.h | 98 ++++++++++++++++---------------
arch/powerpc/include/asm/kvm_book3s_asm.h | 2 +-
arch/powerpc/include/asm/kvm_ppc.h | 2 +-
arch/powerpc/kvm/book3s_hv_builtin.c | 21 +++----
arch/powerpc/kvm/book3s_hv_rm_xics.c | 4 +-
arch/powerpc/platforms/powernv/rng.c | 2 +-
arch/powerpc/sysdev/xics/icp-native.c | 8 +--
7 files changed, 68 insertions(+), 69 deletions(-)
diff --git a/arch/powerpc/include/asm/io.h b/arch/powerpc/include/asm/io.h
index 5ed2924..45c136a 100644
--- a/arch/powerpc/include/asm/io.h
+++ b/arch/powerpc/include/asm/io.h
@@ -192,24 +192,8 @@ DEF_MMIO_OUT_D(out_le32, 32, stw);
#endif /* __BIG_ENDIAN */
-/*
- * Cache inhibitied accessors for use in real mode, you don't want to use these
- * unless you know what you're doing.
- *
- * NB. These use the cpu byte ordering.
- */
-DEF_MMIO_OUT_X(out_rm8, 8, stbcix);
-DEF_MMIO_OUT_X(out_rm16, 16, sthcix);
-DEF_MMIO_OUT_X(out_rm32, 32, stwcix);
-DEF_MMIO_IN_X(in_rm8, 8, lbzcix);
-DEF_MMIO_IN_X(in_rm16, 16, lhzcix);
-DEF_MMIO_IN_X(in_rm32, 32, lwzcix);
-
#ifdef __powerpc64__
-DEF_MMIO_OUT_X(out_rm64, 64, stdcix);
-DEF_MMIO_IN_X(in_rm64, 64, ldcix);
-
#ifdef __BIG_ENDIAN__
DEF_MMIO_OUT_D(out_be64, 64, std);
DEF_MMIO_IN_D(in_be64, 64, ld);
@@ -242,35 +226,6 @@ static inline void out_be64(volatile u64 __iomem *addr, u64 val)
#endif
#endif /* __powerpc64__ */
-
-/*
- * Simple Cache inhibited accessors
- * Unlike the DEF_MMIO_* macros, these don't include any h/w memory
- * barriers, callers need to manage memory barriers on their own.
- * These can only be used in hypervisor real mode.
- */
-
-static inline u32 _lwzcix(unsigned long addr)
-{
- u32 ret;
-
- __asm__ __volatile__("lwzcix %0,0, %1"
- : "=r" (ret) : "r" (addr) : "memory");
- return ret;
-}
-
-static inline void _stbcix(u64 addr, u8 val)
-{
- __asm__ __volatile__("stbcix %0,0,%1"
- : : "r" (val), "r" (addr) : "memory");
-}
-
-static inline void _stwcix(u64 addr, u32 val)
-{
- __asm__ __volatile__("stwcix %0,0,%1"
- : : "r" (val), "r" (addr) : "memory");
-}
-
/*
* Low level IO stream instructions are defined out of line for now
*/
@@ -417,15 +372,64 @@ static inline void __raw_writeq(unsigned long v, volatile void __iomem *addr)
}
/*
- * Real mode version of the above. stdcix is only supposed to be used
- * in hypervisor real mode as per the architecture spec.
+ * Real mode versions of the above. Those instructions are only supposed
+ * to be used in hypervisor real mode as per the architecture spec.
*/
+static inline void __raw_rm_writeb(u8 val, volatile void __iomem *paddr)
+{
+ __asm__ __volatile__("stbcix %0,0,%1"
+ : : "r" (val), "r" (paddr) : "memory");
+}
+
+static inline void __raw_rm_writew(u16 val, volatile void __iomem *paddr)
+{
+ __asm__ __volatile__("sthcix %0,0,%1"
+ : : "r" (val), "r" (paddr) : "memory");
+}
+
+static inline void __raw_rm_writel(u32 val, volatile void __iomem *paddr)
+{
+ __asm__ __volatile__("stwcix %0,0,%1"
+ : : "r" (val), "r" (paddr) : "memory");
+}
+
static inline void __raw_rm_writeq(u64 val, volatile void __iomem *paddr)
{
__asm__ __volatile__("stdcix %0,0,%1"
: : "r" (val), "r" (paddr) : "memory");
}
+static inline u8 __raw_rm_readb(volatile void __iomem *paddr)
+{
+ u8 ret;
+ __asm__ __volatile__("lbzcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
+
+static inline u16 __raw_rm_readw(volatile void __iomem *paddr)
+{
+ u16 ret;
+ __asm__ __volatile__("lhzcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
+
+static inline u32 __raw_rm_readl(volatile void __iomem *paddr)
+{
+ u32 ret;
+ __asm__ __volatile__("lwzcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
+
+static inline u64 __raw_rm_readq(volatile void __iomem *paddr)
+{
+ u64 ret;
+ __asm__ __volatile__("ldcix %0,0, %1"
+ : "=r" (ret) : "r" (paddr) : "memory");
+ return ret;
+}
#endif /* __powerpc64__ */
/*
diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
index d318d43..0593d94 100644
--- a/arch/powerpc/include/asm/kvm_book3s_asm.h
+++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
@@ -110,7 +110,7 @@ struct kvmppc_host_state {
u8 ptid;
struct kvm_vcpu *kvm_vcpu;
struct kvmppc_vcore *kvm_vcore;
- unsigned long xics_phys;
+ void __iomem *xics_phys;
u32 saved_xirr;
u64 dabr;
u64 host_mmcr[7]; /* MMCR 0,1,A, SIAR, SDAR, MMCR2, SIER */
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 0c41865..c387799 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -409,7 +409,7 @@ struct openpic;
extern void kvm_cma_reserve(void) __init;
static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)
{
- paca[cpu].kvm_hstate.xics_phys = addr;
+ paca[cpu].kvm_hstate.xics_phys = (void __iomem *)addr;
}
static inline u32 kvmppc_get_xics_latch(void)
diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c
index 4d6c64b..d48f9b6 100644
--- a/arch/powerpc/kvm/book3s_hv_builtin.c
+++ b/arch/powerpc/kvm/book3s_hv_builtin.c
@@ -193,12 +193,6 @@ long kvmppc_h_random(struct kvm_vcpu *vcpu)
return H_HARDWARE;
}
-static inline void rm_writeb(unsigned long paddr, u8 val)
-{
- __asm__ __volatile__("stbcix %0,0,%1"
- : : "r" (val), "r" (paddr) : "memory");
-}
-
/*
* Send an interrupt or message to another CPU.
* The caller needs to include any barrier needed to order writes
@@ -206,7 +200,7 @@ static inline void rm_writeb(unsigned long paddr, u8 val)
*/
void kvmhv_rm_send_ipi(int cpu)
{
- unsigned long xics_phys;
+ void __iomem *xics_phys;
unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
/* On POWER9 we can use msgsnd for any destination cpu. */
@@ -227,7 +221,7 @@ void kvmhv_rm_send_ipi(int cpu)
/* Else poke the target with an IPI */
xics_phys = paca[cpu].kvm_hstate.xics_phys;
if (xics_phys)
- rm_writeb(xics_phys + XICS_MFRR, IPI_PRIORITY);
+ __raw_rm_writeb(IPI_PRIORITY, xics_phys + XICS_MFRR);
else
opal_int_set_mfrr(get_hard_smp_processor_id(cpu), IPI_PRIORITY);
}
@@ -397,7 +391,7 @@ long kvmppc_read_intr(void)
static long kvmppc_read_one_intr(bool *again)
{
- unsigned long xics_phys;
+ void __iomem *xics_phys;
u32 h_xirr;
__be32 xirr;
u32 xisr;
@@ -415,7 +409,7 @@ static long kvmppc_read_one_intr(bool *again)
if (!xics_phys)
rc = opal_int_get_xirr(&xirr, false);
else
- xirr = _lwzcix(xics_phys + XICS_XIRR);
+ xirr = __raw_rm_readl(xics_phys + XICS_XIRR);
if (rc < 0)
return 1;
@@ -445,8 +439,8 @@ static long kvmppc_read_one_intr(bool *again)
if (xisr = XICS_IPI) {
rc = 0;
if (xics_phys) {
- _stbcix(xics_phys + XICS_MFRR, 0xff);
- _stwcix(xics_phys + XICS_XIRR, xirr);
+ __raw_rm_writeb(0xff, xics_phys + XICS_MFRR);
+ __raw_rm_writel(xirr, xics_phys + XICS_XIRR);
} else {
opal_int_set_mfrr(hard_smp_processor_id(), 0xff);
rc = opal_int_eoi(h_xirr);
@@ -471,7 +465,8 @@ static long kvmppc_read_one_intr(bool *again)
* we need to resend that IPI, bummer
*/
if (xics_phys)
- _stbcix(xics_phys + XICS_MFRR, IPI_PRIORITY);
+ __raw_rm_writeb(IPI_PRIORITY,
+ xics_phys + XICS_MFRR);
else
opal_int_set_mfrr(hard_smp_processor_id(),
IPI_PRIORITY);
diff --git a/arch/powerpc/kvm/book3s_hv_rm_xics.c b/arch/powerpc/kvm/book3s_hv_rm_xics.c
index e78542d..3a1a463 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_xics.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_xics.c
@@ -766,7 +766,7 @@ unsigned long eoi_rc;
static void icp_eoi(struct irq_chip *c, u32 hwirq, __be32 xirr, bool *again)
{
- unsigned long xics_phys;
+ void __iomem *xics_phys;
int64_t rc;
rc = pnv_opal_pci_msi_eoi(c, hwirq);
@@ -779,7 +779,7 @@ static void icp_eoi(struct irq_chip *c, u32 hwirq, __be32 xirr, bool *again)
/* EOI it */
xics_phys = local_paca->kvm_hstate.xics_phys;
if (xics_phys) {
- _stwcix(xics_phys + XICS_XIRR, xirr);
+ __raw_rm_writel(xirr, xics_phys + XICS_XIRR);
} else {
rc = opal_int_eoi(be32_to_cpu(xirr));
*again = rc > 0;
diff --git a/arch/powerpc/platforms/powernv/rng.c b/arch/powerpc/platforms/powernv/rng.c
index 5dcbdea..1a9d843 100644
--- a/arch/powerpc/platforms/powernv/rng.c
+++ b/arch/powerpc/platforms/powernv/rng.c
@@ -62,7 +62,7 @@ int powernv_get_random_real_mode(unsigned long *v)
rng = raw_cpu_read(powernv_rng);
- *v = rng_whiten(rng, in_rm64(rng->regs_real));
+ *v = rng_whiten(rng, __raw_rm_readq(rng->regs_real));
return 1;
}
diff --git a/arch/powerpc/sysdev/xics/icp-native.c b/arch/powerpc/sysdev/xics/icp-native.c
index 8a6a043..f0f3f47 100644
--- a/arch/powerpc/sysdev/xics/icp-native.c
+++ b/arch/powerpc/sysdev/xics/icp-native.c
@@ -168,15 +168,15 @@ void icp_native_cause_ipi_rm(int cpu)
* Need the physical address of the XICS to be
* previously saved in kvm_hstate in the paca.
*/
- unsigned long xics_phys;
+ void __iomem *xics_phys;
/*
* Just like the cause_ipi functions, it is required to
- * include a full barrier (out8 includes a sync) before
- * causing the IPI.
+ * include a full barrier before causing the IPI.
*/
xics_phys = paca[cpu].kvm_hstate.xics_phys;
- out_rm8((u8 *)(xics_phys + XICS_MFRR), IPI_PRIORITY);
+ mb();
+ __raw_rm_writeb(IPI_PRIORITY, xics_phys + XICS_MFRR);
}
#endif
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 11/12] powerpc: Fixup LPCR:PECE and HEIC setting on POWER9
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
We need to set LPES in order for normal external interrupts (0x500)
to be directed to the guest while running in guest state.
We also need HEIC set to prevent them to be sent to the host while
in host state.
With XIVE the host never gets one of these and wouldn't know how to
handle it. All host external interrupts come in via the new
hypervisor virtualization interrupts vector.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/reg.h | 1 +
arch/powerpc/kernel/cpu_setup_power.S | 15 ++++++++++-----
2 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index fc879fd..d0b332b 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -365,6 +365,7 @@
#define LPCR_MER_SH 11
#define LPCR_GTSE ASM_CONST(0x0000000000000400) /* Guest Translation Shootdown Enable */
#define LPCR_TC ASM_CONST(0x0000000000000200) /* Translation control */
+#define LPCR_HEIC ASM_CONST(0x0000000000000010) /* Hypervisor External Interrupt Control */
#define LPCR_LPES 0x0000000c
#define LPCR_LPES0 ASM_CONST(0x0000000000000008) /* LPAR Env selector 0 */
#define LPCR_LPES1 ASM_CONST(0x0000000000000004) /* LPAR Env selector 1 */
diff --git a/arch/powerpc/kernel/cpu_setup_power.S b/arch/powerpc/kernel/cpu_setup_power.S
index 7fe8c79..7013ae3 100644
--- a/arch/powerpc/kernel/cpu_setup_power.S
+++ b/arch/powerpc/kernel/cpu_setup_power.S
@@ -29,6 +29,7 @@ _GLOBAL(__setup_cpu_power7)
li r0,0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
+ li r4,(LPCR_LPES1 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_tlb_power7
mtlr r11
@@ -42,6 +43,7 @@ _GLOBAL(__restore_cpu_power7)
li r0,0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
+ li r4,(LPCR_LPES1 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_tlb_power7
mtlr r11
@@ -59,6 +61,7 @@ _GLOBAL(__setup_cpu_power8)
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
ori r3, r3, LPCR_PECEDH
+ li r4,0 /* LPES = 0 */
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power8
@@ -80,6 +83,7 @@ _GLOBAL(__restore_cpu_power8)
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
ori r3, r3, LPCR_PECEDH
+ li r4,0 /* LPES = 0 */
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power8
@@ -99,10 +103,11 @@ _GLOBAL(__setup_cpu_power9)
mtspr SPRN_PSSCR,r0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
- LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE)
+ LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE | LPCR_HEIC)
or r3, r3, r4
LOAD_REG_IMMEDIATE(r4, LPCR_UPRT | LPCR_HR)
andc r3, r3, r4
+ li r4,(LPCR_LPES0 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power9
@@ -122,10 +127,11 @@ _GLOBAL(__restore_cpu_power9)
mtspr SPRN_PSSCR,r0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
- LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE)
+ LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE | LPCR_HEIC)
or r3, r3, r4
LOAD_REG_IMMEDIATE(r4, LPCR_UPRT | LPCR_HR)
andc r3, r3, r4
+ li r4,(LPCR_LPES0 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power9
@@ -146,7 +152,7 @@ __init_hvmode_206:
__init_LPCR:
/* Setup a sane LPCR:
- * Called with initial LPCR in R3
+ * Called with initial LPCR in R3 and desired LPES 2-bit value in R4
*
* LPES = 0b01 (HSRR0/1 used for 0x500)
* PECE = 0b111
@@ -157,8 +163,7 @@ __init_LPCR:
*
* Other bits untouched for now
*/
- li r5,1
- rldimi r3,r5, LPCR_LPES_SH, 64-LPCR_LPES_SH-2
+ rldimi r3,r4, LPCR_LPES_SH, 64-LPCR_LPES_SH-2
ori r3,r3,(LPCR_PECE0|LPCR_PECE1|LPCR_PECE2)
li r5,4
rldimi r3,r5, LPCR_DPFD_SH, 64-LPCR_DPFD_SH-3
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 11/12] powerpc: Fixup LPCR:PECE and HEIC setting on POWER9
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
We need to set LPES in order for normal external interrupts (0x500)
to be directed to the guest while running in guest state.
We also need HEIC set to prevent them to be sent to the host while
in host state.
With XIVE the host never gets one of these and wouldn't know how to
handle it. All host external interrupts come in via the new
hypervisor virtualization interrupts vector.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/reg.h | 1 +
arch/powerpc/kernel/cpu_setup_power.S | 15 ++++++++++-----
2 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/arch/powerpc/include/asm/reg.h b/arch/powerpc/include/asm/reg.h
index fc879fd..d0b332b 100644
--- a/arch/powerpc/include/asm/reg.h
+++ b/arch/powerpc/include/asm/reg.h
@@ -365,6 +365,7 @@
#define LPCR_MER_SH 11
#define LPCR_GTSE ASM_CONST(0x0000000000000400) /* Guest Translation Shootdown Enable */
#define LPCR_TC ASM_CONST(0x0000000000000200) /* Translation control */
+#define LPCR_HEIC ASM_CONST(0x0000000000000010) /* Hypervisor External Interrupt Control */
#define LPCR_LPES 0x0000000c
#define LPCR_LPES0 ASM_CONST(0x0000000000000008) /* LPAR Env selector 0 */
#define LPCR_LPES1 ASM_CONST(0x0000000000000004) /* LPAR Env selector 1 */
diff --git a/arch/powerpc/kernel/cpu_setup_power.S b/arch/powerpc/kernel/cpu_setup_power.S
index 7fe8c79..7013ae3 100644
--- a/arch/powerpc/kernel/cpu_setup_power.S
+++ b/arch/powerpc/kernel/cpu_setup_power.S
@@ -29,6 +29,7 @@ _GLOBAL(__setup_cpu_power7)
li r0,0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
+ li r4,(LPCR_LPES1 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_tlb_power7
mtlr r11
@@ -42,6 +43,7 @@ _GLOBAL(__restore_cpu_power7)
li r0,0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
+ li r4,(LPCR_LPES1 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_tlb_power7
mtlr r11
@@ -59,6 +61,7 @@ _GLOBAL(__setup_cpu_power8)
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
ori r3, r3, LPCR_PECEDH
+ li r4,0 /* LPES = 0 */
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power8
@@ -80,6 +83,7 @@ _GLOBAL(__restore_cpu_power8)
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
ori r3, r3, LPCR_PECEDH
+ li r4,0 /* LPES = 0 */
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power8
@@ -99,10 +103,11 @@ _GLOBAL(__setup_cpu_power9)
mtspr SPRN_PSSCR,r0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
- LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE)
+ LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE | LPCR_HEIC)
or r3, r3, r4
LOAD_REG_IMMEDIATE(r4, LPCR_UPRT | LPCR_HR)
andc r3, r3, r4
+ li r4,(LPCR_LPES0 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power9
@@ -122,10 +127,11 @@ _GLOBAL(__restore_cpu_power9)
mtspr SPRN_PSSCR,r0
mtspr SPRN_LPID,r0
mfspr r3,SPRN_LPCR
- LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE)
+ LOAD_REG_IMMEDIATE(r4, LPCR_PECEDH | LPCR_PECE_HVEE | LPCR_HVICE | LPCR_HEIC)
or r3, r3, r4
LOAD_REG_IMMEDIATE(r4, LPCR_UPRT | LPCR_HR)
andc r3, r3, r4
+ li r4,(LPCR_LPES0 >> LPCR_LPES_SH)
bl __init_LPCR
bl __init_HFSCR
bl __init_tlb_power9
@@ -146,7 +152,7 @@ __init_hvmode_206:
__init_LPCR:
/* Setup a sane LPCR:
- * Called with initial LPCR in R3
+ * Called with initial LPCR in R3 and desired LPES 2-bit value in R4
*
* LPES = 0b01 (HSRR0/1 used for 0x500)
* PECE = 0b111
@@ -157,8 +163,7 @@ __init_LPCR:
*
* Other bits untouched for now
*/
- li r5,1
- rldimi r3,r5, LPCR_LPES_SH, 64-LPCR_LPES_SH-2
+ rldimi r3,r4, LPCR_LPES_SH, 64-LPCR_LPES_SH-2
ori r3,r3,(LPCR_PECE0|LPCR_PECE1|LPCR_PECE2)
li r5,4
rldimi r3,r5, LPCR_DPFD_SH, 64-LPCR_DPFD_SH-3
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH 12/12] powerpc/kvm: Native usage of the XIVE interrupt controller
2017-03-20 6:49 ` Benjamin Herrenschmidt
` (10 preceding siblings ...)
(?)
@ 2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-28 5:26 ` Paul Mackerras
-1 siblings, 1 reply; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-03-20 6:49 UTC (permalink / raw)
To: linuxppc-dev, kvm-ppc
This patch makes KVM capable of using the XIVE interrupt controller
to provide the standard PAPR "XICS" style hypercalls. It is necessary
for proper operations when the host uses XIVE natively.
This has been lightly tested on an actual system, including PCI
pass-through with a TG3 device.
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
---
arch/powerpc/include/asm/kvm_book3s_asm.h | 2 +
arch/powerpc/include/asm/kvm_host.h | 28 +-
arch/powerpc/include/asm/kvm_ppc.h | 38 +
arch/powerpc/include/asm/xive.h | 11 +-
arch/powerpc/kernel/asm-offsets.c | 10 +
arch/powerpc/kvm/Makefile | 4 +-
arch/powerpc/kvm/book3s.c | 73 +-
arch/powerpc/kvm/book3s_hv.c | 52 +-
arch/powerpc/kvm/book3s_hv_builtin.c | 108 ++
arch/powerpc/kvm/book3s_hv_rm_xics.c | 10 +-
arch/powerpc/kvm/book3s_hv_rm_xive.c | 47 +
arch/powerpc/kvm/book3s_hv_rmhandlers.S | 60 +-
arch/powerpc/kvm/book3s_rtas.c | 21 +-
arch/powerpc/kvm/book3s_xics.c | 35 +-
arch/powerpc/kvm/book3s_xics.h | 5 +
arch/powerpc/kvm/book3s_xive.c | 1898 +++++++++++++++++++++++++++++
arch/powerpc/kvm/book3s_xive.h | 251 ++++
arch/powerpc/kvm/book3s_xive_template.c | 490 ++++++++
arch/powerpc/kvm/irq.h | 1 +
arch/powerpc/kvm/powerpc.c | 17 +-
arch/powerpc/platforms/powernv/opal.c | 1 +
arch/powerpc/sysdev/xive/common.c | 131 +-
arch/powerpc/sysdev/xive/native.c | 92 +-
include/linux/kvm_host.h | 1 -
virt/kvm/kvm_main.c | 4 -
25 files changed, 3305 insertions(+), 85 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_hv_rm_xive.c
create mode 100644 arch/powerpc/kvm/book3s_xive.c
create mode 100644 arch/powerpc/kvm/book3s_xive.h
create mode 100644 arch/powerpc/kvm/book3s_xive_template.c
diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
index 0593d94..e719002 100644
--- a/arch/powerpc/include/asm/kvm_book3s_asm.h
+++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
@@ -111,6 +111,8 @@ struct kvmppc_host_state {
struct kvm_vcpu *kvm_vcpu;
struct kvmppc_vcore *kvm_vcore;
void __iomem *xics_phys;
+ void __iomem *xive_tm_area_phys;
+ void __iomem *xive_tm_area_virt;
u32 saved_xirr;
u64 dabr;
u64 host_mmcr[7]; /* MMCR 0,1,A, SIAR, SDAR, MMCR2, SIER */
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 7bba8f4..fc491ac 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -205,6 +205,12 @@ struct kvmppc_spapr_tce_table {
/* XICS components, defined in book3s_xics.c */
struct kvmppc_xics;
struct kvmppc_icp;
+extern struct kvm_device_ops kvm_xics_ops;
+
+/* XIVE components, defined in book3s_xive.c */
+struct kvmppc_xive;
+struct kvmppc_xive_vcpu;
+extern struct kvm_device_ops kvm_xive_ops;
struct kvmppc_passthru_irqmap;
@@ -293,6 +299,7 @@ struct kvm_arch {
#endif
#ifdef CONFIG_KVM_XICS
struct kvmppc_xics *xics;
+ struct kvmppc_xive *xive;
struct kvmppc_passthru_irqmap *pimap;
#endif
struct kvmppc_ops *kvm_ops;
@@ -421,7 +428,7 @@ struct kvmppc_passthru_irqmap {
#define KVMPPC_IRQ_DEFAULT 0
#define KVMPPC_IRQ_MPIC 1
-#define KVMPPC_IRQ_XICS 2
+#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
#define MMIO_HPTE_CACHE_SIZE 4
@@ -443,6 +450,21 @@ struct mmio_hpte_cache {
struct openpic;
+/* QW0 and QW1 of a context */
+union xive_qw01 {
+ struct {
+ u8 nsr;
+ u8 cppr;
+ u8 ipb;
+ u8 lsmfb;
+ u8 ack;
+ u8 inc;
+ u8 age;
+ u8 pipr;
+ };
+ __be64 qw;
+};
+
struct kvm_vcpu_arch {
ulong host_stack;
u32 host_pid;
@@ -688,6 +710,10 @@ struct kvm_vcpu_arch {
struct openpic *mpic; /* KVM_IRQ_MPIC */
#ifdef CONFIG_KVM_XICS
struct kvmppc_icp *icp; /* XICS presentation controller */
+ struct kvmppc_xive_vcpu *xive_vcpu; /* XIVE virtual CPU data */
+ __be32 xive_cam_word; /* Cooked W2 in proper endian with valid bit */
+ u32 xive_pushed; /* Is the VP pushed on the physical CPU ? */
+ union xive_qw01 xive_saved_state; /* W0..1 of XIVE state */
#endif
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index c387799..2fcf6cf 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -225,6 +225,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq);
extern int kvm_vm_ioctl_rtas_define_token(struct kvm *kvm, void __user *argp);
extern int kvmppc_rtas_hcall(struct kvm_vcpu *vcpu);
extern void kvmppc_rtas_tokens_free(struct kvm *kvm);
+
extern int kvmppc_xics_set_xive(struct kvm *kvm, u32 irq, u32 server,
u32 priority);
extern int kvmppc_xics_get_xive(struct kvm *kvm, u32 irq, u32 *server,
@@ -232,6 +233,15 @@ extern int kvmppc_xics_get_xive(struct kvm *kvm, u32 irq, u32 *server,
extern int kvmppc_xics_int_on(struct kvm *kvm, u32 irq);
extern int kvmppc_xics_int_off(struct kvm *kvm, u32 irq);
+extern int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
+ u32 priority);
+extern int kvmppc_xive_get_xive(struct kvm *kvm, u32 irq, u32 *server,
+ u32 *priority);
+extern int kvmppc_xive_int_on(struct kvm *kvm, u32 irq);
+extern int kvmppc_xive_int_off(struct kvm *kvm, u32 irq);
+extern void kvmppc_xive_init_module(void);
+extern void kvmppc_xive_exit_module(void);
+
void kvmppc_core_dequeue_debug(struct kvm_vcpu *vcpu);
void kvmppc_core_queue_debug(struct kvm_vcpu *vcpu);
@@ -412,6 +422,14 @@ static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)
paca[cpu].kvm_hstate.xics_phys = (void __iomem *)addr;
}
+static inline void kvmppc_set_xive_tm_area(int cpu,
+ unsigned long phys_addr,
+ void __iomem *virt_addr)
+{
+ paca[cpu].kvm_hstate.xive_tm_area_phys = (void __iomem *)phys_addr;
+ paca[cpu].kvm_hstate.xive_tm_area_virt = virt_addr;
+}
+
static inline u32 kvmppc_get_xics_latch(void)
{
u32 xirr;
@@ -442,6 +460,9 @@ static inline void __init kvm_cma_reserve(void)
static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)
{}
+static inline void kvmppc_set_xive_tm_area_phys(int cpu, unsigned long addr)
+{}
+
static inline u32 kvmppc_get_xics_latch(void)
{
return 0;
@@ -492,6 +513,21 @@ extern long kvmppc_deliver_irq_passthru(struct kvm_vcpu *vcpu, __be32 xirr,
struct kvmppc_irq_map *irq_map,
struct kvmppc_passthru_irqmap *pimap,
bool *again);
+extern int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu);
+extern void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu);
+extern int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
+ struct irq_desc *host_desc);
+extern int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
+ struct irq_desc *host_desc);
+extern u64 kvmppc_xive_get_icp(struct kvm_vcpu *vcpu);
+extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
+
+extern int kvmppc_xics_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
+ int level, bool line_status);
+extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
+ int level, bool line_status);
+
extern int h_ipi_redirect;
#else
static inline struct kvmppc_passthru_irqmap *kvmppc_get_passthru_irqmap(
@@ -546,6 +582,8 @@ long kvmppc_h_clear_mod(struct kvm_vcpu *vcpu, unsigned long flags,
long kvmppc_hpte_hv_fault(struct kvm_vcpu *vcpu, unsigned long addr,
unsigned long slb_v, unsigned int status, bool data);
unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu);
+unsigned long kvmppc_rm_h_xirr_x(struct kvm_vcpu *vcpu);
+unsigned long kvmppc_rm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server);
int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
unsigned long mfrr);
int kvmppc_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index b1604b73..94b5cca 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -55,7 +55,8 @@ struct xive_q {
#define XIVE_ESB_SET_PQ_01 0xd00
#define XIVE_ESB_SET_PQ_10 0xe00
#define XIVE_ESB_SET_PQ_11 0xf00
-#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
+#define XIVE_ESB_SOFT_MASK XIVE_ESB_SET_PQ_10
+#define XIVE_ESB_HARD_MASK XIVE_ESB_SET_PQ_01
extern bool __xive_enabled;
@@ -88,11 +89,11 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
__be32 *qpage, u32 order, bool can_escalate);
extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
-extern bool __xive_irq_trigger(struct xive_irq_data *xd);
-extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
-extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
-
+extern void xive_native_sync_source(u32 hw_irq);
extern bool is_xive_irq(struct irq_chip *chip);
+extern int xive_native_enable_vp(u32 vp_id);
+extern int xive_native_disable_vp(u32 vp_id);
+extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
#else
diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
index 4367e7d..59fa705 100644
--- a/arch/powerpc/kernel/asm-offsets.c
+++ b/arch/powerpc/kernel/asm-offsets.c
@@ -630,6 +630,8 @@ int main(void)
HSTATE_FIELD(HSTATE_KVM_VCPU, kvm_vcpu);
HSTATE_FIELD(HSTATE_KVM_VCORE, kvm_vcore);
HSTATE_FIELD(HSTATE_XICS_PHYS, xics_phys);
+ HSTATE_FIELD(HSTATE_XIVE_TM_AREA_PHYS, xive_tm_area_phys);
+ HSTATE_FIELD(HSTATE_XIVE_TM_AREA_VIRT, xive_tm_area_virt);
HSTATE_FIELD(HSTATE_SAVED_XIRR, saved_xirr);
HSTATE_FIELD(HSTATE_HOST_IPI, host_ipi);
HSTATE_FIELD(HSTATE_PTID, ptid);
@@ -715,6 +717,14 @@ int main(void)
OFFSET(VCPU_HOST_MAS6, kvm_vcpu, arch.host_mas6);
#endif
+#ifdef CONFIG_KVM_XICS
+ DEFINE(VCPU_XIVE_SAVED_STATE, offsetof(struct kvm_vcpu,
+ arch.xive_saved_state));
+ DEFINE(VCPU_XIVE_CAM_WORD, offsetof(struct kvm_vcpu,
+ arch.xive_cam_word));
+ DEFINE(VCPU_XIVE_PUSHED, offsetof(struct kvm_vcpu, arch.xive_pushed));
+#endif
+
#ifdef CONFIG_KVM_EXIT_TIMING
OFFSET(VCPU_TIMING_EXIT_TBU, kvm_vcpu, arch.timing_exit.tv32.tbu);
OFFSET(VCPU_TIMING_EXIT_TBL, kvm_vcpu, arch.timing_exit.tv32.tbl);
diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
index b87ccde..ef89c8c 100644
--- a/arch/powerpc/kvm/Makefile
+++ b/arch/powerpc/kvm/Makefile
@@ -74,7 +74,7 @@ kvm-hv-y += \
book3s_64_mmu_radix.o
kvm-book3s_64-builtin-xics-objs-$(CONFIG_KVM_XICS) := \
- book3s_hv_rm_xics.o
+ book3s_hv_rm_xics.o book3s_hv_rm_xive.o
ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \
@@ -87,7 +87,7 @@ kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \
endif
kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
- book3s_xics.o
+ book3s_xics.o book3s_xive.o
kvm-book3s_64-module-objs := \
$(common-objs-y) \
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index aedacef..e459ec4 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -35,6 +35,7 @@
#include <asm/kvm_book3s.h>
#include <asm/mmu_context.h>
#include <asm/page.h>
+#include <asm/xive.h>
#include "book3s.h"
#include "trace.h"
@@ -578,11 +579,14 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
break;
#ifdef CONFIG_KVM_XICS
case KVM_REG_PPC_ICP_STATE:
- if (!vcpu->arch.icp) {
+ if (!vcpu->arch.icp && !vcpu->arch.xive_vcpu) {
r = -ENXIO;
break;
}
- *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
+ if (xive_enabled())
+ *val = get_reg_val(id, kvmppc_xive_get_icp(vcpu));
+ else
+ *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
break;
#endif /* CONFIG_KVM_XICS */
case KVM_REG_PPC_FSCR:
@@ -648,12 +652,14 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
#endif /* CONFIG_VSX */
#ifdef CONFIG_KVM_XICS
case KVM_REG_PPC_ICP_STATE:
- if (!vcpu->arch.icp) {
+ if (!vcpu->arch.icp && !vcpu->arch.xive_vcpu) {
r = -ENXIO;
break;
}
- r = kvmppc_xics_set_icp(vcpu,
- set_reg_val(id, *val));
+ if (xive_enabled())
+ r = kvmppc_xive_set_icp(vcpu, set_reg_val(id, *val));
+ else
+ r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
break;
#endif /* CONFIG_KVM_XICS */
case KVM_REG_PPC_FSCR:
@@ -924,6 +930,50 @@ int kvmppc_book3s_hcall_implemented(struct kvm *kvm, unsigned long hcall)
return kvm->arch.kvm_ops->hcall_implemented(hcall);
}
+#ifdef CONFIG_KVM_XICS
+int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
+ bool line_status)
+{
+ if (xive_enabled())
+ return kvmppc_xive_set_irq(kvm, irq_source_id, irq, level,
+ line_status);
+ else
+ return kvmppc_xics_set_irq(kvm, irq_source_id, irq, level,
+ line_status);
+}
+
+int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *irq_entry,
+ struct kvm *kvm, int irq_source_id,
+ int level, bool line_status)
+{
+ return kvm_set_irq(kvm, irq_source_id, irq_entry->gsi,
+ level, line_status);
+}
+static int kvmppc_book3s_set_irq(struct kvm_kernel_irq_routing_entry *e,
+ struct kvm *kvm, int irq_source_id, int level,
+ bool line_status)
+{
+ return kvm_set_irq(kvm, irq_source_id, e->gsi, level, line_status);
+}
+
+int kvm_irq_map_gsi(struct kvm *kvm,
+ struct kvm_kernel_irq_routing_entry *entries, int gsi)
+{
+ entries->gsi = gsi;
+ entries->type = KVM_IRQ_ROUTING_IRQCHIP;
+ entries->set = kvmppc_book3s_set_irq;
+ entries->irqchip.irqchip = 0;
+ entries->irqchip.pin = gsi;
+ return 1;
+}
+
+int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
+{
+ return pin;
+}
+
+#endif /* CONFIG_KVM_XICS */
+
static int kvmppc_book3s_init(void)
{
int r;
@@ -934,12 +984,23 @@ static int kvmppc_book3s_init(void)
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
r = kvmppc_book3s_init_pr();
#endif
- return r;
+#ifdef CONFIG_KVM_XICS
+ if (xive_enabled()) {
+ kvmppc_xive_init_module();
+ kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
+ } else
+ kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
+#endif
+ return r;
}
static void kvmppc_book3s_exit(void)
{
+#ifdef CONFIG_KVM_XICS
+ if (xive_enabled())
+ kvmppc_xive_exit_module();
+#endif
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
kvmppc_book3s_exit_pr();
#endif
diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
index fadb75a..5c340c2 100644
--- a/arch/powerpc/kvm/book3s_hv.c
+++ b/arch/powerpc/kvm/book3s_hv.c
@@ -67,6 +67,7 @@
#include <asm/mmu.h>
#include <asm/opal.h>
#include <asm/xics.h>
+#include <asm/xive.h>
#include "book3s.h"
@@ -837,6 +838,10 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
case H_IPOLL:
case H_XIRR_X:
if (kvmppc_xics_enabled(vcpu)) {
+ if (xive_enabled()) {
+ ret = H_NOT_AVAILABLE;
+ return RESUME_GUEST;
+ }
ret = kvmppc_xics_hcall(vcpu, req);
break;
}
@@ -2947,8 +2952,12 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu)
r = kvmppc_book3s_hv_page_fault(run, vcpu,
vcpu->arch.fault_dar, vcpu->arch.fault_dsisr);
srcu_read_unlock(&vcpu->kvm->srcu, srcu_idx);
- } else if (r == RESUME_PASSTHROUGH)
- r = kvmppc_xics_rm_complete(vcpu, 0);
+ } else if (r == RESUME_PASSTHROUGH) {
+ if (WARN_ON(xive_enabled()))
+ r = H_SUCCESS;
+ else
+ r = kvmppc_xics_rm_complete(vcpu, 0);
+ }
} while (is_kvmppc_resume_guest(r));
out:
@@ -3400,10 +3409,19 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm)
/*
* On POWER9, VPM0 bit is reserved (VPM0=1 behaviour is assumed)
* Set HVICE bit to enable hypervisor virtualization interrupts.
+ * Set HEIC to prevent OS interrupts to go to hypervisor (should
+ * be unnecessary but better safe than sorry in case we re-enable
+ * EE in HV mode with this LPCR still set)
*/
if (cpu_has_feature(CPU_FTR_ARCH_300)) {
lpcr &= ~LPCR_VPM0;
- lpcr |= LPCR_HVICE;
+ lpcr |= LPCR_HVICE | LPCR_HEIC;
+
+ /* If xive is enabled, we route 0x500 interrupts directly
+ * to the guest
+ */
+ if (xive_enabled())
+ lpcr |= LPCR_LPES;
}
/*
@@ -3533,7 +3551,7 @@ static int kvmppc_set_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
struct kvmppc_irq_map *irq_map;
struct kvmppc_passthru_irqmap *pimap;
struct irq_chip *chip;
- int i;
+ int i, rc = 0;
if (!kvm_irq_bypass)
return 1;
@@ -3558,10 +3576,10 @@ static int kvmppc_set_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
/*
* For now, we only support interrupts for which the EOI operation
* is an OPAL call followed by a write to XIRR, since that's
- * what our real-mode EOI code does.
+ * what our real-mode EOI code does, or a XIVE interrupt
*/
chip = irq_data_get_irq_chip(&desc->irq_data);
- if (!chip || !is_pnv_opal_msi(chip)) {
+ if (!chip || !(is_pnv_opal_msi(chip) || is_xive_irq(chip))) {
pr_warn("kvmppc_set_passthru_irq_hv: Could not assign IRQ map for (%d,%d)\n",
host_irq, guest_gsi);
mutex_unlock(&kvm->lock);
@@ -3603,7 +3621,14 @@ static int kvmppc_set_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
if (i == pimap->n_mapped)
pimap->n_mapped++;
- kvmppc_xics_set_mapped(kvm, guest_gsi, desc->irq_data.hwirq);
+ if (xive_enabled())
+ rc = kvmppc_xive_set_mapped(kvm, guest_gsi, desc);
+ else
+ kvmppc_xics_set_mapped(kvm, guest_gsi, desc->irq_data.hwirq);
+ printk("set mapped for IRQ %d -> %d returned %d\n",
+ host_irq, guest_gsi, rc);
+ if (rc)
+ irq_map->r_hwirq = 0;
mutex_unlock(&kvm->lock);
@@ -3614,7 +3639,7 @@ static int kvmppc_clr_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
{
struct irq_desc *desc;
struct kvmppc_passthru_irqmap *pimap;
- int i;
+ int i, rc = 0;
if (!kvm_irq_bypass)
return 0;
@@ -3641,9 +3666,12 @@ static int kvmppc_clr_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
return -ENODEV;
}
- kvmppc_xics_clr_mapped(kvm, guest_gsi, pimap->mapped[i].r_hwirq);
+ if (xive_enabled())
+ rc = kvmppc_xive_clr_mapped(kvm, guest_gsi, pimap->mapped[i].desc);
+ else
+ kvmppc_xics_clr_mapped(kvm, guest_gsi, pimap->mapped[i].r_hwirq);
- /* invalidate the entry */
+ /* invalidate the entry (what do do on error from the above ?) */
pimap->mapped[i].r_hwirq = 0;
/*
@@ -3652,7 +3680,7 @@ static int kvmppc_clr_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
*/
mutex_unlock(&kvm->lock);
- return 0;
+ return rc;
}
static int kvmppc_irq_bypass_add_producer_hv(struct irq_bypass_consumer *cons,
@@ -3930,7 +3958,7 @@ static int kvmppc_book3s_init_hv(void)
* indirectly, via OPAL.
*/
#ifdef CONFIG_SMP
- if (!get_paca()->kvm_hstate.xics_phys) {
+ if (!xive_enabled() && !get_paca()->kvm_hstate.xics_phys) {
struct device_node *np;
np = of_find_compatible_node(NULL, NULL, "ibm,opal-intc");
diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c
index d48f9b6..8de7ed4 100644
--- a/arch/powerpc/kvm/book3s_hv_builtin.c
+++ b/arch/powerpc/kvm/book3s_hv_builtin.c
@@ -23,6 +23,7 @@
#include <asm/kvm_book3s.h>
#include <asm/archrandom.h>
#include <asm/xics.h>
+#include <asm/xive.h>
#include <asm/dbell.h>
#include <asm/cputhreads.h>
#include <asm/io.h>
@@ -31,6 +32,24 @@
#define KVM_CMA_CHUNK_ORDER 18
+#include "book3s_xics.h"
+#include "book3s_xive.h"
+
+/*
+ * The XIVE module will populate these when it loads
+ */
+unsigned long (*__xive_vm_h_xirr)(struct kvm_vcpu *vcpu);
+unsigned long (*__xive_vm_h_ipoll)(struct kvm_vcpu *vcpu, unsigned long server);
+int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr);
+int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
+int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+EXPORT_SYMBOL_GPL(__xive_vm_h_xirr);
+EXPORT_SYMBOL_GPL(__xive_vm_h_ipoll);
+EXPORT_SYMBOL_GPL(__xive_vm_h_ipi);
+EXPORT_SYMBOL_GPL(__xive_vm_h_cppr);
+EXPORT_SYMBOL_GPL(__xive_vm_h_eoi);
+
/*
* Hash page table alignment on newer cpus(CPU_FTR_ARCH_206)
* should be power of 2.
@@ -209,6 +228,7 @@ void kvmhv_rm_send_ipi(int cpu)
__asm__ __volatile__ (PPC_MSGSND(%0) : : "r" (msg));
return;
}
+
/* On POWER8 for IPIs to threads in the same core, use msgsnd. */
if (cpu_has_feature(CPU_FTR_ARCH_207S) &&
cpu_first_thread_sibling(cpu) ==
@@ -218,6 +238,10 @@ void kvmhv_rm_send_ipi(int cpu)
return;
}
+ /* We should never reach this */
+ if (WARN_ON_ONCE(xive_enabled()))
+ return;
+
/* Else poke the target with an IPI */
xics_phys = paca[cpu].kvm_hstate.xics_phys;
if (xics_phys)
@@ -398,6 +422,9 @@ static long kvmppc_read_one_intr(bool *again)
u8 host_ipi;
int64_t rc;
+ if (xive_enabled())
+ return 1;
+
/* see if a host IPI is pending */
host_ipi = local_paca->kvm_hstate.host_ipi;
if (host_ipi)
@@ -482,3 +509,84 @@ static long kvmppc_read_one_intr(bool *again)
return kvmppc_check_passthru(xisr, xirr, again);
}
+
+static inline bool is_rm(void)
+{
+ return !(mfmsr() & MSR_DR);
+}
+
+/* XXX FIXME: The xive_vm_* calls are in a module... */
+
+unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu)
+{
+ if (xive_enabled()) {
+ if (is_rm())
+ return xive_rm_h_xirr(vcpu);
+ if (unlikely(!__xive_vm_h_xirr))
+ return H_NOT_AVAILABLE;
+ return __xive_vm_h_xirr(vcpu);
+ } else
+ return xics_rm_h_xirr(vcpu);
+}
+
+unsigned long kvmppc_rm_h_xirr_x(struct kvm_vcpu *vcpu)
+{
+ vcpu->arch.gpr[5] = get_tb();
+ if (xive_enabled()) {
+ if (is_rm())
+ return xive_rm_h_xirr(vcpu);
+ if (unlikely(!__xive_vm_h_xirr))
+ return H_NOT_AVAILABLE;
+ return __xive_vm_h_xirr(vcpu);
+ } else
+ return xics_rm_h_xirr(vcpu);
+}
+
+unsigned long kvmppc_rm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server)
+{
+ if (xive_enabled()) {
+ if (is_rm())
+ return xive_rm_h_ipoll(vcpu, server);
+ if (unlikely(!__xive_vm_h_ipoll))
+ return H_NOT_AVAILABLE;
+ return __xive_vm_h_ipoll(vcpu, server);
+ } else
+ return H_TOO_HARD;
+}
+
+int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr)
+{
+ if (xive_enabled()) {
+ if (is_rm())
+ return xive_rm_h_ipi(vcpu, server, mfrr);
+ if (unlikely(!__xive_vm_h_ipi))
+ return H_NOT_AVAILABLE;
+ return __xive_vm_h_ipi(vcpu, server, mfrr);
+ } else
+ return xics_rm_h_ipi(vcpu, server, mfrr);
+}
+
+int kvmppc_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
+{
+ if (xive_enabled()) {
+ if (is_rm())
+ return xive_rm_h_cppr(vcpu, cppr);
+ if (unlikely(!__xive_vm_h_cppr))
+ return H_NOT_AVAILABLE;
+ return __xive_vm_h_cppr(vcpu, cppr);
+ } else
+ return xics_rm_h_cppr(vcpu, cppr);
+}
+
+int kvmppc_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
+{
+ if (xive_enabled()) {
+ if (is_rm())
+ return xive_rm_h_eoi(vcpu, xirr);
+ if (unlikely(!__xive_vm_h_eoi))
+ return H_NOT_AVAILABLE;
+ return __xive_vm_h_eoi(vcpu, xirr);
+ } else
+ return xics_rm_h_eoi(vcpu, xirr);
+}
diff --git a/arch/powerpc/kvm/book3s_hv_rm_xics.c b/arch/powerpc/kvm/book3s_hv_rm_xics.c
index 3a1a463..f806880 100644
--- a/arch/powerpc/kvm/book3s_hv_rm_xics.c
+++ b/arch/powerpc/kvm/book3s_hv_rm_xics.c
@@ -485,7 +485,7 @@ static void icp_rm_down_cppr(struct kvmppc_xics *xics, struct kvmppc_icp *icp,
}
-unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu)
+unsigned long xics_rm_h_xirr(struct kvm_vcpu *vcpu)
{
union kvmppc_icp_state old_state, new_state;
struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
@@ -523,8 +523,8 @@ unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu)
return check_too_hard(xics, icp);
}
-int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
- unsigned long mfrr)
+int xics_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr)
{
union kvmppc_icp_state old_state, new_state;
struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
@@ -610,7 +610,7 @@ int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
return check_too_hard(xics, this_icp);
}
-int kvmppc_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
+int xics_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
{
union kvmppc_icp_state old_state, new_state;
struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
@@ -730,7 +730,7 @@ static int ics_rm_eoi(struct kvm_vcpu *vcpu, u32 irq)
return check_too_hard(xics, icp);
}
-int kvmppc_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
+int xics_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
{
struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
struct kvmppc_icp *icp = vcpu->arch.icp;
diff --git a/arch/powerpc/kvm/book3s_hv_rm_xive.c b/arch/powerpc/kvm/book3s_hv_rm_xive.c
new file mode 100644
index 0000000..6390f71
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_hv_rm_xive.c
@@ -0,0 +1,47 @@
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/kernel_stat.h>
+
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xics.h>
+#include <asm/debug.h>
+#include <asm/synch.h>
+#include <asm/cputhreads.h>
+#include <asm/pgtable.h>
+#include <asm/ppc-opcode.h>
+#include <asm/pnv-pci.h>
+#include <asm/opal.h>
+#include <asm/smp.h>
+#include <asm/asm-prototypes.h>
+#include <asm/xive.h>
+
+#include "book3s_xive.h"
+#include "../sysdev/xive/xive-regs.h"
+
+/* XXX */
+#include <asm/udbg.h>
+//#define DBG(fmt...) udbg_printf(fmt)
+#define DBG(fmt...) do { } while(0)
+
+static inline void __iomem *get_tm_area_phys(void)
+{
+ return local_paca->kvm_hstate.xive_tm_area_phys;
+}
+
+#undef XIVE_RUNTIME_CHECKS
+#define X_PFX xive_rm_
+#define X_STATIC
+#define X_STAT_PFX stat_rm_
+#define __x_tm_area get_tm_area_phys()
+#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_page))
+#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_page))
+#define __x_readb __raw_rm_readb
+#define __x_writeb __raw_rm_writeb
+#define __x_readw __raw_rm_readw
+#define __x_readq __raw_rm_readq
+#define __x_writeq __raw_rm_writeq
+
+#include "book3s_xive_template.c"
diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
index 720b9c0..c06cccd 100644
--- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
+++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
@@ -31,6 +31,8 @@
#include <asm/tm.h>
#include <asm/opal.h>
+#include "../sysdev/xive/xive-regs.h"
+
#define VCPU_GPRS_TM(reg) (((reg) * ULONG_SIZE) + VCPU_GPR_TM)
/* Values in HSTATE_NAPPING(r13) */
@@ -982,6 +984,23 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_ARCH_300)
cmpwi r3, 512 /* 1 microsecond */
blt hdec_soon
+#ifdef CONFIG_KVM_XICS
+ /* We are entering the guest on that thread, push VCPU to XIVE */
+ ld r10, HSTATE_XIVE_TM_AREA_PHYS(r13)
+ cmpldi cr0, r10, r0
+ beq no_xive
+ ld r11, VCPU_XIVE_SAVED_STATE(r4)
+ li r9, TM_QW1_OS
+ stdcix r11,r9,r10
+ eieio
+ lwz r11, VCPU_XIVE_CAM_WORD(r4)
+ li r9, TM_QW1_OS + TM_WORD2
+ stwcix r11,r9,r10
+ li r9, 1
+ stw r9, VCPU_XIVE_PUSHED(r4)
+no_xive:
+#endif /* CONFIG_KVM_XICS */
+
deliver_guest_interrupt:
ld r6, VCPU_CTR(r4)
ld r7, VCPU_XER(r4)
@@ -1319,6 +1338,38 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
blt deliver_guest_interrupt
guest_exit_cont: /* r9 = vcpu, r12 = trap, r13 = paca */
+#ifdef CONFIG_KVM_XICS
+ /* We are exiting, pull the VP from the XIVE */
+ lwz r0, VCPU_XIVE_PUSHED(r9)
+ cmpwi cr0, r0, 0
+ beq 1f
+ li r7, TM_SPC_PULL_OS_CTX
+ li r6, TM_QW1_OS
+ mfmsr r0
+ andi. r0, r0, MSR_IR /* in real mode? */
+ beq 2f
+ ld r10, HSTATE_XIVE_TM_AREA_VIRT(r13)
+ cmpldi cr0, r10, 0
+ beq 1f
+ lwzx r11, r7, r10
+ eieio
+ ldx r11, r6, r10
+ b 3f
+2: ld r10, HSTATE_XIVE_TM_AREA_PHYS(r13)
+ cmpldi cr0, r10, 0
+ beq 1f
+ lwzcix r11, r7, r10
+ eieio
+ ldcix r11, r6, r10
+3: std r11, VCPU_XIVE_SAVED_STATE(r9)
+ /* Fixup some of the state for the next load */
+ li r10, 0
+ li r0, 0xff
+ stw r10, VCPU_XIVE_PUSHED(r9)
+ stb r10, (VCPU_XIVE_SAVED_STATE+3)(r9)
+ stb r0, (VCPU_XIVE_SAVED_STATE+4)(r9)
+1:
+#endif /* CONFIG_KVM_XICS */
/* Save more register state */
mfdar r6
mfdsisr r7
@@ -2035,7 +2086,7 @@ hcall_real_table:
.long DOTSYM(kvmppc_rm_h_eoi) - hcall_real_table
.long DOTSYM(kvmppc_rm_h_cppr) - hcall_real_table
.long DOTSYM(kvmppc_rm_h_ipi) - hcall_real_table
- .long 0 /* 0x70 - H_IPOLL */
+ .long DOTSYM(kvmppc_rm_h_ipoll) - hcall_real_table
.long DOTSYM(kvmppc_rm_h_xirr) - hcall_real_table
#else
.long 0 /* 0x64 - H_EOI */
@@ -2205,7 +2256,11 @@ hcall_real_table:
.long 0 /* 0x2f0 */
.long 0 /* 0x2f4 */
.long 0 /* 0x2f8 */
- .long 0 /* 0x2fc */
+#ifdef CONFIG_KVM_XICS
+ .long DOTSYM(kvmppc_rm_h_xirr_x) - hcall_real_table
+#else
+ .long 0 /* 0x2fc - H_XIRR_X*/
+#endif
.long DOTSYM(kvmppc_h_random) - hcall_real_table
.globl hcall_real_table_end
hcall_real_table_end:
@@ -2980,6 +3035,7 @@ kvmppc_fix_pmao:
isync
blr
+
#ifdef CONFIG_KVM_BOOK3S_HV_EXIT_TIMING
/*
* Start timing an activity
diff --git a/arch/powerpc/kvm/book3s_rtas.c b/arch/powerpc/kvm/book3s_rtas.c
index 20528701..2d3b2b1 100644
--- a/arch/powerpc/kvm/book3s_rtas.c
+++ b/arch/powerpc/kvm/book3s_rtas.c
@@ -16,6 +16,7 @@
#include <asm/kvm_ppc.h>
#include <asm/hvcall.h>
#include <asm/rtas.h>
+#include <asm/xive.h>
#ifdef CONFIG_KVM_XICS
static void kvm_rtas_set_xive(struct kvm_vcpu *vcpu, struct rtas_args *args)
@@ -32,7 +33,10 @@ static void kvm_rtas_set_xive(struct kvm_vcpu *vcpu, struct rtas_args *args)
server = be32_to_cpu(args->args[1]);
priority = be32_to_cpu(args->args[2]);
- rc = kvmppc_xics_set_xive(vcpu->kvm, irq, server, priority);
+ if (xive_enabled())
+ rc = kvmppc_xive_set_xive(vcpu->kvm, irq, server, priority);
+ else
+ rc = kvmppc_xics_set_xive(vcpu->kvm, irq, server, priority);
if (rc)
rc = -3;
out:
@@ -52,7 +56,10 @@ static void kvm_rtas_get_xive(struct kvm_vcpu *vcpu, struct rtas_args *args)
irq = be32_to_cpu(args->args[0]);
server = priority = 0;
- rc = kvmppc_xics_get_xive(vcpu->kvm, irq, &server, &priority);
+ if (xive_enabled())
+ rc = kvmppc_xive_get_xive(vcpu->kvm, irq, &server, &priority);
+ else
+ rc = kvmppc_xics_get_xive(vcpu->kvm, irq, &server, &priority);
if (rc) {
rc = -3;
goto out;
@@ -76,7 +83,10 @@ static void kvm_rtas_int_off(struct kvm_vcpu *vcpu, struct rtas_args *args)
irq = be32_to_cpu(args->args[0]);
- rc = kvmppc_xics_int_off(vcpu->kvm, irq);
+ if (xive_enabled())
+ rc = kvmppc_xive_int_off(vcpu->kvm, irq);
+ else
+ rc = kvmppc_xics_int_off(vcpu->kvm, irq);
if (rc)
rc = -3;
out:
@@ -95,7 +105,10 @@ static void kvm_rtas_int_on(struct kvm_vcpu *vcpu, struct rtas_args *args)
irq = be32_to_cpu(args->args[0]);
- rc = kvmppc_xics_int_on(vcpu->kvm, irq);
+ if (xive_enabled())
+ rc = kvmppc_xive_int_on(vcpu->kvm, irq);
+ else
+ rc = kvmppc_xics_int_on(vcpu->kvm, irq);
if (rc)
rc = -3;
out:
diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
index ef4fd52..e6829c4 100644
--- a/arch/powerpc/kvm/book3s_xics.c
+++ b/arch/powerpc/kvm/book3s_xics.c
@@ -1307,8 +1307,8 @@ static int xics_set_source(struct kvmppc_xics *xics, long irq, u64 addr)
return 0;
}
-int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
- bool line_status)
+int kvmppc_xics_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
+ bool line_status)
{
struct kvmppc_xics *xics = kvm->arch.xics;
@@ -1317,14 +1317,6 @@ int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
return ics_deliver_irq(xics, irq, level);
}
-int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *irq_entry,
- struct kvm *kvm, int irq_source_id,
- int level, bool line_status)
-{
- return kvm_set_irq(kvm, irq_source_id, irq_entry->gsi,
- level, line_status);
-}
-
static int xics_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
{
struct kvmppc_xics *xics = dev->private;
@@ -1458,29 +1450,6 @@ void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu)
vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
}
-static int xics_set_irq(struct kvm_kernel_irq_routing_entry *e,
- struct kvm *kvm, int irq_source_id, int level,
- bool line_status)
-{
- return kvm_set_irq(kvm, irq_source_id, e->gsi, level, line_status);
-}
-
-int kvm_irq_map_gsi(struct kvm *kvm,
- struct kvm_kernel_irq_routing_entry *entries, int gsi)
-{
- entries->gsi = gsi;
- entries->type = KVM_IRQ_ROUTING_IRQCHIP;
- entries->set = xics_set_irq;
- entries->irqchip.irqchip = 0;
- entries->irqchip.pin = gsi;
- return 1;
-}
-
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
-{
- return pin;
-}
-
void kvmppc_xics_set_mapped(struct kvm *kvm, unsigned long irq,
unsigned long host_irq)
{
diff --git a/arch/powerpc/kvm/book3s_xics.h b/arch/powerpc/kvm/book3s_xics.h
index ec5474c..5016676 100644
--- a/arch/powerpc/kvm/book3s_xics.h
+++ b/arch/powerpc/kvm/book3s_xics.h
@@ -144,5 +144,10 @@ static inline struct kvmppc_ics *kvmppc_xics_find_ics(struct kvmppc_xics *xics,
return ics;
}
+extern unsigned long xics_rm_h_xirr(struct kvm_vcpu *vcpu);
+extern int xics_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr);
+extern int xics_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr);
+extern int xics_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr);
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
new file mode 100644
index 0000000..acc882d
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -0,0 +1,1898 @@
+/*
+ * Copyright 2017 Benjamin Herrenschmidt, IBM Corporation.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/percpu.h>
+#include <linux/cpumask.h>
+#include <asm/uaccess.h>
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xics.h>
+#include <asm/xive.h>
+#include <asm/debug.h>
+#include <asm/time.h>
+#include <asm/opal.h>
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+#include "book3s_xive.h"
+#include "../sysdev/xive/xive-regs.h"
+
+//#define DBG(fmt...) printk("KVM/XIVE: " fmt)
+#define DBG(fmt...) do { } while(0)
+
+#ifdef XIVE_RUNTIME_CHECKS
+#define xive_assert(cond) WARN_ON(!(cond))
+#else
+#define xive_assert(cond) (false)
+#endif
+
+/*
+ * Virtual mode variants of the hcalls for use on radix/radix
+ * with AIL. They require the VCPU's VP to be "pushed"
+ *
+ * We still instanciate them here because we use some of the
+ * generated utility functions as well in this file.
+ */
+#define XIVE_RUNTIME_CHECKS
+#define X_PFX xive_vm_
+#define X_STATIC static
+#define X_STAT_PFX stat_vm_
+#define __x_tm_area xive_tm_area
+#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
+#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
+#define __x_readb __raw_readb
+#define __x_writeb __raw_writeb
+#define __x_readw __raw_readw
+#define __x_readq __raw_readq
+#define __x_writeq __raw_writeq
+
+#include "book3s_xive_template.c"
+
+/* We leave a gap of a couple of interrupts in the queue to
+ * account for the IPI and additional safety guard
+ */
+#define XIVE_Q_GAP 2
+
+/*
+ * This is a simple trigger for a generic XIVE IRQ. This must
+ * only be called for interrupts that support a trigger page
+ */
+static bool xive_irq_trigger(struct xive_irq_data *xd)
+{
+ /* This should be only for MSIs */
+ if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
+ return false;
+
+ /* Those interrupts should always have a trigger page */
+ if (WARN_ON(!xd->trig_mmio))
+ return false;
+
+ out_be64(xd->trig_mmio, 0);
+
+ return true;
+}
+
+static irqreturn_t xive_esc_irq(int irq, void *data)
+{
+ struct kvm_vcpu *vcpu = data;
+
+ /* We use the existing H_PROD mechanism to wake up the target */
+ vcpu->arch.prodded = 1;
+ smp_mb();
+ if (vcpu->arch.ceded)
+ kvmppc_fast_vcpu_kick(vcpu);
+
+ return IRQ_HANDLED;
+}
+
+static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q = &xc->queues[prio];
+ char *name = NULL;
+ int rc;
+
+ /* Already there ? */
+ if (xc->esc_virq[prio])
+ return 0;
+
+ /* Hook up the escalation interrupt */
+ xc->esc_virq[prio] = irq_create_mapping(NULL, q->esc_irq);
+ if (!xc->esc_virq[prio]) {
+ pr_err("XIVE-KVM: Failed to map escalation interrupt"
+ " for queue %d of VCPU %d\n",
+ prio, xc->server_num);
+ return -EIO;
+ }
+
+ /*
+ * Future improvement: start with them disabled
+ * and handle DD2 and later scheme of merged escalation
+ * interrupts
+ */
+ name = kasprintf(GFP_KERNEL, "kvm-%d-%d-%d\n",
+ vcpu->kvm->arch.lpid, xc->server_num, prio);
+ if (!name) {
+ pr_err("XIVE-KVM: Failed to allocate escalation irq name"
+ " for queue %d of VCPU %d\n",
+ prio, xc->server_num);
+ rc = -ENOMEM;
+ goto error;
+ }
+ rc = request_irq(xc->esc_virq[prio], xive_esc_irq,
+ IRQF_NO_THREAD, name, vcpu);
+ if (rc) {
+ pr_err("XIVE-KVM: Failed to request escalation interrupt"
+ " for queue %d of VCPU %d\n",
+ prio, xc->server_num);
+ goto error;
+ }
+ xc->esc_virq_names[prio] = name;
+ return 0;
+ error:
+ irq_dispose_mapping(xc->esc_virq[prio]);
+ xc->esc_virq[prio] = 0;
+ kfree(name);
+ return rc;
+}
+
+static int xive_provision_queue(struct kvm_vcpu *vcpu, u8 prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = xc->xive;
+ struct xive_q *q = &xc->queues[prio];
+ void *qpage;
+ int rc;
+
+ if (WARN_ON(q->qpage))
+ return 0;
+
+ /* Allocate the queue and retrieve infos on current node for now */
+ qpage = (__be32 *)__get_free_pages(GFP_KERNEL, xive->q_alloc_order);
+ if (!qpage) {
+ pr_err("XIVE-KVM: Failed to allocate queue %d for VCPU %d\n",
+ prio, xc->server_num);
+ return -ENOMEM;;
+ }
+ memset(qpage, 0, 1 << xive->q_order);
+
+ /*
+ * Reconfigure the queue. This will set q->qpage only once the
+ * queue is fully configured. This is a requirement for prio 0
+ * as we will stop doing EOIs for every IPI as soon as we observe
+ * qpage being non-NULL, and instead will only EOI when we receive
+ * corresponding queue 0 entries
+ */
+ rc = xive_native_configure_queue(xc->vp_id, q, prio, qpage,
+ xive->q_order, true);
+ if (rc)
+ pr_err("XIVE-KVM: Failed to configure queue %d for VCPU %d\n",
+ prio, xc->server_num);
+ return rc;
+}
+
+/* Called with kvm_lock held */
+static int xive_check_provisioning(struct kvm *kvm, u8 prio)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvm_vcpu *vcpu;
+ int i, rc;
+
+ lockdep_assert_held(&kvm->lock);
+
+ /* Already provisioned ? */
+ if (xive->qmap & (1 << prio))
+ return 0;
+
+ DBG("Provisioning prio... %d\n", prio);
+
+ /* Provision each VCPU and enable escalations */
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (!vcpu->arch.xive_vcpu)
+ continue;
+ rc = xive_provision_queue(vcpu, prio);
+ if (rc == 0)
+ xive_attach_escalation(vcpu, prio);
+ if (rc)
+ return rc;
+ }
+
+ /* Order previous stores and mark it as provisioned */
+ mb();
+ xive->qmap |= (1 << prio);
+ return 0;
+}
+
+static void xive_inc_q_pending(struct kvm *kvm, u32 server, u8 prio)
+{
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ struct xive_q *q;
+
+ /* Locate target server */
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_warn("%s: Can't find server %d\n", __func__, server);
+ return;
+ }
+ xc = vcpu->arch.xive_vcpu;
+ if (WARN_ON(!xc))
+ return;
+
+ q = &xc->queues[prio];
+ atomic_inc(&q->pending_count);
+}
+
+static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q;
+ u32 max;
+
+ if (WARN_ON(!xc))
+ return -ENXIO;
+ if (!xc->valid)
+ return -ENXIO;
+
+ q = &xc->queues[prio];
+ if (WARN_ON(!q->qpage))
+ return -ENXIO;
+
+ /* Calculate max number of interrupts in that queue. */
+ max = (q->msk + 1) - XIVE_Q_GAP;
+ return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
+}
+
+static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
+{
+ struct kvm_vcpu *vcpu;
+ int i, rc;
+
+ /* Locate target server */
+ vcpu = kvmppc_xive_find_server(kvm, *server);
+ if (!vcpu) {
+ DBG("Can't find server %d\n", *server);
+ return -EINVAL;
+ }
+
+ DBG("Finding irq target on 0x%x/%d...\n", *server, prio);
+
+ /* Try pick it */
+ rc = xive_try_pick_queue(vcpu, prio);
+ if (rc == 0)
+ return rc;
+
+ DBG(" .. failed, looking up candidate...\n");
+
+ /* Failed, pick another VCPU */
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (!vcpu->arch.xive_vcpu)
+ continue;
+ rc = xive_try_pick_queue(vcpu, prio);
+ if (rc == 0) {
+ *server = vcpu->arch.xive_vcpu->server_num;
+ DBG(" found on 0x%x/%d\n", *server, prio);
+ return rc;
+ }
+ }
+ DBG(" no available target !\n");
+
+ /* No available target ! */
+ return -EBUSY;
+}
+
+static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state)
+{
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u8 old_prio;
+ u64 val;
+
+ /*
+ * Take the lock, set masked, try again if racing
+ * with H_EOI
+ */
+ for (;;) {
+ arch_spin_lock(&sb->lock);
+ old_prio = state->guest_priority;
+ state->guest_priority = MASKED;
+ mb();
+ if (!state->in_eoi)
+ break;
+ state->guest_priority = old_prio;
+ arch_spin_unlock(&sb->lock);
+ }
+
+ /* No change ? Bail */
+ if (old_prio == MASKED)
+ return old_prio;
+
+ /* Get the right irq */
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ /*
+ * If the interrupt is marked as needing masking via
+ * firmware, we do it here. Firmware masking however
+ * is "lossy", it won't return the old p and q bits
+ * and won't set the interrupt to a state where it will
+ * record queued ones. If this is an issue we should do
+ * lazy masking instead.
+ *
+ * For now, we work around this in unmask by forcing
+ * an interrupt whenever we unmask a non-LSI via FW
+ * (if ever).
+ */
+ if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
+ xive_native_configure_irq(hw_num,
+ xive->vp_base + state->act_server,
+ MASKED, state->number);
+ /* set old_p so we can track if an H_EOI was done */
+ state->old_p = true;
+ state->old_q = false;
+ } else {
+ /* Set PQ to 10, return old P and old Q and remember them */
+ val = xive_vm_esb_load(xd, XIVE_ESB_SET_PQ_10);
+ state->old_p = !!(val & 2);
+ state->old_q = !!(val & 1);
+
+ /*
+ * Synchronize hardware to sensure the queues are updated
+ * when masking
+ */
+ xive_native_sync_source(hw_num);
+ }
+
+ return old_prio;
+}
+
+static void xive_lock_for_unmask(struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state)
+{
+ /*
+ * Take the lock try again if racing with H_EOI
+ */
+ for (;;) {
+ arch_spin_lock(&sb->lock);
+ if (!state->in_eoi)
+ break;
+ arch_spin_unlock(&sb->lock);
+ }
+}
+
+static void xive_finish_unmask(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state,
+ u8 prio)
+{
+ struct xive_irq_data *xd;
+ u32 hw_num;
+
+ /* If we aren't changing a thing, move on */
+ if (state->guest_priority != MASKED)
+ goto bail;
+
+ /* Get the right irq */
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ /*
+ * See command in xive_lock_and_mask() concerning masking
+ * via firmware.
+ */
+ if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
+ xive_native_configure_irq(hw_num,
+ xive->vp_base + state->act_server,
+ state->act_priority, state->number);
+ /* If an EOI is needed, do it here */
+ if (!state->old_p)
+ xive_vm_source_eoi(hw_num, xd);
+ /* If this is not an LSI, force a trigger */
+ if (!(xd->flags & OPAL_XIVE_IRQ_LSI))
+ xive_irq_trigger(xd);
+ goto bail;
+ }
+
+ /* Old Q set, set PQ to 11 */
+ if (state->old_q)
+ xive_vm_esb_load(xd, XIVE_ESB_SET_PQ_11);
+
+ /*
+ * If not old P, then perform an "effective" EOI,
+ * on the source. This will handle the cases where
+ * FW EOI is needed.
+ */
+ if (!state->old_p)
+ xive_vm_source_eoi(hw_num, xd);
+
+ /* Synchronize ordering and mark unmasked */
+ mb();
+ bail:
+ state->guest_priority = prio;
+}
+
+/*
+ * Target an interrupt to a given server/prio, this will fallback
+ * to another server if necessary and perform the HW targetting
+ * updates as needed
+ *
+ * NOTE: Must be called with the state lock held
+ */
+static int xive_target_interrupt(struct kvm *kvm,
+ struct kvmppc_xive_irq_state *state,
+ u32 server, u8 prio)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ u32 hw_num;
+ int rc;
+
+ /*
+ * This will return a tentative server and actual
+ * priority. The count for that new target will have
+ * already been incremented.
+ */
+ rc = xive_select_target(kvm, &server, prio);
+
+ /* We failed to find a target ? Not much we can do
+ * at least until we support the GIQ.
+ */
+ if (rc)
+ return rc;
+
+ /*
+ * Increment the old queue pending count if there
+ * was one so that the old queue count gets adjusted later
+ * when observed to be empty.
+ */
+ if (state->act_priority != MASKED)
+ xive_inc_q_pending(kvm,
+ state->act_server,
+ state->act_priority);
+ /*
+ * Update state and HW
+ */
+ state->act_priority = prio;
+ state->act_server = server;
+
+ /* Get the right irq */
+ kvmppc_xive_select_irq(state, &hw_num, NULL);
+
+ return xive_native_configure_irq(hw_num,
+ xive->vp_base + server,
+ prio, state->number);
+}
+
+/*
+ * Targetting rules: In order to avoid losing track of
+ * pending interrupts accross mask and unmask, which would
+ * allow queue overflows, we implement the following rules:
+ *
+ * - Unless it was never enabled (or we run out of capacity)
+ * an interrupt is always targetted at a valid server/queue
+ * pair even when "masked" by the guest. This pair tends to
+ * be the last one used but it can be changed under some
+ * circumstances. That allows us to separate targetting
+ * from masking, we only handle accounting during (re)targetting,
+ * this also allows us to let an interrupt drain into its target
+ * queue after masking, avoiding complex schemes to remove
+ * interrupts out of remote processor queues.
+ *
+ * - When masking, we set PQ to 10 and save the previous value
+ * of P and Q.
+ *
+ * - When unmasking, if saved Q was set, we set PQ to 11
+ * otherwise we leave PQ to the HW state which will be either
+ * 10 if nothing happened or 11 if the interrupt fired while
+ * masked. Effectively we are OR'ing the previous Q into the
+ * HW Q.
+ *
+ * Then if saved P is clear, we do an effective EOI (Q->P->Trigger)
+ * which will unmask the interrupt and shoot a new one if Q was
+ * set.
+ *
+ * Otherwise (saved P is set) we leave PQ unchanged (so 10 or 11,
+ * effectively meaning an H_EOI from the guest is still expected
+ * for that interrupt).
+ *
+ * - If H_EOI occurs while masked, we clear the saved P.
+ *
+ * - When changing target, we account on the new target and
+ * increment a separate "pending" counter on the old one.
+ * This pending counter will be used to decrement the old
+ * target's count when its queue has been observed empty.
+ */
+
+int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
+ u32 priority)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u8 new_act_prio;
+ int rc = 0;
+ u16 idx;
+
+ if (!xive)
+ return -ENODEV;
+
+ DBG("set_xive ! irq 0x%x server 0x%x prio %d\n",
+ irq, server, priority);
+
+ /* First, check provisioning of queues */
+ if (priority != MASKED)
+ rc = xive_check_provisioning(xive->kvm,
+ xive_prio_from_guest(priority));
+ if (rc) {
+ DBG(" provisioning failure %d !\n", rc);
+ return rc;
+ }
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return -EINVAL;
+ state = &sb->irq_state[idx];
+
+ /*
+ * We first handle masking/unmasking since the locking
+ * might need to be retried due to EOIs, we'll handle
+ * targetting changes later. These functions will return
+ * with the SB lock held.
+ *
+ * xive_lock_and_mask() will also set state->guest_priority
+ * but won't otherwise change other fields of the state.
+ *
+ * xive_lock_for_unmask will not actually unmask, this will
+ * be done later by xive_finish_unmask() once the targetting
+ * has been done, so we don't try to unmask an interrupt
+ * that hasn't yet been targetted.
+ */
+ if (priority == MASKED)
+ xive_lock_and_mask(xive, sb, state);
+ else
+ xive_lock_for_unmask(sb, state);
+
+
+ /*
+ * Then we handle targetting.
+ *
+ * First calculate a new "actual priority"
+ */
+ new_act_prio = state->act_priority;
+ if (priority != MASKED)
+ new_act_prio = xive_prio_from_guest(priority);
+
+ DBG(" new_act_prio=%x act_server=%x act_prio=%x\n",
+ new_act_prio, state->act_server, state->act_priority);
+
+ /*
+ * Then check if we actually need to change anything,
+ *
+ * The condition for re-targetting the interrupt is that
+ * we have a valid new priority (new_act_prio is not 0xff)
+ * and either the server or the priority changed.
+ *
+ * Note: If act_priority was ff and the new priority is
+ * also ff, we don't do anything and leave the interrupt
+ * untargetted. An attempt of doing an int_on on an
+ * untargetted interrupt will fail. If that is a problem
+ * we could initialize interrupts with valid default
+ */
+
+ if (new_act_prio != MASKED &&
+ (state->act_server != server ||
+ state->act_priority != new_act_prio))
+ rc = xive_target_interrupt(kvm, state, server, new_act_prio);
+
+ /*
+ * Perform the final unmasking of the interrupt source
+ * if necessary
+ */
+ if (priority != MASKED)
+ xive_finish_unmask(xive, sb, state, priority);
+
+ /*
+ * Finally Update saved_priority to match. Only int_on/off
+ * set this field to a different value.
+ */
+ state->saved_priority = priority;
+
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
+int kvmppc_xive_get_xive(struct kvm *kvm, u32 irq, u32 *server,
+ u32 *priority)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u16 idx;
+
+ if (!xive)
+ return -ENODEV;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return -EINVAL;
+ state = &sb->irq_state[idx];
+ arch_spin_lock(&sb->lock);
+ *server = state->guest_server;
+ *priority = state->guest_priority;
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
+int kvmppc_xive_int_on(struct kvm *kvm, u32 irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u16 idx;
+
+ if (!xive)
+ return -ENODEV;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return -EINVAL;
+ state = &sb->irq_state[idx];
+
+ DBG("int_on(irq=0x%x)\n", irq);
+
+ /*
+ * Check if interrupt was not targetted
+ */
+ if (state->act_priority == MASKED) {
+ DBG("int_on on untargetted interrupt\n");
+ return -EINVAL;
+ }
+
+ /* If saved_priority is 0xff, do nothing */
+ if (state->saved_priority == MASKED)
+ return 0;
+
+ /*
+ * Lock and unmask it.
+ */
+ xive_lock_for_unmask(sb, state);
+ xive_finish_unmask(xive, sb, state, state->saved_priority);
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
+int kvmppc_xive_int_off(struct kvm *kvm, u32 irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u16 idx;
+
+ if (!xive)
+ return -ENODEV;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return -EINVAL;
+ state = &sb->irq_state[idx];
+
+ DBG("int_off(irq=0x%x)\n", irq);
+
+ /*
+ * Lock and mask
+ */
+ state->saved_priority = xive_lock_and_mask(xive, sb, state);
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
+static bool xive_restore_pending_irq(struct kvmppc_xive *xive, u32 irq)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u16 idx;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return false;
+ state = &sb->irq_state[idx];
+ if (!state->valid)
+ return false;
+
+ /*
+ * Trigger the IPI. This assumes we never restore a pass-through
+ * interrupt which should be safe enough
+ */
+ xive_irq_trigger(&state->ipi_data);
+
+ return true;
+}
+
+u64 kvmppc_xive_get_icp(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ return 0;
+
+ /* Return the per-cpu state for state saving/migration */
+ return (u64)xc->cppr << KVM_REG_PPC_ICP_CPPR_SHIFT |
+ (u64)xc->mfrr << KVM_REG_PPC_ICP_MFRR_SHIFT;
+}
+
+int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+ u8 cppr, mfrr;
+ u32 xisr;
+
+ if (!xc || !xive)
+ return -ENOENT;
+
+ /* Grab individual state fields. We don't use pending_pri */
+ cppr = icpval >> KVM_REG_PPC_ICP_CPPR_SHIFT;
+ xisr = (icpval >> KVM_REG_PPC_ICP_XISR_SHIFT) &
+ KVM_REG_PPC_ICP_XISR_MASK;
+ mfrr = icpval >> KVM_REG_PPC_ICP_MFRR_SHIFT;
+
+ DBG("set_icp vcpu %d cppr=0x%x mfrr=0x%x xisr=0x%x\n",
+ xc->server_num, cppr, mfrr, xisr);
+
+ /*
+ * We can't update the state of a "pushed" VCPU, but that
+ * shouldn't happen.
+ */
+ if (WARN_ON(vcpu->arch.xive_pushed))
+ return -EIO;
+
+ /* Update VCPU HW saved state */
+ vcpu->arch.xive_saved_state.cppr = cppr;
+ xc->hw_cppr = xc->cppr = cppr;
+
+ /*
+ * Update MFRR state. If it's not 0xff, we mark the VCPU as
+ * having a pending MFRR change, which will re-evaluate the
+ * target. The VCPU will thus potentially get a spurious
+ * interrupt but that's not a big deal.
+ */
+ xc->mfrr = mfrr;
+ if (mfrr < cppr)
+ xive_irq_trigger(&xc->vp_ipi_data);
+
+ /*
+ * Now saved XIRR is "interesting". It means there's something in
+ * the legacy "1 element" queue... for an IPI we simply ignore it,
+ * as the MFRR restore will handle that. For anything else we need
+ * to force a resend of the source.
+ * However the source may not have been setup yet. If that's the
+ * case, we keep that info and increment a counter in the xive to
+ * tell subsequent xive_set_source() to go look.
+ */
+ if (xisr > XICS_IPI && !xive_restore_pending_irq(xive, xisr)) {
+ xc->delayed_irq = xisr;
+ xive->delayed_irqs++;
+ DBG(" xisr restore delayed\n");
+ }
+
+ return 0;
+}
+
+int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
+ struct irq_desc *host_desc)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct irq_data *host_data = irq_desc_get_irq_data(host_desc);
+ unsigned int host_irq = irq_desc_get_irq(host_desc);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(host_data);
+ u16 idx;
+ u8 prio;
+ int rc;
+
+ if (!xive)
+ return -ENODEV;
+
+ DBG("set_mapped girq 0x%lx host HW irq 0x%x...\n", guest_irq, hw_irq);
+
+ sb = kvmppc_xive_find_source(xive, guest_irq, &idx);
+ if (!sb)
+ return -EINVAL;
+ state = &sb->irq_state[idx];
+
+ /*
+ * Mark the passed-through interrupt as going to a VCPU,
+ * this will prevent further EOIs and similar operations
+ * from the XIVE code. It will also mask the interrupt
+ * to either PQ=10 or 11 state, the latter if the interrupt
+ * is pending. This will allow us to unmask or retrigger it
+ * after routing it to the guest with a simple EOI.
+ *
+ * The "state" argument is a "token", all it needs is to be
+ * non-NULL to switch to passed-through or NULL for the
+ * other way around. We may not yet have an actual VCPU
+ * target here and we don't really care.
+ */
+ rc = irq_set_vcpu_affinity(host_irq, state);
+ if (rc) {
+ pr_err("Failed to set VCPU affinity for irq %d\n", host_irq);
+ return rc;
+ }
+
+ /*
+ * Mask and read state of IPI. We need to know if its P bit
+ * is set as that means it's potentially already using a
+ * queue entry in the target
+ */
+ prio = xive_lock_and_mask(xive, sb, state);
+ DBG(" old IPI prio %02x P:%d Q:%d\n", prio, state->old_p, state->old_q);
+
+ /* Turn the IPI hard off */
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+
+ /* Grab info about irq */
+ state->pt_number = hw_irq;
+ state->pt_data = irq_data_get_irq_handler_data(host_data);
+
+ /*
+ * Configure the IRQ to match the existing configuration of
+ * the IPI if it was already targetted. Otherwise this will
+ * mask the interrupt in a lossy way (act_priority is 0xff)
+ * which is fine for a never started interrupt.
+ */
+ xive_native_configure_irq(hw_irq,
+ xive->vp_base + state->act_server,
+ state->act_priority, state->number);
+
+ /*
+ * We do an EOI to enable the interrupt (and retrigger if needed)
+ * if the guest has the interrupt unmasked and the P bit was *not*
+ * set in the IPI. If it was set, we know a slot may still be in
+ * use in the target queue thus we have to wait for a guest
+ * originated EOI
+ */
+ if (prio != MASKED && !state->old_p)
+ xive_vm_source_eoi(hw_irq, state->pt_data);
+
+ /* Clear old_p/old_q as they are no longer relevant */
+ state->old_p = state->old_q = false;
+
+ /* Restore guest prio (unlocks EOI) */
+ mb();
+ state->guest_priority = prio;
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvmppc_xive_set_mapped);
+
+int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
+ struct irq_desc *host_desc)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ unsigned int host_irq = irq_desc_get_irq(host_desc);
+ u16 idx;
+ u8 prio;
+ int rc;
+
+ if (!xive)
+ return -ENODEV;
+
+ DBG("clr_mapped girq 0x%lx...\n", guest_irq);
+
+ sb = kvmppc_xive_find_source(xive, guest_irq, &idx);
+ if (!sb)
+ return -EINVAL;
+ state = &sb->irq_state[idx];
+
+ /*
+ * Mask and read state of IRQ. We need to know if its P bit
+ * is set as that means it's potentially already using a
+ * queue entry in the target
+ */
+ prio = xive_lock_and_mask(xive, sb, state);
+ DBG(" old IRQ prio %02x P:%d Q:%d\n", prio, state->old_p, state->old_q);
+
+ /*
+ * If old_p is set, the interrupt is pending, we switch it to
+ * PQ=11. This will force a resend in the host so the interrupt
+ * isn't lost to whatver host driver may pick it up
+ */
+ if (state->old_p)
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_11);
+
+ /* Relase the passed-through interrupt to the host */
+ rc = irq_set_vcpu_affinity(host_irq, NULL);
+ if (rc) {
+ pr_err("Failed to clr VCPU affinity for irq %d\n", host_irq);
+ return rc;
+ }
+
+ /* Forget about the IRQ */
+ state->pt_number = 0;
+ state->pt_data = NULL;
+
+ /* Reconfigure the IPI */
+ xive_native_configure_irq(state->ipi_number,
+ xive->vp_base + state->act_server,
+ state->act_priority, state->number);
+
+ /*
+ * If old_p is set (we have a queue entry potentially
+ * occupied) or the interrupt is masked, we set the IPI
+ * to PQ=10 state. Otherwise we just re-enable it (PQ=00).
+ */
+ if (prio == MASKED || state->old_p)
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_10);
+ else
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_00);
+
+ /* Restore guest prio (unlocks EOI) */
+ mb();
+ state->guest_priority = prio;
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
+
+static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvm *kvm = vcpu->kvm;
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ int i, j;
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (!sb)
+ continue;
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
+
+ if (!state->valid)
+ continue;
+ if (state->act_priority == MASKED)
+ continue;
+ if (state->act_server != xc->server_num)
+ continue;
+
+ /* Clean it up */
+ arch_spin_lock(&sb->lock);
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+ if (state->pt_number) {
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->pt_number, 0, MASKED, 0);
+ }
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+}
+
+void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = xc->xive;
+ int i;
+
+ DBG("cleanup_vcpu(cpu=%d)\n", xc->server_num);
+
+ /* Ensure no interrupt is still routed to that VP */
+ xc->valid = false;
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ /* Mask the VP IPI */
+ xive_vm_esb_load(&xc->vp_ipi_data, XIVE_ESB_SET_PQ_01);
+
+ /* Disable the VP */
+ xive_native_disable_vp(xc->vp_id);
+
+ /* Free the queues & associated interrupts */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+
+ /* Free the escalation irq */
+ if (xc->esc_virq[i]) {
+ free_irq(xc->esc_virq[i], vcpu);
+ irq_dispose_mapping(xc->esc_virq[i]);
+ kfree(xc->esc_virq_names[i]);
+ }
+ /* Free the queue */
+ xive_native_disable_queue(xc->vp_id, q, i);
+ if (q->qpage) {
+ free_pages((unsigned long)q->qpage,
+ xive->q_alloc_order);
+ q->qpage = NULL;
+ }
+ }
+
+ /* Free the IPI */
+ if (xc->vp_ipi) {
+ xive_cleanup_irq_data(&xc->vp_ipi_data);
+ xive_native_free_irq(xc->vp_ipi);
+ }
+ /* Free the VP */
+ kfree(xc);
+}
+
+int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_vcpu *xc;
+ int i, r = -EBUSY;
+
+ DBG("connect_vcpu(cpu=%d)\n", cpu);
+
+ if (dev->ops != &kvm_xive_ops) {
+ DBG("Wrong ops !\n");
+ return -EPERM;
+ }
+ if (xive->kvm != vcpu->kvm)
+ return -EPERM;
+ if (vcpu->arch.irq_type)
+ return -EBUSY;
+ if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
+ DBG("Duplicate !\n");
+ return -EEXIST;
+ }
+ if (cpu >= KVM_MAX_VCPUS) {
+ DBG("Out of bounds !\n");
+ return -EINVAL;
+ }
+ xc = kzalloc(sizeof(*xc), GFP_KERNEL);
+ if (!xc)
+ return -ENOMEM;
+
+ /* We need to synchronize with queue provisioning */
+ mutex_lock(&vcpu->kvm->lock);
+ vcpu->arch.xive_vcpu = xc;
+ xc->xive = xive;
+ xc->vcpu = vcpu;
+ xc->server_num = cpu;
+ xc->vp_id = xive->vp_base + cpu;
+ xc->mfrr = 0xff;
+ xc->valid = true;
+
+ r = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
+ if (r)
+ goto bail;
+
+ /* Configure VCPU fields for use by assembly push/pull */
+ vcpu->arch.xive_saved_state.qw = cpu_to_be64(0xff000000);
+ vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
+
+ /* Allocate IPI */
+ xc->vp_ipi = xive_native_alloc_irq();
+ if (!xc->vp_ipi) {
+ r = -EIO;
+ goto bail;
+ }
+ DBG(" IPI=0x%x\n", xc->vp_ipi);
+
+ r = xive_native_populate_irq_data(xc->vp_ipi, &xc->vp_ipi_data);
+ if (r)
+ goto bail;
+
+ /*
+ * Initialize queues. Initially we set them all for no queueing
+ * and we enable escalation for queue 0 only which we'll use for
+ * our mfrr change notifications. If the VCPU is hot-plugged, we
+ * do handle provisioning however.
+ */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+
+ /* Is queue already enabled ? Provision it */
+ if (xive->qmap & (1 << i)) {
+ r = xive_provision_queue(vcpu, i);
+ if (r == 0)
+ xive_attach_escalation(vcpu, i);
+ if (r)
+ goto bail;
+ } else {
+ r = xive_native_configure_queue(xc->vp_id,
+ q, i, NULL, 0, true);
+ if (r) {
+ pr_err("XIVE-KVM: Failed to configure queue %d"
+ " for VCPU %d\n",
+ i, cpu);
+ goto bail;
+ }
+ }
+ }
+
+ /* If not done above, attach priority 0 escalation */
+ r = xive_attach_escalation(vcpu, 0);
+ if (r)
+ goto bail;
+
+ /* Enable the VP */
+ r = xive_native_enable_vp(xc->vp_id);
+ if (r)
+ goto bail;
+
+ /* Route the IPI */
+ r = xive_native_configure_irq(xc->vp_ipi, xc->vp_id, 0, XICS_IPI);
+ if (!r)
+ xive_vm_esb_load(&xc->vp_ipi_data, XIVE_ESB_SET_PQ_00);
+
+ bail:
+ mutex_unlock(&vcpu->kvm->lock);
+ if (r) {
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ return r;
+ }
+
+ vcpu->arch.irq_type = KVMPPC_IRQ_XICS;
+ return 0;
+}
+
+/*
+ * Scanning of queues before/after migration save
+ */
+static void xive_pre_save_set_queued(struct kvmppc_xive *xive, u32 irq)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u16 idx;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return;
+
+ state = &sb->irq_state[idx];
+
+ /* Some sanity checking */
+ if (!state->valid) {
+ pr_err("XIVE/XIVE: invalid irq 0x%x in cpu queue!\n", irq);
+ return;
+ }
+
+ /*
+ * If the interrupt is in a queue it should have P set.
+ * We warn so that gets reported. A backtrace isn't useful
+ * so no need to use a WARN_ON.
+ */
+ if (!state->saved_p)
+ pr_err("KVM/XIVE: Interrupt 0x%x is marked in a queue"
+ " but P not set !\n", irq);
+
+ /* Set flag */
+ state->in_queue = true;
+}
+
+static void xive_pre_scan_mask_irq(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ u32 irq)
+{
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[irq];
+
+ if (!state->valid)
+ return;
+
+ /* Mask and save state, this will also sync HW queues */
+ state->saved_scan_prio = xive_lock_and_mask(xive, sb, state);
+
+ /* Transfer P and Q */
+ state->saved_p = state->old_p;
+ state->saved_q = state->old_q;
+
+ /* Unlock */
+ arch_spin_unlock(&sb->lock);
+}
+
+static void xive_pre_scan_unmask_irq(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ u32 irq)
+{
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[irq];
+
+ if (!state->valid)
+ return;
+
+ /*
+ * Lock / exclude EOI (not technically necessary if the
+ * guest isn't running concurrently. If this becomes a
+ * performance issue we can probably remove the lock.
+ */
+ xive_lock_for_unmask(sb, state);
+
+ /* Restore mask/prio if it wasn't masked */
+ if (state->saved_scan_prio != MASKED)
+ xive_finish_unmask(xive, sb, state, state->saved_scan_prio);
+
+ /* Unlock */
+ arch_spin_unlock(&sb->lock);
+}
+
+static void xive_pre_save_queue(struct kvmppc_xive *xive, struct xive_q *q)
+{
+ u32 idx = q->idx;
+ u32 toggle = q->toggle;
+ u32 irq;
+
+ do {
+ irq = __xive_read_eq(q->qpage, q->msk, &idx, &toggle);
+ if (irq > XICS_IPI)
+ xive_pre_save_set_queued(xive, irq);
+ } while(irq);
+}
+
+static void xive_pre_save_scan(struct kvmppc_xive *xive)
+{
+ struct kvm_vcpu *vcpu = NULL;
+ int i, j;
+
+ /*
+ * See comment in xive_get_source() about how this
+ * work. Collect a stable state for all interrupts
+ */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+ if (!sb)
+ continue;
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++)
+ xive_pre_scan_mask_irq(xive, sb, j);
+ }
+
+ /* Then scan the queues and update the "in_queue" flag */
+ kvm_for_each_vcpu(i, vcpu, xive->kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ if (!xc)
+ continue;
+ for (j = 0; j < KVMPPC_XIVE_Q_COUNT; j++) {
+ if (xc->queues[i].qpage)
+ xive_pre_save_queue(xive, &xc->queues[i]);
+ }
+ }
+
+ /* Finally restore interrupt states */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+ if (!sb)
+ continue;
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++)
+ xive_pre_scan_unmask_irq(xive, sb, j);
+ }
+}
+
+static void xive_post_save_scan(struct kvmppc_xive *xive)
+{
+ u32 i, j;
+
+ /* Clear all the in_queue flags */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+ if (!sb)
+ continue;
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++)
+ sb->irq_state[j].in_queue = false;
+ }
+
+ /* Next get_source() will do a new scan */
+ xive->saved_src_count = 0;
+}
+
+/*
+ * This returns the source configuration and state to user space.
+ */
+static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u64 val, prio;
+ u16 idx;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[idx];
+
+ if (!state->valid)
+ return -ENOENT;
+
+ DBG("get_source(%ld)...\n", irq);
+
+ /*
+ * So to properly save the state into something that looks like a
+ * XICS migration stream we cannot treat interrupts individually.
+ *
+ * We need, instead, mask them all (& save their previous PQ state)
+ * to get a stable state in the HW, then sync them to ensure that
+ * any interrupt that had already fired hits its queue, and finally
+ * scan all the queues to collect which interrupts are still present
+ * in the queues, so we can set the "pending" flag on them and
+ * they can be resent on restore.
+ *
+ * So we do it all when the "first" interrupt gets saved, all the
+ * state is collected at that point, the rest of xive_get_source()
+ * will merely collect and convert that state to the expected
+ * userspace bit mask.
+ */
+ if (xive->saved_src_count == 0)
+ xive_pre_save_scan(xive);
+ xive->saved_src_count++;
+
+ /* Convert saved state into something compatible with xics */
+ val = state->guest_server;
+ prio = state->saved_scan_prio;
+
+ if (prio == MASKED) {
+ val |= KVM_XICS_MASKED;
+ prio = state->saved_priority;
+ }
+ val |= prio << KVM_XICS_PRIORITY_SHIFT;
+ if (state->lsi) {
+ val |= KVM_XICS_LEVEL_SENSITIVE;
+ if (state->saved_p)
+ val |= KVM_XICS_PENDING;
+ } else {
+ if (state->saved_p)
+ val |= KVM_XICS_PRESENTED;
+
+ if (state->saved_q)
+ val |= KVM_XICS_QUEUED;
+
+ /*
+ * We mark it pending (which will attempt a re-delivery)
+ * if we are in a queue *or* we were masked and had
+ * Q set which is equivalent to the XICS "masked pending"
+ * state
+ */
+ if (state->in_queue || (prio == MASKED && state->saved_q))
+ val |= KVM_XICS_PENDING;
+ }
+
+ /*
+ * If that was the last interrupt saved, reset the
+ * in_queue flags
+ */
+ if (xive->saved_src_count == xive->src_count)
+ xive_post_save_scan(xive);
+
+ /* Copy the result to userspace */
+ if (put_user(val, ubufp))
+ return -EFAULT;
+
+ return 0;
+}
+
+static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
+ int irq)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvmppc_xive_src_block *sb;
+ int i, bid;
+
+ bid = irq >> KVMPPC_XICS_ICS_SHIFT;
+
+ mutex_lock(&kvm->lock);
+
+ /* block already exists - somebody else got here first */
+ if (xive->src_blocks[bid])
+ goto out;
+
+ /* Create the ICS */
+ sb = kzalloc(sizeof(*sb), GFP_KERNEL);
+ if (!sb)
+ goto out;
+
+ sb->id = bid;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
+ sb->irq_state[i].guest_priority = MASKED;
+ sb->irq_state[i].saved_priority = MASKED;
+ sb->irq_state[i].act_priority = MASKED;
+ }
+ smp_wmb();
+ xive->src_blocks[bid] = sb;
+
+ if (bid > xive->max_sbid)
+ xive->max_sbid = bid;
+
+ out:
+ mutex_unlock(&kvm->lock);
+ return xive->src_blocks[bid];
+}
+
+static bool xive_check_delayed_irq(struct kvmppc_xive *xive, u32 irq)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu = NULL;
+ int i;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ if (xc->delayed_irq == irq) {
+ xc->delayed_irq = 0;
+ xive->delayed_irqs--;
+ return true;
+ }
+ }
+ return false;
+}
+
+static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u16 idx;
+ u64 val;
+ u8 act_prio, guest_prio;
+ u32 server;
+ int rc = 0;
+
+ if (irq < KVMPPC_XICS_FIRST_IRQ || irq >= KVMPPC_XICS_NR_IRQS)
+ return -ENOENT;
+
+ DBG("set_source(irq=0x%lx)\n", irq);
+
+ /* Find the source */
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb) {
+ DBG("No source, creating source block...\n");
+ sb = xive_create_src_block(xive, irq);
+ if (!sb) {
+ DBG("Failed to create block...\n");
+ return -ENOMEM;
+ }
+ }
+ state = &sb->irq_state[idx];
+
+ /* Read user passed data */
+ if (get_user(val, ubufp)) {
+ DBG("fault getting user info !\n");
+ return -EFAULT;
+ }
+
+ server = val & KVM_XICS_DESTINATION_MASK;
+ guest_prio = val >> KVM_XICS_PRIORITY_SHIFT;
+
+ DBG(" val=0x016%llx (server=0x%x, guest_prio=%d)\n",
+ val, server, guest_prio);
+ /*
+ * If the source doesn't already have an IPI, allocate
+ * one and get the corresponding data
+ */
+ if (!state->ipi_number) {
+ state->ipi_number = xive_native_alloc_irq();
+ if (state->ipi_number == 0) {
+ DBG("Failed to allocate IPI !\n");
+ return -ENOMEM;
+ }
+ xive_native_populate_irq_data(state->ipi_number, &state->ipi_data);
+ DBG(" src_ipi=0x%x\n", state->ipi_number);
+ }
+
+ /*
+ * We use lock_and_mask() to set us in the right masked
+ * state. We will override that state from the saved state
+ * further down, but this will handle the cases of interrupts
+ * that need FW masking. We set the initial guest_priority to
+ * 0 before calling it to ensure it actually performs the masking.
+ */
+ state->guest_priority = 0;
+ xive_lock_and_mask(xive, sb, state);
+
+ /*
+ * Now, we select a target if we have one. If we don't we
+ * leave the interrupt untargetted. It means that an interrupt
+ * can become "untargetted" accross migration if it was masked
+ * by set_xive() but there is little we can do about it.
+ */
+
+ /* First convert prio and mark interrupt as untargetted */
+ act_prio = xive_prio_from_guest(guest_prio);
+ state->act_priority = MASKED;
+ state->guest_server = server;
+
+ /*
+ * We need to drop the lock due to the mutex below. Hopefully
+ * nothing is touching that interrupt yet since it hasn't been
+ * advertized to a running guest yet
+ */
+ arch_spin_unlock(&sb->lock);
+
+ /* If we have a priority target the interrupt */
+ if (act_prio != MASKED) {
+ /* First, check provisioning of queues */
+ mutex_lock(&xive->kvm->lock);
+ rc = xive_check_provisioning(xive->kvm, act_prio);
+ mutex_unlock(&xive->kvm->lock);
+
+ /* Target interrupt */
+ if (rc == 0)
+ rc = xive_target_interrupt(xive->kvm, state,
+ server, act_prio);
+ /*
+ * If provisioning or targetting failed, leave it
+ * alone and masked. It will remain disabled until
+ * the guest re-targets it.
+ */
+ }
+
+ /*
+ * Find out if this was a delayed irq stashed in an ICP,
+ * in which case, treat it as pending
+ */
+ if (xive->delayed_irqs && xive_check_delayed_irq(xive, irq)) {
+ val |= KVM_XICS_PENDING;
+ DBG(" Found delayed ! forcing PENDING !\n");
+ }
+
+ /* Cleanup the SW state */
+ state->old_p = false;
+ state->old_q = false;
+ state->lsi = false;
+ state->asserted = false;
+
+ /* Restore LSI state */
+ if (val & KVM_XICS_LEVEL_SENSITIVE) {
+ state->lsi = true;
+ if (val & KVM_XICS_PENDING)
+ state->asserted = true;
+ DBG(" LSI ! Asserted=%d\n", state->asserted);
+ }
+
+ /*
+ * Restore P and Q. If the interrupt was pending, we
+ * force both P and Q, which will trigger a resend.
+ *
+ * That means that a guest that had both an interrupt
+ * pending (queued) and Q set will restore with only
+ * one instance of that interrupt instead of 2, but that
+ * is perfectly fine as coalescing interrupts that haven't
+ * been presented yet is always allowed.
+ */
+ if (val & KVM_XICS_PRESENTED || val & KVM_XICS_PENDING)
+ state->old_p = true;
+ if (val & KVM_XICS_QUEUED || val & KVM_XICS_PENDING)
+ state->old_q = true;
+
+ DBG(" P=%d, Q=%d\n", state->old_p, state->old_q);
+
+ /*
+ * If the interrupt was unmasked, update guest priority and
+ * perform the appropriate state transition and do a
+ * re-trigger if necessary.
+ */
+ if (val & KVM_XICS_MASKED) {
+ DBG(" masked, saving prio\n");
+ state->guest_priority = MASKED;
+ state->saved_priority = guest_prio;
+ } else {
+ DBG(" unmasked, restoring to prio %d\n", guest_prio);
+ xive_finish_unmask(xive, sb, state, guest_prio);
+ state->saved_priority = guest_prio;
+ }
+
+ /* Increment the number of valid sources and mark this one valid */
+ if (!state->valid)
+ xive->src_count++;
+ state->valid = true;
+
+ return 0;
+}
+
+int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
+ bool line_status)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u16 idx;
+
+ if (!xive)
+ return -ENODEV;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb)
+ return -EINVAL;
+
+ /* Perform locklessly .... (we need to do some RCUisms here...) */
+ state = &sb->irq_state[idx];
+ if (!state->valid)
+ return -EINVAL;
+
+ /* We don't allow a trigger on a passed-through interrupt */
+ if (state->pt_number)
+ return -EINVAL;
+
+ if ((level == 1 && state->lsi) || level == KVM_INTERRUPT_SET_LEVEL)
+ state->asserted = 1;
+ else if (level == 0 || level == KVM_INTERRUPT_UNSET) {
+ state->asserted = 0;
+ return 0;
+ }
+
+ /* Trigger the IPI */
+ xive_irq_trigger(&state->ipi_data);
+
+ return 0;
+}
+
+static int xive_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
+{
+ struct kvmppc_xive *xive = dev->private;
+
+ /* We honor the existing XICS ioctl */
+ switch (attr->group) {
+ case KVM_DEV_XICS_GRP_SOURCES:
+ return xive_set_source(xive, attr->attr, attr->addr);
+ }
+ return -ENXIO;
+}
+
+static int xive_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
+{
+ struct kvmppc_xive *xive = dev->private;
+
+ /* We honor the existing XICS ioctl */
+ switch (attr->group) {
+ case KVM_DEV_XICS_GRP_SOURCES:
+ return xive_get_source(xive, attr->attr, attr->addr);
+ }
+ return -ENXIO;
+}
+
+static int xive_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
+{
+ /* We honor the same limits as XICS, at least for now */
+ switch (attr->group) {
+ case KVM_DEV_XICS_GRP_SOURCES:
+ if (attr->attr >= KVMPPC_XICS_FIRST_IRQ &&
+ attr->attr < KVMPPC_XICS_NR_IRQS)
+ return 0;
+ break;
+ }
+ return -ENXIO;
+}
+
+static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
+{
+ xive_vm_esb_load(xd, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(hw_num, 0, MASKED, 0);
+ xive_cleanup_irq_data(xd);
+}
+
+static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
+{
+ int i;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
+
+ if (!state->valid)
+ continue;
+
+ kvmppc_xive_cleanup_irq(state->ipi_number, &state->ipi_data);
+ xive_native_free_irq(state->ipi_number);
+
+ /* Pass-through, cleanup too */
+ if (state->pt_number)
+ kvmppc_xive_cleanup_irq(state->pt_number, state->pt_data);
+
+ state->valid = false;
+ }
+}
+
+static void kvmppc_xive_free(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ int i;
+
+ debugfs_remove(xive->dentry);
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ /* Mask and free interrupts */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+
+ kfree(xive);
+ kfree(dev);
+}
+
+static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
+{
+ struct kvmppc_xive *xive;
+ struct kvm *kvm = dev->kvm;
+ int ret = 0;
+
+ DBG("Creating xive for partition\n");
+
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ if (!xive)
+ return -ENOMEM;
+
+ dev->private = xive;
+ xive->dev = dev;
+ xive->kvm = kvm;
+
+ /* Already there ? */
+ if (kvm->arch.xive)
+ ret = -EEXIST;
+ else
+ kvm->arch.xive = xive;
+
+ /* We use the default queue size set by the host */
+ xive->q_order = xive_native_default_eq_shift();
+ if (xive->q_order < PAGE_SHIFT)
+ xive->q_alloc_order = 0;
+ else
+ xive->q_alloc_order = xive->q_order - PAGE_SHIFT;
+
+ /* Allocate a bunch of VPs */
+ xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
+ DBG("VP_Base=%x\n", xive->vp_base);
+ if (xive->vp_base == XIVE_INVALID_VP)
+ ret = -ENOMEM;
+
+ if (ret) {
+ kfree(xive);
+ return ret;
+ }
+
+ return 0;
+}
+
+
+static int xive_debug_show(struct seq_file *m, void *private)
+{
+ struct kvmppc_xive *xive = m->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ u64 t_rm_h_xirr = 0;
+ u64 t_rm_h_ipoll = 0;
+ u64 t_rm_h_cppr = 0;
+ u64 t_rm_h_eoi = 0;
+ u64 t_rm_h_ipi = 0;
+ u64 t_vm_h_xirr = 0;
+ u64 t_vm_h_ipoll = 0;
+ u64 t_vm_h_cppr = 0;
+ u64 t_vm_h_eoi = 0;
+ u64 t_vm_h_ipi = 0;
+ unsigned int i;
+
+ if (!kvm)
+ return 0;
+
+ seq_printf(m, "=========\nVCPU state\n=========\n");
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ seq_printf(m, "cpu server %#x CPPR:%#x HWCPPR:%#x"
+ " MFRR:%#x PEND:%#x h_xirr: R=%lld V=%lld\n",
+ xc->server_num, xc->cppr, xc->hw_cppr,
+ xc->mfrr, xc->pending,
+ xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
+ t_rm_h_xirr += xc->stat_rm_h_xirr;
+ t_rm_h_ipoll += xc->stat_rm_h_ipoll;
+ t_rm_h_cppr += xc->stat_rm_h_cppr;
+ t_rm_h_eoi += xc->stat_rm_h_eoi;
+ t_rm_h_ipi += xc->stat_rm_h_ipi;
+ t_vm_h_xirr += xc->stat_vm_h_xirr;
+ t_vm_h_ipoll += xc->stat_vm_h_ipoll;
+ t_vm_h_cppr += xc->stat_vm_h_cppr;
+ t_vm_h_eoi += xc->stat_vm_h_eoi;
+ t_vm_h_ipi += xc->stat_vm_h_ipi;
+ }
+
+ seq_printf(m, "Hcalls totals\n");
+ seq_printf(m, " H_XIRR R=%10lld V=%10lld\n", t_rm_h_xirr, t_vm_h_xirr);
+ seq_printf(m, " H_IPOLL R=%10lld V=%10lld\n", t_rm_h_ipoll, t_vm_h_ipoll);
+ seq_printf(m, " H_CPPR R=%10lld V=%10lld\n", t_rm_h_cppr, t_vm_h_cppr);
+ seq_printf(m, " H_EOI R=%10lld V=%10lld\n", t_rm_h_eoi, t_vm_h_eoi);
+ seq_printf(m, " H_IPI R=%10lld V=%10lld\n", t_rm_h_ipi, t_vm_h_ipi);
+
+ return 0;
+}
+
+static int xive_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, xive_debug_show, inode->i_private);
+}
+
+static const struct file_operations xive_debug_fops = {
+ .open = xive_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void xive_debugfs_init(struct kvmppc_xive *xive)
+{
+ char *name;
+
+ name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
+ if (!name) {
+ pr_err("%s: no memory for name\n", __func__);
+ return;
+ }
+
+ xive->dentry = debugfs_create_file(name, S_IRUGO, powerpc_debugfs_root,
+ xive, &xive_debug_fops);
+
+ pr_debug("%s: created %s\n", __func__, name);
+ kfree(name);
+}
+
+static void kvmppc_xive_init(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
+
+ /* Register some debug interfaces */
+ xive_debugfs_init(xive);
+}
+
+struct kvm_device_ops kvm_xive_ops = {
+ .name = "kvm-xive",
+ .create = kvmppc_xive_create,
+ .init = kvmppc_xive_init,
+ .destroy = kvmppc_xive_free,
+ .set_attr = xive_set_attr,
+ .get_attr = xive_get_attr,
+ .has_attr = xive_has_attr,
+};
+
+void kvmppc_xive_init_module(void)
+{
+ __xive_vm_h_xirr = xive_vm_h_xirr;
+ __xive_vm_h_ipoll = xive_vm_h_ipoll;
+ __xive_vm_h_ipi = xive_vm_h_ipi;
+ __xive_vm_h_cppr = xive_vm_h_cppr;
+ __xive_vm_h_eoi = xive_vm_h_eoi;
+}
+
+void kvmppc_xive_exit_module(void)
+{
+ __xive_vm_h_xirr = NULL;
+ __xive_vm_h_ipoll = NULL;
+ __xive_vm_h_ipi = NULL;
+ __xive_vm_h_cppr = NULL;
+ __xive_vm_h_eoi = NULL;
+}
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
new file mode 100644
index 0000000..2b7fdbd
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -0,0 +1,251 @@
+/*
+ * Copyright 2017 Benjamin Herrenschmidt, IBM Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef _KVM_PPC_BOOK3S_XIVE_H
+#define _KVM_PPC_BOOK3S_XIVE_H
+
+#include "book3s_xics.h"
+
+/* State for one guest irq source.
+ *
+ * For each guest source we allocate a HW interrupt in the XIVE
+ * which we use for all SW triggers. It will be unused for
+ * pass-through but it's easier to keep around as the same
+ * guest interrupt can alternatively be emulated or pass-through
+ * if a physical device is hot unplugged and replaced with an
+ * emulated one.
+ *
+ * This state structure is very similar to the XICS one with
+ * additional XIVE specific tracking.
+ */
+struct kvmppc_xive_irq_state {
+ bool valid; /* Interrupt entry is valid */
+
+ u32 number; /* Guest IRQ number */
+ u32 ipi_number; /* XIVE IPI HW number */
+ struct xive_irq_data ipi_data; /* XIVE IPI associated data */
+ u32 pt_number; /* XIVE Pass-through number if any */
+ struct xive_irq_data *pt_data; /* XIVE Pass-through associated data */
+
+ /* Targetting as set by guest */
+ u32 guest_server; /* Current guest selected target */
+ u8 guest_priority; /* Guest set priority */
+ u8 saved_priority; /* Saved priority when masking */
+
+ /* Actual targetting */
+ u32 act_server; /* Actual server */
+ u8 act_priority; /* Actual priority */
+
+ /* Various state bits */
+ bool in_eoi; /* Synchronize with H_EOI */
+ bool old_p; /* P bit state when masking */
+ bool old_q; /* Q bit state when masking */
+ bool lsi; /* level-sensitive interrupt */
+ bool asserted; /* Only for emulated LSI: current state */
+
+ /* Saved for migration state */
+ bool in_queue;
+ bool saved_p;
+ bool saved_q;
+ u8 saved_scan_prio;
+};
+
+/* Select the "right" interrupt (IPI vs. passthrough) */
+static inline void kvmppc_xive_select_irq(struct kvmppc_xive_irq_state *state,
+ u32 *out_hw_irq,
+ struct xive_irq_data **out_xd)
+{
+ if (state->pt_number) {
+ if (out_hw_irq)
+ *out_hw_irq = state->pt_number;
+ if (out_xd)
+ *out_xd = state->pt_data;
+ } else {
+ if (out_hw_irq)
+ *out_hw_irq = state->ipi_number;
+ if (out_xd)
+ *out_xd = &state->ipi_data;
+ }
+}
+
+/* This corresponds to an "ICS" in XICS terminology, we use it
+ * as a mean to break up source information into multiple structures
+ */
+struct kvmppc_xive_src_block {
+ arch_spinlock_t lock;
+ u16 id;
+ struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
+};
+
+
+struct kvmppc_xive {
+ struct kvm *kvm;
+ struct kvm_device *dev;
+ struct dentry *dentry;
+
+ /* VP block associated with the VM */
+ u32 vp_base;
+
+ /* Blocks of sources */
+ struct kvmppc_xive_src_block *src_blocks[KVMPPC_XICS_MAX_ICS_ID + 1];
+ u32 max_sbid;
+
+ /*
+ * For state save, we lazily scan the queues on the first interrupt
+ * being migrated. We don't have a clean way to reset that flags
+ * so we keep track of the number of valid sources and how many of
+ * them were migrated so we can reset when all of them have been
+ * processed.
+ */
+ u32 src_count;
+ u32 saved_src_count;
+
+ /*
+ * Some irqs are delayed on restore until the source is created,
+ * keep track here of how many of them
+ */
+ u32 delayed_irqs;
+
+ /* Which queues (priorities) are in use by the guest */
+ u8 qmap;
+
+ /* Queue orders */
+ u32 q_order;
+ u32 q_alloc_order;
+
+};
+
+#define KVMPPC_XIVE_Q_COUNT 8
+
+struct kvmppc_xive_vcpu {
+ struct kvmppc_xive *xive;
+ struct kvm_vcpu *vcpu;
+ bool valid;
+
+ /* Server number. This is the HW CPU ID from a guest perspective */
+ u32 server_num;
+
+ /* HW VP corresponding to this VCPU. This is the base of the VP
+ * block plus the server number
+ */
+ u32 vp_id;
+ u32 vp_chip_id;
+ u32 vp_cam;
+
+ /* IPI used for sending ... IPIs */
+ u32 vp_ipi;
+ struct xive_irq_data vp_ipi_data;
+
+ /* Local emulation state */
+ uint8_t cppr; /* guest CPPR */
+ uint8_t hw_cppr;/* Hardware CPPR */
+ uint8_t mfrr;
+ uint8_t pending;
+
+ /* Each VP has 8 queues though we only provision some */
+ struct xive_q queues[KVMPPC_XIVE_Q_COUNT];
+ u32 esc_virq[KVMPPC_XIVE_Q_COUNT];
+ char *esc_virq_names[KVMPPC_XIVE_Q_COUNT];
+
+ /* Stash a delayed irq on restore from migration (see set_icp) */
+ u32 delayed_irq;
+
+ /* Stats */
+ u64 stat_rm_h_xirr;
+ u64 stat_rm_h_ipoll;
+ u64 stat_rm_h_cppr;
+ u64 stat_rm_h_eoi;
+ u64 stat_rm_h_ipi;
+ u64 stat_vm_h_xirr;
+ u64 stat_vm_h_ipoll;
+ u64 stat_vm_h_cppr;
+ u64 stat_vm_h_eoi;
+ u64 stat_vm_h_ipi;
+};
+
+static inline struct kvm_vcpu *kvmppc_xive_find_server(struct kvm *kvm, u32 nr)
+{
+ struct kvm_vcpu *vcpu = NULL;
+ int i;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (vcpu->arch.xive_vcpu && nr == vcpu->arch.xive_vcpu->server_num)
+ return vcpu;
+ }
+ return NULL;
+}
+
+static inline struct kvmppc_xive_src_block *kvmppc_xive_find_source(struct kvmppc_xive *xive,
+ u32 irq, u16 *source)
+{
+ u32 bid = irq >> KVMPPC_XICS_ICS_SHIFT;
+ u16 src = irq & KVMPPC_XICS_SRC_MASK;
+
+ if (source)
+ *source = src;
+ if (bid > KVMPPC_XICS_MAX_ICS_ID)
+ return NULL;
+ return xive->src_blocks[bid];
+}
+
+/*
+ * Mapping between guest priorities and host priorities
+ * is as follow.
+ *
+ * Guest request for 0...6 are honored. Guest request for anything
+ * higher results in a priority of 7 being applied.
+ *
+ * However, when XIRR is returned via H_XIRR, 7 is translated to 0xb
+ * in order to match AIX expectations
+ *
+ * Similar mapping is done for CPPR values
+ */
+static inline u8 xive_prio_from_guest(u8 prio)
+{
+ if (prio == 0xff || prio < 8)
+ return prio;
+ return 7;
+}
+
+static inline u8 xive_prio_to_guest(u8 prio)
+{
+ if (prio == 0xff || prio < 7)
+ return prio;
+ return 0xb;
+}
+
+static inline u32 __xive_read_eq(__be32 *qpage, u32 msk, u32 *idx, u32 *toggle)
+{
+ u32 cur;
+
+ if (!qpage)
+ return 0;
+ cur = be32_to_cpup(qpage + *idx);
+ if ((cur >> 31) == *toggle)
+ return 0;
+ *idx = (*idx + 1) & msk;
+ if (*idx == 0)
+ (*toggle) ^= 1;
+ return cur & 0x7fffffff;
+}
+
+extern unsigned long xive_rm_h_xirr(struct kvm_vcpu *vcpu);
+extern unsigned long xive_rm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server);
+extern int xive_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr);
+extern int xive_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr);
+extern int xive_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr);
+
+extern unsigned long (*__xive_vm_h_xirr)(struct kvm_vcpu *vcpu);
+extern unsigned long (*__xive_vm_h_ipoll)(struct kvm_vcpu *vcpu, unsigned long server);
+extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr);
+extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
+extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+
+#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive_template.c b/arch/powerpc/kvm/book3s_xive_template.c
new file mode 100644
index 0000000..b28c264
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive_template.c
@@ -0,0 +1,490 @@
+/*
+ * Copyright 2017 Benjamin Herrenschmidt, IBM Corporation
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License, version 2, as
+ * published by the Free Software Foundation.
+ */
+
+/* File to be included by other .c files */
+
+#define XGLUE(a,b) a##b
+#define GLUE(a,b) XGLUE(a,b)
+
+static void GLUE(X_PFX,ack_pending)(struct kvmppc_xive_vcpu *xc)
+{
+ u8 cppr;
+ u16 ack;
+
+ /* XXX DD1 bug workaround: Check PIPR vs. CPPR first ! */
+
+ /* Perform the acknowledge OS to register cycle. */
+ ack = be16_to_cpu(__x_readw(__x_tm_area + TM_SPC_ACK_OS_REG));
+
+ /* Synchronize subsequent queue accesses */
+ mb();
+
+ /* XXX Check grouping level */
+
+ /* Anything ? */
+ if (!((ack >> 8) & TM_QW1_NSR_EO))
+ return;
+
+ /* Grab CPPR of the most favored pending interrupt */
+ cppr = ack & 0xff;
+ if (cppr < 8)
+ xc->pending |= 1 << cppr;
+
+#ifdef XIVE_RUNTIME_CHECKS
+ /* Check consistency */
+ if (cppr >= xc->hw_cppr)
+ pr_warn("KVM-XIVE: CPU %d odd ack CPPR, got %d at %d\n",
+ smp_processor_id(), cppr, xc->hw_cppr);
+#endif
+
+ /* Update our image of the HW CPPR. We don't yet modify
+ * xc->cppr, this will be done as we scan for interrupts
+ * in the queues.
+ */
+ xc->hw_cppr = cppr;
+}
+
+static u8 GLUE(X_PFX,esb_load)(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val =__x_readq(__x_eoi_page(xd) + offset);
+#ifdef __LITTLE_ENDIAN__
+ val >>= 64-8;
+#endif
+ return (u8)val;
+}
+
+
+static void GLUE(X_PFX,source_eoi)(u32 hw_irq, struct xive_irq_data *xd)
+{
+ /* If the XIVE supports the new "store EOI facility, use it */
+ if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
+ __x_writeq(0, __x_eoi_page(xd));
+ else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
+ opal_int_eoi(hw_irq);
+ } else {
+ uint64_t eoi_val;
+
+ /* Otherwise for EOI, we use the special MMIO that does
+ * a clear of both P and Q and returns the old Q.
+ *
+ * This allows us to then do a re-trigger if Q was set
+ * rather than synthetizing an interrupt in software
+ */
+ eoi_val = GLUE(X_PFX,esb_load)(xd, XIVE_ESB_SET_PQ_00);
+ if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
+ return;
+
+ /* Re-trigger */
+ if (__x_trig_page(xd))
+ __x_writeq(0, __x_trig_page(xd));
+ }
+
+}
+
+enum {
+ scan_fetch,
+ scan_poll,
+ scan_eoi,
+};
+
+static u32 GLUE(X_PFX,scan_interrupts)(struct kvmppc_xive_vcpu *xc,
+ u8 pending, int scan_type)
+{
+ u32 hirq = 0;
+ u8 prio = 0xff;
+
+ /* Find highest pending priority */
+ while ((xc->mfrr != 0xff || pending != 0) && hirq == 0) {
+ struct xive_q *q;
+ u32 idx, toggle;
+ __be32 *qpage;
+
+ /*
+ * If pending is 0 this will return 0xff which is what
+ * we want
+ */
+ prio = ffs(pending) - 1;
+
+ /*
+ * If the most favoured prio we found pending is less
+ * favored (or equal) than a pending IPI, we return
+ * the IPI instead.
+ *
+ * Note: If pending was 0 and mfrr is 0xff, we will
+ * not spurriously take an IPI because mfrr cannot
+ * then be smaller than cppr.
+ */
+ if (prio >= xc->mfrr && xc->mfrr < xc->cppr) {
+ prio = xc->mfrr;
+ hirq = XICS_IPI;
+ break;
+ }
+
+ /* Don't scan past the guest cppr */
+ if (prio >= xc->cppr || prio > 7)
+ break;
+
+ /* Grab queue and pointers */
+ q = &xc->queues[prio];
+ idx = q->idx;
+ toggle = q->toggle;
+
+ /*
+ * Snapshot the queue page. The test further down for EOI
+ * must use the same "copy" that was used by __xive_read_eq
+ * since qpage can be set concurrently and we don't want
+ * to miss an EOI.
+ */
+ qpage = READ_ONCE(q->qpage);
+
+ skip_ipi:
+ /* Try to fetch from the queue. Will return 0 for a
+ * non-queueing priority (ie, qpage = 0)
+ */
+ hirq = __xive_read_eq(qpage, q->msk, &idx, &toggle);
+
+ /*
+ * If this was a signal for an MFFR change done by
+ * H_IPI we skip it. Additionally, if we were fetching
+ * we EOI it now, thus re-enabling reception of a new
+ * such signal.
+ *
+ * We also need to do that if prio is 0 and we had no
+ * page for the queue. In this case, we have non-queued
+ * IPI that needs to be EOId.
+ *
+ * This is safe because if we have another pending MFRR
+ * change that wasn't observed above, the Q bit will have
+ * been set and another occurrence of the IPI will trigger.
+ */
+ if (hirq == XICS_IPI || (prio == 0 && !qpage)) {
+ if (scan_type == scan_fetch)
+ GLUE(X_PFX,source_eoi)(xc->vp_ipi,
+ &xc->vp_ipi_data);
+ /* Loop back on same queue with updated idx/toggle */
+#ifdef XIVE_RUNTIME_CHECKS
+ WARN_ON(hirq && hirq != XICS_IPI);
+#endif
+ if (hirq)
+ goto skip_ipi;
+ }
+
+ /* If fetching, update queue pointers */
+ if (scan_type == scan_fetch) {
+ q->idx = idx;
+ q->toggle = toggle;
+ }
+
+ /* Something found, stop searching */
+ if (hirq)
+ break;
+
+ /* Clear the pending bit on the now empty queue */
+ pending &= ~(1 << prio);
+
+ /*
+ * Check if the queue count needs adjusting due to
+ * interrupts being moved away.
+ */
+ if (atomic_read(&q->pending_count)) {
+ int p = atomic_xchg(&q->pending_count, 0);
+ if (p) {
+#ifdef XIVE_RUNTIME_CHECKS
+ WARN_ON(p > atomic_read(&q->count));
+#endif
+ atomic_sub(p, &q->count);
+ }
+ }
+ }
+
+ /* If we are just taking a "peek", do nothing else */
+ if (scan_type == scan_poll)
+ return hirq;
+
+ /* Update the pending bits */
+ xc->pending = pending;
+
+ /* If this is an EOI that's it, no CPPR adjustment done here,
+ * all we needed was cleanup the stale pending bits and check
+ * if there's anything left.
+ */
+ if (scan_type == scan_eoi)
+ return hirq;
+
+ /* If we found an interrupt, adjust what the guest CPPR should
+ * be as if we had just fetched that interrupt from HW
+ */
+ if (hirq)
+ xc->cppr = prio;
+ /*
+ * If it was an IPI the HW CPPR might have been lowered too much
+ * as the HW interrupt we use for IPIs is routed to priority 0.
+ *
+ * We re-sync it here.
+ */
+ if (xc->cppr != xc->hw_cppr) {
+ xc->hw_cppr = xc->cppr;
+ __x_writeb(xc->cppr, __x_tm_area + TM_QW1_OS + TM_CPPR);
+ }
+
+ return hirq;
+}
+
+X_STATIC unsigned long GLUE(X_PFX,h_xirr)(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u8 old_cppr;
+ u32 hirq;
+
+ DBG("H_XIRR\n");
+
+ xc->GLUE(X_STAT_PFX,h_xirr)++;
+
+ /* First collect pending bits from HW */
+ GLUE(X_PFX,ack_pending)(xc);
+
+ /* Cleanup the old-style bits if needed (they may have been
+ * set by pull or an escalation interrupts)
+ */
+ if (test_bit(BOOK3S_IRQPRIO_EXTERNAL, &vcpu->arch.pending_exceptions))
+ clear_bit(BOOK3S_IRQPRIO_EXTERNAL_LEVEL,
+ &vcpu->arch.pending_exceptions);
+
+ DBG(" new pending=0x%02x hw_cppr=%d cppr=%d\n",
+ xc->pending, xc->hw_cppr, xc->cppr);
+
+ /* Grab previous CPPR and reverse map it */
+ old_cppr = xive_prio_to_guest(xc->cppr);
+
+ /* Scan for actual interrupts */
+ hirq = GLUE(X_PFX,scan_interrupts)(xc, xc->pending, scan_fetch);
+
+ DBG(" got hirq=0x%x hw_cppr=%d cppr=%d\n",
+ hirq, xc->hw_cppr, xc->cppr);
+
+#ifdef XIVE_RUNTIME_CHECKS
+ /* That should never hit */
+ if (hirq & 0xff000000)
+ pr_warn("XIVE: Weird guest interrupt number 0x%08x\n", hirq);
+#endif
+
+ /*
+ * XXX We could check if the interrupt is masked here and
+ * filter it. If we chose to do so, we would need to do:
+ *
+ * if (masked) {
+ * lock();
+ * if (masked) {
+ * old_Q = true;
+ * hirq = 0;
+ * }
+ * unlock();
+ * }
+ */
+
+ /* Return interrupt and old CPPR in GPR4 */
+ vcpu->arch.gpr[4] = hirq | (old_cppr << 24);
+
+ return H_SUCCESS;
+}
+
+X_STATIC unsigned long GLUE(X_PFX,h_ipoll)(struct kvm_vcpu *vcpu, unsigned long server)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u8 pending = xc->pending;
+ u32 hirq;
+ u8 pipr;
+
+ DBG("H_IPOLL(server=%ld)\n", server);
+
+ xc->GLUE(X_STAT_PFX,h_ipoll)++;
+
+ /* Grab the target VCPU if not the current one */
+ if (xc->server_num != server) {
+ vcpu = kvmppc_xive_find_server(vcpu->kvm, server);
+ if (!vcpu)
+ return H_PARAMETER;
+ xc = vcpu->arch.xive_vcpu;
+
+ /* Scan all priorities */
+ pending = 0xff;
+ } else {
+ /* Grab pending interrupt if any */
+ pipr = __x_readb(__x_tm_area + TM_QW1_OS + TM_PIPR);
+ if (pipr < 8)
+ pending |= 1 << pipr;
+ }
+
+ hirq = GLUE(X_PFX,scan_interrupts)(xc, pending, scan_poll);
+
+ /* Return interrupt and old CPPR in GPR4 */
+ vcpu->arch.gpr[4] = hirq | (xc->cppr << 24);
+
+ return H_SUCCESS;
+}
+
+static void GLUE(X_PFX,push_pending_to_hw)(struct kvmppc_xive_vcpu *xc)
+{
+ u8 pending, prio;
+
+ pending = xc->pending;
+ if (xc->mfrr != 0xff) {
+ if (xc->mfrr < 8)
+ pending |= 1 << xc->mfrr;
+ else
+ pending |= 0x80;
+ }
+ if (!pending)
+ return;
+ prio = ffs(pending) - 1;
+
+ __x_writeb(prio, __x_tm_area + TM_SPC_SET_OS_PENDING);
+}
+
+X_STATIC int GLUE(X_PFX,h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u8 old_cppr;
+
+ DBG("H_CPPR(cppr=%ld)\n", cppr);
+
+ xc->GLUE(X_STAT_PFX,h_cppr)++;
+
+ /* Map CPPR */
+ cppr = xive_prio_from_guest(cppr);
+
+ /* Remember old and update SW state */
+ old_cppr = xc->cppr;
+ xc->cppr = cppr;
+
+ /*
+ * We are masking less, we need to look for pending things
+ * to deliver and set VP pending bits accordingly to trigger
+ * a new interrupt otherwise we might miss MFRR changes for
+ * which we have optimized out sending an IPI signal.
+ */
+ if (cppr > old_cppr)
+ GLUE(X_PFX,push_pending_to_hw)(xc);
+
+ /* Apply new CPPR */
+ xc->hw_cppr = cppr;
+ __x_writeb(cppr, __x_tm_area + TM_QW1_OS + TM_CPPR);
+
+ return H_SUCCESS;
+}
+
+X_STATIC int GLUE(X_PFX,h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr)
+{
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_irq_data *xd;
+ u8 new_cppr = xirr >> 24;
+ u32 irq = xirr & 0x00ffffff, hw_num;
+ u16 src;
+ int rc = 0;
+
+ DBG("H_EOI(xirr=%08lx)\n", xirr);
+
+ xc->GLUE(X_STAT_PFX,h_eoi)++;
+
+ xc->cppr = xive_prio_from_guest(new_cppr);
+
+ /*
+ * IPIs are synthetized from MFRR and thus don't need
+ * any special EOI handling. The underlying interrupt
+ * used to signal MFRR changes is EOId when fetched from
+ * the queue.
+ */
+ if (irq == XICS_IPI || irq == 0)
+ goto bail;
+
+ /* Find interrupt source */
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb) {
+ DBG(" source not found !\n");
+ rc = H_PARAMETER;
+ goto bail;
+ }
+ state = &sb->irq_state[src];
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ state->in_eoi = true;
+ mb();
+
+ again:
+ if (state->guest_priority == MASKED) {
+ arch_spin_lock(&sb->lock);
+ if (state->guest_priority != MASKED) {
+ arch_spin_unlock(&sb->lock);
+ goto again;
+ }
+ DBG(" EOI on saved P...\n");
+
+ /* Clear old_p, that will cause unmask to perform an EOI */
+ state->old_p = false;
+
+ arch_spin_unlock(&sb->lock);
+ } else {
+ DBG(" EOI on source...\n");
+
+ /* Perform EOI on the source */
+ GLUE(X_PFX,source_eoi)(hw_num, xd);
+
+ /* If it's an emulated LSI, check level and resend */
+ if (state->lsi && state->asserted)
+ __x_writeq(0, __x_trig_page(xd));
+
+ }
+
+ mb();
+ state->in_eoi = false;
+ bail:
+
+ /* Re-evaluate pending IRQs and update HW */
+ GLUE(X_PFX,scan_interrupts)(xc, xc->pending, scan_eoi);
+ GLUE(X_PFX,push_pending_to_hw)(xc);
+ DBG(" after scan pending=%02x\n", xc->pending);
+
+ /* Apply new CPPR */
+ xc->hw_cppr = xc->cppr;
+ __x_writeb(xc->cppr, __x_tm_area + TM_QW1_OS + TM_CPPR);
+
+ return rc;
+}
+
+X_STATIC int GLUE(X_PFX,h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
+ unsigned long mfrr)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ DBG("H_IPI(server=%08lx,mfrr=%ld)\n", server, mfrr);
+
+ xc->GLUE(X_STAT_PFX,h_ipi)++;
+
+ /* Find target */
+ vcpu = kvmppc_xive_find_server(vcpu->kvm, server);
+ if (!vcpu)
+ return H_PARAMETER;
+ xc = vcpu->arch.xive_vcpu;
+
+ /* Locklessly write over MFRR */
+ xc->mfrr = mfrr;
+
+ /* Shoot the IPI if most favored than target cppr */
+ if (mfrr < xc->cppr)
+ __x_writeq(0, __x_trig_page(&xc->vp_ipi_data));
+
+ return H_SUCCESS;
+}
diff --git a/arch/powerpc/kvm/irq.h b/arch/powerpc/kvm/irq.h
index 5a9a10b..3f1be85 100644
--- a/arch/powerpc/kvm/irq.h
+++ b/arch/powerpc/kvm/irq.h
@@ -12,6 +12,7 @@ static inline int irqchip_in_kernel(struct kvm *kvm)
#endif
#ifdef CONFIG_KVM_XICS
ret = ret || (kvm->arch.xics != NULL);
+ ret = ret || (kvm->arch.xive != NULL);
#endif
smp_rmb();
return ret;
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 95c91a9..de79bd72 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -37,6 +37,8 @@
#include <asm/cputhreads.h>
#include <asm/irqflags.h>
#include <asm/iommu.h>
+#include <asm/xive.h>
+
#include "timing.h"
#include "irq.h"
#include "../mm/mmu_decl.h"
@@ -699,7 +701,10 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
kvmppc_mpic_disconnect_vcpu(vcpu->arch.mpic, vcpu);
break;
case KVMPPC_IRQ_XICS:
- kvmppc_xics_free_icp(vcpu);
+ if (xive_enabled())
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ else
+ kvmppc_xics_free_icp(vcpu);
break;
}
@@ -1219,8 +1224,12 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
r = -EPERM;
dev = kvm_device_from_filp(f.file);
- if (dev)
- r = kvmppc_xics_connect_vcpu(dev, vcpu, cap->args[1]);
+ if (dev) {
+ if (xive_enabled())
+ r = kvmppc_xive_connect_vcpu(dev, vcpu, cap->args[1]);
+ else
+ r = kvmppc_xics_connect_vcpu(dev, vcpu, cap->args[1]);
+ }
fdput(f);
break;
@@ -1244,7 +1253,7 @@ bool kvm_arch_intc_initialized(struct kvm *kvm)
return true;
#endif
#ifdef CONFIG_KVM_XICS
- if (kvm->arch.xics)
+ if (kvm->arch.xics || kvm->arch.xive)
return true;
#endif
return false;
diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
index e0f856b..d71cd77 100644
--- a/arch/powerpc/platforms/powernv/opal.c
+++ b/arch/powerpc/platforms/powernv/opal.c
@@ -890,3 +890,4 @@ EXPORT_SYMBOL_GPL(opal_leds_set_ind);
EXPORT_SYMBOL_GPL(opal_write_oppanel_async);
/* Export this for KVM */
EXPORT_SYMBOL_GPL(opal_int_set_mfrr);
+EXPORT_SYMBOL_GPL(opal_int_eoi);
diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
index 96037e0..6429cd3 100644
--- a/arch/powerpc/sysdev/xive/common.c
+++ b/arch/powerpc/sysdev/xive/common.c
@@ -45,12 +45,14 @@
#endif
bool __xive_enabled;
+EXPORT_SYMBOL_GPL(__xive_enabled);
bool xive_cmdline_disabled;
/* We use only one priority for now */
static u8 xive_irq_priority;
void __iomem *xive_tm_area;
+EXPORT_SYMBOL_GPL(xive_tm_area);
u32 xive_tm_offset;
static const struct xive_ops *xive_ops;
static struct irq_domain *xive_irq_domain;
@@ -304,7 +306,7 @@ static void xive_irq_eoi(struct irq_data *d)
DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
d->irq, irqd_to_hwirq(d), xc->pending_prio);
- if (!irqd_irq_disabled(d))
+ if (!irqd_irq_disabled(d) && !irqd_is_forwarded_to_vcpu(d))
xive_do_source_eoi(irqd_to_hwirq(d), xd);
/*
@@ -579,9 +581,10 @@ static int xive_irq_set_affinity(struct irq_data *d,
* Only configure the irq if it's not currently passed-through to
* a KVM guest
*/
- rc = xive_ops->configure_irq(hw_irq,
- get_hard_smp_processor_id(target),
- xive_irq_priority, d->irq);
+ if (!irqd_is_forwarded_to_vcpu(d))
+ rc = xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(target),
+ xive_irq_priority, d->irq);
if (rc < 0) {
pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
return rc;
@@ -661,6 +664,123 @@ static int xive_irq_retrigger(struct irq_data *d)
return 1;
}
+static int xive_irq_set_vcpu_affinity(struct irq_data *d, void *state)
+{
+ struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
+ unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
+ int rc;
+ u8 pq;
+
+ /*
+ * We only support this on interrupts that do not require
+ * firmware calls for masking and unmasking
+ */
+ if (xd->flags & XIVE_IRQ_FLAG_MASK_FW)
+ return -EIO;
+
+ /*
+ * This is called by KVM with state non-NULL for enabling
+ * pass-through or NULL for disabling it
+ */
+ if (state) {
+ irqd_set_forwarded_to_vcpu(d);
+
+ /* Set it to PQ=10 state to prevent further sends */
+ pq = xive_poke_esb(xd, 0xe00);
+
+ /* No target ? nothing to do */
+ if (xd->target == XIVE_INVALID_TARGET) {
+ /*
+ * An untargetted interrupt should have been
+ * also masked at the source
+ */
+ WARN_ON(pq & 2);
+
+ return 0;
+ }
+
+ /*
+ * If P was set, adjust state to PQ=11 to indicate
+ * that a resend is needed for the interrupt to reach
+ * the guest. Also remember the value of P.
+ *
+ * This also tells us that it's in flight to a host queue
+ * or has already been fetched but hasn't been EOIed yet
+ * by the host. This it's potentially using up a host
+ * queue slot. This is important to know because as long
+ * as this is the case, we must not hard-unmask it when
+ * "returning" that interrupt to the host.
+ *
+ * This saved_p is cleared by the host EOI, when we know
+ * for sure the queue slot is no longer in use.
+ */
+ if (pq & 2) {
+ pq = xive_poke_esb(xd, 0xf00);
+ xd->saved_p = true;
+
+ /*
+ * Sync the XIVE source HW to ensure the interrupt
+ * has gone through the EAS before we change its
+ * target to the guest. That should guarantee us
+ * that we *will* eventually get an EOI for it on
+ * the host. Otherwise there would be a small window
+ * for P to be seen here but the interrupt going
+ * to the guest queue.
+ */
+ if (xive_ops->sync_source)
+ xive_ops->sync_source(hw_irq);
+ } else
+ xd->saved_p = false;
+ } else {
+ irqd_clr_forwarded_to_vcpu(d);
+
+ /* No host target ? hard mask and return */
+ if (xd->target == XIVE_INVALID_TARGET) {
+ xive_do_source_set_mask(xd, true);
+ return 0;
+ }
+
+ /*
+ * Sync the XIVE source HW to ensure the interrupt
+ * has gone through the EAS before we change its
+ * target to the host.
+ */
+ if (xive_ops->sync_source)
+ xive_ops->sync_source(hw_irq);
+
+ /*
+ * By convention we are called with the interrupt in
+ * a PQ=10 or PQ=11 state, ie, it won't fire and will
+ * have latched in Q whether there's a pending HW
+ * interrupt or not.
+ *
+ * First reconfigure the target.
+ */
+ rc = xive_ops->configure_irq(hw_irq,
+ get_hard_smp_processor_id(xd->target),
+ xive_irq_priority, d->irq);
+ if (rc)
+ return rc;
+
+ /*
+ * Then if saved_p is not set, effectively re-enable the
+ * interrupt with an EOI. If it is set, we know there is
+ * still a message in a host queue somewhere that will be
+ * EOId eventually.
+ *
+ * Note: We don't check irqd_irq_disabled(). Effectively,
+ * we *will* let the irq get through even if masked if the
+ * HW is still firing it in order to deal with the whole
+ * saved_p business properly. If the interrupt triggers
+ * while masked, the generic code will re-mask it anyway.
+ */
+ if (!xd->saved_p)
+ xive_do_source_eoi(hw_irq, xd);
+
+ }
+ return 0;
+}
+
static struct irq_chip xive_irq_chip = {
.name = "XIVE-IRQ",
.irq_startup = xive_irq_startup,
@@ -671,12 +791,14 @@ static struct irq_chip xive_irq_chip = {
.irq_set_affinity = xive_irq_set_affinity,
.irq_set_type = xive_irq_set_type,
.irq_retrigger = xive_irq_retrigger,
+ .irq_set_vcpu_affinity = xive_irq_set_vcpu_affinity,
};
bool is_xive_irq(struct irq_chip *chip)
{
return chip == &xive_irq_chip;
}
+EXPORT_SYMBOL_GPL(is_xive_irq);
void xive_cleanup_irq_data(struct xive_irq_data *xd)
{
@@ -691,6 +813,7 @@ void xive_cleanup_irq_data(struct xive_irq_data *xd)
xd->trig_mmio = NULL;
}
}
+EXPORT_SYMBOL_GPL(xive_cleanup_irq_data);
static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
{
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 26cc6bf..0130af8 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -27,6 +27,7 @@
#include <asm/errno.h>
#include <asm/xive.h>
#include <asm/opal.h>
+#include <asm/kvm_ppc.h>
#include "xive-regs.h"
#include "xive-internal.h"
@@ -98,6 +99,7 @@ int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
}
return 0;
}
+EXPORT_SYMBOL_GPL(xive_native_populate_irq_data);
int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
{
@@ -111,6 +113,8 @@ int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
}
return rc == 0 ? 0 : -ENXIO;
}
+EXPORT_SYMBOL_GPL(xive_native_configure_irq);
+
/* This can be called multiple time to change a queue configuration */
int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
@@ -187,6 +191,7 @@ int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
fail:
return rc;
}
+EXPORT_SYMBOL_GPL(xive_native_configure_queue);
static void __xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
{
@@ -211,6 +216,7 @@ void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
iounmap(q->eoi_mmio);
q->eoi_mmio = NULL;
}
+EXPORT_SYMBOL_GPL(xive_native_disable_queue);
static int xive_native_setup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
{
@@ -297,6 +303,7 @@ u32 xive_native_alloc_irq(void)
return 0;
return rc;
}
+EXPORT_SYMBOL_GPL(xive_native_alloc_irq);
void xive_native_free_irq(u32 irq)
{
@@ -307,6 +314,7 @@ void xive_native_free_irq(u32 irq)
msleep(1);
}
}
+EXPORT_SYMBOL_GPL(xive_native_free_irq);
static void xive_native_put_ipi(unsigned int cpu, struct xive_cpu *xc)
{
@@ -406,10 +414,11 @@ static void xive_native_teardown_cpu(unsigned int cpu, struct xive_cpu *xc)
}
}
-static void xive_native_sync_source(u32 hw_irq)
+void xive_native_sync_source(u32 hw_irq)
{
opal_xive_sync(XIVE_SYNC_EAS, hw_irq);
}
+EXPORT_SYMBOL_GPL(xive_native_sync_source);
static const struct xive_ops xive_native_ops = {
.populate_irq_data = xive_native_populate_irq_data,
@@ -468,10 +477,38 @@ static bool xive_parse_provisioning(struct device_node *np)
return true;
}
+static void xive_native_setup_pools(void)
+{
+ u32 max_pir = 0;
+ unsigned int cpu;
+
+ /*
+ * The HW won't let us enable OS VPs for KVM is we don't
+ * have enabled pool VPs so let's do that. First we find
+ * out our highest HW CPU ID
+ */
+ for_each_possible_cpu(cpu) {
+ u32 hw_id = get_hard_smp_processor_id(cpu);
+ if (hw_id > max_pir)
+ max_pir = hw_id;
+ }
+
+ /* Allocate a pool big enough */
+ pr_debug("XIVE: Allocating VP block for pool size %d\n",
+ max_pir + 1);
+ xive_pool_vps = xive_native_alloc_vp_block(max_pir + 1);
+ if (WARN_ON(xive_pool_vps == XIVE_INVALID_VP))
+ pr_err("XIVE: No pool VPsvp KVM might not function\n");
+
+ pr_debug("XIVE: Pool VPs allocated at 0x%x for max_pir 0x%x\n",
+ xive_pool_vps, max_pir);
+}
+
u32 xive_native_default_eq_shift(void)
{
return xive_queue_shift;
}
+EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
bool xive_native_init(void)
{
@@ -481,7 +518,7 @@ bool xive_native_init(void)
struct property *prop;
u8 max_prio = 7;
const __be32 *p;
- u32 val;
+ u32 val, cpu;
s64 rc;
if (xive_cmdline_disabled)
@@ -517,6 +554,10 @@ bool xive_native_init(void)
break;
}
+ /* Configure TM areas for KVM */
+ for_each_possible_cpu(cpu)
+ kvmppc_set_xive_tm_area(cpu, r.start, tm_area);
+
/* Grab size of provisionning pages */
xive_parse_provisioning(np);
@@ -528,6 +569,9 @@ bool xive_native_init(void)
return false;
}
+ /* Setup some dummy HV pool VPs */
+ xive_native_setup_pools();
+
/* Initialize XIVE core with our backend */
if (!xive_core_init(&xive_native_ops, tm_area, TM_QW3_HV_PHYS,
max_prio)) {
@@ -602,3 +646,47 @@ void xive_native_free_vp_block(u32 vp_base)
pr_warn("XIVE: OPAL error %lld freeing VP block\n", rc);
}
EXPORT_SYMBOL_GPL(xive_native_free_vp_block);
+
+int xive_native_enable_vp(u32 vp_id)
+{
+ s64 rc;
+
+ for (;;) {
+ rc = opal_xive_set_vp_info(vp_id, OPAL_XIVE_VP_ENABLED, 0);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ return rc ? -EIO : 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_enable_vp);
+
+int xive_native_disable_vp(u32 vp_id)
+{
+ s64 rc;
+
+ for (;;) {
+ rc = opal_xive_set_vp_info(vp_id, 0, 0);
+ if (rc != OPAL_BUSY)
+ break;
+ msleep(1);
+ }
+ return rc ? -EIO : 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_disable_vp);
+
+int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id)
+{
+ __be64 vp_cam_be;
+ __be32 vp_chip_id_be;
+ s64 rc;
+
+ rc = opal_xive_get_vp_info(vp_id, NULL, &vp_cam_be, NULL, &vp_chip_id_be);
+ if (rc)
+ return -EIO;
+ *out_cam_id = be64_to_cpu(vp_cam_be) & 0xffffffffu;
+ *out_chip_id = be32_to_cpu(vp_chip_id_be);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_vp_info);
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 2c14ad9..d1a6e55 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1165,7 +1165,6 @@ int kvm_register_device_ops(struct kvm_device_ops *ops, u32 type);
void kvm_unregister_device_ops(u32 type);
extern struct kvm_device_ops kvm_mpic_ops;
-extern struct kvm_device_ops kvm_xics_ops;
extern struct kvm_device_ops kvm_arm_vgic_v2_ops;
extern struct kvm_device_ops kvm_arm_vgic_v3_ops;
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index a17d787..1b0da57 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2839,10 +2839,6 @@ static struct kvm_device_ops *kvm_device_ops_table[KVM_DEV_TYPE_MAX] = {
[KVM_DEV_TYPE_FSL_MPIC_20] = &kvm_mpic_ops,
[KVM_DEV_TYPE_FSL_MPIC_42] = &kvm_mpic_ops,
#endif
-
-#ifdef CONFIG_KVM_XICS
- [KVM_DEV_TYPE_XICS] = &kvm_xics_ops,
-#endif
};
int kvm_register_device_ops(struct kvm_device_ops *ops, u32 type)
--
2.9.3
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-24 5:22 ` Paul Mackerras
-1 siblings, 0 replies; 38+ messages in thread
From: Paul Mackerras @ 2017-03-24 5:22 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev, kvm-ppc
On Mon, Mar 20, 2017 at 05:49:08PM +1100, Benjamin Herrenschmidt wrote:
> The XIVE interrupt controller is the new interrupt controller
> found in POWER9. It supports advanced virtualization capabilities
> among other things.
>
> Currently we use a set of firmware calls that simulate the old
> "XICS" interrupt controller but this is fairly inefficient.
>
> This adds the framework for using XIVE along with a native
> backend which OPAL for configuration. Later, a backend allowing
> the use in a KVM or PowerVM guest will also be provided.
>
> This disables some fast path for interrupts in KVM when XIVE is
> enabled as these rely on the firmware emulation code which is no
> longer available when the XIVE is used natively by Linux.
>
> A latter patch will make KVM also directly exploit the XIVE, thus
> recovering the lost performance (and more).
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Mostly looks fine, a few minor comments below...
> ---
> arch/powerpc/include/asm/xive.h | 116 +++
> arch/powerpc/include/asm/xmon.h | 2 +
> arch/powerpc/platforms/powernv/Kconfig | 2 +
> arch/powerpc/platforms/powernv/setup.c | 15 +-
> arch/powerpc/platforms/powernv/smp.c | 39 +-
> arch/powerpc/sysdev/Kconfig | 1 +
> arch/powerpc/sysdev/Makefile | 1 +
> arch/powerpc/sysdev/xive/Kconfig | 7 +
> arch/powerpc/sysdev/xive/Makefile | 4 +
> arch/powerpc/sysdev/xive/common.c | 1175 ++++++++++++++++++++++++++++++
> arch/powerpc/sysdev/xive/native.c | 604 +++++++++++++++
> arch/powerpc/sysdev/xive/xive-internal.h | 51 ++
> arch/powerpc/sysdev/xive/xive-regs.h | 88 +++
> arch/powerpc/xmon/xmon.c | 93 ++-
> 14 files changed, 2186 insertions(+), 12 deletions(-)
> create mode 100644 arch/powerpc/include/asm/xive.h
> create mode 100644 arch/powerpc/sysdev/xive/Kconfig
> create mode 100644 arch/powerpc/sysdev/xive/Makefile
> create mode 100644 arch/powerpc/sysdev/xive/common.c
> create mode 100644 arch/powerpc/sysdev/xive/native.c
> create mode 100644 arch/powerpc/sysdev/xive/xive-internal.h
> create mode 100644 arch/powerpc/sysdev/xive/xive-regs.h
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> new file mode 100644
> index 0000000..b1604b73
> --- /dev/null
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -0,0 +1,116 @@
> +#ifndef _ASM_POWERPC_XIVE_H
> +#define _ASM_POWERPC_XIVE_H
> +
> +#define XIVE_INVALID_VP 0xffffffff
> +
> +#ifdef CONFIG_PPC_XIVE
> +
> +extern void __iomem *xive_tm_area;
> +extern u32 xive_tm_offset;
> +
> +/*
> + * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
> + * have it stored in the xive_cpu structure. We also cache
> + * for normal interrupts the current target CPU.
> + */
> +struct xive_irq_data {
> + /* Setup by backend */
> + u64 flags;
> +#define XIVE_IRQ_FLAG_STORE_EOI 0x01
> +#define XIVE_IRQ_FLAG_LSI 0x02
> +#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
> +#define XIVE_IRQ_FLAG_MASK_FW 0x08
> +#define XIVE_IRQ_FLAG_EOI_FW 0x10
> + u64 eoi_page;
> + void __iomem *eoi_mmio;
> + u64 trig_page;
> + void __iomem *trig_mmio;
> + u32 esb_shift;
> + int src_chip;
> +
> + /* Setup/used by frontend */
> + int target;
> + bool saved_p;
> +};
> +#define XIVE_INVALID_CHIP_ID -1
> +
> +/* A queue tracking structure in a CPU */
> +struct xive_q {
> + __be32 *qpage;
> + u32 msk;
> + u32 idx;
> + u32 toggle;
> + u64 eoi_phys;
> + void __iomem *eoi_mmio;
> + u32 esc_irq;
> + atomic_t count;
> + atomic_t pending_count;
> +};
> +
> +/*
> + * "magic" ESB MMIO offsets
> + */
> +#define XIVE_ESB_GET 0x800
> +#define XIVE_ESB_SET_PQ_00 0xc00
> +#define XIVE_ESB_SET_PQ_01 0xd00
> +#define XIVE_ESB_SET_PQ_10 0xe00
> +#define XIVE_ESB_SET_PQ_11 0xf00
> +#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> +
> +extern bool __xive_enabled;
> +
> +static inline bool xive_enabled(void) { return __xive_enabled; }
> +
> +extern bool xive_native_init(void);
> +extern void xive_smp_probe(void);
> +extern int xive_smp_prepare_cpu(unsigned int cpu);
> +extern void xive_smp_setup_cpu(void);
> +extern void xive_smp_disable_cpu(void);
> +extern void xive_kexec_teardown_cpu(int secondary);
> +extern void xive_shutdown(void);
> +extern void xive_flush_interrupt(void);
> +
> +/* xmon hook */
> +extern void xmon_xive_do_dump(int cpu);
> +
> +/* APIs used by KVM */
> +extern u32 xive_native_default_eq_shift(void);
> +extern u32 xive_native_alloc_vp_block(u32 max_vcpus);
> +extern void xive_native_free_vp_block(u32 vp_base);
> +extern int xive_native_populate_irq_data(u32 hw_irq,
> + struct xive_irq_data *data);
> +extern void xive_cleanup_irq_data(struct xive_irq_data *xd);
> +extern u32 xive_native_alloc_irq(void);
> +extern void xive_native_free_irq(u32 irq);
> +extern int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
> +
> +extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate);
> +extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
> +
> +extern bool __xive_irq_trigger(struct xive_irq_data *xd);
> +extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
> +extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
> +
> +extern bool is_xive_irq(struct irq_chip *chip);
> +
> +#else
> +
> +static inline bool xive_enabled(void) { return false; }
> +
> +static inline bool xive_native_init(void) { return false; }
> +static inline void xive_smp_probe(void) { }
> +extern inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
> +static inline void xive_smp_setup_cpu(void) { }
> +static inline void xive_smp_disable_cpu(void) { }
> +static inline void xive_kexec_teardown_cpu(int secondary) { }
> +static inline void xive_shutdown(void) { }
> +static inline void xive_flush_interrupt(void) { }
> +
> +static inline u32 xive_native_alloc_vp_block(u32 max_vcpus)
> + { return XIVE_INVALID_VP; }
> +static inline void xive_native_free_vp_block(u32 vp_base) { }
> +
> +#endif
> +
> +#endif /* _ASM_POWERPC_XIVE_H */
> diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h
> index 5eb8e59..eb42a0c 100644
> --- a/arch/powerpc/include/asm/xmon.h
> +++ b/arch/powerpc/include/asm/xmon.h
> @@ -29,5 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { };
> extern int cpus_are_in_xmon(void);
> #endif
>
> +extern void xmon_printf(const char *format, ...);
> +
> #endif /* __KERNEL __ */
> #endif /* __ASM_POWERPC_XMON_H */
> diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
> index 3a07e4d..81ee2ed 100644
> --- a/arch/powerpc/platforms/powernv/Kconfig
> +++ b/arch/powerpc/platforms/powernv/Kconfig
> @@ -4,6 +4,8 @@ config PPC_POWERNV
> select PPC_NATIVE
> select PPC_XICS
> select PPC_ICP_NATIVE
> + select PPC_XIVE
> + select PPC_XIVE_NATIVE
> select PPC_P7_NAP
> select PCI
> select PCI_MSI
> diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
> index d50c7d9..adceac9 100644
> --- a/arch/powerpc/platforms/powernv/setup.c
> +++ b/arch/powerpc/platforms/powernv/setup.c
> @@ -32,6 +32,7 @@
> #include <asm/machdep.h>
> #include <asm/firmware.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/kexec.h>
> #include <asm/smp.h>
> @@ -76,7 +77,9 @@ static void __init pnv_init(void)
>
> static void __init pnv_init_IRQ(void)
> {
> - xics_init();
> + /* Try using a XIVE if available, otherwise use a XICS */
> + if (!xive_native_init())
> + xics_init();
>
> WARN_ON(!ppc_md.get_irq);
> }
> @@ -218,10 +221,12 @@ static void pnv_kexec_wait_secondaries_down(void)
>
> static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> {
> - xics_kexec_teardown_cpu(secondary);
> + if (xive_enabled())
> + xive_kexec_teardown_cpu(secondary);
> + else
> + xics_kexec_teardown_cpu(secondary);
>
> /* On OPAL, we return all CPUs to firmware */
> -
> if (!firmware_has_feature(FW_FEATURE_OPAL))
> return;
>
> @@ -237,6 +242,10 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> /* Primary waits for the secondaries to have reached OPAL */
> pnv_kexec_wait_secondaries_down();
>
> + /* Switch XIVE back to emulation mode */
> + if (xive_enabled())
> + xive_shutdown();
> +
> /*
> * We might be running as little-endian - now that interrupts
> * are disabled, reset the HILE bit to big-endian so we don't
> diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
> index 8b67e1e..f571955 100644
> --- a/arch/powerpc/platforms/powernv/smp.c
> +++ b/arch/powerpc/platforms/powernv/smp.c
> @@ -29,6 +29,7 @@
> #include <asm/vdso_datapage.h>
> #include <asm/cputhreads.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/runlatch.h>
> #include <asm/code-patching.h>
> @@ -47,7 +48,9 @@
>
> static void pnv_smp_setup_cpu(int cpu)
> {
> - if (cpu != boot_cpuid)
> + if (xive_enabled())
> + xive_smp_setup_cpu();
> + else if (cpu != boot_cpuid)
> xics_setup_cpu();
>
> #ifdef CONFIG_PPC_DOORBELL
> @@ -132,7 +135,10 @@ static int pnv_smp_cpu_disable(void)
> vdso_data->processorCount--;
> if (cpu == boot_cpuid)
> boot_cpuid = cpumask_any(cpu_online_mask);
> - xics_migrate_irqs_away();
> + if (xive_enabled())
> + xive_smp_disable_cpu();
> + else
> + xics_migrate_irqs_away();
> return 0;
> }
>
> @@ -213,9 +219,12 @@ static void pnv_smp_cpu_kill_self(void)
> if (((srr1 & wmask) == SRR1_WAKEEE) ||
> ((srr1 & wmask) == SRR1_WAKEHVI) ||
> (local_paca->irq_happened & PACA_IRQ_EE)) {
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - icp_opal_flush_interrupt();
> - else
> + if (cpu_has_feature(CPU_FTR_ARCH_300)) {
> + if (xive_enabled())
> + xive_flush_interrupt();
> + else
> + icp_opal_flush_interrupt();
> + } else
> icp_native_flush_interrupt();
> } else if ((srr1 & wmask) == SRR1_WAKEHDBELL) {
> unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
> @@ -252,10 +261,26 @@ static int pnv_cpu_bootable(unsigned int nr)
> return smp_generic_cpu_bootable(nr);
> }
>
> +static int pnv_smp_prepare_cpu(int cpu)
> +{
> + if (xive_enabled())
> + return xive_smp_prepare_cpu(cpu);
> + return 0;
> +}
> +
> +static void __init pnv_smp_probe(void)
> +{
> + if (xive_enabled())
> + xive_smp_probe();
> + else
> + xics_smp_probe();
> +}
> +
> static struct smp_ops_t pnv_smp_ops = {
> .message_pass = smp_muxed_ipi_message_pass,
> - .cause_ipi = NULL, /* Filled at runtime by xics_smp_probe() */
> - .probe = xics_smp_probe,
> + .cause_ipi = NULL, /* Filled at runtime by xi{cs,ve}_smp_probe() */
> + .probe = pnv_smp_probe,
> + .prepare_cpu = pnv_smp_prepare_cpu,
> .kick_cpu = pnv_smp_kick_cpu,
> .setup_cpu = pnv_smp_setup_cpu,
> .cpu_bootable = pnv_cpu_bootable,
> diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig
> index 52dc165..caf882e 100644
> --- a/arch/powerpc/sysdev/Kconfig
> +++ b/arch/powerpc/sysdev/Kconfig
> @@ -28,6 +28,7 @@ config PPC_MSI_BITMAP
> default y if PPC_POWERNV
>
> source "arch/powerpc/sysdev/xics/Kconfig"
> +source "arch/powerpc/sysdev/xive/Kconfig"
>
> config PPC_SCOM
> bool
> diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile
> index a254824..c0ae11d 100644
> --- a/arch/powerpc/sysdev/Makefile
> +++ b/arch/powerpc/sysdev/Makefile
> @@ -71,5 +71,6 @@ obj-$(CONFIG_PPC_EARLY_DEBUG_MEMCONS) += udbg_memcons.o
> subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
>
> obj-$(CONFIG_PPC_XICS) += xics/
> +obj-$(CONFIG_PPC_XIVE) += xive/
>
> obj-$(CONFIG_GE_FPGA) += ge/
> diff --git a/arch/powerpc/sysdev/xive/Kconfig b/arch/powerpc/sysdev/xive/Kconfig
> new file mode 100644
> index 0000000..c8816c8
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Kconfig
> @@ -0,0 +1,7 @@
> +config PPC_XIVE
> + def_bool n
> + select PPC_SMP_MUXED_IPI
> + select HARDIRQS_SW_RESEND
> +
> +config PPC_XIVE_NATIVE
> + def_bool n
> diff --git a/arch/powerpc/sysdev/xive/Makefile b/arch/powerpc/sysdev/xive/Makefile
> new file mode 100644
> index 0000000..3fab303
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Makefile
> @@ -0,0 +1,4 @@
> +subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
> +
> +obj-y += common.o
> +obj-$(CONFIG_PPC_XIVE_NATIVE) += native.o
> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
> new file mode 100644
> index 0000000..96037e0
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/common.c
> @@ -0,0 +1,1175 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +#include <linux/types.h>
> +#include <linux/threads.h>
> +#include <linux/kernel.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
> +#include <linux/init.h>
> +#include <linux/cpu.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/msi.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/machdep.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/xmon.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#undef DEBUG_FLUSH
> +#undef DEBUG_ALL
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +#ifdef DEBUG_ALL
> +#define DBG_VERBOSE(fmt...) pr_devel("XIVE: " fmt)
> +#else
> +#define DBG_VERBOSE(fmt...) do { } while(0)
> +#endif
> +
> +bool __xive_enabled;
> +bool xive_cmdline_disabled;
> +
> +/* We use only one priority for now */
> +static u8 xive_irq_priority;
> +
> +void __iomem *xive_tm_area;
> +u32 xive_tm_offset;
> +static const struct xive_ops *xive_ops;
> +static struct irq_domain *xive_irq_domain;
> +
> +/* The IPIs all use the same logical irq number */
> +static u32 xive_ipi_irq;
> +
> +/* Xive state for each CPU */
> +static DEFINE_PER_CPU(struct xive_cpu *, xive_cpu);
> +
> +/*
> + * A "disabled" interrupt should never fire, to catch problems
> + * we set its logical number to this
> + */
> +#define XIVE_BAD_IRQ 0x7fffffff
> +#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
> +
> +/* An invalid CPU target */
> +#define XIVE_INVALID_TARGET (-1)
> +
> +static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
A brief comment here explaining what the function does would be
helpful. Likewise for xive_scan_interrupts(), etc. etc.
The 'prio' argument doesn't seem to be used.
> +{
> + u32 cur;
> +
> + if (!q->qpage)
> + return 0;
> + cur = be32_to_cpup(q->qpage + q->idx);
> + if ((cur >> 31) == q->toggle)
> + return 0;
> + if (!just_peek) {
> + q->idx = (q->idx + 1) & q->msk;
> + if (q->idx == 0)
> + q->toggle ^= 1;
> + }
> + return cur & 0x7fffffff;
> +}
> +
> +static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
> +{
> + u32 hirq = 0;
> + u8 prio;
> +
> + /* Find highest pending priority */
> + while (xc->pending_prio != 0) {
> + struct xive_q *q;
> +
> + prio = ffs(xc->pending_prio) - 1;
> + DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
> +
> + /* Try to fetch */
> + hirq = xive_read_eq(&xc->queue[prio], prio, just_peek);
> +
> + /* Found something ? That's it */
> + if (hirq)
> + break;
> +
> + /* Clear pending bits */
> + xc->pending_prio &= ~(1 << prio);
> +
> + /*
> + * Check if the queue count needs adjusting due to
> + * interrupts being moved away.
> + */
> + q = &xc->queue[prio];
> + if (atomic_read(&q->pending_count)) {
> + int p = atomic_xchg(&q->pending_count, 0);
> + if (p) {
> + WARN_ON(p > atomic_read(&q->count));
> + atomic_sub(p, &q->count);
> + }
> + }
> + }
> +
> + /* If nothing was found, set CPPR to 0xff */
> + if (hirq == 0)
> + prio = 0xff;
> +
> + /* Update HW CPPR to match if necessary */
> + if (prio != xc->cppr) {
> + DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n", prio);
> + xc->cppr = prio;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, prio);
> + }
> +
> + return hirq;
> +}
> +
> +#ifdef CONFIG_XMON
> +static void xive_dump_eq(const char *name, struct xive_q *q)
> +{
> + u32 i0, i1, idx;
> +
> + if (!q->qpage)
> + return;
> + idx = q->idx;
> + i0 = be32_to_cpup(q->qpage + idx);
> + idx = (idx + 1) & q->msk;
> + i1 = be32_to_cpup(q->qpage + idx);
> + xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
> + q->toggle, i0, i1);
> +}
> +
> +void xmon_xive_do_dump(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_irq_data *xd;
> + uint64_t val, offset;
> +
> + xmon_printf("XIVE state for CPU %d:\n", cpu);
> + xmon_printf(" pp=%02x cppr=%02x\n", xc->pending_prio, xc->cppr);
> + xive_dump_eq("IRQ", &xc->queue[xive_irq_priority]);
> + xd = &xc->ipi_data;
> + offset = 0x800;
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> + val = in_be64(xd->eoi_mmio + offset);
> + xmon_printf(" IPI state: %x:%c%c\n", xc->hw_ipi,
> + val & 2 ? 'P' : 'p',
> + val & 1 ? 'Q' : 'q');
> +}
> +#endif /* CONFIG_XMON */
> +
> +static void xive_update_pending_irqs(struct xive_cpu *xc)
> +{
> + u8 he, cppr;
> + u16 ack;
> +
> + /* Perform the acknowledge hypervisor to register cycle */
> + ack = be16_to_cpu(__raw_readw(xive_tm_area + TM_SPC_ACK_HV_REG));
This sounds like it's something that only the hypervisor should do, so
it's not clear why it's in common code.
> + /* Synchronize subsequent queue accesses */
> + mb();
> +
> + DBG_VERBOSE("CPU %d get_irq, ack=%04x\n", smp_processor_id(), ack);
> +
> + /* Check the HE field */
What is a "HE" field?
> + cppr = ack & 0xff;
> + he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8));
> + switch(he) {
> + case TM_QW3_NSR_HE_NONE:
> + break;
> + case TM_QW3_NSR_HE_PHYS:
> + if (cppr == 0xff)
> + return;
> + xc->pending_prio |= 1 << cppr;
> + if (cppr >= xc->cppr)
> + pr_err("XIVE: CPU %d odd ack CPPR, got %d at %d\n",
> + smp_processor_id(), cppr, xc->cppr);
> + xc->cppr = cppr;
> + break;
> + case TM_QW3_NSR_HE_POOL:
> + case TM_QW3_NSR_HE_LSI:
> + pr_err("XIVE: CPU %d got unexpected interrupt type HE=%d\n",
> + smp_processor_id(), he);
> + return;
> + }
> +}
> +
> +static unsigned int xive_get_irq(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + u32 hirq;
> +
> + /*
> + * This can be called either as a result of a HW interrupt or
> + * as a "replay" because EOI decided there was still something
> + * in one of the queues.
> + *
> + * First we perform an ACK cycle in order to update our mask
> + * of pending priorities. This will also have the effect of
> + * updating the CPPR to the most favored pending interrupts.
> + *
> + * In the future, if we have a way to differenciate a first
> + * entry (on HW interrupt) from a replay triggered by EOI,
> + * we could skip this on replays unless we soft-mask tells us
> + * that a new HW interrupt occurred.
> + */
> + xive_update_pending_irqs(xc);
> +
> + DBG_VERBOSE("get_irq: pending=%02x\n", xc->pending_prio);
> +
> + hirq = xive_scan_interrupts(xc, false);
> +
> + DBG_VERBOSE("get_irq: got irq 0x%x, new pending=0x%02x\n",
> + hirq, xc->pending_prio);
> +
> + /* Return pending interrupt if any */
> + if (hirq == XIVE_BAD_IRQ)
> + return 0;
> + return hirq;
> +}
> +
> +
> +static void xive_do_queue_eoi(struct xive_cpu *xc)
> +{
> + if (xive_scan_interrupts(xc, true) != 0) {
> + DBG_VERBOSE("eoi: pending=0x%02x\n", xc->pending_prio);
> + force_external_irq_replay();
> + }
> +}
> +
> +static u8 xive_poke_esb(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = in_be64(xd->eoi_mmio + offset);
> +
> + return (u8)val;
> +}
> +
> +static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd)
> +{
> + /* If the XIVE supports the new "store EOI facility, use it */
> + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> + out_be64(xd->eoi_mmio, 0);
> + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> + if (WARN_ON_ONCE(!xive_ops->eoi))
> + return;
> + xive_ops->eoi(hw_irq);
> + } else {
> + uint8_t eoi_val;
> +
> + /*
> + * Otherwise for EOI, we use the special MMIO that does
> + * a clear of both P and Q and returns the old Q.
> + *
> + * This allows us to then do a re-trigger if Q was set
> + * rather than synthetizing an interrupt in software
^^^^^^^^^^^^ synthesizing
> + */
> + eoi_val = xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> + DBG_VERBOSE("eoi_val=%x\n", offset, eoi_val);
> +
> + if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
> + return;
> +
> + /* Re-trigger */
> + if (xd->trig_mmio)
> + out_be64(xd->trig_mmio, 0);
> + }
> +
> +}
> +
> +static void xive_irq_eoi(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
> + d->irq, irqd_to_hwirq(d), xc->pending_prio);
> +
> + if (!irqd_irq_disabled(d))
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> +
> + /*
> + * Clear saved_p to indicate that it's no longer occupying
> + * a queue slot on the target queue
> + */
> + xd->saved_p = false;
> +
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_do_source_set_mask(struct xive_irq_data *xd,
> + bool masked)
> +{
> + if (masked)
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_01);
> + else
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> +}
> +
> +static bool xive_try_pick_target(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> + int max;
> +
> + /* Calculate max number of interrupts in that queue.
> + *
> + * We leave a gap of 1 just in case...
> + */
> + max = (q->msk + 1) - 1;
> + return !!atomic_add_unless(&q->count, 1, max);
> +}
> +
> +static void xive_dec_target_count(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> +
> + if (WARN_ON(cpu < 0))
> + return;
> +
> + /*
> + * We increment the "pending count" which will be used
> + * to decrement the target queue count whenever it's next
> + * processed and found empty. This ensure that we don't
> + * decrement while we still have the interrupt there
> + * occupying a slot.
> + */
> + atomic_inc(&q->pending_count);
> +}
> +
> +static int xive_find_target_in_mask(const struct cpumask *mask,
> + unsigned int fuzz)
> +{
> + int cpu, first, num, i;
> +
> + /* Pick up a starting point CPU in the mask based on fuzz */
> + num = cpumask_weight(mask);
> + first = (fuzz++) % num;
Incrementing fuzz here is not very useful, given it's a parameter and
it's not used subsequently in this function.
> +
> + /* Locate it */
> + cpu = cpumask_first(mask);
> + for (i = 0; i < first; i++)
> + cpu = cpumask_next(cpu, mask);
> + first = cpu;
> +
> + /*
> + * Now go through the entire mask until we find a valid
> + * target.
> + */
> + for (;;) {
> + /*
> + * We re-check online as the fallback case passes us
> + * an untested affinity mask
> + */
> + if (cpu_online(cpu) && xive_try_pick_target(cpu))
> + return cpu;
> + cpu = cpumask_next(cpu, mask);
Don't we need to wrap around from the end of the mask to the beginning
here? i.e. if (cpu == (none found)) cpu = cpumask_first(mask)
> + if (cpu == first)
> + break;
> + }
> + return -1;
> +}
> +
> +static int xive_pick_irq_target(struct irq_data *d,
> + const struct cpumask *affinity)
> +{
> + static unsigned int fuzz;
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + cpumask_var_t mask;
> + int cpu = -1;
> +
> + /*
> + * Pick a target CPU for an interrupt. This is done at
> + * startup or if the affinity is changed in a way that
> + * invalidates the current target.
> + */
> +
> + /* If we have chip IDs, first we try to build a mask of
> + * CPUs matching ther CPU and find a target in there
> + */
> + if (xd->src_chip != XIVE_INVALID_CHIP_ID &&
> + zalloc_cpumask_var(&mask, GFP_ATOMIC)) {
> + /* Build a mask of matching chip IDs */
> + for_each_cpu_and(cpu, affinity, cpu_online_mask) {
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + if (xc->chip_id == xd->src_chip)
> + cpumask_set_cpu(cpu, mask);
> + }
> + /* Try to find a target */
> + if (!cpumask_empty(mask))
> + cpu = xive_find_target_in_mask(mask, fuzz++);
> + free_cpumask_var(mask);
> + if (cpu >= 0)
> + return cpu;
> + fuzz--;
> + }
> +
> + /* No chip IDs, fallback to using the affinity mask */
> + return xive_find_target_in_mask(affinity, fuzz++);
> +}
> +
> +static unsigned int xive_irq_startup(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + int target, rc;
> +
> + DBG("xive_irq_startup: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> +#ifdef CONFIG_PCI_MSI
> + /*
> + * The generic MSI code returns with the interrupt disabled on the
> + * card, using the MSI mask bits. Firmware doesn't appear to unmask
> + * at that level, so we do it here by hand.
> + */
> + if (irq_data_get_msi_desc(d))
> + pci_msi_unmask_irq(d);
> +#endif
> +
> + /* Pick a target */
> + target = xive_pick_irq_target(d, irq_data_get_affinity_mask(d));
> + if (target == XIVE_INVALID_TARGET) {
> + /* Try again breaking affinity */
> + target = xive_pick_irq_target(d, cpu_online_mask);
> + if (target == XIVE_INVALID_TARGET)
> + return -ENXIO;
> + pr_warn("XIVE: irq %d started with broken affinity\n",
> + d->irq);
> + }
> + xd->target = target;
> +
> + /*
> + * Configure the logical number to be the Linux IRQ number
> + * and set the target queue
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc)
> + return rc;
> +
> + /* Unmask the ESB */
> + xive_do_source_set_mask(xd, false);
> +
> + return 0;
> +}
> +
> +static void xive_irq_shutdown(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + DBG("xive_irq_shutdown: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> + if (WARN_ON(xd->target == XIVE_INVALID_TARGET))
> + return;
> +
> + /* Mask the interrupt at the source */
> + xive_do_source_set_mask(xd, true);
> +
> + /* Mask the interrupt in HW in the IVT/EAS */
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, hw_irq);
> +
> + xive_dec_target_count(xd->target);
> + xd->target = XIVE_INVALID_TARGET;
> +}
> +
> +static void xive_irq_unmask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_unmask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call FW to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + xive_irq_priority, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, false);
> +}
> +
> +static void xive_irq_mask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_mask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call OPAL to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, true);
> +}
> +
> +static int xive_irq_set_affinity(struct irq_data *d,
> + const struct cpumask *cpumask,
> + bool force)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + u32 target, old_target;
> + int rc = 0;
> +
> + DBG("xive_irq_set_affinity: irq %d\n", d->irq);
> +
> + /* Is this valid ? */
> + if (cpumask_any_and(cpumask, cpu_online_mask) >= nr_cpu_ids)
> + return -EINVAL;
> +
> + /* If existing target is already in the new mask, and is
> + * online then do nothing.
> + */
> + if (cpu_online(xd->target) &&
> + cpumask_test_cpu(xd->target, cpumask))
> + return IRQ_SET_MASK_OK;
> +
> + /* Pick a new target */
> + target = xive_pick_irq_target(d, cpumask);
> +
> + /* No target found */
> + if (target == XIVE_INVALID_TARGET)
> + return -ENXIO;
> +
> + old_target = xd->target;
> +
> + /*
> + * Only configure the irq if it's not currently passed-through to
> + * a KVM guest
Does the code here implement that, or is it a requirement on
xive_ops->configure_irq, or is it not implemented in this patch but
comes along in a later patch?
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
> + return rc;
> + }
> +
> + DBG(" target: 0x%x\n", target);
> + xd->target = target;
> +
> + /* Give up previous target */
> + if (old_target != XIVE_INVALID_TARGET)
> + xive_dec_target_count(old_target);
> +
> + return IRQ_SET_MASK_OK;
> +}
> +
> +static int xive_irq_set_type(struct irq_data *d, unsigned int flow_type)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /*
> + * We only support these. This has really no effect other than setting
> + * the corresponding descriptor bits mind you but those will in turn
> + * affect the resend function when re-enabling an edge interrupt.
> + *
> + * Set set the default to edge as explained in map().
> + */
> + if (flow_type == IRQ_TYPE_DEFAULT || flow_type == IRQ_TYPE_NONE)
> + flow_type = IRQ_TYPE_EDGE_RISING;
> +
> + if (flow_type != IRQ_TYPE_EDGE_RISING &&
> + flow_type != IRQ_TYPE_LEVEL_LOW)
> + return -EINVAL;
> +
> + irqd_set_trigger_type(d, flow_type);
> +
> + /*
> + * Double check it matches what the FW thinks
> + *
> + * NOTE: We don't know yet if the PAPR interface will provide
> + * the LSI vs MSI information appart from the device-tree so
^^^^^^ apart
> + * this check might have to move into an optional backend call
> + * that is specific to the native backend
> + */
> + if ((flow_type == IRQ_TYPE_LEVEL_LOW) !=
> + !!(xd->flags & XIVE_IRQ_FLAG_LSI))
> + pr_warn("XIVE: Interrupt %d (HW 0x%x) type mismatch,"
> + " Linux says %s, FW says %s\n",
> + d->irq, (u32)irqd_to_hwirq(d),
> + (flow_type == IRQ_TYPE_LEVEL_LOW) ? "Level" : "Edge",
> + (xd->flags & XIVE_IRQ_FLAG_LSI) ? "Level" : "Edge");
> +
> + return IRQ_SET_MASK_OK_NOCOPY;
> +}
> +
> +static int xive_irq_retrigger(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /* This should be only for MSIs */
> + if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
> + return 0;
> +
> + /*
> + * To perform a retrigger, we first set the PQ bits to
> + * 11, then perform an EOI.
> + */
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_11);
> +
> + /*
> + * Note: We pass "0" to the hw_irq argument in order to
> + * avoid calling into the backend EOI code which we don't
> + * want to do in the case of a re-trigger. Backends typically
> + * only do EOI for LSIs anyway.
> + */
> + xive_do_source_eoi(0, xd);
> +
> + return 1;
> +}
> +
> +static struct irq_chip xive_irq_chip = {
> + .name = "XIVE-IRQ",
> + .irq_startup = xive_irq_startup,
> + .irq_shutdown = xive_irq_shutdown,
> + .irq_eoi = xive_irq_eoi,
> + .irq_mask = xive_irq_mask,
> + .irq_unmask = xive_irq_unmask,
> + .irq_set_affinity = xive_irq_set_affinity,
> + .irq_set_type = xive_irq_set_type,
> + .irq_retrigger = xive_irq_retrigger,
> +};
> +
> +bool is_xive_irq(struct irq_chip *chip)
> +{
> + return chip == &xive_irq_chip;
> +}
> +
> +void xive_cleanup_irq_data(struct xive_irq_data *xd)
> +{
> + if (xd->eoi_mmio) {
> + iounmap(xd->eoi_mmio);
> + if (xd->eoi_mmio == xd->trig_mmio)
> + xd->trig_mmio = NULL;
> + xd->eoi_mmio = NULL;
> + }
> + if (xd->trig_mmio) {
> + iounmap(xd->trig_mmio);
> + xd->trig_mmio = NULL;
> + }
> +}
> +
> +static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
> +{
> + struct xive_irq_data *xd;
> + int rc;
> +
> + xd = kzalloc(sizeof(struct xive_irq_data), GFP_KERNEL);
> + if (!xd)
> + return -ENOMEM;
> + rc = xive_ops->populate_irq_data(hw, xd);
> + if (rc) {
> + kfree(xd);
> + return rc;
> + }
> + xd->target = XIVE_INVALID_TARGET;
> + irq_set_handler_data(virq, xd);
> +
> + return 0;
> +}
> +
> +static void xive_irq_free_data(unsigned int virq)
> +{
> + struct xive_irq_data *xd = irq_get_handler_data(virq);
> +
> + if (!xd)
> + return;
> + irq_set_handler_data(virq, NULL);
> + xive_cleanup_irq_data(xd);
> + kfree(xd);
> +}
> +
> +#ifdef CONFIG_SMP
> +
> +static void xive_cause_ipi(int cpu, unsigned long msg)
> +{
> + struct xive_cpu *xc;
> + struct xive_irq_data *xd;
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + DBG_VERBOSE("IPI msg#%ld CPU %d -> %d (HW IRQ 0x%x)\n",
> + msg, smp_processor_id(), cpu, xc->hw_ipi);
> +
> + xd = &xc->ipi_data;
> + if (WARN_ON(!xd->trig_mmio))
> + return;
> + out_be64(xd->trig_mmio, 0);
> +}
> +
> +static irqreturn_t xive_muxed_ipi_action(int irq, void *dev_id)
> +{
> + return smp_ipi_demux();
> +}
> +
> +static void xive_ipi_eoi(struct irq_data *d)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Handle possible race with unplug and drop stale IPIs */
> + if (!xc)
> + return;
> + xive_do_source_eoi(xc->hw_ipi, &xc->ipi_data);
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_ipi_unmask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static void xive_ipi_mask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static struct irq_chip xive_ipi_chip = {
> + .name = "XIVE-IPI",
> + .irq_eoi = xive_ipi_eoi,
> + .irq_mask = xive_ipi_mask,
> + .irq_unmask = xive_ipi_unmask,
> +};
> +
> +static void __init xive_request_ipi(void)
> +{
> + unsigned int virq;
> +
> + /* Initialize it */
> + virq = irq_create_mapping(xive_irq_domain, 0);
> + xive_ipi_irq = virq;
> +
> + BUG_ON(request_irq(virq, xive_muxed_ipi_action,
> + IRQF_PERCPU | IRQF_NO_THREAD, "IPI", NULL));
> +}
> +
> +static int xive_setup_cpu_ipi(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> + int rc;
> +
> + pr_debug("XIVE: Setting up IPI for CPU %d\n", cpu);
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + /* Check if we are already setup */
> + if (xc->hw_ipi != 0)
> + return 0;
> +
> + /* Grab an IPI from the backend, this will populate xc->hw_ipi */
> + if (xive_ops->get_ipi(cpu, xc))
> + return -EIO;
> +
> + /* Populate the IRQ data in the xive_cpu structure and
> + * configure the HW / enable the IPIs
> + */
> + rc = xive_ops->populate_irq_data(xc->hw_ipi, &xc->ipi_data);
> + if (rc) {
> + pr_err("XIVE: Failed to populate IPI data on CPU %d\n", cpu);
> + return -EIO;
> + }
> + rc = xive_ops->configure_irq(xc->hw_ipi,
> + get_hard_smp_processor_id(cpu),
> + xive_irq_priority, xive_ipi_irq);
> + if (rc) {
> + pr_err("XIVE: Failed to map IPI CPU %d\n", cpu);
> + return -EIO;
> + }
> + DBG("XIVE: CPU %d HW IPI %x, virq %d, trig_mmio=%p\n", cpu,
> + xc->hw_ipi, xive_ipi_irq, xc->ipi_data.trig_mmio);
> +
> + /* Unmask it */
> + xive_do_source_set_mask(&xc->ipi_data, false);
> +
> + return 0;
> +}
> +
> +static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + /* Disable the IPI and free the IRQ data */
> +
> + /* Already cleaned up ? */
> + if (xc->hw_ipi == 0)
> + return;
> +
> + /* Mask the IPI */
> + xive_do_source_set_mask(&xc->ipi_data, true);
> +
> + /*
> + * Note: We don't call xive_cleanup_irq_data() to free
> + * the mappings as this is called from an IPI on kexec
> + * which is not a safe environment to call iounmap()
> + */
> +
> + /* Deconfigure/mask in the backend */
> + xive_ops->configure_irq(xc->hw_ipi, hard_smp_processor_id(),
> + 0xff, xive_ipi_irq);
> +
> + /* Free the IPIs in the backend */
> + xive_ops->put_ipi(cpu, xc);
> +}
> +
> +void __init xive_smp_probe(void)
> +{
> + smp_ops->cause_ipi = xive_cause_ipi;
> +
> + /* Register the IPI */
> + xive_request_ipi();
> +
> + /* Allocate and setup IPI for the boot CPU */
> + xive_setup_cpu_ipi(smp_processor_id());
> +}
> +
> +#endif /* CONFIG_SMP */
> +
> +static int xive_irq_domain_map(struct irq_domain *h, unsigned int virq,
> + irq_hw_number_t hw)
> +{
> + int rc;
> +
> + /*
> + * Mark interrupts as edge sensitive by default so that resend
> + * actually works. Will fix that up below if needed.
> + */
> + irq_clear_status_flags(virq, IRQ_LEVEL);
> +
> + /* IPIs are special and come up with HW number 0 */
> + if (hw == 0) {
> + /*
> + * IPIs are marked per-cpu. We use separate HW interrupts under
> + * the hood but associated with the same "linux" interrupt
> + */
> + irq_set_chip_and_handler(virq, &xive_ipi_chip,
> + handle_percpu_irq);
> + return 0;
> + }
> +
> + rc = xive_irq_alloc_data(virq, hw);
> + if (rc)
> + return rc;
> +
> + irq_set_chip_and_handler(virq, &xive_irq_chip, handle_fasteoi_irq);
> +
> + return 0;
> +}
> +
> +static void xive_irq_domain_unmap(struct irq_domain *d, unsigned int virq)
> +{
> + struct irq_data *data = irq_get_irq_data(virq);
> + unsigned int hw_irq;
> +
> + if (!data)
> + return;
> + hw_irq = (unsigned int)irqd_to_hwirq(data);
> + if (hw_irq)
> + xive_irq_free_data(virq);
> +}
> +
> +static int xive_irq_domain_xlate(struct irq_domain *h, struct device_node *ct,
> + const u32 *intspec, unsigned int intsize,
> + irq_hw_number_t *out_hwirq, unsigned int *out_flags)
> +
> +{
> + *out_hwirq = intspec[0];
> +
> + /*
> + * If intsize is at least 2, we look for the type in the second cell,
> + * we assume the LSB indicates a level interrupt.
> + */
> + if (intsize > 1) {
> + if (intspec[1] & 1)
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> + else
> + *out_flags = IRQ_TYPE_EDGE_RISING;
> + } else
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> +
> + return 0;
> +}
> +
> +static int xive_irq_domain_match(struct irq_domain *h, struct device_node *node,
> + enum irq_domain_bus_token bus_token)
> +{
> + return xive_ops->match(node);
> +}
> +
> +static const struct irq_domain_ops xive_irq_domain_ops = {
> + .match = xive_irq_domain_match,
> + .map = xive_irq_domain_map,
> + .unmap = xive_irq_domain_unmap,
> + .xlate = xive_irq_domain_xlate,
> +};
> +
> +static void __init xive_init_host(void)
> +{
> + xive_irq_domain = irq_domain_add_nomap(NULL, XIVE_MAX_IRQ,
> + &xive_irq_domain_ops, NULL);
> + BUG_ON(xive_irq_domain == NULL);
> + irq_set_default_host(xive_irq_domain);
> +}
> +
> +static void xive_cleanup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + if (xc->queue[xive_irq_priority].qpage)
> + xive_ops->cleanup_queue(cpu, xc, xive_irq_priority);
> +}
> +
> +static int xive_setup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + int rc = 0;
> +
> + /* We setup 1 queues for now with a 64k page */
> + if (!xc->queue[xive_irq_priority].qpage)
> + rc = xive_ops->setup_queue(cpu, xc, xive_irq_priority);
> +
> + return rc;
> +}
> +
> +static int xive_prepare_cpu(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> +
> + xc = per_cpu(xive_cpu, cpu);
> + if (!xc) {
> + struct device_node *np;
> +
> + xc = kzalloc_node(sizeof(struct xive_cpu),
> + GFP_KERNEL, cpu_to_node(cpu));
> + if (!xc)
> + return -ENOMEM;
> + np = of_get_cpu_node(cpu, NULL);
> + if (np)
> + xc->chip_id = of_get_ibm_chip_id(np);
> + of_node_put(np);
> +
> + per_cpu(xive_cpu, cpu) = xc;
> + }
> +
> + /* Setup EQs if not already */
> + return xive_setup_cpu_queues(cpu, xc);
> +}
> +
> +static void xive_setup_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Debug: Dump the TM state */
> + DBG("CPU %d [HW 0x%02x] VT=%02x\n",
> + smp_processor_id(), hard_smp_processor_id(),
> + in_8(xive_tm_area + xive_tm_offset + TM_WORD2));
> +
> + /* The backend might have additional things to do */
> + if (xive_ops->setup_cpu)
> + xive_ops->setup_cpu(smp_processor_id(), xc);
> +
> + /* Set CPPR to 0xff to enable flow of interrupts */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +#ifdef CONFIG_SMP
> +void xive_smp_setup_cpu(void)
> +{
> + DBG("XIVE: SMP setup CPU %d\n", smp_processor_id());
> +
> + /* This will have already been done on the boot CPU */
> + if (smp_processor_id() != boot_cpuid)
> + xive_setup_cpu();
> +
> +}
> +
> +int xive_smp_prepare_cpu(unsigned int cpu)
> +{
> + int rc;
> +
> + /* Allocate per-CPU data and queues */
> + rc = xive_prepare_cpu(cpu);
> + if (rc)
> + return rc;
> +
> + /* Allocate and setup IPI for the new CPU */
> + return xive_setup_cpu_ipi(cpu);
> +}
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +static void xive_flush_cpu_queue(unsigned int cpu, struct xive_cpu *xc)
> +{
> + u32 irq;
> +
> + /* We assume local irqs are disabled */
> + WARN_ON(!irqs_disabled());
> +
> + /* Check what's already in the CPU queue */
> + while ((irq = xive_scan_interrupts(xc, false)) != 0) {
> + /*
> + * We need to re-route that interrupt to its new distination.
destination ^^^^^^^^^^^
> + * First get and lock the descriptor
> + */
> + struct irq_desc *desc = irq_to_desc(irq);
> + struct irq_data *d = irq_desc_get_irq_data(desc);
> + struct xive_irq_data *xd;
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + /*
> + * Ignore anything that isn't a XIVE irq and ignore
> + * IPIs, so can just be dropped.
> + */
> + if (d->domain != xive_irq_domain || hw_irq == 0)
> + continue;
> +#ifdef DEBUG_FLUSH
> + pr_info("CPU %d: Got irq %d while offline, re-routing...\n",
> + cpu, irq);
> +#endif
> + raw_spin_lock(&desc->lock);
> + xd = irq_desc_get_handler_data(desc);
What actually does the rerouting here? Nothing in this loop has a
name that would lead me to think that it is the thing that is doing
the rerouting.
> +
> + /* For LSIs, we EOI, this will cause a resend if it's
> + * still asserted. Otherwise do an MSI retrigger
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_LSI)
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> + else
> + xive_irq_retrigger(d);
> + raw_spin_unlock(&desc->lock);
> + }
> +}
> +
> +void xive_smp_disable_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Migrate interrupts away from the CPU */
> + irq_migrate_all_off_this_cpu();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Flush everything still in the queue */
> + xive_flush_cpu_queue(cpu, xc);
> +
> + /* Re-enable CPPR */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +void xive_flush_interrupt(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Called if an interrupt occurs while the CPU is hot unplugged */
> + xive_flush_cpu_queue(cpu, xc);
> +}
> +
> +#endif /* CONFIG_HOTPLUG_CPU */
> +
> +#endif /* CONFIG_SMP */
> +
> +void xive_kexec_teardown_cpu(int secondary)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Backend cleanup if any */
> + if (xive_ops->teardown_cpu)
> + xive_ops->teardown_cpu(cpu, xc);
> +
> + /* Get rid of IPI */
> + xive_cleanup_cpu_ipi(cpu, xc);
> +
> + /* Disable and free the queues */
> + xive_cleanup_cpu_queues(cpu, xc);
> +}
> +
> +void xive_shutdown(void)
> +{
> + xive_ops->shutdown();
> +}
> +
> +bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
> + u8 max_prio)
> +{
> + xive_tm_area = area;
> + xive_tm_offset = offset;
> + xive_ops = ops;
> + xive_irq_priority = max_prio;
> +
> + ppc_md.get_irq = xive_get_irq;
> + __xive_enabled = true;
> +
> + DBG("Initializing host..\n");
> + xive_init_host();
> +
> + DBG("Initializing boot CPU..\n");
> +
> + /* Allocate per-CPU data and queues */
> + xive_prepare_cpu(smp_processor_id());
> +
> + /* Get ready for interrupts */
> + xive_setup_cpu();
> +
> + pr_info("XIVE: Interrupt handling intialized with %s backend\n",
> + xive_ops->name);
> + pr_info("XIVE: Using priority %d for all interrupts\n", max_prio);
> +
> + return true;
> +}
> +
> +static int __init xive_off(char *arg)
> +{
> + xive_cmdline_disabled = true;
> + return 0;
> +}
> +__setup("xive=off", xive_off);
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> new file mode 100644
> index 0000000..26cc6bf
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -0,0 +1,604 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +#include <linux/types.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
> +#include <linux/init.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/cpumask.h>
> +#include <linux/mm.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/opal.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +/* Enable this for using queue MMIO page for EOI. We don't currently
> + * use it as we always notify
> + */
> +#undef USE_QUEUE_MMIO
> +
> +static u32 xive_provision_size;
> +static u32 *xive_provision_chips;
> +static u32 xive_provision_chip_count;
> +static u32 xive_queue_shift;
> +static u32 xive_pool_vps = XIVE_INVALID_VP;
> +static struct kmem_cache *xive_provision_cache;
> +
> +int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
> +{
> + __be64 flags, eoi_page, trig_page;
> + __be32 esb_shift, src_chip;
> + u64 opal_flags;
> + s64 rc;
> +
> + memset(data, 0, sizeof(*data));
> +
> + rc = opal_xive_get_irq_info(hw_irq, &flags, &eoi_page, &trig_page,
> + &esb_shift, &src_chip);
> + if (rc) {
> + pr_err("XIVE: opal_xive_get_irq_info(0x%x) returned %lld\n",
> + hw_irq, rc);
> + return -EINVAL;
> + }
> +
> + opal_flags = be64_to_cpu(flags);
> + if (opal_flags & OPAL_XIVE_IRQ_STORE_EOI)
> + data->flags |= XIVE_IRQ_FLAG_STORE_EOI;
> + if (opal_flags & OPAL_XIVE_IRQ_LSI)
> + data->flags |= XIVE_IRQ_FLAG_LSI;
> + if (opal_flags & OPAL_XIVE_IRQ_SHIFT_BUG)
> + data->flags |= XIVE_IRQ_FLAG_SHIFT_BUG;
> + if (opal_flags & OPAL_XIVE_IRQ_MASK_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_MASK_FW;
> + if (opal_flags & OPAL_XIVE_IRQ_EOI_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_EOI_FW;
> + data->eoi_page = be64_to_cpu(eoi_page);
> + data->trig_page = be64_to_cpu(trig_page);
> + data->esb_shift = be32_to_cpu(esb_shift);
> + data->src_chip = be32_to_cpu(src_chip);
> +
> + data->eoi_mmio = ioremap(data->eoi_page, 1u << data->esb_shift);
> + if (!data->eoi_mmio) {
> + pr_err("XIVE: Failed to map EOI page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> +
> + if (!data->trig_page)
> + return 0;
> + if (data->trig_page == data->eoi_page) {
> + data->trig_mmio = data->eoi_mmio;
> + return 0;
> + }
> +
> + data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
> + if (!data->trig_mmio) {
> + pr_err("XIVE: Failed to map trigger page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> + return 0;
> +}
> +
> +int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_set_irq_config(hw_irq, target, prio, sw_irq);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + return rc == 0 ? 0 : -ENXIO;
> +}
> +
> +/* This can be called multiple time to change a queue configuration */
> +int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate)
> +{
> + s64 rc = 0;
> + __be64 qeoi_page_be;
> + __be32 esc_irq_be;
> + u64 flags, qpage_phys;
> +
> + /* If there's an actual queue page, clean it */
> + if (order) {
> + BUG_ON(!qpage);
> + qpage_phys = __pa(qpage);
> + } else
> + qpage_phys = 0;
> +
> + /* Initialize the rest of the fields */
> + q->msk = order ? ((1u << (order - 2)) - 1) : 0;
> + q->idx = 0;
> + q->toggle = 0;
> +
> + rc = opal_xive_get_queue_info(vp_id, prio, NULL, NULL,
> + &qeoi_page_be,
> + &esc_irq_be,
> + NULL);
> + if (rc) {
> + pr_err("XIVE: Error %lld getting queue info prio %d\n",
> + rc, prio);
> + rc = -EIO;
> + goto fail;
> + }
> + q->eoi_phys = be64_to_cpu(qeoi_page_be);
> +
> +#ifdef USE_QUEUE_MMIO
> + if (!q->eoi_mmio)
> + q->eoi_mmio = ioremap(q->eoi_phys, PAGE_SIZE);
> + if (!q->eoi_mmio) {
> + pr_err("XIVE: Failed to map queue MMIO prio %d CPU %d\n",
> + rc, prio, cpu);
> + rc = -ENOMEM;
> + goto fail;
> + }
> +#endif /* USE_QUEUE_MMIO */
> +
> + /* Default flags */
> + flags = OPAL_XIVE_EQ_ALWAYS_NOTIFY | OPAL_XIVE_EQ_ENABLED;
> +
> + /* Escalation needed ? */
> + if (can_escalate) {
> + q->esc_irq = be32_to_cpu(esc_irq_be);
> + flags |= OPAL_XIVE_EQ_ESCALATE;
> + }
> +
> + /* Configure and enable the queue in HW */
> + for (;;) {
> + rc = opal_xive_set_queue_info(vp_id, prio, qpage_phys, order, flags);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + if (rc) {
> + pr_err("XIVE: Error %lld setting queue for prio %d\n",
> + rc, prio);
> + rc = -EIO;
> + } else {
> + /*
> + * KVM code requires all of the above to be visible before
> + * q->qpage is set due to how it manages IPI EOIs
> + */
> + wmb();
> + q->qpage = qpage;
> + }
> + fail:
> + return rc;
> +}
> +
> +static void __xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
> +{
> + s64 rc;
> +
> + /* Disable the queue in HW */
> + for (;;) {
> + rc = opal_xive_set_queue_info(vp_id, prio, 0, 0, 0);
> + break;
> + msleep(1);
> + }
> + if (rc)
> + pr_err("XIVE: Error %lld disabling queue for prio %d\n",
> + rc, prio);
> +}
> +
> +void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
> +{
> + __xive_native_disable_queue(vp_id, q, prio);
> +
> + if (q->eoi_mmio)
> + iounmap(q->eoi_mmio);
> + q->eoi_mmio = NULL;
> +}
> +
> +static int xive_native_setup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
> +{
> + struct xive_q *q = &xc->queue[prio];
> + unsigned int alloc_order;
> + struct page *pages;
> + __be32 *qpage;
> +
> + alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
> + (xive_queue_shift - PAGE_SHIFT) : 0;
> + pages = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL, alloc_order);
> + if (!pages)
> + return -ENOMEM;
> + qpage = (__be32 *)page_address(pages);
> + memset(qpage, 0, 1 << xive_queue_shift);
> + return xive_native_configure_queue(get_hard_smp_processor_id(cpu),
> + q, prio, qpage, xive_queue_shift, false);
> +}
> +
> +static void xive_native_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
> +{
> + struct xive_q *q = &xc->queue[prio];
> + unsigned int alloc_order;
> +
> + /*
> + * We use the variant with no iounmap as this is called on exec
> + * from an IPI and iounmap isn't safe
> + */
> + __xive_native_disable_queue(get_hard_smp_processor_id(cpu), q, prio);
> + alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
> + (xive_queue_shift - PAGE_SHIFT) : 0;
> + free_pages((unsigned long)q->qpage, alloc_order);
> + q->qpage = NULL;
> +}
> +
> +static bool xive_native_match(struct device_node *node)
> +{
> + return of_device_is_compatible(node, "ibm,opal-xive-vc");
> +}
> +
> +#ifdef CONFIG_SMP
> +static int xive_native_get_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + struct device_node *np;
> + unsigned int chip_id;
> + s64 irq;
> +
> + /* Find the chip ID */
> + np = of_get_cpu_node(cpu, NULL);
> + if (np) {
> + if (of_property_read_u32(np, "ibm,chip-id", &chip_id) < 0)
> + chip_id = 0;
> + }
> +
> + /* Allocate an IPI and populate info about it */
> + for (;;) {
> + irq = opal_xive_allocate_irq(chip_id);
> + if (irq == OPAL_BUSY) {
> + msleep(1);
> + continue;
> + }
> + if (irq < 0) {
> + pr_err("XIVE: Failed to allocate IPI on CPU %d\n",
> + cpu);
> + return -ENXIO;
> + }
> + xc->hw_ipi = irq;
> + break;
> + }
> + return 0;
> +}
> +
> +u32 xive_native_alloc_irq(void)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_allocate_irq(OPAL_XIVE_ANY_CHIP);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + if (rc < 0)
> + return 0;
> + return rc;
> +}
> +
> +void xive_native_free_irq(u32 irq)
> +{
> + for (;;) {
> + s64 rc = opal_xive_free_irq(irq);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> +}
> +
> +static void xive_native_put_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + s64 rc;
> +
> + /* Free the IPI */
> + if (!xc->hw_ipi)
> + return;
> + for (;;) {
> + rc = opal_xive_free_irq(xc->hw_ipi);
> + if (rc == OPAL_BUSY) {
> + msleep(1);
> + continue;
> + }
> + xc->hw_ipi = 0;
> + break;
> + }
> +}
> +#endif /* CONFIG_SMP */
> +
> +static void xive_native_shutdown(void)
> +{
> + /* Switch the XIVE to emulation mode */
> + opal_xive_reset(OPAL_XIVE_MODE_EMU);
> +}
> +
> +static void xive_native_eoi(u32 hw_irq)
> +{
> + /* Not normally used except if specific interrupts need
> + * a workaround on EOI
> + */
> + opal_int_eoi(hw_irq);
> +}
> +
> +static void xive_native_setup_cpu(unsigned int cpu, struct xive_cpu *xc)
> +{
> + s64 rc;
> + u32 vp;
> + __be64 vp_cam_be;
> + u64 vp_cam;
> +
> + if (xive_pool_vps == XIVE_INVALID_VP)
> + return;
> +
> + /* Enable the pool VP */
> + vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
> + pr_debug("XIVE: CPU %d setting up pool VP 0x%x\n", cpu, vp);
> + for (;;) {
> + rc = opal_xive_set_vp_info(vp, OPAL_XIVE_VP_ENABLED, 0);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + if (rc) {
> + pr_err("XIVE: Failed to enable pool VP on CPU %d\n", cpu);
> + return;
> + }
> +
> + /* Grab it's CAM value */
> + rc = opal_xive_get_vp_info(vp, NULL, &vp_cam_be, NULL, NULL);
> + if (rc) {
> + pr_err("XIVE: Failed to get pool VP info CPU %d\n", cpu);
> + return;
> + }
> + vp_cam = be64_to_cpu(vp_cam_be);
> +
> + pr_debug("XIVE: VP CAM = %llx\n", vp_cam);
> +
> + /* Push it on the CPU (set LSMFB to 0xff to skip backlog scan) */
> + pr_debug("XIVE: (Old HW value: %08x)\n",
> + in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
> + out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD0, 0xff);
> + out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2,
> + TM_QW2W2_VP | vp_cam);
> + pr_debug("XIVE: (New HW value: %08x)\n",
> + in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
> +}
> +
> +static void xive_native_teardown_cpu(unsigned int cpu, struct xive_cpu *xc)
> +{
> + s64 rc;
> + u32 vp;
> +
> + if (xive_pool_vps == XIVE_INVALID_VP)
> + return;
> +
> + /* Pull the pool VP from the CPU */
> + in_be64(xive_tm_area + TM_SPC_PULL_POOL_CTX);
> +
> + /* Disable it */
> + vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
> + for (;;) {
> + rc = opal_xive_set_vp_info(vp, 0, 0);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> +}
> +
> +static void xive_native_sync_source(u32 hw_irq)
> +{
> + opal_xive_sync(XIVE_SYNC_EAS, hw_irq);
> +}
> +
> +static const struct xive_ops xive_native_ops = {
> + .populate_irq_data = xive_native_populate_irq_data,
> + .configure_irq = xive_native_configure_irq,
> + .setup_queue = xive_native_setup_queue,
> + .cleanup_queue = xive_native_cleanup_queue,
> + .match = xive_native_match,
> + .shutdown = xive_native_shutdown,
> + .eoi = xive_native_eoi,
> + .setup_cpu = xive_native_setup_cpu,
> + .teardown_cpu = xive_native_teardown_cpu,
> + .sync_source = xive_native_sync_source,
> +#ifdef CONFIG_SMP
> + .get_ipi = xive_native_get_ipi,
> + .put_ipi = xive_native_put_ipi,
> +#endif /* CONFIG_SMP */
> + .name = "native",
> +};
> +
> +static bool xive_parse_provisioning(struct device_node *np)
> +{
> + int rc;
> +
> + if (of_property_read_u32(np, "ibm,xive-provision-page-size",
> + &xive_provision_size) < 0)
> + return true;
> + rc = of_property_count_elems_of_size(np, "ibm,xive-provision-chips", 4);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d getting provision chips array\n", rc);
> + return false;
> + }
> + xive_provision_chip_count = rc;
> + if (rc == 0)
> + return true;
> +
> + xive_provision_chips = kzalloc(4 * xive_provision_chip_count,
> + GFP_KERNEL);
> + BUG_ON(!xive_provision_chips);
> +
> + rc = of_property_read_u32_array(np, "ibm,xive-provision-chips",
> + xive_provision_chips,
> + xive_provision_chip_count);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d reading provision chips array\n", rc);
> + return false;
> + }
> +
> + xive_provision_cache = kmem_cache_create("xive-provision",
> + xive_provision_size,
> + xive_provision_size,
> + 0, NULL);
> + if (!xive_provision_cache) {
> + pr_err("XIVE: Failed to allocate provision cache\n");
> + return false;
> + }
> + return true;
> +}
> +
> +u32 xive_native_default_eq_shift(void)
> +{
> + return xive_queue_shift;
> +}
> +
> +bool xive_native_init(void)
> +{
> + struct device_node *np;
> + struct resource r;
> + void __iomem *tm_area;
> + struct property *prop;
> + u8 max_prio = 7;
> + const __be32 *p;
> + u32 val;
> + s64 rc;
> +
> + if (xive_cmdline_disabled)
> + return false;
> +
> + DBG("xive_native_init()\n");
> + np = of_find_compatible_node(NULL, NULL, "ibm,opal-xive-pe");
> + if (!np) {
> + DBG("not found !\n");
> + return false;
> + }
> + DBG("Found %s\n", np->full_name);
> +
> + /* Resource 1 is HV window */
> + if (of_address_to_resource(np, 1, &r)) {
> + pr_err("XIVE: Failed to get TM area resource\n");
"TM" is confusing since it would usually mean "Transactional memory".
I couldn't see anywhere in this patch where you spell out that it
means "Thread Management". For messages that are printed out it's
probably better to say "thread mgmt area" rather than "TM area".
> + return false;
> + }
> + tm_area = ioremap(r.start, resource_size(&r));
> + if (!tm_area) {
> + pr_err("XIVE: Failed to map TM area\n");
> + return false;
> + }
> +
> + /* Read number of priorities */
> + if (of_property_read_u32(np, "ibm,xive-#priorities", &val) == 0)
> + max_prio = val - 1;
> +
> + /* Iterate the EQ sizes and pick one */
> + of_property_for_each_u32(np, "ibm,xive-eq-sizes", prop, p, val) {
> + xive_queue_shift = val;
> + if (val == PAGE_SHIFT)
> + break;
> + }
> +
> + /* Grab size of provisionning pages */
^^^^^^^^^^^^^ provisioning
> + xive_parse_provisioning(np);
> +
> + /* Switch the XIVE to exploitation mode */
> + rc = opal_xive_reset(OPAL_XIVE_MODE_EXPL);
> + if (rc) {
> + pr_err("XIVE: Switch to exploitation mode failed"
> + " with error %lld\n", rc);
> + return false;
> + }
> +
> + /* Initialize XIVE core with our backend */
> + if (!xive_core_init(&xive_native_ops, tm_area, TM_QW3_HV_PHYS,
> + max_prio)) {
> + opal_xive_reset(OPAL_XIVE_MODE_EMU);
> + return false;
> + }
> + pr_info("XIVE: Using %dkB queues\n", 1 << (xive_queue_shift - 10));
> + return true;
> +}
> +
> +static bool xive_native_provision_pages(void)
> +{
> + u32 i;
> + void *p;
> +
> + for (i = 0; i < xive_provision_chip_count; i++) {
> + u32 chip = xive_provision_chips[i];
> +
> + /* XXX TODO: Try to make the allocation local to the node where
> + * the chip reside
> + */
> + p = kmem_cache_alloc(xive_provision_cache, GFP_KERNEL);
> + if (!p) {
> + pr_err("XIVE: Failed to allocate provisioning page\n");
> + return false;
> + }
> + opal_xive_donate_page(chip, __pa(p));
> + }
> + return true;
> +}
> +
> +u32 xive_native_alloc_vp_block(u32 max_vcpus)
> +{
> + s64 rc;
> + u32 order;
> +
> + order = fls(max_vcpus) - 1;
> + pr_info("XIVE: VP block alloc, for max VCPUs %d"
> + " use order %d\n", max_vcpus, order);
> + for (;;) {
> + rc = opal_xive_alloc_vp_block(order);
> + switch (rc) {
> + case OPAL_BUSY:
> + msleep(1);
> + break;
> + case OPAL_XIVE_PROVISIONING:
> + if (!xive_native_provision_pages())
> + return XIVE_INVALID_VP;
> + break;
> + default:
> + if (rc < 0) {
> + pr_err("XIVE: OPAL failed to allocate VCPUs"
> + " order %d, err %lld\n",
> + order, rc);
> + return XIVE_INVALID_VP;
> + }
> + return rc;
> + }
> + }
> +}
> +EXPORT_SYMBOL_GPL(xive_native_alloc_vp_block);
> +
> +void xive_native_free_vp_block(u32 vp_base)
> +{
> + s64 rc;
> +
> + if (vp_base == XIVE_INVALID_VP)
> + return;
> +
> + rc = opal_xive_free_vp_block(vp_base);
> + if (rc < 0)
> + pr_warn("XIVE: OPAL error %lld freeing VP block\n", rc);
> +}
> +EXPORT_SYMBOL_GPL(xive_native_free_vp_block);
> diff --git a/arch/powerpc/sysdev/xive/xive-internal.h b/arch/powerpc/sysdev/xive/xive-internal.h
> new file mode 100644
> index 0000000..e736fc5
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-internal.h
> @@ -0,0 +1,51 @@
> +#ifndef __XIVE_INTERNAL_H
> +#define __XIVE_INTERNAL_H
> +
> +/* Each CPU carry one of these with various per-CPU state */
> +struct xive_cpu {
> +#ifdef CONFIG_SMP
> + /* HW irq number and data of IPI */
> + u32 hw_ipi;
> + struct xive_irq_data ipi_data;
> +#endif /* CONFIG_SMP */
> +
> + int chip_id;
> +
> + /* Queue datas. Only one is populated */
> +#define XIVE_MAX_QUEUES 8
> + struct xive_q queue[XIVE_MAX_QUEUES];
> +
> + /* Pending mask. Each bit corresponds to a priority that
> + * potentially has pending interrupts
> + */
> + u8 pending_prio;
> +
> + /* Cache of HW CPPR */
> + u8 cppr;
> +};
> +
> +/* Backend ops */
> +struct xive_ops {
> + int (*populate_irq_data)(u32 hw_irq, struct xive_irq_data *data);
> + int (*configure_irq)(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
> + int (*setup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
> + void (*cleanup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
> + void (*setup_cpu)(unsigned int cpu, struct xive_cpu *xc);
> + void (*teardown_cpu)(unsigned int cpu, struct xive_cpu *xc);
> + bool (*match)(struct device_node *np);
> + void (*shutdown)(void);
> + void (*eoi)(u32 hw_irq);
> + void (*sync_source)(u32 hw_irq);
> +#ifdef CONFIG_SMP
> + int (*get_ipi)(unsigned int cpu, struct xive_cpu *xc);
> + void (*put_ipi)(unsigned int cpu, struct xive_cpu *xc);
> +#endif
> + const char *name;
> +};
> +
> +bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
> + u8 max_prio);
> +
> +extern bool xive_cmdline_disabled;
> +
> +#endif /* __XIVE_INTERNAL_H */
> diff --git a/arch/powerpc/sysdev/xive/xive-regs.h b/arch/powerpc/sysdev/xive/xive-regs.h
> new file mode 100644
> index 0000000..f1edb23
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-regs.h
> @@ -0,0 +1,88 @@
> +#ifndef __XIVE_REGS_H__
> +#define __XIVE_REGS_H__
> +
> +/*
> + * TM registers
"Thread Management (TM) registers"
> + */
> +
> +/* TM register offsets */
> +#define TM_QW0_USER 0x000 /* All rings */
> +#define TM_QW1_OS 0x010 /* Ring 0..2 */
> +#define TM_QW2_HV_POOL 0x020 /* Ring 0..1 */
> +#define TM_QW3_HV_PHYS 0x030 /* Ring 0..1 */
> +
> +/* Byte offsets inside a QW QW0 QW1 QW2 QW3 */
> +#define TM_NSR 0x0 /* + + - + */
> +#define TM_CPPR 0x1 /* - + - + */
> +#define TM_IPB 0x2 /* - + + + */
> +#define TM_LSMFB 0x3 /* - + + + */
> +#define TM_ACK_CNT 0x4 /* - + - - */
> +#define TM_INC 0x5 /* - + - + */
> +#define TM_AGE 0x6 /* - + - + */
> +#define TM_PIPR 0x7 /* - + - + */
> +
> +#define TM_WORD0 0x0
> +#define TM_WORD1 0x4
> +
> +/* QW word 2 contains the valid bit at the top and other fields
> + * depending on the QW
> + */
> +#define TM_WORD2 0x8
> +#define TM_QW0W2_VU PPC_BIT32(0)
> +#define TM_QW0W2_LOGIC_SERV PPC_BITMASK32(1,31) // XX 2,31 ?
> +#define TM_QW1W2_VO PPC_BIT32(0)
> +#define TM_QW1W2_OS_CAM PPC_BITMASK32(8,31)
> +#define TM_QW2W2_VP PPC_BIT32(0)
> +#define TM_QW2W2_POOL_CAM PPC_BITMASK32(8,31)
> +#define TM_QW3W2_VT PPC_BIT32(0)
> +#define TM_QW3W2_LP PPC_BIT32(6)
> +#define TM_QW3W2_LE PPC_BIT32(7)
> +#define TM_QW3W2_T PPC_BIT32(31)
> +
> +/* In addition to normal loads to "peek" and writes (only when invalid)
> + * using 4 and 8 bytes accesses, the above registers support these
> + * "special" byte operations:
> + *
> + * - Byte load from QW0[NSR] - User level NSR (EBB)
> + * - Byte store to QW0[NSR] - User level NSR (EBB)
> + * - Byte load/store to QW1[CPPR] and QW3[CPPR] - CPPR access
> + * - Byte load from QW3[TM_WORD2] - Read VT||00000||LP||LE on thrd 0
> + * otherwise VT||0000000
> + * - Byte store to QW3[TM_WORD2] - Set VT bit (and LP/LE if present)
> + *
> + * Then we have all these "special" CI ops at these offset that trigger
> + * all sorts of side effects:
> + */
> +#define TM_SPC_ACK_EBB 0x800 /* Load8 ack EBB to reg*/
> +#define TM_SPC_ACK_OS_REG 0x810 /* Load16 ack OS irq to reg */
> +#define TM_SPC_PUSH_USR_CTX 0x808 /* Store32 Push/Validate user context */
> +#define TM_SPC_PULL_USR_CTX 0x808 /* Load32 Pull/Invalidate user context */
> +#define TM_SPC_SET_OS_PENDING 0x812 /* Store8 Set OS irq pending bit */
> +#define TM_SPC_PULL_OS_CTX 0x818 /* Load32/Load64 Pull/Invalidate OS context to reg */
> +#define TM_SPC_PULL_POOL_CTX 0x828 /* Load32/Load64 Pull/Invalidate Pool context to reg*/
> +#define TM_SPC_ACK_HV_REG 0x830 /* Load16 ack HV irq to reg */
> +#define TM_SPC_PULL_USR_CTX_OL 0xc08 /* Store8 Pull/Inval usr ctx to odd line */
> +#define TM_SPC_ACK_OS_EL 0xc10 /* Store8 ack OS irq to even line */
> +#define TM_SPC_ACK_HV_POOL_EL 0xc20 /* Store8 ack HV evt pool to even line */
> +#define TM_SPC_ACK_HV_EL 0xc30 /* Store8 ack HV irq to even line */
> +/* XXX more... */
> +
> +/* NSR fields for the various QW ack types */
> +#define TM_QW0_NSR_EB PPC_BIT8(0)
> +#define TM_QW1_NSR_EO PPC_BIT8(0)
> +#define TM_QW3_NSR_HE PPC_BITMASK8(0,1)
> +#define TM_QW3_NSR_HE_NONE 0
> +#define TM_QW3_NSR_HE_POOL 1
> +#define TM_QW3_NSR_HE_PHYS 2
> +#define TM_QW3_NSR_HE_LSI 3
> +#define TM_QW3_NSR_I PPC_BIT8(2)
> +#define TM_QW3_NSR_GRP_LVL PPC_BIT8(3,7)
> +
> +/* Utilities to manipulate these (originaly from OPAL) */
> +#define MASK_TO_LSH(m) (__builtin_ffsl(m) - 1)
> +#define GETFIELD(m, v) (((v) & (m)) >> MASK_TO_LSH(m))
> +#define SETFIELD(m, v, val) \
> + (((v) & ~(m)) | ((((typeof(v))(val)) << MASK_TO_LSH(m)) & (m)))
> +
> +
> +#endif /* __XIVE_H__ */
> diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> index 16321ad..c71e919 100644
> --- a/arch/powerpc/xmon/xmon.c
> +++ b/arch/powerpc/xmon/xmon.c
> @@ -48,7 +48,7 @@
> #include <asm/reg.h>
> #include <asm/debug.h>
> #include <asm/hw_breakpoint.h>
> -
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/firmware.h>
>
> @@ -232,7 +232,13 @@ Commands:\n\
> "\
> dr dump stream of raw bytes\n\
> dt dump the tracing buffers (uses printk)\n\
> - e print exception information\n\
> +"
> +#ifdef CONFIG_PPC_POWERNV
> +" dx# dump xive on CPU\n\
> + dxi# dump xive irq state\n\
> + dxa dump xive on all CPUs\n"
> +#endif
> +" e print exception information\n\
> f flush cache\n\
> la lookup symbol+offset of specified address\n\
> ls lookup address of specified symbol\n\
> @@ -2338,6 +2344,81 @@ static void dump_pacas(void)
> }
> #endif
>
> +#ifdef CONFIG_PPC_POWERNV
> +static void dump_one_xive(int cpu)
> +{
> + unsigned int hwid = get_hard_smp_processor_id(cpu);
> +
> + opal_xive_dump(XIVE_DUMP_TM_HYP, hwid);
> + opal_xive_dump(XIVE_DUMP_TM_POOL, hwid);
> + opal_xive_dump(XIVE_DUMP_TM_OS, hwid);
> + opal_xive_dump(XIVE_DUMP_TM_USER, hwid);
> + opal_xive_dump(XIVE_DUMP_VP, hwid);
> + opal_xive_dump(XIVE_DUMP_EMU_STATE, hwid);
> +
> + if (setjmp(bus_error_jmp) != 0) {
> + catch_memory_errors = 0;
> + printf("*** Error dumping xive on cpu %d\n", cpu);
> + return;
> + }
> +
> + catch_memory_errors = 1;
> + sync();
> + xmon_xive_do_dump(cpu);
> + sync();
> + __delay(200);
> + catch_memory_errors = 0;
> +}
> +
> +static void dump_all_xives(void)
> +{
> + int cpu;
> +
> + if (num_possible_cpus() == 0) {
> + printf("No possible cpus, use 'dx #' to dump individual cpus\n");
> + return;
> + }
> +
> + for_each_possible_cpu(cpu)
> + dump_one_xive(cpu);
> +}
> +
> +static void dump_one_xive_irq(uint32_t num)
> +{
> + int64_t rc;
> + __be64 vp;
> + uint8_t prio;
> + __be32 lirq;
> +
> + rc = opal_xive_get_irq_config(num, &vp, &prio, &lirq);
> + xmon_printf("IRQ 0x%x config: vp=0x%llx prio=%d lirq=0x%x (rc=%lld)\n",
> + num, be64_to_cpu(vp), prio, be32_to_cpu(lirq), rc);
> +}
> +
> +static void dump_xives(void)
> +{
> + unsigned long num;
> + int c;
> +
> + c = inchar();
> + if (c == 'a') {
> + dump_all_xives();
> + return;
> + } else if (c == 'i') {
> + if (scanhex(&num))
> + dump_one_xive_irq(num);
> + return;
> + }
> +
> + termch = c; /* Put c back, it wasn't 'a' */
> +
> + if (scanhex(&num))
> + dump_one_xive(num);
> + else
> + dump_one_xive(xmon_owner);
> +}
> +#endif /* CONFIG_PPC_POWERNV */
> +
> static void dump_by_size(unsigned long addr, long count, int size)
> {
> unsigned char temp[16];
> @@ -2386,6 +2467,14 @@ dump(void)
> return;
> }
> #endif
> +#ifdef CONFIG_PPC_POWERNV
> + if (c == 'x') {
> + xmon_start_pagination();
> + dump_xives();
> + xmon_end_pagination();
> + return;
> + }
> +#endif
>
> if (c == '\n')
> termch = c;
> --
> 2.9.3
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Paul.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
@ 2017-03-24 5:22 ` Paul Mackerras
0 siblings, 0 replies; 38+ messages in thread
From: Paul Mackerras @ 2017-03-24 5:22 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev, kvm-ppc
On Mon, Mar 20, 2017 at 05:49:08PM +1100, Benjamin Herrenschmidt wrote:
> The XIVE interrupt controller is the new interrupt controller
> found in POWER9. It supports advanced virtualization capabilities
> among other things.
>
> Currently we use a set of firmware calls that simulate the old
> "XICS" interrupt controller but this is fairly inefficient.
>
> This adds the framework for using XIVE along with a native
> backend which OPAL for configuration. Later, a backend allowing
> the use in a KVM or PowerVM guest will also be provided.
>
> This disables some fast path for interrupts in KVM when XIVE is
> enabled as these rely on the firmware emulation code which is no
> longer available when the XIVE is used natively by Linux.
>
> A latter patch will make KVM also directly exploit the XIVE, thus
> recovering the lost performance (and more).
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Mostly looks fine, a few minor comments below...
> ---
> arch/powerpc/include/asm/xive.h | 116 +++
> arch/powerpc/include/asm/xmon.h | 2 +
> arch/powerpc/platforms/powernv/Kconfig | 2 +
> arch/powerpc/platforms/powernv/setup.c | 15 +-
> arch/powerpc/platforms/powernv/smp.c | 39 +-
> arch/powerpc/sysdev/Kconfig | 1 +
> arch/powerpc/sysdev/Makefile | 1 +
> arch/powerpc/sysdev/xive/Kconfig | 7 +
> arch/powerpc/sysdev/xive/Makefile | 4 +
> arch/powerpc/sysdev/xive/common.c | 1175 ++++++++++++++++++++++++++++++
> arch/powerpc/sysdev/xive/native.c | 604 +++++++++++++++
> arch/powerpc/sysdev/xive/xive-internal.h | 51 ++
> arch/powerpc/sysdev/xive/xive-regs.h | 88 +++
> arch/powerpc/xmon/xmon.c | 93 ++-
> 14 files changed, 2186 insertions(+), 12 deletions(-)
> create mode 100644 arch/powerpc/include/asm/xive.h
> create mode 100644 arch/powerpc/sysdev/xive/Kconfig
> create mode 100644 arch/powerpc/sysdev/xive/Makefile
> create mode 100644 arch/powerpc/sysdev/xive/common.c
> create mode 100644 arch/powerpc/sysdev/xive/native.c
> create mode 100644 arch/powerpc/sysdev/xive/xive-internal.h
> create mode 100644 arch/powerpc/sysdev/xive/xive-regs.h
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> new file mode 100644
> index 0000000..b1604b73
> --- /dev/null
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -0,0 +1,116 @@
> +#ifndef _ASM_POWERPC_XIVE_H
> +#define _ASM_POWERPC_XIVE_H
> +
> +#define XIVE_INVALID_VP 0xffffffff
> +
> +#ifdef CONFIG_PPC_XIVE
> +
> +extern void __iomem *xive_tm_area;
> +extern u32 xive_tm_offset;
> +
> +/*
> + * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
> + * have it stored in the xive_cpu structure. We also cache
> + * for normal interrupts the current target CPU.
> + */
> +struct xive_irq_data {
> + /* Setup by backend */
> + u64 flags;
> +#define XIVE_IRQ_FLAG_STORE_EOI 0x01
> +#define XIVE_IRQ_FLAG_LSI 0x02
> +#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
> +#define XIVE_IRQ_FLAG_MASK_FW 0x08
> +#define XIVE_IRQ_FLAG_EOI_FW 0x10
> + u64 eoi_page;
> + void __iomem *eoi_mmio;
> + u64 trig_page;
> + void __iomem *trig_mmio;
> + u32 esb_shift;
> + int src_chip;
> +
> + /* Setup/used by frontend */
> + int target;
> + bool saved_p;
> +};
> +#define XIVE_INVALID_CHIP_ID -1
> +
> +/* A queue tracking structure in a CPU */
> +struct xive_q {
> + __be32 *qpage;
> + u32 msk;
> + u32 idx;
> + u32 toggle;
> + u64 eoi_phys;
> + void __iomem *eoi_mmio;
> + u32 esc_irq;
> + atomic_t count;
> + atomic_t pending_count;
> +};
> +
> +/*
> + * "magic" ESB MMIO offsets
> + */
> +#define XIVE_ESB_GET 0x800
> +#define XIVE_ESB_SET_PQ_00 0xc00
> +#define XIVE_ESB_SET_PQ_01 0xd00
> +#define XIVE_ESB_SET_PQ_10 0xe00
> +#define XIVE_ESB_SET_PQ_11 0xf00
> +#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> +
> +extern bool __xive_enabled;
> +
> +static inline bool xive_enabled(void) { return __xive_enabled; }
> +
> +extern bool xive_native_init(void);
> +extern void xive_smp_probe(void);
> +extern int xive_smp_prepare_cpu(unsigned int cpu);
> +extern void xive_smp_setup_cpu(void);
> +extern void xive_smp_disable_cpu(void);
> +extern void xive_kexec_teardown_cpu(int secondary);
> +extern void xive_shutdown(void);
> +extern void xive_flush_interrupt(void);
> +
> +/* xmon hook */
> +extern void xmon_xive_do_dump(int cpu);
> +
> +/* APIs used by KVM */
> +extern u32 xive_native_default_eq_shift(void);
> +extern u32 xive_native_alloc_vp_block(u32 max_vcpus);
> +extern void xive_native_free_vp_block(u32 vp_base);
> +extern int xive_native_populate_irq_data(u32 hw_irq,
> + struct xive_irq_data *data);
> +extern void xive_cleanup_irq_data(struct xive_irq_data *xd);
> +extern u32 xive_native_alloc_irq(void);
> +extern void xive_native_free_irq(u32 irq);
> +extern int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
> +
> +extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate);
> +extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
> +
> +extern bool __xive_irq_trigger(struct xive_irq_data *xd);
> +extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
> +extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
> +
> +extern bool is_xive_irq(struct irq_chip *chip);
> +
> +#else
> +
> +static inline bool xive_enabled(void) { return false; }
> +
> +static inline bool xive_native_init(void) { return false; }
> +static inline void xive_smp_probe(void) { }
> +extern inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
> +static inline void xive_smp_setup_cpu(void) { }
> +static inline void xive_smp_disable_cpu(void) { }
> +static inline void xive_kexec_teardown_cpu(int secondary) { }
> +static inline void xive_shutdown(void) { }
> +static inline void xive_flush_interrupt(void) { }
> +
> +static inline u32 xive_native_alloc_vp_block(u32 max_vcpus)
> + { return XIVE_INVALID_VP; }
> +static inline void xive_native_free_vp_block(u32 vp_base) { }
> +
> +#endif
> +
> +#endif /* _ASM_POWERPC_XIVE_H */
> diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h
> index 5eb8e59..eb42a0c 100644
> --- a/arch/powerpc/include/asm/xmon.h
> +++ b/arch/powerpc/include/asm/xmon.h
> @@ -29,5 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { };
> extern int cpus_are_in_xmon(void);
> #endif
>
> +extern void xmon_printf(const char *format, ...);
> +
> #endif /* __KERNEL __ */
> #endif /* __ASM_POWERPC_XMON_H */
> diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
> index 3a07e4d..81ee2ed 100644
> --- a/arch/powerpc/platforms/powernv/Kconfig
> +++ b/arch/powerpc/platforms/powernv/Kconfig
> @@ -4,6 +4,8 @@ config PPC_POWERNV
> select PPC_NATIVE
> select PPC_XICS
> select PPC_ICP_NATIVE
> + select PPC_XIVE
> + select PPC_XIVE_NATIVE
> select PPC_P7_NAP
> select PCI
> select PCI_MSI
> diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
> index d50c7d9..adceac9 100644
> --- a/arch/powerpc/platforms/powernv/setup.c
> +++ b/arch/powerpc/platforms/powernv/setup.c
> @@ -32,6 +32,7 @@
> #include <asm/machdep.h>
> #include <asm/firmware.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/kexec.h>
> #include <asm/smp.h>
> @@ -76,7 +77,9 @@ static void __init pnv_init(void)
>
> static void __init pnv_init_IRQ(void)
> {
> - xics_init();
> + /* Try using a XIVE if available, otherwise use a XICS */
> + if (!xive_native_init())
> + xics_init();
>
> WARN_ON(!ppc_md.get_irq);
> }
> @@ -218,10 +221,12 @@ static void pnv_kexec_wait_secondaries_down(void)
>
> static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> {
> - xics_kexec_teardown_cpu(secondary);
> + if (xive_enabled())
> + xive_kexec_teardown_cpu(secondary);
> + else
> + xics_kexec_teardown_cpu(secondary);
>
> /* On OPAL, we return all CPUs to firmware */
> -
> if (!firmware_has_feature(FW_FEATURE_OPAL))
> return;
>
> @@ -237,6 +242,10 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> /* Primary waits for the secondaries to have reached OPAL */
> pnv_kexec_wait_secondaries_down();
>
> + /* Switch XIVE back to emulation mode */
> + if (xive_enabled())
> + xive_shutdown();
> +
> /*
> * We might be running as little-endian - now that interrupts
> * are disabled, reset the HILE bit to big-endian so we don't
> diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
> index 8b67e1e..f571955 100644
> --- a/arch/powerpc/platforms/powernv/smp.c
> +++ b/arch/powerpc/platforms/powernv/smp.c
> @@ -29,6 +29,7 @@
> #include <asm/vdso_datapage.h>
> #include <asm/cputhreads.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/runlatch.h>
> #include <asm/code-patching.h>
> @@ -47,7 +48,9 @@
>
> static void pnv_smp_setup_cpu(int cpu)
> {
> - if (cpu != boot_cpuid)
> + if (xive_enabled())
> + xive_smp_setup_cpu();
> + else if (cpu != boot_cpuid)
> xics_setup_cpu();
>
> #ifdef CONFIG_PPC_DOORBELL
> @@ -132,7 +135,10 @@ static int pnv_smp_cpu_disable(void)
> vdso_data->processorCount--;
> if (cpu = boot_cpuid)
> boot_cpuid = cpumask_any(cpu_online_mask);
> - xics_migrate_irqs_away();
> + if (xive_enabled())
> + xive_smp_disable_cpu();
> + else
> + xics_migrate_irqs_away();
> return 0;
> }
>
> @@ -213,9 +219,12 @@ static void pnv_smp_cpu_kill_self(void)
> if (((srr1 & wmask) = SRR1_WAKEEE) ||
> ((srr1 & wmask) = SRR1_WAKEHVI) ||
> (local_paca->irq_happened & PACA_IRQ_EE)) {
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - icp_opal_flush_interrupt();
> - else
> + if (cpu_has_feature(CPU_FTR_ARCH_300)) {
> + if (xive_enabled())
> + xive_flush_interrupt();
> + else
> + icp_opal_flush_interrupt();
> + } else
> icp_native_flush_interrupt();
> } else if ((srr1 & wmask) = SRR1_WAKEHDBELL) {
> unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
> @@ -252,10 +261,26 @@ static int pnv_cpu_bootable(unsigned int nr)
> return smp_generic_cpu_bootable(nr);
> }
>
> +static int pnv_smp_prepare_cpu(int cpu)
> +{
> + if (xive_enabled())
> + return xive_smp_prepare_cpu(cpu);
> + return 0;
> +}
> +
> +static void __init pnv_smp_probe(void)
> +{
> + if (xive_enabled())
> + xive_smp_probe();
> + else
> + xics_smp_probe();
> +}
> +
> static struct smp_ops_t pnv_smp_ops = {
> .message_pass = smp_muxed_ipi_message_pass,
> - .cause_ipi = NULL, /* Filled at runtime by xics_smp_probe() */
> - .probe = xics_smp_probe,
> + .cause_ipi = NULL, /* Filled at runtime by xi{cs,ve}_smp_probe() */
> + .probe = pnv_smp_probe,
> + .prepare_cpu = pnv_smp_prepare_cpu,
> .kick_cpu = pnv_smp_kick_cpu,
> .setup_cpu = pnv_smp_setup_cpu,
> .cpu_bootable = pnv_cpu_bootable,
> diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig
> index 52dc165..caf882e 100644
> --- a/arch/powerpc/sysdev/Kconfig
> +++ b/arch/powerpc/sysdev/Kconfig
> @@ -28,6 +28,7 @@ config PPC_MSI_BITMAP
> default y if PPC_POWERNV
>
> source "arch/powerpc/sysdev/xics/Kconfig"
> +source "arch/powerpc/sysdev/xive/Kconfig"
>
> config PPC_SCOM
> bool
> diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile
> index a254824..c0ae11d 100644
> --- a/arch/powerpc/sysdev/Makefile
> +++ b/arch/powerpc/sysdev/Makefile
> @@ -71,5 +71,6 @@ obj-$(CONFIG_PPC_EARLY_DEBUG_MEMCONS) += udbg_memcons.o
> subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
>
> obj-$(CONFIG_PPC_XICS) += xics/
> +obj-$(CONFIG_PPC_XIVE) += xive/
>
> obj-$(CONFIG_GE_FPGA) += ge/
> diff --git a/arch/powerpc/sysdev/xive/Kconfig b/arch/powerpc/sysdev/xive/Kconfig
> new file mode 100644
> index 0000000..c8816c8
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Kconfig
> @@ -0,0 +1,7 @@
> +config PPC_XIVE
> + def_bool n
> + select PPC_SMP_MUXED_IPI
> + select HARDIRQS_SW_RESEND
> +
> +config PPC_XIVE_NATIVE
> + def_bool n
> diff --git a/arch/powerpc/sysdev/xive/Makefile b/arch/powerpc/sysdev/xive/Makefile
> new file mode 100644
> index 0000000..3fab303
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Makefile
> @@ -0,0 +1,4 @@
> +subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
> +
> +obj-y += common.o
> +obj-$(CONFIG_PPC_XIVE_NATIVE) += native.o
> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
> new file mode 100644
> index 0000000..96037e0
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/common.c
> @@ -0,0 +1,1175 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +#include <linux/types.h>
> +#include <linux/threads.h>
> +#include <linux/kernel.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
> +#include <linux/init.h>
> +#include <linux/cpu.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/msi.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/machdep.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/xmon.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#undef DEBUG_FLUSH
> +#undef DEBUG_ALL
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +#ifdef DEBUG_ALL
> +#define DBG_VERBOSE(fmt...) pr_devel("XIVE: " fmt)
> +#else
> +#define DBG_VERBOSE(fmt...) do { } while(0)
> +#endif
> +
> +bool __xive_enabled;
> +bool xive_cmdline_disabled;
> +
> +/* We use only one priority for now */
> +static u8 xive_irq_priority;
> +
> +void __iomem *xive_tm_area;
> +u32 xive_tm_offset;
> +static const struct xive_ops *xive_ops;
> +static struct irq_domain *xive_irq_domain;
> +
> +/* The IPIs all use the same logical irq number */
> +static u32 xive_ipi_irq;
> +
> +/* Xive state for each CPU */
> +static DEFINE_PER_CPU(struct xive_cpu *, xive_cpu);
> +
> +/*
> + * A "disabled" interrupt should never fire, to catch problems
> + * we set its logical number to this
> + */
> +#define XIVE_BAD_IRQ 0x7fffffff
> +#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
> +
> +/* An invalid CPU target */
> +#define XIVE_INVALID_TARGET (-1)
> +
> +static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
A brief comment here explaining what the function does would be
helpful. Likewise for xive_scan_interrupts(), etc. etc.
The 'prio' argument doesn't seem to be used.
> +{
> + u32 cur;
> +
> + if (!q->qpage)
> + return 0;
> + cur = be32_to_cpup(q->qpage + q->idx);
> + if ((cur >> 31) = q->toggle)
> + return 0;
> + if (!just_peek) {
> + q->idx = (q->idx + 1) & q->msk;
> + if (q->idx = 0)
> + q->toggle ^= 1;
> + }
> + return cur & 0x7fffffff;
> +}
> +
> +static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
> +{
> + u32 hirq = 0;
> + u8 prio;
> +
> + /* Find highest pending priority */
> + while (xc->pending_prio != 0) {
> + struct xive_q *q;
> +
> + prio = ffs(xc->pending_prio) - 1;
> + DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
> +
> + /* Try to fetch */
> + hirq = xive_read_eq(&xc->queue[prio], prio, just_peek);
> +
> + /* Found something ? That's it */
> + if (hirq)
> + break;
> +
> + /* Clear pending bits */
> + xc->pending_prio &= ~(1 << prio);
> +
> + /*
> + * Check if the queue count needs adjusting due to
> + * interrupts being moved away.
> + */
> + q = &xc->queue[prio];
> + if (atomic_read(&q->pending_count)) {
> + int p = atomic_xchg(&q->pending_count, 0);
> + if (p) {
> + WARN_ON(p > atomic_read(&q->count));
> + atomic_sub(p, &q->count);
> + }
> + }
> + }
> +
> + /* If nothing was found, set CPPR to 0xff */
> + if (hirq = 0)
> + prio = 0xff;
> +
> + /* Update HW CPPR to match if necessary */
> + if (prio != xc->cppr) {
> + DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n", prio);
> + xc->cppr = prio;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, prio);
> + }
> +
> + return hirq;
> +}
> +
> +#ifdef CONFIG_XMON
> +static void xive_dump_eq(const char *name, struct xive_q *q)
> +{
> + u32 i0, i1, idx;
> +
> + if (!q->qpage)
> + return;
> + idx = q->idx;
> + i0 = be32_to_cpup(q->qpage + idx);
> + idx = (idx + 1) & q->msk;
> + i1 = be32_to_cpup(q->qpage + idx);
> + xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
> + q->toggle, i0, i1);
> +}
> +
> +void xmon_xive_do_dump(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_irq_data *xd;
> + uint64_t val, offset;
> +
> + xmon_printf("XIVE state for CPU %d:\n", cpu);
> + xmon_printf(" pp=%02x cppr=%02x\n", xc->pending_prio, xc->cppr);
> + xive_dump_eq("IRQ", &xc->queue[xive_irq_priority]);
> + xd = &xc->ipi_data;
> + offset = 0x800;
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> + val = in_be64(xd->eoi_mmio + offset);
> + xmon_printf(" IPI state: %x:%c%c\n", xc->hw_ipi,
> + val & 2 ? 'P' : 'p',
> + val & 1 ? 'Q' : 'q');
> +}
> +#endif /* CONFIG_XMON */
> +
> +static void xive_update_pending_irqs(struct xive_cpu *xc)
> +{
> + u8 he, cppr;
> + u16 ack;
> +
> + /* Perform the acknowledge hypervisor to register cycle */
> + ack = be16_to_cpu(__raw_readw(xive_tm_area + TM_SPC_ACK_HV_REG));
This sounds like it's something that only the hypervisor should do, so
it's not clear why it's in common code.
> + /* Synchronize subsequent queue accesses */
> + mb();
> +
> + DBG_VERBOSE("CPU %d get_irq, ack=%04x\n", smp_processor_id(), ack);
> +
> + /* Check the HE field */
What is a "HE" field?
> + cppr = ack & 0xff;
> + he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8));
> + switch(he) {
> + case TM_QW3_NSR_HE_NONE:
> + break;
> + case TM_QW3_NSR_HE_PHYS:
> + if (cppr = 0xff)
> + return;
> + xc->pending_prio |= 1 << cppr;
> + if (cppr >= xc->cppr)
> + pr_err("XIVE: CPU %d odd ack CPPR, got %d at %d\n",
> + smp_processor_id(), cppr, xc->cppr);
> + xc->cppr = cppr;
> + break;
> + case TM_QW3_NSR_HE_POOL:
> + case TM_QW3_NSR_HE_LSI:
> + pr_err("XIVE: CPU %d got unexpected interrupt type HE=%d\n",
> + smp_processor_id(), he);
> + return;
> + }
> +}
> +
> +static unsigned int xive_get_irq(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + u32 hirq;
> +
> + /*
> + * This can be called either as a result of a HW interrupt or
> + * as a "replay" because EOI decided there was still something
> + * in one of the queues.
> + *
> + * First we perform an ACK cycle in order to update our mask
> + * of pending priorities. This will also have the effect of
> + * updating the CPPR to the most favored pending interrupts.
> + *
> + * In the future, if we have a way to differenciate a first
> + * entry (on HW interrupt) from a replay triggered by EOI,
> + * we could skip this on replays unless we soft-mask tells us
> + * that a new HW interrupt occurred.
> + */
> + xive_update_pending_irqs(xc);
> +
> + DBG_VERBOSE("get_irq: pending=%02x\n", xc->pending_prio);
> +
> + hirq = xive_scan_interrupts(xc, false);
> +
> + DBG_VERBOSE("get_irq: got irq 0x%x, new pending=0x%02x\n",
> + hirq, xc->pending_prio);
> +
> + /* Return pending interrupt if any */
> + if (hirq = XIVE_BAD_IRQ)
> + return 0;
> + return hirq;
> +}
> +
> +
> +static void xive_do_queue_eoi(struct xive_cpu *xc)
> +{
> + if (xive_scan_interrupts(xc, true) != 0) {
> + DBG_VERBOSE("eoi: pending=0x%02x\n", xc->pending_prio);
> + force_external_irq_replay();
> + }
> +}
> +
> +static u8 xive_poke_esb(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = in_be64(xd->eoi_mmio + offset);
> +
> + return (u8)val;
> +}
> +
> +static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd)
> +{
> + /* If the XIVE supports the new "store EOI facility, use it */
> + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> + out_be64(xd->eoi_mmio, 0);
> + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> + if (WARN_ON_ONCE(!xive_ops->eoi))
> + return;
> + xive_ops->eoi(hw_irq);
> + } else {
> + uint8_t eoi_val;
> +
> + /*
> + * Otherwise for EOI, we use the special MMIO that does
> + * a clear of both P and Q and returns the old Q.
> + *
> + * This allows us to then do a re-trigger if Q was set
> + * rather than synthetizing an interrupt in software
^^^^^^^^^^^^ synthesizing
> + */
> + eoi_val = xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> + DBG_VERBOSE("eoi_val=%x\n", offset, eoi_val);
> +
> + if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
> + return;
> +
> + /* Re-trigger */
> + if (xd->trig_mmio)
> + out_be64(xd->trig_mmio, 0);
> + }
> +
> +}
> +
> +static void xive_irq_eoi(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
> + d->irq, irqd_to_hwirq(d), xc->pending_prio);
> +
> + if (!irqd_irq_disabled(d))
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> +
> + /*
> + * Clear saved_p to indicate that it's no longer occupying
> + * a queue slot on the target queue
> + */
> + xd->saved_p = false;
> +
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_do_source_set_mask(struct xive_irq_data *xd,
> + bool masked)
> +{
> + if (masked)
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_01);
> + else
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> +}
> +
> +static bool xive_try_pick_target(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> + int max;
> +
> + /* Calculate max number of interrupts in that queue.
> + *
> + * We leave a gap of 1 just in case...
> + */
> + max = (q->msk + 1) - 1;
> + return !!atomic_add_unless(&q->count, 1, max);
> +}
> +
> +static void xive_dec_target_count(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> +
> + if (WARN_ON(cpu < 0))
> + return;
> +
> + /*
> + * We increment the "pending count" which will be used
> + * to decrement the target queue count whenever it's next
> + * processed and found empty. This ensure that we don't
> + * decrement while we still have the interrupt there
> + * occupying a slot.
> + */
> + atomic_inc(&q->pending_count);
> +}
> +
> +static int xive_find_target_in_mask(const struct cpumask *mask,
> + unsigned int fuzz)
> +{
> + int cpu, first, num, i;
> +
> + /* Pick up a starting point CPU in the mask based on fuzz */
> + num = cpumask_weight(mask);
> + first = (fuzz++) % num;
Incrementing fuzz here is not very useful, given it's a parameter and
it's not used subsequently in this function.
> +
> + /* Locate it */
> + cpu = cpumask_first(mask);
> + for (i = 0; i < first; i++)
> + cpu = cpumask_next(cpu, mask);
> + first = cpu;
> +
> + /*
> + * Now go through the entire mask until we find a valid
> + * target.
> + */
> + for (;;) {
> + /*
> + * We re-check online as the fallback case passes us
> + * an untested affinity mask
> + */
> + if (cpu_online(cpu) && xive_try_pick_target(cpu))
> + return cpu;
> + cpu = cpumask_next(cpu, mask);
Don't we need to wrap around from the end of the mask to the beginning
here? i.e. if (cpu = (none found)) cpu = cpumask_first(mask)
> + if (cpu = first)
> + break;
> + }
> + return -1;
> +}
> +
> +static int xive_pick_irq_target(struct irq_data *d,
> + const struct cpumask *affinity)
> +{
> + static unsigned int fuzz;
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + cpumask_var_t mask;
> + int cpu = -1;
> +
> + /*
> + * Pick a target CPU for an interrupt. This is done at
> + * startup or if the affinity is changed in a way that
> + * invalidates the current target.
> + */
> +
> + /* If we have chip IDs, first we try to build a mask of
> + * CPUs matching ther CPU and find a target in there
> + */
> + if (xd->src_chip != XIVE_INVALID_CHIP_ID &&
> + zalloc_cpumask_var(&mask, GFP_ATOMIC)) {
> + /* Build a mask of matching chip IDs */
> + for_each_cpu_and(cpu, affinity, cpu_online_mask) {
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + if (xc->chip_id = xd->src_chip)
> + cpumask_set_cpu(cpu, mask);
> + }
> + /* Try to find a target */
> + if (!cpumask_empty(mask))
> + cpu = xive_find_target_in_mask(mask, fuzz++);
> + free_cpumask_var(mask);
> + if (cpu >= 0)
> + return cpu;
> + fuzz--;
> + }
> +
> + /* No chip IDs, fallback to using the affinity mask */
> + return xive_find_target_in_mask(affinity, fuzz++);
> +}
> +
> +static unsigned int xive_irq_startup(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + int target, rc;
> +
> + DBG("xive_irq_startup: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> +#ifdef CONFIG_PCI_MSI
> + /*
> + * The generic MSI code returns with the interrupt disabled on the
> + * card, using the MSI mask bits. Firmware doesn't appear to unmask
> + * at that level, so we do it here by hand.
> + */
> + if (irq_data_get_msi_desc(d))
> + pci_msi_unmask_irq(d);
> +#endif
> +
> + /* Pick a target */
> + target = xive_pick_irq_target(d, irq_data_get_affinity_mask(d));
> + if (target = XIVE_INVALID_TARGET) {
> + /* Try again breaking affinity */
> + target = xive_pick_irq_target(d, cpu_online_mask);
> + if (target = XIVE_INVALID_TARGET)
> + return -ENXIO;
> + pr_warn("XIVE: irq %d started with broken affinity\n",
> + d->irq);
> + }
> + xd->target = target;
> +
> + /*
> + * Configure the logical number to be the Linux IRQ number
> + * and set the target queue
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc)
> + return rc;
> +
> + /* Unmask the ESB */
> + xive_do_source_set_mask(xd, false);
> +
> + return 0;
> +}
> +
> +static void xive_irq_shutdown(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + DBG("xive_irq_shutdown: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> + if (WARN_ON(xd->target = XIVE_INVALID_TARGET))
> + return;
> +
> + /* Mask the interrupt at the source */
> + xive_do_source_set_mask(xd, true);
> +
> + /* Mask the interrupt in HW in the IVT/EAS */
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, hw_irq);
> +
> + xive_dec_target_count(xd->target);
> + xd->target = XIVE_INVALID_TARGET;
> +}
> +
> +static void xive_irq_unmask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_unmask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call FW to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + xive_irq_priority, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, false);
> +}
> +
> +static void xive_irq_mask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_mask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call OPAL to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, true);
> +}
> +
> +static int xive_irq_set_affinity(struct irq_data *d,
> + const struct cpumask *cpumask,
> + bool force)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + u32 target, old_target;
> + int rc = 0;
> +
> + DBG("xive_irq_set_affinity: irq %d\n", d->irq);
> +
> + /* Is this valid ? */
> + if (cpumask_any_and(cpumask, cpu_online_mask) >= nr_cpu_ids)
> + return -EINVAL;
> +
> + /* If existing target is already in the new mask, and is
> + * online then do nothing.
> + */
> + if (cpu_online(xd->target) &&
> + cpumask_test_cpu(xd->target, cpumask))
> + return IRQ_SET_MASK_OK;
> +
> + /* Pick a new target */
> + target = xive_pick_irq_target(d, cpumask);
> +
> + /* No target found */
> + if (target = XIVE_INVALID_TARGET)
> + return -ENXIO;
> +
> + old_target = xd->target;
> +
> + /*
> + * Only configure the irq if it's not currently passed-through to
> + * a KVM guest
Does the code here implement that, or is it a requirement on
xive_ops->configure_irq, or is it not implemented in this patch but
comes along in a later patch?
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
> + return rc;
> + }
> +
> + DBG(" target: 0x%x\n", target);
> + xd->target = target;
> +
> + /* Give up previous target */
> + if (old_target != XIVE_INVALID_TARGET)
> + xive_dec_target_count(old_target);
> +
> + return IRQ_SET_MASK_OK;
> +}
> +
> +static int xive_irq_set_type(struct irq_data *d, unsigned int flow_type)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /*
> + * We only support these. This has really no effect other than setting
> + * the corresponding descriptor bits mind you but those will in turn
> + * affect the resend function when re-enabling an edge interrupt.
> + *
> + * Set set the default to edge as explained in map().
> + */
> + if (flow_type = IRQ_TYPE_DEFAULT || flow_type = IRQ_TYPE_NONE)
> + flow_type = IRQ_TYPE_EDGE_RISING;
> +
> + if (flow_type != IRQ_TYPE_EDGE_RISING &&
> + flow_type != IRQ_TYPE_LEVEL_LOW)
> + return -EINVAL;
> +
> + irqd_set_trigger_type(d, flow_type);
> +
> + /*
> + * Double check it matches what the FW thinks
> + *
> + * NOTE: We don't know yet if the PAPR interface will provide
> + * the LSI vs MSI information appart from the device-tree so
^^^^^^ apart
> + * this check might have to move into an optional backend call
> + * that is specific to the native backend
> + */
> + if ((flow_type = IRQ_TYPE_LEVEL_LOW) !> + !!(xd->flags & XIVE_IRQ_FLAG_LSI))
> + pr_warn("XIVE: Interrupt %d (HW 0x%x) type mismatch,"
> + " Linux says %s, FW says %s\n",
> + d->irq, (u32)irqd_to_hwirq(d),
> + (flow_type = IRQ_TYPE_LEVEL_LOW) ? "Level" : "Edge",
> + (xd->flags & XIVE_IRQ_FLAG_LSI) ? "Level" : "Edge");
> +
> + return IRQ_SET_MASK_OK_NOCOPY;
> +}
> +
> +static int xive_irq_retrigger(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /* This should be only for MSIs */
> + if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
> + return 0;
> +
> + /*
> + * To perform a retrigger, we first set the PQ bits to
> + * 11, then perform an EOI.
> + */
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_11);
> +
> + /*
> + * Note: We pass "0" to the hw_irq argument in order to
> + * avoid calling into the backend EOI code which we don't
> + * want to do in the case of a re-trigger. Backends typically
> + * only do EOI for LSIs anyway.
> + */
> + xive_do_source_eoi(0, xd);
> +
> + return 1;
> +}
> +
> +static struct irq_chip xive_irq_chip = {
> + .name = "XIVE-IRQ",
> + .irq_startup = xive_irq_startup,
> + .irq_shutdown = xive_irq_shutdown,
> + .irq_eoi = xive_irq_eoi,
> + .irq_mask = xive_irq_mask,
> + .irq_unmask = xive_irq_unmask,
> + .irq_set_affinity = xive_irq_set_affinity,
> + .irq_set_type = xive_irq_set_type,
> + .irq_retrigger = xive_irq_retrigger,
> +};
> +
> +bool is_xive_irq(struct irq_chip *chip)
> +{
> + return chip = &xive_irq_chip;
> +}
> +
> +void xive_cleanup_irq_data(struct xive_irq_data *xd)
> +{
> + if (xd->eoi_mmio) {
> + iounmap(xd->eoi_mmio);
> + if (xd->eoi_mmio = xd->trig_mmio)
> + xd->trig_mmio = NULL;
> + xd->eoi_mmio = NULL;
> + }
> + if (xd->trig_mmio) {
> + iounmap(xd->trig_mmio);
> + xd->trig_mmio = NULL;
> + }
> +}
> +
> +static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
> +{
> + struct xive_irq_data *xd;
> + int rc;
> +
> + xd = kzalloc(sizeof(struct xive_irq_data), GFP_KERNEL);
> + if (!xd)
> + return -ENOMEM;
> + rc = xive_ops->populate_irq_data(hw, xd);
> + if (rc) {
> + kfree(xd);
> + return rc;
> + }
> + xd->target = XIVE_INVALID_TARGET;
> + irq_set_handler_data(virq, xd);
> +
> + return 0;
> +}
> +
> +static void xive_irq_free_data(unsigned int virq)
> +{
> + struct xive_irq_data *xd = irq_get_handler_data(virq);
> +
> + if (!xd)
> + return;
> + irq_set_handler_data(virq, NULL);
> + xive_cleanup_irq_data(xd);
> + kfree(xd);
> +}
> +
> +#ifdef CONFIG_SMP
> +
> +static void xive_cause_ipi(int cpu, unsigned long msg)
> +{
> + struct xive_cpu *xc;
> + struct xive_irq_data *xd;
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + DBG_VERBOSE("IPI msg#%ld CPU %d -> %d (HW IRQ 0x%x)\n",
> + msg, smp_processor_id(), cpu, xc->hw_ipi);
> +
> + xd = &xc->ipi_data;
> + if (WARN_ON(!xd->trig_mmio))
> + return;
> + out_be64(xd->trig_mmio, 0);
> +}
> +
> +static irqreturn_t xive_muxed_ipi_action(int irq, void *dev_id)
> +{
> + return smp_ipi_demux();
> +}
> +
> +static void xive_ipi_eoi(struct irq_data *d)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Handle possible race with unplug and drop stale IPIs */
> + if (!xc)
> + return;
> + xive_do_source_eoi(xc->hw_ipi, &xc->ipi_data);
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_ipi_unmask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static void xive_ipi_mask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static struct irq_chip xive_ipi_chip = {
> + .name = "XIVE-IPI",
> + .irq_eoi = xive_ipi_eoi,
> + .irq_mask = xive_ipi_mask,
> + .irq_unmask = xive_ipi_unmask,
> +};
> +
> +static void __init xive_request_ipi(void)
> +{
> + unsigned int virq;
> +
> + /* Initialize it */
> + virq = irq_create_mapping(xive_irq_domain, 0);
> + xive_ipi_irq = virq;
> +
> + BUG_ON(request_irq(virq, xive_muxed_ipi_action,
> + IRQF_PERCPU | IRQF_NO_THREAD, "IPI", NULL));
> +}
> +
> +static int xive_setup_cpu_ipi(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> + int rc;
> +
> + pr_debug("XIVE: Setting up IPI for CPU %d\n", cpu);
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + /* Check if we are already setup */
> + if (xc->hw_ipi != 0)
> + return 0;
> +
> + /* Grab an IPI from the backend, this will populate xc->hw_ipi */
> + if (xive_ops->get_ipi(cpu, xc))
> + return -EIO;
> +
> + /* Populate the IRQ data in the xive_cpu structure and
> + * configure the HW / enable the IPIs
> + */
> + rc = xive_ops->populate_irq_data(xc->hw_ipi, &xc->ipi_data);
> + if (rc) {
> + pr_err("XIVE: Failed to populate IPI data on CPU %d\n", cpu);
> + return -EIO;
> + }
> + rc = xive_ops->configure_irq(xc->hw_ipi,
> + get_hard_smp_processor_id(cpu),
> + xive_irq_priority, xive_ipi_irq);
> + if (rc) {
> + pr_err("XIVE: Failed to map IPI CPU %d\n", cpu);
> + return -EIO;
> + }
> + DBG("XIVE: CPU %d HW IPI %x, virq %d, trig_mmio=%p\n", cpu,
> + xc->hw_ipi, xive_ipi_irq, xc->ipi_data.trig_mmio);
> +
> + /* Unmask it */
> + xive_do_source_set_mask(&xc->ipi_data, false);
> +
> + return 0;
> +}
> +
> +static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + /* Disable the IPI and free the IRQ data */
> +
> + /* Already cleaned up ? */
> + if (xc->hw_ipi = 0)
> + return;
> +
> + /* Mask the IPI */
> + xive_do_source_set_mask(&xc->ipi_data, true);
> +
> + /*
> + * Note: We don't call xive_cleanup_irq_data() to free
> + * the mappings as this is called from an IPI on kexec
> + * which is not a safe environment to call iounmap()
> + */
> +
> + /* Deconfigure/mask in the backend */
> + xive_ops->configure_irq(xc->hw_ipi, hard_smp_processor_id(),
> + 0xff, xive_ipi_irq);
> +
> + /* Free the IPIs in the backend */
> + xive_ops->put_ipi(cpu, xc);
> +}
> +
> +void __init xive_smp_probe(void)
> +{
> + smp_ops->cause_ipi = xive_cause_ipi;
> +
> + /* Register the IPI */
> + xive_request_ipi();
> +
> + /* Allocate and setup IPI for the boot CPU */
> + xive_setup_cpu_ipi(smp_processor_id());
> +}
> +
> +#endif /* CONFIG_SMP */
> +
> +static int xive_irq_domain_map(struct irq_domain *h, unsigned int virq,
> + irq_hw_number_t hw)
> +{
> + int rc;
> +
> + /*
> + * Mark interrupts as edge sensitive by default so that resend
> + * actually works. Will fix that up below if needed.
> + */
> + irq_clear_status_flags(virq, IRQ_LEVEL);
> +
> + /* IPIs are special and come up with HW number 0 */
> + if (hw = 0) {
> + /*
> + * IPIs are marked per-cpu. We use separate HW interrupts under
> + * the hood but associated with the same "linux" interrupt
> + */
> + irq_set_chip_and_handler(virq, &xive_ipi_chip,
> + handle_percpu_irq);
> + return 0;
> + }
> +
> + rc = xive_irq_alloc_data(virq, hw);
> + if (rc)
> + return rc;
> +
> + irq_set_chip_and_handler(virq, &xive_irq_chip, handle_fasteoi_irq);
> +
> + return 0;
> +}
> +
> +static void xive_irq_domain_unmap(struct irq_domain *d, unsigned int virq)
> +{
> + struct irq_data *data = irq_get_irq_data(virq);
> + unsigned int hw_irq;
> +
> + if (!data)
> + return;
> + hw_irq = (unsigned int)irqd_to_hwirq(data);
> + if (hw_irq)
> + xive_irq_free_data(virq);
> +}
> +
> +static int xive_irq_domain_xlate(struct irq_domain *h, struct device_node *ct,
> + const u32 *intspec, unsigned int intsize,
> + irq_hw_number_t *out_hwirq, unsigned int *out_flags)
> +
> +{
> + *out_hwirq = intspec[0];
> +
> + /*
> + * If intsize is at least 2, we look for the type in the second cell,
> + * we assume the LSB indicates a level interrupt.
> + */
> + if (intsize > 1) {
> + if (intspec[1] & 1)
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> + else
> + *out_flags = IRQ_TYPE_EDGE_RISING;
> + } else
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> +
> + return 0;
> +}
> +
> +static int xive_irq_domain_match(struct irq_domain *h, struct device_node *node,
> + enum irq_domain_bus_token bus_token)
> +{
> + return xive_ops->match(node);
> +}
> +
> +static const struct irq_domain_ops xive_irq_domain_ops = {
> + .match = xive_irq_domain_match,
> + .map = xive_irq_domain_map,
> + .unmap = xive_irq_domain_unmap,
> + .xlate = xive_irq_domain_xlate,
> +};
> +
> +static void __init xive_init_host(void)
> +{
> + xive_irq_domain = irq_domain_add_nomap(NULL, XIVE_MAX_IRQ,
> + &xive_irq_domain_ops, NULL);
> + BUG_ON(xive_irq_domain = NULL);
> + irq_set_default_host(xive_irq_domain);
> +}
> +
> +static void xive_cleanup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + if (xc->queue[xive_irq_priority].qpage)
> + xive_ops->cleanup_queue(cpu, xc, xive_irq_priority);
> +}
> +
> +static int xive_setup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + int rc = 0;
> +
> + /* We setup 1 queues for now with a 64k page */
> + if (!xc->queue[xive_irq_priority].qpage)
> + rc = xive_ops->setup_queue(cpu, xc, xive_irq_priority);
> +
> + return rc;
> +}
> +
> +static int xive_prepare_cpu(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> +
> + xc = per_cpu(xive_cpu, cpu);
> + if (!xc) {
> + struct device_node *np;
> +
> + xc = kzalloc_node(sizeof(struct xive_cpu),
> + GFP_KERNEL, cpu_to_node(cpu));
> + if (!xc)
> + return -ENOMEM;
> + np = of_get_cpu_node(cpu, NULL);
> + if (np)
> + xc->chip_id = of_get_ibm_chip_id(np);
> + of_node_put(np);
> +
> + per_cpu(xive_cpu, cpu) = xc;
> + }
> +
> + /* Setup EQs if not already */
> + return xive_setup_cpu_queues(cpu, xc);
> +}
> +
> +static void xive_setup_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Debug: Dump the TM state */
> + DBG("CPU %d [HW 0x%02x] VT=%02x\n",
> + smp_processor_id(), hard_smp_processor_id(),
> + in_8(xive_tm_area + xive_tm_offset + TM_WORD2));
> +
> + /* The backend might have additional things to do */
> + if (xive_ops->setup_cpu)
> + xive_ops->setup_cpu(smp_processor_id(), xc);
> +
> + /* Set CPPR to 0xff to enable flow of interrupts */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +#ifdef CONFIG_SMP
> +void xive_smp_setup_cpu(void)
> +{
> + DBG("XIVE: SMP setup CPU %d\n", smp_processor_id());
> +
> + /* This will have already been done on the boot CPU */
> + if (smp_processor_id() != boot_cpuid)
> + xive_setup_cpu();
> +
> +}
> +
> +int xive_smp_prepare_cpu(unsigned int cpu)
> +{
> + int rc;
> +
> + /* Allocate per-CPU data and queues */
> + rc = xive_prepare_cpu(cpu);
> + if (rc)
> + return rc;
> +
> + /* Allocate and setup IPI for the new CPU */
> + return xive_setup_cpu_ipi(cpu);
> +}
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +static void xive_flush_cpu_queue(unsigned int cpu, struct xive_cpu *xc)
> +{
> + u32 irq;
> +
> + /* We assume local irqs are disabled */
> + WARN_ON(!irqs_disabled());
> +
> + /* Check what's already in the CPU queue */
> + while ((irq = xive_scan_interrupts(xc, false)) != 0) {
> + /*
> + * We need to re-route that interrupt to its new distination.
destination ^^^^^^^^^^^
> + * First get and lock the descriptor
> + */
> + struct irq_desc *desc = irq_to_desc(irq);
> + struct irq_data *d = irq_desc_get_irq_data(desc);
> + struct xive_irq_data *xd;
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + /*
> + * Ignore anything that isn't a XIVE irq and ignore
> + * IPIs, so can just be dropped.
> + */
> + if (d->domain != xive_irq_domain || hw_irq = 0)
> + continue;
> +#ifdef DEBUG_FLUSH
> + pr_info("CPU %d: Got irq %d while offline, re-routing...\n",
> + cpu, irq);
> +#endif
> + raw_spin_lock(&desc->lock);
> + xd = irq_desc_get_handler_data(desc);
What actually does the rerouting here? Nothing in this loop has a
name that would lead me to think that it is the thing that is doing
the rerouting.
> +
> + /* For LSIs, we EOI, this will cause a resend if it's
> + * still asserted. Otherwise do an MSI retrigger
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_LSI)
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> + else
> + xive_irq_retrigger(d);
> + raw_spin_unlock(&desc->lock);
> + }
> +}
> +
> +void xive_smp_disable_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Migrate interrupts away from the CPU */
> + irq_migrate_all_off_this_cpu();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Flush everything still in the queue */
> + xive_flush_cpu_queue(cpu, xc);
> +
> + /* Re-enable CPPR */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +void xive_flush_interrupt(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Called if an interrupt occurs while the CPU is hot unplugged */
> + xive_flush_cpu_queue(cpu, xc);
> +}
> +
> +#endif /* CONFIG_HOTPLUG_CPU */
> +
> +#endif /* CONFIG_SMP */
> +
> +void xive_kexec_teardown_cpu(int secondary)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Backend cleanup if any */
> + if (xive_ops->teardown_cpu)
> + xive_ops->teardown_cpu(cpu, xc);
> +
> + /* Get rid of IPI */
> + xive_cleanup_cpu_ipi(cpu, xc);
> +
> + /* Disable and free the queues */
> + xive_cleanup_cpu_queues(cpu, xc);
> +}
> +
> +void xive_shutdown(void)
> +{
> + xive_ops->shutdown();
> +}
> +
> +bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
> + u8 max_prio)
> +{
> + xive_tm_area = area;
> + xive_tm_offset = offset;
> + xive_ops = ops;
> + xive_irq_priority = max_prio;
> +
> + ppc_md.get_irq = xive_get_irq;
> + __xive_enabled = true;
> +
> + DBG("Initializing host..\n");
> + xive_init_host();
> +
> + DBG("Initializing boot CPU..\n");
> +
> + /* Allocate per-CPU data and queues */
> + xive_prepare_cpu(smp_processor_id());
> +
> + /* Get ready for interrupts */
> + xive_setup_cpu();
> +
> + pr_info("XIVE: Interrupt handling intialized with %s backend\n",
> + xive_ops->name);
> + pr_info("XIVE: Using priority %d for all interrupts\n", max_prio);
> +
> + return true;
> +}
> +
> +static int __init xive_off(char *arg)
> +{
> + xive_cmdline_disabled = true;
> + return 0;
> +}
> +__setup("xive=off", xive_off);
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> new file mode 100644
> index 0000000..26cc6bf
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -0,0 +1,604 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +#include <linux/types.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
> +#include <linux/init.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/cpumask.h>
> +#include <linux/mm.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/opal.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +/* Enable this for using queue MMIO page for EOI. We don't currently
> + * use it as we always notify
> + */
> +#undef USE_QUEUE_MMIO
> +
> +static u32 xive_provision_size;
> +static u32 *xive_provision_chips;
> +static u32 xive_provision_chip_count;
> +static u32 xive_queue_shift;
> +static u32 xive_pool_vps = XIVE_INVALID_VP;
> +static struct kmem_cache *xive_provision_cache;
> +
> +int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
> +{
> + __be64 flags, eoi_page, trig_page;
> + __be32 esb_shift, src_chip;
> + u64 opal_flags;
> + s64 rc;
> +
> + memset(data, 0, sizeof(*data));
> +
> + rc = opal_xive_get_irq_info(hw_irq, &flags, &eoi_page, &trig_page,
> + &esb_shift, &src_chip);
> + if (rc) {
> + pr_err("XIVE: opal_xive_get_irq_info(0x%x) returned %lld\n",
> + hw_irq, rc);
> + return -EINVAL;
> + }
> +
> + opal_flags = be64_to_cpu(flags);
> + if (opal_flags & OPAL_XIVE_IRQ_STORE_EOI)
> + data->flags |= XIVE_IRQ_FLAG_STORE_EOI;
> + if (opal_flags & OPAL_XIVE_IRQ_LSI)
> + data->flags |= XIVE_IRQ_FLAG_LSI;
> + if (opal_flags & OPAL_XIVE_IRQ_SHIFT_BUG)
> + data->flags |= XIVE_IRQ_FLAG_SHIFT_BUG;
> + if (opal_flags & OPAL_XIVE_IRQ_MASK_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_MASK_FW;
> + if (opal_flags & OPAL_XIVE_IRQ_EOI_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_EOI_FW;
> + data->eoi_page = be64_to_cpu(eoi_page);
> + data->trig_page = be64_to_cpu(trig_page);
> + data->esb_shift = be32_to_cpu(esb_shift);
> + data->src_chip = be32_to_cpu(src_chip);
> +
> + data->eoi_mmio = ioremap(data->eoi_page, 1u << data->esb_shift);
> + if (!data->eoi_mmio) {
> + pr_err("XIVE: Failed to map EOI page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> +
> + if (!data->trig_page)
> + return 0;
> + if (data->trig_page = data->eoi_page) {
> + data->trig_mmio = data->eoi_mmio;
> + return 0;
> + }
> +
> + data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
> + if (!data->trig_mmio) {
> + pr_err("XIVE: Failed to map trigger page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> + return 0;
> +}
> +
> +int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_set_irq_config(hw_irq, target, prio, sw_irq);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + return rc = 0 ? 0 : -ENXIO;
> +}
> +
> +/* This can be called multiple time to change a queue configuration */
> +int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate)
> +{
> + s64 rc = 0;
> + __be64 qeoi_page_be;
> + __be32 esc_irq_be;
> + u64 flags, qpage_phys;
> +
> + /* If there's an actual queue page, clean it */
> + if (order) {
> + BUG_ON(!qpage);
> + qpage_phys = __pa(qpage);
> + } else
> + qpage_phys = 0;
> +
> + /* Initialize the rest of the fields */
> + q->msk = order ? ((1u << (order - 2)) - 1) : 0;
> + q->idx = 0;
> + q->toggle = 0;
> +
> + rc = opal_xive_get_queue_info(vp_id, prio, NULL, NULL,
> + &qeoi_page_be,
> + &esc_irq_be,
> + NULL);
> + if (rc) {
> + pr_err("XIVE: Error %lld getting queue info prio %d\n",
> + rc, prio);
> + rc = -EIO;
> + goto fail;
> + }
> + q->eoi_phys = be64_to_cpu(qeoi_page_be);
> +
> +#ifdef USE_QUEUE_MMIO
> + if (!q->eoi_mmio)
> + q->eoi_mmio = ioremap(q->eoi_phys, PAGE_SIZE);
> + if (!q->eoi_mmio) {
> + pr_err("XIVE: Failed to map queue MMIO prio %d CPU %d\n",
> + rc, prio, cpu);
> + rc = -ENOMEM;
> + goto fail;
> + }
> +#endif /* USE_QUEUE_MMIO */
> +
> + /* Default flags */
> + flags = OPAL_XIVE_EQ_ALWAYS_NOTIFY | OPAL_XIVE_EQ_ENABLED;
> +
> + /* Escalation needed ? */
> + if (can_escalate) {
> + q->esc_irq = be32_to_cpu(esc_irq_be);
> + flags |= OPAL_XIVE_EQ_ESCALATE;
> + }
> +
> + /* Configure and enable the queue in HW */
> + for (;;) {
> + rc = opal_xive_set_queue_info(vp_id, prio, qpage_phys, order, flags);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + if (rc) {
> + pr_err("XIVE: Error %lld setting queue for prio %d\n",
> + rc, prio);
> + rc = -EIO;
> + } else {
> + /*
> + * KVM code requires all of the above to be visible before
> + * q->qpage is set due to how it manages IPI EOIs
> + */
> + wmb();
> + q->qpage = qpage;
> + }
> + fail:
> + return rc;
> +}
> +
> +static void __xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
> +{
> + s64 rc;
> +
> + /* Disable the queue in HW */
> + for (;;) {
> + rc = opal_xive_set_queue_info(vp_id, prio, 0, 0, 0);
> + break;
> + msleep(1);
> + }
> + if (rc)
> + pr_err("XIVE: Error %lld disabling queue for prio %d\n",
> + rc, prio);
> +}
> +
> +void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
> +{
> + __xive_native_disable_queue(vp_id, q, prio);
> +
> + if (q->eoi_mmio)
> + iounmap(q->eoi_mmio);
> + q->eoi_mmio = NULL;
> +}
> +
> +static int xive_native_setup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
> +{
> + struct xive_q *q = &xc->queue[prio];
> + unsigned int alloc_order;
> + struct page *pages;
> + __be32 *qpage;
> +
> + alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
> + (xive_queue_shift - PAGE_SHIFT) : 0;
> + pages = alloc_pages_node(cpu_to_node(cpu), GFP_KERNEL, alloc_order);
> + if (!pages)
> + return -ENOMEM;
> + qpage = (__be32 *)page_address(pages);
> + memset(qpage, 0, 1 << xive_queue_shift);
> + return xive_native_configure_queue(get_hard_smp_processor_id(cpu),
> + q, prio, qpage, xive_queue_shift, false);
> +}
> +
> +static void xive_native_cleanup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
> +{
> + struct xive_q *q = &xc->queue[prio];
> + unsigned int alloc_order;
> +
> + /*
> + * We use the variant with no iounmap as this is called on exec
> + * from an IPI and iounmap isn't safe
> + */
> + __xive_native_disable_queue(get_hard_smp_processor_id(cpu), q, prio);
> + alloc_order = (xive_queue_shift > PAGE_SHIFT) ?
> + (xive_queue_shift - PAGE_SHIFT) : 0;
> + free_pages((unsigned long)q->qpage, alloc_order);
> + q->qpage = NULL;
> +}
> +
> +static bool xive_native_match(struct device_node *node)
> +{
> + return of_device_is_compatible(node, "ibm,opal-xive-vc");
> +}
> +
> +#ifdef CONFIG_SMP
> +static int xive_native_get_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + struct device_node *np;
> + unsigned int chip_id;
> + s64 irq;
> +
> + /* Find the chip ID */
> + np = of_get_cpu_node(cpu, NULL);
> + if (np) {
> + if (of_property_read_u32(np, "ibm,chip-id", &chip_id) < 0)
> + chip_id = 0;
> + }
> +
> + /* Allocate an IPI and populate info about it */
> + for (;;) {
> + irq = opal_xive_allocate_irq(chip_id);
> + if (irq = OPAL_BUSY) {
> + msleep(1);
> + continue;
> + }
> + if (irq < 0) {
> + pr_err("XIVE: Failed to allocate IPI on CPU %d\n",
> + cpu);
> + return -ENXIO;
> + }
> + xc->hw_ipi = irq;
> + break;
> + }
> + return 0;
> +}
> +
> +u32 xive_native_alloc_irq(void)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_allocate_irq(OPAL_XIVE_ANY_CHIP);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + if (rc < 0)
> + return 0;
> + return rc;
> +}
> +
> +void xive_native_free_irq(u32 irq)
> +{
> + for (;;) {
> + s64 rc = opal_xive_free_irq(irq);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> +}
> +
> +static void xive_native_put_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + s64 rc;
> +
> + /* Free the IPI */
> + if (!xc->hw_ipi)
> + return;
> + for (;;) {
> + rc = opal_xive_free_irq(xc->hw_ipi);
> + if (rc = OPAL_BUSY) {
> + msleep(1);
> + continue;
> + }
> + xc->hw_ipi = 0;
> + break;
> + }
> +}
> +#endif /* CONFIG_SMP */
> +
> +static void xive_native_shutdown(void)
> +{
> + /* Switch the XIVE to emulation mode */
> + opal_xive_reset(OPAL_XIVE_MODE_EMU);
> +}
> +
> +static void xive_native_eoi(u32 hw_irq)
> +{
> + /* Not normally used except if specific interrupts need
> + * a workaround on EOI
> + */
> + opal_int_eoi(hw_irq);
> +}
> +
> +static void xive_native_setup_cpu(unsigned int cpu, struct xive_cpu *xc)
> +{
> + s64 rc;
> + u32 vp;
> + __be64 vp_cam_be;
> + u64 vp_cam;
> +
> + if (xive_pool_vps = XIVE_INVALID_VP)
> + return;
> +
> + /* Enable the pool VP */
> + vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
> + pr_debug("XIVE: CPU %d setting up pool VP 0x%x\n", cpu, vp);
> + for (;;) {
> + rc = opal_xive_set_vp_info(vp, OPAL_XIVE_VP_ENABLED, 0);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + if (rc) {
> + pr_err("XIVE: Failed to enable pool VP on CPU %d\n", cpu);
> + return;
> + }
> +
> + /* Grab it's CAM value */
> + rc = opal_xive_get_vp_info(vp, NULL, &vp_cam_be, NULL, NULL);
> + if (rc) {
> + pr_err("XIVE: Failed to get pool VP info CPU %d\n", cpu);
> + return;
> + }
> + vp_cam = be64_to_cpu(vp_cam_be);
> +
> + pr_debug("XIVE: VP CAM = %llx\n", vp_cam);
> +
> + /* Push it on the CPU (set LSMFB to 0xff to skip backlog scan) */
> + pr_debug("XIVE: (Old HW value: %08x)\n",
> + in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
> + out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD0, 0xff);
> + out_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2,
> + TM_QW2W2_VP | vp_cam);
> + pr_debug("XIVE: (New HW value: %08x)\n",
> + in_be32(xive_tm_area + TM_QW2_HV_POOL + TM_WORD2));
> +}
> +
> +static void xive_native_teardown_cpu(unsigned int cpu, struct xive_cpu *xc)
> +{
> + s64 rc;
> + u32 vp;
> +
> + if (xive_pool_vps = XIVE_INVALID_VP)
> + return;
> +
> + /* Pull the pool VP from the CPU */
> + in_be64(xive_tm_area + TM_SPC_PULL_POOL_CTX);
> +
> + /* Disable it */
> + vp = xive_pool_vps + get_hard_smp_processor_id(cpu);
> + for (;;) {
> + rc = opal_xive_set_vp_info(vp, 0, 0);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> +}
> +
> +static void xive_native_sync_source(u32 hw_irq)
> +{
> + opal_xive_sync(XIVE_SYNC_EAS, hw_irq);
> +}
> +
> +static const struct xive_ops xive_native_ops = {
> + .populate_irq_data = xive_native_populate_irq_data,
> + .configure_irq = xive_native_configure_irq,
> + .setup_queue = xive_native_setup_queue,
> + .cleanup_queue = xive_native_cleanup_queue,
> + .match = xive_native_match,
> + .shutdown = xive_native_shutdown,
> + .eoi = xive_native_eoi,
> + .setup_cpu = xive_native_setup_cpu,
> + .teardown_cpu = xive_native_teardown_cpu,
> + .sync_source = xive_native_sync_source,
> +#ifdef CONFIG_SMP
> + .get_ipi = xive_native_get_ipi,
> + .put_ipi = xive_native_put_ipi,
> +#endif /* CONFIG_SMP */
> + .name = "native",
> +};
> +
> +static bool xive_parse_provisioning(struct device_node *np)
> +{
> + int rc;
> +
> + if (of_property_read_u32(np, "ibm,xive-provision-page-size",
> + &xive_provision_size) < 0)
> + return true;
> + rc = of_property_count_elems_of_size(np, "ibm,xive-provision-chips", 4);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d getting provision chips array\n", rc);
> + return false;
> + }
> + xive_provision_chip_count = rc;
> + if (rc = 0)
> + return true;
> +
> + xive_provision_chips = kzalloc(4 * xive_provision_chip_count,
> + GFP_KERNEL);
> + BUG_ON(!xive_provision_chips);
> +
> + rc = of_property_read_u32_array(np, "ibm,xive-provision-chips",
> + xive_provision_chips,
> + xive_provision_chip_count);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d reading provision chips array\n", rc);
> + return false;
> + }
> +
> + xive_provision_cache = kmem_cache_create("xive-provision",
> + xive_provision_size,
> + xive_provision_size,
> + 0, NULL);
> + if (!xive_provision_cache) {
> + pr_err("XIVE: Failed to allocate provision cache\n");
> + return false;
> + }
> + return true;
> +}
> +
> +u32 xive_native_default_eq_shift(void)
> +{
> + return xive_queue_shift;
> +}
> +
> +bool xive_native_init(void)
> +{
> + struct device_node *np;
> + struct resource r;
> + void __iomem *tm_area;
> + struct property *prop;
> + u8 max_prio = 7;
> + const __be32 *p;
> + u32 val;
> + s64 rc;
> +
> + if (xive_cmdline_disabled)
> + return false;
> +
> + DBG("xive_native_init()\n");
> + np = of_find_compatible_node(NULL, NULL, "ibm,opal-xive-pe");
> + if (!np) {
> + DBG("not found !\n");
> + return false;
> + }
> + DBG("Found %s\n", np->full_name);
> +
> + /* Resource 1 is HV window */
> + if (of_address_to_resource(np, 1, &r)) {
> + pr_err("XIVE: Failed to get TM area resource\n");
"TM" is confusing since it would usually mean "Transactional memory".
I couldn't see anywhere in this patch where you spell out that it
means "Thread Management". For messages that are printed out it's
probably better to say "thread mgmt area" rather than "TM area".
> + return false;
> + }
> + tm_area = ioremap(r.start, resource_size(&r));
> + if (!tm_area) {
> + pr_err("XIVE: Failed to map TM area\n");
> + return false;
> + }
> +
> + /* Read number of priorities */
> + if (of_property_read_u32(np, "ibm,xive-#priorities", &val) = 0)
> + max_prio = val - 1;
> +
> + /* Iterate the EQ sizes and pick one */
> + of_property_for_each_u32(np, "ibm,xive-eq-sizes", prop, p, val) {
> + xive_queue_shift = val;
> + if (val = PAGE_SHIFT)
> + break;
> + }
> +
> + /* Grab size of provisionning pages */
^^^^^^^^^^^^^ provisioning
> + xive_parse_provisioning(np);
> +
> + /* Switch the XIVE to exploitation mode */
> + rc = opal_xive_reset(OPAL_XIVE_MODE_EXPL);
> + if (rc) {
> + pr_err("XIVE: Switch to exploitation mode failed"
> + " with error %lld\n", rc);
> + return false;
> + }
> +
> + /* Initialize XIVE core with our backend */
> + if (!xive_core_init(&xive_native_ops, tm_area, TM_QW3_HV_PHYS,
> + max_prio)) {
> + opal_xive_reset(OPAL_XIVE_MODE_EMU);
> + return false;
> + }
> + pr_info("XIVE: Using %dkB queues\n", 1 << (xive_queue_shift - 10));
> + return true;
> +}
> +
> +static bool xive_native_provision_pages(void)
> +{
> + u32 i;
> + void *p;
> +
> + for (i = 0; i < xive_provision_chip_count; i++) {
> + u32 chip = xive_provision_chips[i];
> +
> + /* XXX TODO: Try to make the allocation local to the node where
> + * the chip reside
> + */
> + p = kmem_cache_alloc(xive_provision_cache, GFP_KERNEL);
> + if (!p) {
> + pr_err("XIVE: Failed to allocate provisioning page\n");
> + return false;
> + }
> + opal_xive_donate_page(chip, __pa(p));
> + }
> + return true;
> +}
> +
> +u32 xive_native_alloc_vp_block(u32 max_vcpus)
> +{
> + s64 rc;
> + u32 order;
> +
> + order = fls(max_vcpus) - 1;
> + pr_info("XIVE: VP block alloc, for max VCPUs %d"
> + " use order %d\n", max_vcpus, order);
> + for (;;) {
> + rc = opal_xive_alloc_vp_block(order);
> + switch (rc) {
> + case OPAL_BUSY:
> + msleep(1);
> + break;
> + case OPAL_XIVE_PROVISIONING:
> + if (!xive_native_provision_pages())
> + return XIVE_INVALID_VP;
> + break;
> + default:
> + if (rc < 0) {
> + pr_err("XIVE: OPAL failed to allocate VCPUs"
> + " order %d, err %lld\n",
> + order, rc);
> + return XIVE_INVALID_VP;
> + }
> + return rc;
> + }
> + }
> +}
> +EXPORT_SYMBOL_GPL(xive_native_alloc_vp_block);
> +
> +void xive_native_free_vp_block(u32 vp_base)
> +{
> + s64 rc;
> +
> + if (vp_base = XIVE_INVALID_VP)
> + return;
> +
> + rc = opal_xive_free_vp_block(vp_base);
> + if (rc < 0)
> + pr_warn("XIVE: OPAL error %lld freeing VP block\n", rc);
> +}
> +EXPORT_SYMBOL_GPL(xive_native_free_vp_block);
> diff --git a/arch/powerpc/sysdev/xive/xive-internal.h b/arch/powerpc/sysdev/xive/xive-internal.h
> new file mode 100644
> index 0000000..e736fc5
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-internal.h
> @@ -0,0 +1,51 @@
> +#ifndef __XIVE_INTERNAL_H
> +#define __XIVE_INTERNAL_H
> +
> +/* Each CPU carry one of these with various per-CPU state */
> +struct xive_cpu {
> +#ifdef CONFIG_SMP
> + /* HW irq number and data of IPI */
> + u32 hw_ipi;
> + struct xive_irq_data ipi_data;
> +#endif /* CONFIG_SMP */
> +
> + int chip_id;
> +
> + /* Queue datas. Only one is populated */
> +#define XIVE_MAX_QUEUES 8
> + struct xive_q queue[XIVE_MAX_QUEUES];
> +
> + /* Pending mask. Each bit corresponds to a priority that
> + * potentially has pending interrupts
> + */
> + u8 pending_prio;
> +
> + /* Cache of HW CPPR */
> + u8 cppr;
> +};
> +
> +/* Backend ops */
> +struct xive_ops {
> + int (*populate_irq_data)(u32 hw_irq, struct xive_irq_data *data);
> + int (*configure_irq)(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
> + int (*setup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
> + void (*cleanup_queue)(unsigned int cpu, struct xive_cpu *xc, u8 prio);
> + void (*setup_cpu)(unsigned int cpu, struct xive_cpu *xc);
> + void (*teardown_cpu)(unsigned int cpu, struct xive_cpu *xc);
> + bool (*match)(struct device_node *np);
> + void (*shutdown)(void);
> + void (*eoi)(u32 hw_irq);
> + void (*sync_source)(u32 hw_irq);
> +#ifdef CONFIG_SMP
> + int (*get_ipi)(unsigned int cpu, struct xive_cpu *xc);
> + void (*put_ipi)(unsigned int cpu, struct xive_cpu *xc);
> +#endif
> + const char *name;
> +};
> +
> +bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
> + u8 max_prio);
> +
> +extern bool xive_cmdline_disabled;
> +
> +#endif /* __XIVE_INTERNAL_H */
> diff --git a/arch/powerpc/sysdev/xive/xive-regs.h b/arch/powerpc/sysdev/xive/xive-regs.h
> new file mode 100644
> index 0000000..f1edb23
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-regs.h
> @@ -0,0 +1,88 @@
> +#ifndef __XIVE_REGS_H__
> +#define __XIVE_REGS_H__
> +
> +/*
> + * TM registers
"Thread Management (TM) registers"
> + */
> +
> +/* TM register offsets */
> +#define TM_QW0_USER 0x000 /* All rings */
> +#define TM_QW1_OS 0x010 /* Ring 0..2 */
> +#define TM_QW2_HV_POOL 0x020 /* Ring 0..1 */
> +#define TM_QW3_HV_PHYS 0x030 /* Ring 0..1 */
> +
> +/* Byte offsets inside a QW QW0 QW1 QW2 QW3 */
> +#define TM_NSR 0x0 /* + + - + */
> +#define TM_CPPR 0x1 /* - + - + */
> +#define TM_IPB 0x2 /* - + + + */
> +#define TM_LSMFB 0x3 /* - + + + */
> +#define TM_ACK_CNT 0x4 /* - + - - */
> +#define TM_INC 0x5 /* - + - + */
> +#define TM_AGE 0x6 /* - + - + */
> +#define TM_PIPR 0x7 /* - + - + */
> +
> +#define TM_WORD0 0x0
> +#define TM_WORD1 0x4
> +
> +/* QW word 2 contains the valid bit at the top and other fields
> + * depending on the QW
> + */
> +#define TM_WORD2 0x8
> +#define TM_QW0W2_VU PPC_BIT32(0)
> +#define TM_QW0W2_LOGIC_SERV PPC_BITMASK32(1,31) // XX 2,31 ?
> +#define TM_QW1W2_VO PPC_BIT32(0)
> +#define TM_QW1W2_OS_CAM PPC_BITMASK32(8,31)
> +#define TM_QW2W2_VP PPC_BIT32(0)
> +#define TM_QW2W2_POOL_CAM PPC_BITMASK32(8,31)
> +#define TM_QW3W2_VT PPC_BIT32(0)
> +#define TM_QW3W2_LP PPC_BIT32(6)
> +#define TM_QW3W2_LE PPC_BIT32(7)
> +#define TM_QW3W2_T PPC_BIT32(31)
> +
> +/* In addition to normal loads to "peek" and writes (only when invalid)
> + * using 4 and 8 bytes accesses, the above registers support these
> + * "special" byte operations:
> + *
> + * - Byte load from QW0[NSR] - User level NSR (EBB)
> + * - Byte store to QW0[NSR] - User level NSR (EBB)
> + * - Byte load/store to QW1[CPPR] and QW3[CPPR] - CPPR access
> + * - Byte load from QW3[TM_WORD2] - Read VT||00000||LP||LE on thrd 0
> + * otherwise VT||0000000
> + * - Byte store to QW3[TM_WORD2] - Set VT bit (and LP/LE if present)
> + *
> + * Then we have all these "special" CI ops at these offset that trigger
> + * all sorts of side effects:
> + */
> +#define TM_SPC_ACK_EBB 0x800 /* Load8 ack EBB to reg*/
> +#define TM_SPC_ACK_OS_REG 0x810 /* Load16 ack OS irq to reg */
> +#define TM_SPC_PUSH_USR_CTX 0x808 /* Store32 Push/Validate user context */
> +#define TM_SPC_PULL_USR_CTX 0x808 /* Load32 Pull/Invalidate user context */
> +#define TM_SPC_SET_OS_PENDING 0x812 /* Store8 Set OS irq pending bit */
> +#define TM_SPC_PULL_OS_CTX 0x818 /* Load32/Load64 Pull/Invalidate OS context to reg */
> +#define TM_SPC_PULL_POOL_CTX 0x828 /* Load32/Load64 Pull/Invalidate Pool context to reg*/
> +#define TM_SPC_ACK_HV_REG 0x830 /* Load16 ack HV irq to reg */
> +#define TM_SPC_PULL_USR_CTX_OL 0xc08 /* Store8 Pull/Inval usr ctx to odd line */
> +#define TM_SPC_ACK_OS_EL 0xc10 /* Store8 ack OS irq to even line */
> +#define TM_SPC_ACK_HV_POOL_EL 0xc20 /* Store8 ack HV evt pool to even line */
> +#define TM_SPC_ACK_HV_EL 0xc30 /* Store8 ack HV irq to even line */
> +/* XXX more... */
> +
> +/* NSR fields for the various QW ack types */
> +#define TM_QW0_NSR_EB PPC_BIT8(0)
> +#define TM_QW1_NSR_EO PPC_BIT8(0)
> +#define TM_QW3_NSR_HE PPC_BITMASK8(0,1)
> +#define TM_QW3_NSR_HE_NONE 0
> +#define TM_QW3_NSR_HE_POOL 1
> +#define TM_QW3_NSR_HE_PHYS 2
> +#define TM_QW3_NSR_HE_LSI 3
> +#define TM_QW3_NSR_I PPC_BIT8(2)
> +#define TM_QW3_NSR_GRP_LVL PPC_BIT8(3,7)
> +
> +/* Utilities to manipulate these (originaly from OPAL) */
> +#define MASK_TO_LSH(m) (__builtin_ffsl(m) - 1)
> +#define GETFIELD(m, v) (((v) & (m)) >> MASK_TO_LSH(m))
> +#define SETFIELD(m, v, val) \
> + (((v) & ~(m)) | ((((typeof(v))(val)) << MASK_TO_LSH(m)) & (m)))
> +
> +
> +#endif /* __XIVE_H__ */
> diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> index 16321ad..c71e919 100644
> --- a/arch/powerpc/xmon/xmon.c
> +++ b/arch/powerpc/xmon/xmon.c
> @@ -48,7 +48,7 @@
> #include <asm/reg.h>
> #include <asm/debug.h>
> #include <asm/hw_breakpoint.h>
> -
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/firmware.h>
>
> @@ -232,7 +232,13 @@ Commands:\n\
> "\
> dr dump stream of raw bytes\n\
> dt dump the tracing buffers (uses printk)\n\
> - e print exception information\n\
> +"
> +#ifdef CONFIG_PPC_POWERNV
> +" dx# dump xive on CPU\n\
> + dxi# dump xive irq state\n\
> + dxa dump xive on all CPUs\n"
> +#endif
> +" e print exception information\n\
> f flush cache\n\
> la lookup symbol+offset of specified address\n\
> ls lookup address of specified symbol\n\
> @@ -2338,6 +2344,81 @@ static void dump_pacas(void)
> }
> #endif
>
> +#ifdef CONFIG_PPC_POWERNV
> +static void dump_one_xive(int cpu)
> +{
> + unsigned int hwid = get_hard_smp_processor_id(cpu);
> +
> + opal_xive_dump(XIVE_DUMP_TM_HYP, hwid);
> + opal_xive_dump(XIVE_DUMP_TM_POOL, hwid);
> + opal_xive_dump(XIVE_DUMP_TM_OS, hwid);
> + opal_xive_dump(XIVE_DUMP_TM_USER, hwid);
> + opal_xive_dump(XIVE_DUMP_VP, hwid);
> + opal_xive_dump(XIVE_DUMP_EMU_STATE, hwid);
> +
> + if (setjmp(bus_error_jmp) != 0) {
> + catch_memory_errors = 0;
> + printf("*** Error dumping xive on cpu %d\n", cpu);
> + return;
> + }
> +
> + catch_memory_errors = 1;
> + sync();
> + xmon_xive_do_dump(cpu);
> + sync();
> + __delay(200);
> + catch_memory_errors = 0;
> +}
> +
> +static void dump_all_xives(void)
> +{
> + int cpu;
> +
> + if (num_possible_cpus() = 0) {
> + printf("No possible cpus, use 'dx #' to dump individual cpus\n");
> + return;
> + }
> +
> + for_each_possible_cpu(cpu)
> + dump_one_xive(cpu);
> +}
> +
> +static void dump_one_xive_irq(uint32_t num)
> +{
> + int64_t rc;
> + __be64 vp;
> + uint8_t prio;
> + __be32 lirq;
> +
> + rc = opal_xive_get_irq_config(num, &vp, &prio, &lirq);
> + xmon_printf("IRQ 0x%x config: vp=0x%llx prio=%d lirq=0x%x (rc=%lld)\n",
> + num, be64_to_cpu(vp), prio, be32_to_cpu(lirq), rc);
> +}
> +
> +static void dump_xives(void)
> +{
> + unsigned long num;
> + int c;
> +
> + c = inchar();
> + if (c = 'a') {
> + dump_all_xives();
> + return;
> + } else if (c = 'i') {
> + if (scanhex(&num))
> + dump_one_xive_irq(num);
> + return;
> + }
> +
> + termch = c; /* Put c back, it wasn't 'a' */
> +
> + if (scanhex(&num))
> + dump_one_xive(num);
> + else
> + dump_one_xive(xmon_owner);
> +}
> +#endif /* CONFIG_PPC_POWERNV */
> +
> static void dump_by_size(unsigned long addr, long count, int size)
> {
> unsigned char temp[16];
> @@ -2386,6 +2467,14 @@ dump(void)
> return;
> }
> #endif
> +#ifdef CONFIG_PPC_POWERNV
> + if (c = 'x') {
> + xmon_start_pagination();
> + dump_xives();
> + xmon_end_pagination();
> + return;
> + }
> +#endif
>
> if (c = '\n')
> termch = c;
> --
> 2.9.3
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm-ppc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
Paul.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 12/12] powerpc/kvm: Native usage of the XIVE interrupt controller
2017-03-20 6:49 ` [PATCH 12/12] powerpc/kvm: Native usage of the XIVE interrupt controller Benjamin Herrenschmidt
@ 2017-03-28 5:26 ` Paul Mackerras
2017-04-03 2:25 ` Benjamin Herrenschmidt
0 siblings, 1 reply; 38+ messages in thread
From: Paul Mackerras @ 2017-03-28 5:26 UTC (permalink / raw)
To: Benjamin Herrenschmidt; +Cc: linuxppc-dev, kvm-ppc
On Mon, Mar 20, 2017 at 05:49:14PM +1100, Benjamin Herrenschmidt wrote:
> This patch makes KVM capable of using the XIVE interrupt controller
> to provide the standard PAPR "XICS" style hypercalls. It is necessary
> for proper operations when the host uses XIVE natively.
>
> This has been lightly tested on an actual system, including PCI
> pass-through with a TG3 device.
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Looks good overall, some comments below...
> ---
> arch/powerpc/include/asm/kvm_book3s_asm.h | 2 +
> arch/powerpc/include/asm/kvm_host.h | 28 +-
> arch/powerpc/include/asm/kvm_ppc.h | 38 +
> arch/powerpc/include/asm/xive.h | 11 +-
> arch/powerpc/kernel/asm-offsets.c | 10 +
> arch/powerpc/kvm/Makefile | 4 +-
> arch/powerpc/kvm/book3s.c | 73 +-
> arch/powerpc/kvm/book3s_hv.c | 52 +-
> arch/powerpc/kvm/book3s_hv_builtin.c | 108 ++
> arch/powerpc/kvm/book3s_hv_rm_xics.c | 10 +-
> arch/powerpc/kvm/book3s_hv_rm_xive.c | 47 +
> arch/powerpc/kvm/book3s_hv_rmhandlers.S | 60 +-
> arch/powerpc/kvm/book3s_rtas.c | 21 +-
> arch/powerpc/kvm/book3s_xics.c | 35 +-
> arch/powerpc/kvm/book3s_xics.h | 5 +
> arch/powerpc/kvm/book3s_xive.c | 1898 +++++++++++++++++++++++++++++
> arch/powerpc/kvm/book3s_xive.h | 251 ++++
> arch/powerpc/kvm/book3s_xive_template.c | 490 ++++++++
> arch/powerpc/kvm/irq.h | 1 +
> arch/powerpc/kvm/powerpc.c | 17 +-
> arch/powerpc/platforms/powernv/opal.c | 1 +
> arch/powerpc/sysdev/xive/common.c | 131 +-
> arch/powerpc/sysdev/xive/native.c | 92 +-
> include/linux/kvm_host.h | 1 -
> virt/kvm/kvm_main.c | 4 -
> 25 files changed, 3305 insertions(+), 85 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_hv_rm_xive.c
> create mode 100644 arch/powerpc/kvm/book3s_xive.c
> create mode 100644 arch/powerpc/kvm/book3s_xive.h
> create mode 100644 arch/powerpc/kvm/book3s_xive_template.c
>
> diff --git a/arch/powerpc/include/asm/kvm_book3s_asm.h b/arch/powerpc/include/asm/kvm_book3s_asm.h
> index 0593d94..e719002 100644
> --- a/arch/powerpc/include/asm/kvm_book3s_asm.h
> +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
> @@ -111,6 +111,8 @@ struct kvmppc_host_state {
> struct kvm_vcpu *kvm_vcpu;
> struct kvmppc_vcore *kvm_vcore;
> void __iomem *xics_phys;
> + void __iomem *xive_tm_area_phys;
> + void __iomem *xive_tm_area_virt;
Does this cause the paca to become a cacheline larger? (Not that
there is much alternative to having these fields.)
> u32 saved_xirr;
> u64 dabr;
> u64 host_mmcr[7]; /* MMCR 0,1,A, SIAR, SDAR, MMCR2, SIER */
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 7bba8f4..fc491ac 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -205,6 +205,12 @@ struct kvmppc_spapr_tce_table {
> /* XICS components, defined in book3s_xics.c */
> struct kvmppc_xics;
> struct kvmppc_icp;
> +extern struct kvm_device_ops kvm_xics_ops;
> +
> +/* XIVE components, defined in book3s_xive.c */
> +struct kvmppc_xive;
> +struct kvmppc_xive_vcpu;
> +extern struct kvm_device_ops kvm_xive_ops;
>
> struct kvmppc_passthru_irqmap;
>
> @@ -293,6 +299,7 @@ struct kvm_arch {
> #endif
> #ifdef CONFIG_KVM_XICS
> struct kvmppc_xics *xics;
> + struct kvmppc_xive *xive;
> struct kvmppc_passthru_irqmap *pimap;
> #endif
> struct kvmppc_ops *kvm_ops;
> @@ -421,7 +428,7 @@ struct kvmppc_passthru_irqmap {
>
> #define KVMPPC_IRQ_DEFAULT 0
> #define KVMPPC_IRQ_MPIC 1
> -#define KVMPPC_IRQ_XICS 2
> +#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
>
> #define MMIO_HPTE_CACHE_SIZE 4
>
> @@ -443,6 +450,21 @@ struct mmio_hpte_cache {
>
> struct openpic;
>
> +/* QW0 and QW1 of a context */
> +union xive_qw01 {
> + struct {
> + u8 nsr;
> + u8 cppr;
> + u8 ipb;
> + u8 lsmfb;
> + u8 ack;
> + u8 inc;
> + u8 age;
> + u8 pipr;
> + };
> + __be64 qw;
> +};
This is slightly confusing because a "QW" (quadword) would normally be
128 bits, but this union is 64 bits.
> +
> struct kvm_vcpu_arch {
> ulong host_stack;
> u32 host_pid;
> @@ -688,6 +710,10 @@ struct kvm_vcpu_arch {
> struct openpic *mpic; /* KVM_IRQ_MPIC */
> #ifdef CONFIG_KVM_XICS
> struct kvmppc_icp *icp; /* XICS presentation controller */
> + struct kvmppc_xive_vcpu *xive_vcpu; /* XIVE virtual CPU data */
> + __be32 xive_cam_word; /* Cooked W2 in proper endian with valid bit */
> + u32 xive_pushed; /* Is the VP pushed on the physical CPU ? */
> + union xive_qw01 xive_saved_state; /* W0..1 of XIVE state */
> #endif
>
> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index c387799..2fcf6cf 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -225,6 +225,7 @@ int kvm_vcpu_ioctl_interrupt(struct kvm_vcpu *vcpu, struct kvm_interrupt *irq);
> extern int kvm_vm_ioctl_rtas_define_token(struct kvm *kvm, void __user *argp);
> extern int kvmppc_rtas_hcall(struct kvm_vcpu *vcpu);
> extern void kvmppc_rtas_tokens_free(struct kvm *kvm);
> +
> extern int kvmppc_xics_set_xive(struct kvm *kvm, u32 irq, u32 server,
> u32 priority);
> extern int kvmppc_xics_get_xive(struct kvm *kvm, u32 irq, u32 *server,
> @@ -232,6 +233,15 @@ extern int kvmppc_xics_get_xive(struct kvm *kvm, u32 irq, u32 *server,
> extern int kvmppc_xics_int_on(struct kvm *kvm, u32 irq);
> extern int kvmppc_xics_int_off(struct kvm *kvm, u32 irq);
>
> +extern int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> + u32 priority);
> +extern int kvmppc_xive_get_xive(struct kvm *kvm, u32 irq, u32 *server,
> + u32 *priority);
Might be worth a comment here to explain that the first xive is
eXternal Interrupt Virtualization Engine and the second xive is
eXternal Interrupt Vector Entry.
> +extern int kvmppc_xive_int_on(struct kvm *kvm, u32 irq);
> +extern int kvmppc_xive_int_off(struct kvm *kvm, u32 irq);
> +extern void kvmppc_xive_init_module(void);
> +extern void kvmppc_xive_exit_module(void);
> +
> void kvmppc_core_dequeue_debug(struct kvm_vcpu *vcpu);
> void kvmppc_core_queue_debug(struct kvm_vcpu *vcpu);
>
> @@ -412,6 +422,14 @@ static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)
> paca[cpu].kvm_hstate.xics_phys = (void __iomem *)addr;
> }
>
> +static inline void kvmppc_set_xive_tm_area(int cpu,
> + unsigned long phys_addr,
> + void __iomem *virt_addr)
> +{
> + paca[cpu].kvm_hstate.xive_tm_area_phys = (void __iomem *)phys_addr;
> + paca[cpu].kvm_hstate.xive_tm_area_virt = virt_addr;
> +}
> +
> static inline u32 kvmppc_get_xics_latch(void)
> {
> u32 xirr;
> @@ -442,6 +460,9 @@ static inline void __init kvm_cma_reserve(void)
> static inline void kvmppc_set_xics_phys(int cpu, unsigned long addr)
> {}
>
> +static inline void kvmppc_set_xive_tm_area_phys(int cpu, unsigned long addr)
> +{}
Shouldn't this be kvmppc_set_xive_tm_area to match the other definition?
> +
> static inline u32 kvmppc_get_xics_latch(void)
> {
> return 0;
> @@ -492,6 +513,21 @@ extern long kvmppc_deliver_irq_passthru(struct kvm_vcpu *vcpu, __be32 xirr,
> struct kvmppc_irq_map *irq_map,
> struct kvmppc_passthru_irqmap *pimap,
> bool *again);
> +extern int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu);
> +extern void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu);
> +extern int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
> + struct irq_desc *host_desc);
> +extern int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
> + struct irq_desc *host_desc);
> +extern u64 kvmppc_xive_get_icp(struct kvm_vcpu *vcpu);
> +extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
> +
> +extern int kvmppc_xics_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> + int level, bool line_status);
> +extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> + int level, bool line_status);
> +
> extern int h_ipi_redirect;
> #else
> static inline struct kvmppc_passthru_irqmap *kvmppc_get_passthru_irqmap(
> @@ -546,6 +582,8 @@ long kvmppc_h_clear_mod(struct kvm_vcpu *vcpu, unsigned long flags,
> long kvmppc_hpte_hv_fault(struct kvm_vcpu *vcpu, unsigned long addr,
> unsigned long slb_v, unsigned int status, bool data);
> unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu);
> +unsigned long kvmppc_rm_h_xirr_x(struct kvm_vcpu *vcpu);
> +unsigned long kvmppc_rm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server);
> int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> unsigned long mfrr);
> int kvmppc_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr);
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index b1604b73..94b5cca 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -55,7 +55,8 @@ struct xive_q {
> #define XIVE_ESB_SET_PQ_01 0xd00
> #define XIVE_ESB_SET_PQ_10 0xe00
> #define XIVE_ESB_SET_PQ_11 0xf00
> -#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> +#define XIVE_ESB_SOFT_MASK XIVE_ESB_SET_PQ_10
> +#define XIVE_ESB_HARD_MASK XIVE_ESB_SET_PQ_01
What's the difference between a "soft" mask and a "hard" mask?
>
> extern bool __xive_enabled;
>
> @@ -88,11 +89,11 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> __be32 *qpage, u32 order, bool can_escalate);
> extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
>
> -extern bool __xive_irq_trigger(struct xive_irq_data *xd);
> -extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
> -extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
> -
> +extern void xive_native_sync_source(u32 hw_irq);
> extern bool is_xive_irq(struct irq_chip *chip);
> +extern int xive_native_enable_vp(u32 vp_id);
> +extern int xive_native_disable_vp(u32 vp_id);
> +extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
>
> #else
>
> diff --git a/arch/powerpc/kernel/asm-offsets.c b/arch/powerpc/kernel/asm-offsets.c
> index 4367e7d..59fa705 100644
> --- a/arch/powerpc/kernel/asm-offsets.c
> +++ b/arch/powerpc/kernel/asm-offsets.c
> @@ -630,6 +630,8 @@ int main(void)
> HSTATE_FIELD(HSTATE_KVM_VCPU, kvm_vcpu);
> HSTATE_FIELD(HSTATE_KVM_VCORE, kvm_vcore);
> HSTATE_FIELD(HSTATE_XICS_PHYS, xics_phys);
> + HSTATE_FIELD(HSTATE_XIVE_TM_AREA_PHYS, xive_tm_area_phys);
> + HSTATE_FIELD(HSTATE_XIVE_TM_AREA_VIRT, xive_tm_area_virt);
> HSTATE_FIELD(HSTATE_SAVED_XIRR, saved_xirr);
> HSTATE_FIELD(HSTATE_HOST_IPI, host_ipi);
> HSTATE_FIELD(HSTATE_PTID, ptid);
> @@ -715,6 +717,14 @@ int main(void)
> OFFSET(VCPU_HOST_MAS6, kvm_vcpu, arch.host_mas6);
> #endif
>
> +#ifdef CONFIG_KVM_XICS
> + DEFINE(VCPU_XIVE_SAVED_STATE, offsetof(struct kvm_vcpu,
> + arch.xive_saved_state));
> + DEFINE(VCPU_XIVE_CAM_WORD, offsetof(struct kvm_vcpu,
> + arch.xive_cam_word));
> + DEFINE(VCPU_XIVE_PUSHED, offsetof(struct kvm_vcpu, arch.xive_pushed));
> +#endif
> +
> #ifdef CONFIG_KVM_EXIT_TIMING
> OFFSET(VCPU_TIMING_EXIT_TBU, kvm_vcpu, arch.timing_exit.tv32.tbu);
> OFFSET(VCPU_TIMING_EXIT_TBL, kvm_vcpu, arch.timing_exit.tv32.tbl);
> diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
> index b87ccde..ef89c8c 100644
> --- a/arch/powerpc/kvm/Makefile
> +++ b/arch/powerpc/kvm/Makefile
> @@ -74,7 +74,7 @@ kvm-hv-y += \
> book3s_64_mmu_radix.o
>
> kvm-book3s_64-builtin-xics-objs-$(CONFIG_KVM_XICS) := \
> - book3s_hv_rm_xics.o
> + book3s_hv_rm_xics.o book3s_hv_rm_xive.o
>
> ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \
> @@ -87,7 +87,7 @@ kvm-book3s_64-builtin-objs-$(CONFIG_KVM_BOOK3S_64_HANDLER) += \
> endif
>
> kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
> - book3s_xics.o
> + book3s_xics.o book3s_xive.o
>
> kvm-book3s_64-module-objs := \
> $(common-objs-y) \
> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> index aedacef..e459ec4 100644
> --- a/arch/powerpc/kvm/book3s.c
> +++ b/arch/powerpc/kvm/book3s.c
> @@ -35,6 +35,7 @@
> #include <asm/kvm_book3s.h>
> #include <asm/mmu_context.h>
> #include <asm/page.h>
> +#include <asm/xive.h>
>
> #include "book3s.h"
> #include "trace.h"
> @@ -578,11 +579,14 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
> break;
> #ifdef CONFIG_KVM_XICS
> case KVM_REG_PPC_ICP_STATE:
> - if (!vcpu->arch.icp) {
> + if (!vcpu->arch.icp && !vcpu->arch.xive_vcpu) {
> r = -ENXIO;
> break;
> }
> - *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
> + if (xive_enabled())
> + *val = get_reg_val(id, kvmppc_xive_get_icp(vcpu));
> + else
> + *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
> break;
> #endif /* CONFIG_KVM_XICS */
> case KVM_REG_PPC_FSCR:
> @@ -648,12 +652,14 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
> #endif /* CONFIG_VSX */
> #ifdef CONFIG_KVM_XICS
> case KVM_REG_PPC_ICP_STATE:
> - if (!vcpu->arch.icp) {
> + if (!vcpu->arch.icp && !vcpu->arch.xive_vcpu) {
> r = -ENXIO;
> break;
> }
> - r = kvmppc_xics_set_icp(vcpu,
> - set_reg_val(id, *val));
> + if (xive_enabled())
> + r = kvmppc_xive_set_icp(vcpu, set_reg_val(id, *val));
> + else
> + r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
> break;
> #endif /* CONFIG_KVM_XICS */
> case KVM_REG_PPC_FSCR:
> @@ -924,6 +930,50 @@ int kvmppc_book3s_hcall_implemented(struct kvm *kvm, unsigned long hcall)
> return kvm->arch.kvm_ops->hcall_implemented(hcall);
> }
>
> +#ifdef CONFIG_KVM_XICS
> +int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
> + bool line_status)
> +{
> + if (xive_enabled())
> + return kvmppc_xive_set_irq(kvm, irq_source_id, irq, level,
> + line_status);
> + else
> + return kvmppc_xics_set_irq(kvm, irq_source_id, irq, level,
> + line_status);
> +}
> +
> +int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *irq_entry,
> + struct kvm *kvm, int irq_source_id,
> + int level, bool line_status)
> +{
> + return kvm_set_irq(kvm, irq_source_id, irq_entry->gsi,
> + level, line_status);
> +}
> +static int kvmppc_book3s_set_irq(struct kvm_kernel_irq_routing_entry *e,
> + struct kvm *kvm, int irq_source_id, int level,
> + bool line_status)
> +{
> + return kvm_set_irq(kvm, irq_source_id, e->gsi, level, line_status);
> +}
> +
> +int kvm_irq_map_gsi(struct kvm *kvm,
> + struct kvm_kernel_irq_routing_entry *entries, int gsi)
> +{
> + entries->gsi = gsi;
> + entries->type = KVM_IRQ_ROUTING_IRQCHIP;
> + entries->set = kvmppc_book3s_set_irq;
> + entries->irqchip.irqchip = 0;
> + entries->irqchip.pin = gsi;
> + return 1;
> +}
> +
> +int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
> +{
> + return pin;
> +}
> +
> +#endif /* CONFIG_KVM_XICS */
> +
> static int kvmppc_book3s_init(void)
> {
> int r;
> @@ -934,12 +984,23 @@ static int kvmppc_book3s_init(void)
> #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
> r = kvmppc_book3s_init_pr();
> #endif
> - return r;
>
> +#ifdef CONFIG_KVM_XICS
> + if (xive_enabled()) {
> + kvmppc_xive_init_module();
> + kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
> + } else
> + kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
> +#endif
> + return r;
> }
>
> static void kvmppc_book3s_exit(void)
> {
> +#ifdef CONFIG_KVM_XICS
> + if (xive_enabled())
> + kvmppc_xive_exit_module();
> +#endif
> #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
> kvmppc_book3s_exit_pr();
> #endif
> diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c
> index fadb75a..5c340c2 100644
> --- a/arch/powerpc/kvm/book3s_hv.c
> +++ b/arch/powerpc/kvm/book3s_hv.c
> @@ -67,6 +67,7 @@
> #include <asm/mmu.h>
> #include <asm/opal.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
>
> #include "book3s.h"
>
> @@ -837,6 +838,10 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu)
> case H_IPOLL:
> case H_XIRR_X:
> if (kvmppc_xics_enabled(vcpu)) {
> + if (xive_enabled()) {
> + ret = H_NOT_AVAILABLE;
> + return RESUME_GUEST;
> + }
> ret = kvmppc_xics_hcall(vcpu, req);
> break;
> }
> @@ -2947,8 +2952,12 @@ static int kvmppc_vcpu_run_hv(struct kvm_run *run, struct kvm_vcpu *vcpu)
> r = kvmppc_book3s_hv_page_fault(run, vcpu,
> vcpu->arch.fault_dar, vcpu->arch.fault_dsisr);
> srcu_read_unlock(&vcpu->kvm->srcu, srcu_idx);
> - } else if (r == RESUME_PASSTHROUGH)
> - r = kvmppc_xics_rm_complete(vcpu, 0);
> + } else if (r == RESUME_PASSTHROUGH) {
> + if (WARN_ON(xive_enabled()))
> + r = H_SUCCESS;
> + else
> + r = kvmppc_xics_rm_complete(vcpu, 0);
> + }
> } while (is_kvmppc_resume_guest(r));
>
> out:
> @@ -3400,10 +3409,19 @@ static int kvmppc_core_init_vm_hv(struct kvm *kvm)
> /*
> * On POWER9, VPM0 bit is reserved (VPM0=1 behaviour is assumed)
> * Set HVICE bit to enable hypervisor virtualization interrupts.
> + * Set HEIC to prevent OS interrupts to go to hypervisor (should
> + * be unnecessary but better safe than sorry in case we re-enable
> + * EE in HV mode with this LPCR still set)
> */
> if (cpu_has_feature(CPU_FTR_ARCH_300)) {
> lpcr &= ~LPCR_VPM0;
> - lpcr |= LPCR_HVICE;
> + lpcr |= LPCR_HVICE | LPCR_HEIC;
> +
> + /* If xive is enabled, we route 0x500 interrupts directly
> + * to the guest
> + */
> + if (xive_enabled())
> + lpcr |= LPCR_LPES;
> }
>
> /*
> @@ -3533,7 +3551,7 @@ static int kvmppc_set_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
> struct kvmppc_irq_map *irq_map;
> struct kvmppc_passthru_irqmap *pimap;
> struct irq_chip *chip;
> - int i;
> + int i, rc = 0;
>
> if (!kvm_irq_bypass)
> return 1;
> @@ -3558,10 +3576,10 @@ static int kvmppc_set_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
> /*
> * For now, we only support interrupts for which the EOI operation
> * is an OPAL call followed by a write to XIRR, since that's
> - * what our real-mode EOI code does.
> + * what our real-mode EOI code does, or a XIVE interrupt
> */
> chip = irq_data_get_irq_chip(&desc->irq_data);
> - if (!chip || !is_pnv_opal_msi(chip)) {
> + if (!chip || !(is_pnv_opal_msi(chip) || is_xive_irq(chip))) {
> pr_warn("kvmppc_set_passthru_irq_hv: Could not assign IRQ map for (%d,%d)\n",
> host_irq, guest_gsi);
> mutex_unlock(&kvm->lock);
> @@ -3603,7 +3621,14 @@ static int kvmppc_set_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
> if (i == pimap->n_mapped)
> pimap->n_mapped++;
>
> - kvmppc_xics_set_mapped(kvm, guest_gsi, desc->irq_data.hwirq);
> + if (xive_enabled())
> + rc = kvmppc_xive_set_mapped(kvm, guest_gsi, desc);
> + else
> + kvmppc_xics_set_mapped(kvm, guest_gsi, desc->irq_data.hwirq);
> + printk("set mapped for IRQ %d -> %d returned %d\n",
> + host_irq, guest_gsi, rc);
This seems like a debugging thing that should be removed or turned
into a DBG().
> + if (rc)
> + irq_map->r_hwirq = 0;
>
> mutex_unlock(&kvm->lock);
>
> @@ -3614,7 +3639,7 @@ static int kvmppc_clr_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
> {
> struct irq_desc *desc;
> struct kvmppc_passthru_irqmap *pimap;
> - int i;
> + int i, rc = 0;
>
> if (!kvm_irq_bypass)
> return 0;
> @@ -3641,9 +3666,12 @@ static int kvmppc_clr_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
> return -ENODEV;
> }
>
> - kvmppc_xics_clr_mapped(kvm, guest_gsi, pimap->mapped[i].r_hwirq);
> + if (xive_enabled())
> + rc = kvmppc_xive_clr_mapped(kvm, guest_gsi, pimap->mapped[i].desc);
> + else
> + kvmppc_xics_clr_mapped(kvm, guest_gsi, pimap->mapped[i].r_hwirq);
>
> - /* invalidate the entry */
> + /* invalidate the entry (what do do on error from the above ?) */
> pimap->mapped[i].r_hwirq = 0;
>
> /*
> @@ -3652,7 +3680,7 @@ static int kvmppc_clr_passthru_irq(struct kvm *kvm, int host_irq, int guest_gsi)
> */
>
> mutex_unlock(&kvm->lock);
> - return 0;
> + return rc;
> }
>
> static int kvmppc_irq_bypass_add_producer_hv(struct irq_bypass_consumer *cons,
> @@ -3930,7 +3958,7 @@ static int kvmppc_book3s_init_hv(void)
> * indirectly, via OPAL.
> */
> #ifdef CONFIG_SMP
> - if (!get_paca()->kvm_hstate.xics_phys) {
> + if (!xive_enabled() && !get_paca()->kvm_hstate.xics_phys) {
> struct device_node *np;
>
> np = of_find_compatible_node(NULL, NULL, "ibm,opal-intc");
> diff --git a/arch/powerpc/kvm/book3s_hv_builtin.c b/arch/powerpc/kvm/book3s_hv_builtin.c
> index d48f9b6..8de7ed4 100644
> --- a/arch/powerpc/kvm/book3s_hv_builtin.c
> +++ b/arch/powerpc/kvm/book3s_hv_builtin.c
> @@ -23,6 +23,7 @@
> #include <asm/kvm_book3s.h>
> #include <asm/archrandom.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/dbell.h>
> #include <asm/cputhreads.h>
> #include <asm/io.h>
> @@ -31,6 +32,24 @@
>
> #define KVM_CMA_CHUNK_ORDER 18
>
> +#include "book3s_xics.h"
> +#include "book3s_xive.h"
> +
> +/*
> + * The XIVE module will populate these when it loads
> + */
> +unsigned long (*__xive_vm_h_xirr)(struct kvm_vcpu *vcpu);
> +unsigned long (*__xive_vm_h_ipoll)(struct kvm_vcpu *vcpu, unsigned long server);
> +int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr);
> +int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
> +int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> +EXPORT_SYMBOL_GPL(__xive_vm_h_xirr);
> +EXPORT_SYMBOL_GPL(__xive_vm_h_ipoll);
> +EXPORT_SYMBOL_GPL(__xive_vm_h_ipi);
> +EXPORT_SYMBOL_GPL(__xive_vm_h_cppr);
> +EXPORT_SYMBOL_GPL(__xive_vm_h_eoi);
> +
> /*
> * Hash page table alignment on newer cpus(CPU_FTR_ARCH_206)
> * should be power of 2.
> @@ -209,6 +228,7 @@ void kvmhv_rm_send_ipi(int cpu)
> __asm__ __volatile__ (PPC_MSGSND(%0) : : "r" (msg));
> return;
> }
> +
> /* On POWER8 for IPIs to threads in the same core, use msgsnd. */
> if (cpu_has_feature(CPU_FTR_ARCH_207S) &&
> cpu_first_thread_sibling(cpu) ==
> @@ -218,6 +238,10 @@ void kvmhv_rm_send_ipi(int cpu)
> return;
> }
>
> + /* We should never reach this */
> + if (WARN_ON_ONCE(xive_enabled()))
> + return;
> +
> /* Else poke the target with an IPI */
> xics_phys = paca[cpu].kvm_hstate.xics_phys;
> if (xics_phys)
> @@ -398,6 +422,9 @@ static long kvmppc_read_one_intr(bool *again)
> u8 host_ipi;
> int64_t rc;
>
> + if (xive_enabled())
> + return 1;
Why not do this in kvmppc_read_intr() rather than here?
> +
> /* see if a host IPI is pending */
> host_ipi = local_paca->kvm_hstate.host_ipi;
> if (host_ipi)
> @@ -482,3 +509,84 @@ static long kvmppc_read_one_intr(bool *again)
>
> return kvmppc_check_passthru(xisr, xirr, again);
> }
> +
> +static inline bool is_rm(void)
> +{
> + return !(mfmsr() & MSR_DR);
> +}
> +
> +/* XXX FIXME: The xive_vm_* calls are in a module... */
> +
> +unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu)
> +{
> + if (xive_enabled()) {
> + if (is_rm())
> + return xive_rm_h_xirr(vcpu);
> + if (unlikely(!__xive_vm_h_xirr))
> + return H_NOT_AVAILABLE;
> + return __xive_vm_h_xirr(vcpu);
> + } else
> + return xics_rm_h_xirr(vcpu);
> +}
> +
> +unsigned long kvmppc_rm_h_xirr_x(struct kvm_vcpu *vcpu)
> +{
> + vcpu->arch.gpr[5] = get_tb();
> + if (xive_enabled()) {
> + if (is_rm())
> + return xive_rm_h_xirr(vcpu);
> + if (unlikely(!__xive_vm_h_xirr))
> + return H_NOT_AVAILABLE;
> + return __xive_vm_h_xirr(vcpu);
> + } else
> + return xics_rm_h_xirr(vcpu);
> +}
> +
> +unsigned long kvmppc_rm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server)
> +{
> + if (xive_enabled()) {
> + if (is_rm())
> + return xive_rm_h_ipoll(vcpu, server);
> + if (unlikely(!__xive_vm_h_ipoll))
> + return H_NOT_AVAILABLE;
> + return __xive_vm_h_ipoll(vcpu, server);
> + } else
> + return H_TOO_HARD;
> +}
> +
> +int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr)
> +{
> + if (xive_enabled()) {
> + if (is_rm())
> + return xive_rm_h_ipi(vcpu, server, mfrr);
> + if (unlikely(!__xive_vm_h_ipi))
> + return H_NOT_AVAILABLE;
> + return __xive_vm_h_ipi(vcpu, server, mfrr);
> + } else
> + return xics_rm_h_ipi(vcpu, server, mfrr);
> +}
> +
> +int kvmppc_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
> +{
> + if (xive_enabled()) {
> + if (is_rm())
> + return xive_rm_h_cppr(vcpu, cppr);
> + if (unlikely(!__xive_vm_h_cppr))
> + return H_NOT_AVAILABLE;
> + return __xive_vm_h_cppr(vcpu, cppr);
> + } else
> + return xics_rm_h_cppr(vcpu, cppr);
> +}
> +
> +int kvmppc_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
> +{
> + if (xive_enabled()) {
> + if (is_rm())
> + return xive_rm_h_eoi(vcpu, xirr);
> + if (unlikely(!__xive_vm_h_eoi))
> + return H_NOT_AVAILABLE;
> + return __xive_vm_h_eoi(vcpu, xirr);
> + } else
> + return xics_rm_h_eoi(vcpu, xirr);
> +}
> diff --git a/arch/powerpc/kvm/book3s_hv_rm_xics.c b/arch/powerpc/kvm/book3s_hv_rm_xics.c
> index 3a1a463..f806880 100644
> --- a/arch/powerpc/kvm/book3s_hv_rm_xics.c
> +++ b/arch/powerpc/kvm/book3s_hv_rm_xics.c
> @@ -485,7 +485,7 @@ static void icp_rm_down_cppr(struct kvmppc_xics *xics, struct kvmppc_icp *icp,
> }
>
>
> -unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu)
> +unsigned long xics_rm_h_xirr(struct kvm_vcpu *vcpu)
> {
> union kvmppc_icp_state old_state, new_state;
> struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
> @@ -523,8 +523,8 @@ unsigned long kvmppc_rm_h_xirr(struct kvm_vcpu *vcpu)
> return check_too_hard(xics, icp);
> }
>
> -int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> - unsigned long mfrr)
> +int xics_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr)
> {
> union kvmppc_icp_state old_state, new_state;
> struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
> @@ -610,7 +610,7 @@ int kvmppc_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> return check_too_hard(xics, this_icp);
> }
>
> -int kvmppc_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
> +int xics_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr)
> {
> union kvmppc_icp_state old_state, new_state;
> struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
> @@ -730,7 +730,7 @@ static int ics_rm_eoi(struct kvm_vcpu *vcpu, u32 irq)
> return check_too_hard(xics, icp);
> }
>
> -int kvmppc_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
> +int xics_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr)
> {
> struct kvmppc_xics *xics = vcpu->kvm->arch.xics;
> struct kvmppc_icp *icp = vcpu->arch.icp;
> diff --git a/arch/powerpc/kvm/book3s_hv_rm_xive.c b/arch/powerpc/kvm/book3s_hv_rm_xive.c
> new file mode 100644
> index 0000000..6390f71
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_hv_rm_xive.c
> @@ -0,0 +1,47 @@
> +#include <linux/kernel.h>
> +#include <linux/kvm_host.h>
> +#include <linux/err.h>
> +#include <linux/kernel_stat.h>
> +
> +#include <asm/kvm_book3s.h>
> +#include <asm/kvm_ppc.h>
> +#include <asm/hvcall.h>
> +#include <asm/xics.h>
> +#include <asm/debug.h>
> +#include <asm/synch.h>
> +#include <asm/cputhreads.h>
> +#include <asm/pgtable.h>
> +#include <asm/ppc-opcode.h>
> +#include <asm/pnv-pci.h>
> +#include <asm/opal.h>
> +#include <asm/smp.h>
> +#include <asm/asm-prototypes.h>
> +#include <asm/xive.h>
> +
> +#include "book3s_xive.h"
> +#include "../sysdev/xive/xive-regs.h"
> +
> +/* XXX */
> +#include <asm/udbg.h>
> +//#define DBG(fmt...) udbg_printf(fmt)
> +#define DBG(fmt...) do { } while(0)
> +
> +static inline void __iomem *get_tm_area_phys(void)
> +{
> + return local_paca->kvm_hstate.xive_tm_area_phys;
> +}
> +
> +#undef XIVE_RUNTIME_CHECKS
> +#define X_PFX xive_rm_
> +#define X_STATIC
> +#define X_STAT_PFX stat_rm_
> +#define __x_tm_area get_tm_area_phys()
> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_page))
> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_page))
> +#define __x_readb __raw_rm_readb
> +#define __x_writeb __raw_rm_writeb
> +#define __x_readw __raw_rm_readw
> +#define __x_readq __raw_rm_readq
> +#define __x_writeq __raw_rm_writeq
> +
> +#include "book3s_xive_template.c"
> diff --git a/arch/powerpc/kvm/book3s_hv_rmhandlers.S b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> index 720b9c0..c06cccd 100644
> --- a/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> +++ b/arch/powerpc/kvm/book3s_hv_rmhandlers.S
> @@ -31,6 +31,8 @@
> #include <asm/tm.h>
> #include <asm/opal.h>
>
> +#include "../sysdev/xive/xive-regs.h"
> +
> #define VCPU_GPRS_TM(reg) (((reg) * ULONG_SIZE) + VCPU_GPR_TM)
>
> /* Values in HSTATE_NAPPING(r13) */
> @@ -982,6 +984,23 @@ ALT_FTR_SECTION_END_IFCLR(CPU_FTR_ARCH_300)
> cmpwi r3, 512 /* 1 microsecond */
> blt hdec_soon
>
> +#ifdef CONFIG_KVM_XICS
> + /* We are entering the guest on that thread, push VCPU to XIVE */
> + ld r10, HSTATE_XIVE_TM_AREA_PHYS(r13)
> + cmpldi cr0, r10, r0
> + beq no_xive
> + ld r11, VCPU_XIVE_SAVED_STATE(r4)
> + li r9, TM_QW1_OS
> + stdcix r11,r9,r10
> + eieio
> + lwz r11, VCPU_XIVE_CAM_WORD(r4)
> + li r9, TM_QW1_OS + TM_WORD2
> + stwcix r11,r9,r10
> + li r9, 1
> + stw r9, VCPU_XIVE_PUSHED(r4)
> +no_xive:
> +#endif /* CONFIG_KVM_XICS */
> +
> deliver_guest_interrupt:
> ld r6, VCPU_CTR(r4)
> ld r7, VCPU_XER(r4)
> @@ -1319,6 +1338,38 @@ END_FTR_SECTION_IFSET(CPU_FTR_HAS_PPR)
> blt deliver_guest_interrupt
>
> guest_exit_cont: /* r9 = vcpu, r12 = trap, r13 = paca */
> +#ifdef CONFIG_KVM_XICS
> + /* We are exiting, pull the VP from the XIVE */
> + lwz r0, VCPU_XIVE_PUSHED(r9)
> + cmpwi cr0, r0, 0
> + beq 1f
> + li r7, TM_SPC_PULL_OS_CTX
> + li r6, TM_QW1_OS
> + mfmsr r0
> + andi. r0, r0, MSR_IR /* in real mode? */
> + beq 2f
> + ld r10, HSTATE_XIVE_TM_AREA_VIRT(r13)
> + cmpldi cr0, r10, 0
> + beq 1f
> + lwzx r11, r7, r10
> + eieio
> + ldx r11, r6, r10
I assume you meant to do these two loads into the same target
register, but I don't know why, so a comment would be useful.
> + b 3f
> +2: ld r10, HSTATE_XIVE_TM_AREA_PHYS(r13)
> + cmpldi cr0, r10, 0
> + beq 1f
> + lwzcix r11, r7, r10
> + eieio
> + ldcix r11, r6, r10
> +3: std r11, VCPU_XIVE_SAVED_STATE(r9)
> + /* Fixup some of the state for the next load */
> + li r10, 0
> + li r0, 0xff
> + stw r10, VCPU_XIVE_PUSHED(r9)
> + stb r10, (VCPU_XIVE_SAVED_STATE+3)(r9)
> + stb r0, (VCPU_XIVE_SAVED_STATE+4)(r9)
> +1:
> +#endif /* CONFIG_KVM_XICS */
> /* Save more register state */
> mfdar r6
> mfdsisr r7
> @@ -2035,7 +2086,7 @@ hcall_real_table:
> .long DOTSYM(kvmppc_rm_h_eoi) - hcall_real_table
> .long DOTSYM(kvmppc_rm_h_cppr) - hcall_real_table
> .long DOTSYM(kvmppc_rm_h_ipi) - hcall_real_table
> - .long 0 /* 0x70 - H_IPOLL */
> + .long DOTSYM(kvmppc_rm_h_ipoll) - hcall_real_table
> .long DOTSYM(kvmppc_rm_h_xirr) - hcall_real_table
> #else
> .long 0 /* 0x64 - H_EOI */
> @@ -2205,7 +2256,11 @@ hcall_real_table:
> .long 0 /* 0x2f0 */
> .long 0 /* 0x2f4 */
> .long 0 /* 0x2f8 */
> - .long 0 /* 0x2fc */
> +#ifdef CONFIG_KVM_XICS
> + .long DOTSYM(kvmppc_rm_h_xirr_x) - hcall_real_table
> +#else
> + .long 0 /* 0x2fc - H_XIRR_X*/
> +#endif
> .long DOTSYM(kvmppc_h_random) - hcall_real_table
> .globl hcall_real_table_end
> hcall_real_table_end:
> @@ -2980,6 +3035,7 @@ kvmppc_fix_pmao:
> isync
> blr
>
> +
Gratuitous extra blank line.
> #ifdef CONFIG_KVM_BOOK3S_HV_EXIT_TIMING
> /*
> * Start timing an activity
> diff --git a/arch/powerpc/kvm/book3s_rtas.c b/arch/powerpc/kvm/book3s_rtas.c
> index 20528701..2d3b2b1 100644
> --- a/arch/powerpc/kvm/book3s_rtas.c
> +++ b/arch/powerpc/kvm/book3s_rtas.c
> @@ -16,6 +16,7 @@
> #include <asm/kvm_ppc.h>
> #include <asm/hvcall.h>
> #include <asm/rtas.h>
> +#include <asm/xive.h>
>
> #ifdef CONFIG_KVM_XICS
> static void kvm_rtas_set_xive(struct kvm_vcpu *vcpu, struct rtas_args *args)
> @@ -32,7 +33,10 @@ static void kvm_rtas_set_xive(struct kvm_vcpu *vcpu, struct rtas_args *args)
> server = be32_to_cpu(args->args[1]);
> priority = be32_to_cpu(args->args[2]);
>
> - rc = kvmppc_xics_set_xive(vcpu->kvm, irq, server, priority);
> + if (xive_enabled())
> + rc = kvmppc_xive_set_xive(vcpu->kvm, irq, server, priority);
> + else
> + rc = kvmppc_xics_set_xive(vcpu->kvm, irq, server, priority);
> if (rc)
> rc = -3;
> out:
> @@ -52,7 +56,10 @@ static void kvm_rtas_get_xive(struct kvm_vcpu *vcpu, struct rtas_args *args)
> irq = be32_to_cpu(args->args[0]);
>
> server = priority = 0;
> - rc = kvmppc_xics_get_xive(vcpu->kvm, irq, &server, &priority);
> + if (xive_enabled())
> + rc = kvmppc_xive_get_xive(vcpu->kvm, irq, &server, &priority);
> + else
> + rc = kvmppc_xics_get_xive(vcpu->kvm, irq, &server, &priority);
> if (rc) {
> rc = -3;
> goto out;
> @@ -76,7 +83,10 @@ static void kvm_rtas_int_off(struct kvm_vcpu *vcpu, struct rtas_args *args)
>
> irq = be32_to_cpu(args->args[0]);
>
> - rc = kvmppc_xics_int_off(vcpu->kvm, irq);
> + if (xive_enabled())
> + rc = kvmppc_xive_int_off(vcpu->kvm, irq);
> + else
> + rc = kvmppc_xics_int_off(vcpu->kvm, irq);
> if (rc)
> rc = -3;
> out:
> @@ -95,7 +105,10 @@ static void kvm_rtas_int_on(struct kvm_vcpu *vcpu, struct rtas_args *args)
>
> irq = be32_to_cpu(args->args[0]);
>
> - rc = kvmppc_xics_int_on(vcpu->kvm, irq);
> + if (xive_enabled())
> + rc = kvmppc_xive_int_on(vcpu->kvm, irq);
> + else
> + rc = kvmppc_xics_int_on(vcpu->kvm, irq);
> if (rc)
> rc = -3;
> out:
> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
> index ef4fd52..e6829c4 100644
> --- a/arch/powerpc/kvm/book3s_xics.c
> +++ b/arch/powerpc/kvm/book3s_xics.c
> @@ -1307,8 +1307,8 @@ static int xics_set_source(struct kvmppc_xics *xics, long irq, u64 addr)
> return 0;
> }
>
> -int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
> - bool line_status)
> +int kvmppc_xics_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
> + bool line_status)
> {
> struct kvmppc_xics *xics = kvm->arch.xics;
>
> @@ -1317,14 +1317,6 @@ int kvm_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
> return ics_deliver_irq(xics, irq, level);
> }
>
> -int kvm_arch_set_irq_inatomic(struct kvm_kernel_irq_routing_entry *irq_entry,
> - struct kvm *kvm, int irq_source_id,
> - int level, bool line_status)
> -{
> - return kvm_set_irq(kvm, irq_source_id, irq_entry->gsi,
> - level, line_status);
> -}
> -
> static int xics_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
> {
> struct kvmppc_xics *xics = dev->private;
> @@ -1458,29 +1450,6 @@ void kvmppc_xics_free_icp(struct kvm_vcpu *vcpu)
> vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
> }
>
> -static int xics_set_irq(struct kvm_kernel_irq_routing_entry *e,
> - struct kvm *kvm, int irq_source_id, int level,
> - bool line_status)
> -{
> - return kvm_set_irq(kvm, irq_source_id, e->gsi, level, line_status);
> -}
> -
> -int kvm_irq_map_gsi(struct kvm *kvm,
> - struct kvm_kernel_irq_routing_entry *entries, int gsi)
> -{
> - entries->gsi = gsi;
> - entries->type = KVM_IRQ_ROUTING_IRQCHIP;
> - entries->set = xics_set_irq;
> - entries->irqchip.irqchip = 0;
> - entries->irqchip.pin = gsi;
> - return 1;
> -}
> -
> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
> -{
> - return pin;
> -}
> -
> void kvmppc_xics_set_mapped(struct kvm *kvm, unsigned long irq,
> unsigned long host_irq)
> {
> diff --git a/arch/powerpc/kvm/book3s_xics.h b/arch/powerpc/kvm/book3s_xics.h
> index ec5474c..5016676 100644
> --- a/arch/powerpc/kvm/book3s_xics.h
> +++ b/arch/powerpc/kvm/book3s_xics.h
> @@ -144,5 +144,10 @@ static inline struct kvmppc_ics *kvmppc_xics_find_ics(struct kvmppc_xics *xics,
> return ics;
> }
>
> +extern unsigned long xics_rm_h_xirr(struct kvm_vcpu *vcpu);
> +extern int xics_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr);
> +extern int xics_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr);
> +extern int xics_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr);
>
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> new file mode 100644
> index 0000000..acc882d
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -0,0 +1,1898 @@
> +/*
> + * Copyright 2017 Benjamin Herrenschmidt, IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation.
> + */
> +
> +#include <linux/kernel.h>
> +#include <linux/kvm_host.h>
> +#include <linux/err.h>
> +#include <linux/gfp.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/percpu.h>
> +#include <linux/cpumask.h>
> +#include <asm/uaccess.h>
> +#include <asm/kvm_book3s.h>
> +#include <asm/kvm_ppc.h>
> +#include <asm/hvcall.h>
> +#include <asm/xics.h>
> +#include <asm/xive.h>
> +#include <asm/debug.h>
> +#include <asm/time.h>
> +#include <asm/opal.h>
> +
> +#include <linux/debugfs.h>
> +#include <linux/seq_file.h>
> +
> +#include "book3s_xive.h"
> +#include "../sysdev/xive/xive-regs.h"
> +
> +//#define DBG(fmt...) printk("KVM/XIVE: " fmt)
> +#define DBG(fmt...) do { } while(0)
> +
> +#ifdef XIVE_RUNTIME_CHECKS
> +#define xive_assert(cond) WARN_ON(!(cond))
> +#else
> +#define xive_assert(cond) (false)
> +#endif
> +
> +/*
> + * Virtual mode variants of the hcalls for use on radix/radix
> + * with AIL. They require the VCPU's VP to be "pushed"
> + *
> + * We still instanciate them here because we use some of the
> + * generated utility functions as well in this file.
> + */
> +#define XIVE_RUNTIME_CHECKS
> +#define X_PFX xive_vm_
> +#define X_STATIC static
> +#define X_STAT_PFX stat_vm_
> +#define __x_tm_area xive_tm_area
> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> +#define __x_readb __raw_readb
> +#define __x_writeb __raw_writeb
> +#define __x_readw __raw_readw
> +#define __x_readq __raw_readq
> +#define __x_writeq __raw_writeq
> +
> +#include "book3s_xive_template.c"
> +
> +/* We leave a gap of a couple of interrupts in the queue to
> + * account for the IPI and additional safety guard
> + */
> +#define XIVE_Q_GAP 2
> +
> +/*
> + * This is a simple trigger for a generic XIVE IRQ. This must
> + * only be called for interrupts that support a trigger page
> + */
> +static bool xive_irq_trigger(struct xive_irq_data *xd)
> +{
> + /* This should be only for MSIs */
> + if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
> + return false;
> +
> + /* Those interrupts should always have a trigger page */
> + if (WARN_ON(!xd->trig_mmio))
> + return false;
> +
> + out_be64(xd->trig_mmio, 0);
> +
> + return true;
> +}
> +
> +static irqreturn_t xive_esc_irq(int irq, void *data)
> +{
> + struct kvm_vcpu *vcpu = data;
> +
> + /* We use the existing H_PROD mechanism to wake up the target */
> + vcpu->arch.prodded = 1;
> + smp_mb();
> + if (vcpu->arch.ceded)
> + kvmppc_fast_vcpu_kick(vcpu);
> +
> + return IRQ_HANDLED;
> +}
> +
> +static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct xive_q *q = &xc->queues[prio];
> + char *name = NULL;
> + int rc;
> +
> + /* Already there ? */
> + if (xc->esc_virq[prio])
> + return 0;
> +
> + /* Hook up the escalation interrupt */
> + xc->esc_virq[prio] = irq_create_mapping(NULL, q->esc_irq);
> + if (!xc->esc_virq[prio]) {
> + pr_err("XIVE-KVM: Failed to map escalation interrupt"
> + " for queue %d of VCPU %d\n",
> + prio, xc->server_num);
> + return -EIO;
> + }
> +
> + /*
> + * Future improvement: start with them disabled
> + * and handle DD2 and later scheme of merged escalation
> + * interrupts
> + */
> + name = kasprintf(GFP_KERNEL, "kvm-%d-%d-%d\n",
> + vcpu->kvm->arch.lpid, xc->server_num, prio);
> + if (!name) {
> + pr_err("XIVE-KVM: Failed to allocate escalation irq name"
> + " for queue %d of VCPU %d\n",
> + prio, xc->server_num);
> + rc = -ENOMEM;
> + goto error;
> + }
> + rc = request_irq(xc->esc_virq[prio], xive_esc_irq,
> + IRQF_NO_THREAD, name, vcpu);
> + if (rc) {
> + pr_err("XIVE-KVM: Failed to request escalation interrupt"
> + " for queue %d of VCPU %d\n",
> + prio, xc->server_num);
> + goto error;
> + }
> + xc->esc_virq_names[prio] = name;
> + return 0;
> + error:
> + irq_dispose_mapping(xc->esc_virq[prio]);
> + xc->esc_virq[prio] = 0;
> + kfree(name);
> + return rc;
> +}
> +
> +static int xive_provision_queue(struct kvm_vcpu *vcpu, u8 prio)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct kvmppc_xive *xive = xc->xive;
> + struct xive_q *q = &xc->queues[prio];
> + void *qpage;
> + int rc;
> +
> + if (WARN_ON(q->qpage))
> + return 0;
> +
> + /* Allocate the queue and retrieve infos on current node for now */
> + qpage = (__be32 *)__get_free_pages(GFP_KERNEL, xive->q_alloc_order);
Possibly q_page_order would be a better name than q_alloc_order.
> + if (!qpage) {
> + pr_err("XIVE-KVM: Failed to allocate queue %d for VCPU %d\n",
> + prio, xc->server_num);
> + return -ENOMEM;;
> + }
> + memset(qpage, 0, 1 << xive->q_order);
> +
> + /*
> + * Reconfigure the queue. This will set q->qpage only once the
> + * queue is fully configured. This is a requirement for prio 0
> + * as we will stop doing EOIs for every IPI as soon as we observe
> + * qpage being non-NULL, and instead will only EOI when we receive
> + * corresponding queue 0 entries
> + */
> + rc = xive_native_configure_queue(xc->vp_id, q, prio, qpage,
> + xive->q_order, true);
> + if (rc)
> + pr_err("XIVE-KVM: Failed to configure queue %d for VCPU %d\n",
> + prio, xc->server_num);
> + return rc;
> +}
> +
> +/* Called with kvm_lock held */
> +static int xive_check_provisioning(struct kvm *kvm, u8 prio)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvm_vcpu *vcpu;
> + int i, rc;
> +
> + lockdep_assert_held(&kvm->lock);
> +
> + /* Already provisioned ? */
> + if (xive->qmap & (1 << prio))
> + return 0;
> +
> + DBG("Provisioning prio... %d\n", prio);
> +
> + /* Provision each VCPU and enable escalations */
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + if (!vcpu->arch.xive_vcpu)
> + continue;
> + rc = xive_provision_queue(vcpu, prio);
> + if (rc == 0)
> + xive_attach_escalation(vcpu, prio);
> + if (rc)
> + return rc;
> + }
> +
> + /* Order previous stores and mark it as provisioned */
> + mb();
> + xive->qmap |= (1 << prio);
> + return 0;
> +}
> +
> +static void xive_inc_q_pending(struct kvm *kvm, u32 server, u8 prio)
> +{
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + struct xive_q *q;
> +
> + /* Locate target server */
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_warn("%s: Can't find server %d\n", __func__, server);
> + return;
> + }
> + xc = vcpu->arch.xive_vcpu;
> + if (WARN_ON(!xc))
> + return;
> +
> + q = &xc->queues[prio];
> + atomic_inc(&q->pending_count);
> +}
> +
> +static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct xive_q *q;
> + u32 max;
> +
> + if (WARN_ON(!xc))
> + return -ENXIO;
> + if (!xc->valid)
> + return -ENXIO;
> +
> + q = &xc->queues[prio];
> + if (WARN_ON(!q->qpage))
> + return -ENXIO;
> +
> + /* Calculate max number of interrupts in that queue. */
> + max = (q->msk + 1) - XIVE_Q_GAP;
> + return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
> +}
> +
> +static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
> +{
> + struct kvm_vcpu *vcpu;
> + int i, rc;
> +
> + /* Locate target server */
> + vcpu = kvmppc_xive_find_server(kvm, *server);
> + if (!vcpu) {
> + DBG("Can't find server %d\n", *server);
> + return -EINVAL;
> + }
> +
> + DBG("Finding irq target on 0x%x/%d...\n", *server, prio);
> +
> + /* Try pick it */
> + rc = xive_try_pick_queue(vcpu, prio);
> + if (rc == 0)
> + return rc;
> +
> + DBG(" .. failed, looking up candidate...\n");
> +
> + /* Failed, pick another VCPU */
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + if (!vcpu->arch.xive_vcpu)
> + continue;
> + rc = xive_try_pick_queue(vcpu, prio);
> + if (rc == 0) {
> + *server = vcpu->arch.xive_vcpu->server_num;
> + DBG(" found on 0x%x/%d\n", *server, prio);
> + return rc;
> + }
> + }
> + DBG(" no available target !\n");
> +
> + /* No available target ! */
> + return -EBUSY;
> +}
> +
> +static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
> + struct kvmppc_xive_src_block *sb,
> + struct kvmppc_xive_irq_state *state)
> +{
> + struct xive_irq_data *xd;
> + u32 hw_num;
> + u8 old_prio;
> + u64 val;
> +
> + /*
> + * Take the lock, set masked, try again if racing
> + * with H_EOI
> + */
> + for (;;) {
> + arch_spin_lock(&sb->lock);
> + old_prio = state->guest_priority;
> + state->guest_priority = MASKED;
> + mb();
> + if (!state->in_eoi)
> + break;
> + state->guest_priority = old_prio;
> + arch_spin_unlock(&sb->lock);
> + }
> +
> + /* No change ? Bail */
> + if (old_prio == MASKED)
> + return old_prio;
> +
> + /* Get the right irq */
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> +
> + /*
> + * If the interrupt is marked as needing masking via
> + * firmware, we do it here. Firmware masking however
> + * is "lossy", it won't return the old p and q bits
> + * and won't set the interrupt to a state where it will
> + * record queued ones. If this is an issue we should do
> + * lazy masking instead.
> + *
> + * For now, we work around this in unmask by forcing
> + * an interrupt whenever we unmask a non-LSI via FW
> + * (if ever).
> + */
> + if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
> + xive_native_configure_irq(hw_num,
> + xive->vp_base + state->act_server,
> + MASKED, state->number);
> + /* set old_p so we can track if an H_EOI was done */
> + state->old_p = true;
> + state->old_q = false;
> + } else {
> + /* Set PQ to 10, return old P and old Q and remember them */
> + val = xive_vm_esb_load(xd, XIVE_ESB_SET_PQ_10);
> + state->old_p = !!(val & 2);
> + state->old_q = !!(val & 1);
> +
> + /*
> + * Synchronize hardware to sensure the queues are updated
> + * when masking
> + */
> + xive_native_sync_source(hw_num);
> + }
> +
> + return old_prio;
> +}
> +
> +static void xive_lock_for_unmask(struct kvmppc_xive_src_block *sb,
> + struct kvmppc_xive_irq_state *state)
> +{
> + /*
> + * Take the lock try again if racing with H_EOI
> + */
> + for (;;) {
> + arch_spin_lock(&sb->lock);
> + if (!state->in_eoi)
> + break;
> + arch_spin_unlock(&sb->lock);
> + }
> +}
> +
> +static void xive_finish_unmask(struct kvmppc_xive *xive,
> + struct kvmppc_xive_src_block *sb,
> + struct kvmppc_xive_irq_state *state,
> + u8 prio)
> +{
> + struct xive_irq_data *xd;
> + u32 hw_num;
> +
> + /* If we aren't changing a thing, move on */
> + if (state->guest_priority != MASKED)
> + goto bail;
> +
> + /* Get the right irq */
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> +
> + /*
> + * See command in xive_lock_and_mask() concerning masking
> + * via firmware.
> + */
> + if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
> + xive_native_configure_irq(hw_num,
> + xive->vp_base + state->act_server,
> + state->act_priority, state->number);
> + /* If an EOI is needed, do it here */
> + if (!state->old_p)
> + xive_vm_source_eoi(hw_num, xd);
> + /* If this is not an LSI, force a trigger */
> + if (!(xd->flags & OPAL_XIVE_IRQ_LSI))
> + xive_irq_trigger(xd);
> + goto bail;
> + }
> +
> + /* Old Q set, set PQ to 11 */
> + if (state->old_q)
> + xive_vm_esb_load(xd, XIVE_ESB_SET_PQ_11);
> +
> + /*
> + * If not old P, then perform an "effective" EOI,
> + * on the source. This will handle the cases where
> + * FW EOI is needed.
> + */
> + if (!state->old_p)
> + xive_vm_source_eoi(hw_num, xd);
> +
> + /* Synchronize ordering and mark unmasked */
> + mb();
> + bail:
> + state->guest_priority = prio;
> +}
> +
> +/*
> + * Target an interrupt to a given server/prio, this will fallback
> + * to another server if necessary and perform the HW targetting
> + * updates as needed
> + *
> + * NOTE: Must be called with the state lock held
> + */
> +static int xive_target_interrupt(struct kvm *kvm,
> + struct kvmppc_xive_irq_state *state,
> + u32 server, u8 prio)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + u32 hw_num;
> + int rc;
> +
> + /*
> + * This will return a tentative server and actual
> + * priority. The count for that new target will have
> + * already been incremented.
> + */
> + rc = xive_select_target(kvm, &server, prio);
> +
> + /* We failed to find a target ? Not much we can do
> + * at least until we support the GIQ.
> + */
> + if (rc)
> + return rc;
> +
> + /*
> + * Increment the old queue pending count if there
> + * was one so that the old queue count gets adjusted later
> + * when observed to be empty.
> + */
> + if (state->act_priority != MASKED)
> + xive_inc_q_pending(kvm,
> + state->act_server,
> + state->act_priority);
> + /*
> + * Update state and HW
> + */
> + state->act_priority = prio;
> + state->act_server = server;
> +
> + /* Get the right irq */
> + kvmppc_xive_select_irq(state, &hw_num, NULL);
> +
> + return xive_native_configure_irq(hw_num,
> + xive->vp_base + server,
> + prio, state->number);
> +}
> +
> +/*
> + * Targetting rules: In order to avoid losing track of
> + * pending interrupts accross mask and unmask, which would
> + * allow queue overflows, we implement the following rules:
> + *
> + * - Unless it was never enabled (or we run out of capacity)
> + * an interrupt is always targetted at a valid server/queue
> + * pair even when "masked" by the guest. This pair tends to
> + * be the last one used but it can be changed under some
> + * circumstances. That allows us to separate targetting
> + * from masking, we only handle accounting during (re)targetting,
> + * this also allows us to let an interrupt drain into its target
> + * queue after masking, avoiding complex schemes to remove
> + * interrupts out of remote processor queues.
> + *
> + * - When masking, we set PQ to 10 and save the previous value
> + * of P and Q.
> + *
> + * - When unmasking, if saved Q was set, we set PQ to 11
> + * otherwise we leave PQ to the HW state which will be either
> + * 10 if nothing happened or 11 if the interrupt fired while
> + * masked. Effectively we are OR'ing the previous Q into the
> + * HW Q.
> + *
> + * Then if saved P is clear, we do an effective EOI (Q->P->Trigger)
> + * which will unmask the interrupt and shoot a new one if Q was
> + * set.
> + *
> + * Otherwise (saved P is set) we leave PQ unchanged (so 10 or 11,
> + * effectively meaning an H_EOI from the guest is still expected
> + * for that interrupt).
> + *
> + * - If H_EOI occurs while masked, we clear the saved P.
> + *
> + * - When changing target, we account on the new target and
> + * increment a separate "pending" counter on the old one.
> + * This pending counter will be used to decrement the old
> + * target's count when its queue has been observed empty.
> + */
> +
> +int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> + u32 priority)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u8 new_act_prio;
> + int rc = 0;
> + u16 idx;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + DBG("set_xive ! irq 0x%x server 0x%x prio %d\n",
> + irq, server, priority);
> +
> + /* First, check provisioning of queues */
> + if (priority != MASKED)
> + rc = xive_check_provisioning(xive->kvm,
> + xive_prio_from_guest(priority));
> + if (rc) {
> + DBG(" provisioning failure %d !\n", rc);
> + return rc;
> + }
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return -EINVAL;
> + state = &sb->irq_state[idx];
> +
> + /*
> + * We first handle masking/unmasking since the locking
> + * might need to be retried due to EOIs, we'll handle
> + * targetting changes later. These functions will return
> + * with the SB lock held.
> + *
> + * xive_lock_and_mask() will also set state->guest_priority
> + * but won't otherwise change other fields of the state.
> + *
> + * xive_lock_for_unmask will not actually unmask, this will
> + * be done later by xive_finish_unmask() once the targetting
> + * has been done, so we don't try to unmask an interrupt
> + * that hasn't yet been targetted.
> + */
> + if (priority == MASKED)
> + xive_lock_and_mask(xive, sb, state);
> + else
> + xive_lock_for_unmask(sb, state);
> +
> +
> + /*
> + * Then we handle targetting.
> + *
> + * First calculate a new "actual priority"
> + */
> + new_act_prio = state->act_priority;
> + if (priority != MASKED)
> + new_act_prio = xive_prio_from_guest(priority);
> +
> + DBG(" new_act_prio=%x act_server=%x act_prio=%x\n",
> + new_act_prio, state->act_server, state->act_priority);
> +
> + /*
> + * Then check if we actually need to change anything,
> + *
> + * The condition for re-targetting the interrupt is that
> + * we have a valid new priority (new_act_prio is not 0xff)
> + * and either the server or the priority changed.
> + *
> + * Note: If act_priority was ff and the new priority is
> + * also ff, we don't do anything and leave the interrupt
> + * untargetted. An attempt of doing an int_on on an
> + * untargetted interrupt will fail. If that is a problem
> + * we could initialize interrupts with valid default
> + */
> +
> + if (new_act_prio != MASKED &&
> + (state->act_server != server ||
> + state->act_priority != new_act_prio))
> + rc = xive_target_interrupt(kvm, state, server, new_act_prio);
> +
> + /*
> + * Perform the final unmasking of the interrupt source
> + * if necessary
> + */
> + if (priority != MASKED)
> + xive_finish_unmask(xive, sb, state, priority);
> +
> + /*
> + * Finally Update saved_priority to match. Only int_on/off
> + * set this field to a different value.
> + */
> + state->saved_priority = priority;
> +
> + arch_spin_unlock(&sb->lock);
> + return rc;
> +}
> +
> +int kvmppc_xive_get_xive(struct kvm *kvm, u32 irq, u32 *server,
> + u32 *priority)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u16 idx;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return -EINVAL;
> + state = &sb->irq_state[idx];
> + arch_spin_lock(&sb->lock);
> + *server = state->guest_server;
> + *priority = state->guest_priority;
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> +int kvmppc_xive_int_on(struct kvm *kvm, u32 irq)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u16 idx;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return -EINVAL;
> + state = &sb->irq_state[idx];
> +
> + DBG("int_on(irq=0x%x)\n", irq);
> +
> + /*
> + * Check if interrupt was not targetted
> + */
> + if (state->act_priority == MASKED) {
> + DBG("int_on on untargetted interrupt\n");
> + return -EINVAL;
> + }
> +
> + /* If saved_priority is 0xff, do nothing */
> + if (state->saved_priority == MASKED)
> + return 0;
> +
> + /*
> + * Lock and unmask it.
> + */
> + xive_lock_for_unmask(sb, state);
> + xive_finish_unmask(xive, sb, state, state->saved_priority);
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> +int kvmppc_xive_int_off(struct kvm *kvm, u32 irq)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u16 idx;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return -EINVAL;
> + state = &sb->irq_state[idx];
> +
> + DBG("int_off(irq=0x%x)\n", irq);
> +
> + /*
> + * Lock and mask
> + */
> + state->saved_priority = xive_lock_and_mask(xive, sb, state);
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> +static bool xive_restore_pending_irq(struct kvmppc_xive *xive, u32 irq)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u16 idx;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return false;
> + state = &sb->irq_state[idx];
> + if (!state->valid)
> + return false;
> +
> + /*
> + * Trigger the IPI. This assumes we never restore a pass-through
> + * interrupt which should be safe enough
> + */
> + xive_irq_trigger(&state->ipi_data);
> +
> + return true;
> +}
> +
> +u64 kvmppc_xive_get_icp(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> +
> + if (!xc)
> + return 0;
> +
> + /* Return the per-cpu state for state saving/migration */
> + return (u64)xc->cppr << KVM_REG_PPC_ICP_CPPR_SHIFT |
> + (u64)xc->mfrr << KVM_REG_PPC_ICP_MFRR_SHIFT;
> +}
> +
> +int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
> + u8 cppr, mfrr;
> + u32 xisr;
> +
> + if (!xc || !xive)
> + return -ENOENT;
> +
> + /* Grab individual state fields. We don't use pending_pri */
> + cppr = icpval >> KVM_REG_PPC_ICP_CPPR_SHIFT;
> + xisr = (icpval >> KVM_REG_PPC_ICP_XISR_SHIFT) &
> + KVM_REG_PPC_ICP_XISR_MASK;
> + mfrr = icpval >> KVM_REG_PPC_ICP_MFRR_SHIFT;
> +
> + DBG("set_icp vcpu %d cppr=0x%x mfrr=0x%x xisr=0x%x\n",
> + xc->server_num, cppr, mfrr, xisr);
> +
> + /*
> + * We can't update the state of a "pushed" VCPU, but that
> + * shouldn't happen.
> + */
> + if (WARN_ON(vcpu->arch.xive_pushed))
> + return -EIO;
> +
> + /* Update VCPU HW saved state */
> + vcpu->arch.xive_saved_state.cppr = cppr;
> + xc->hw_cppr = xc->cppr = cppr;
> +
> + /*
> + * Update MFRR state. If it's not 0xff, we mark the VCPU as
> + * having a pending MFRR change, which will re-evaluate the
> + * target. The VCPU will thus potentially get a spurious
> + * interrupt but that's not a big deal.
> + */
> + xc->mfrr = mfrr;
> + if (mfrr < cppr)
> + xive_irq_trigger(&xc->vp_ipi_data);
> +
> + /*
> + * Now saved XIRR is "interesting". It means there's something in
> + * the legacy "1 element" queue... for an IPI we simply ignore it,
> + * as the MFRR restore will handle that. For anything else we need
> + * to force a resend of the source.
> + * However the source may not have been setup yet. If that's the
> + * case, we keep that info and increment a counter in the xive to
> + * tell subsequent xive_set_source() to go look.
> + */
> + if (xisr > XICS_IPI && !xive_restore_pending_irq(xive, xisr)) {
> + xc->delayed_irq = xisr;
> + xive->delayed_irqs++;
> + DBG(" xisr restore delayed\n");
> + }
> +
> + return 0;
> +}
> +
> +int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
> + struct irq_desc *host_desc)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + struct irq_data *host_data = irq_desc_get_irq_data(host_desc);
> + unsigned int host_irq = irq_desc_get_irq(host_desc);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(host_data);
> + u16 idx;
> + u8 prio;
> + int rc;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + DBG("set_mapped girq 0x%lx host HW irq 0x%x...\n", guest_irq, hw_irq);
> +
> + sb = kvmppc_xive_find_source(xive, guest_irq, &idx);
> + if (!sb)
> + return -EINVAL;
> + state = &sb->irq_state[idx];
> +
> + /*
> + * Mark the passed-through interrupt as going to a VCPU,
> + * this will prevent further EOIs and similar operations
> + * from the XIVE code. It will also mask the interrupt
> + * to either PQ=10 or 11 state, the latter if the interrupt
> + * is pending. This will allow us to unmask or retrigger it
> + * after routing it to the guest with a simple EOI.
> + *
> + * The "state" argument is a "token", all it needs is to be
> + * non-NULL to switch to passed-through or NULL for the
> + * other way around. We may not yet have an actual VCPU
> + * target here and we don't really care.
> + */
> + rc = irq_set_vcpu_affinity(host_irq, state);
> + if (rc) {
> + pr_err("Failed to set VCPU affinity for irq %d\n", host_irq);
> + return rc;
> + }
> +
> + /*
> + * Mask and read state of IPI. We need to know if its P bit
> + * is set as that means it's potentially already using a
> + * queue entry in the target
> + */
> + prio = xive_lock_and_mask(xive, sb, state);
> + DBG(" old IPI prio %02x P:%d Q:%d\n", prio, state->old_p, state->old_q);
> +
> + /* Turn the IPI hard off */
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> +
> + /* Grab info about irq */
> + state->pt_number = hw_irq;
> + state->pt_data = irq_data_get_irq_handler_data(host_data);
> +
> + /*
> + * Configure the IRQ to match the existing configuration of
> + * the IPI if it was already targetted. Otherwise this will
> + * mask the interrupt in a lossy way (act_priority is 0xff)
> + * which is fine for a never started interrupt.
> + */
> + xive_native_configure_irq(hw_irq,
> + xive->vp_base + state->act_server,
> + state->act_priority, state->number);
> +
> + /*
> + * We do an EOI to enable the interrupt (and retrigger if needed)
> + * if the guest has the interrupt unmasked and the P bit was *not*
> + * set in the IPI. If it was set, we know a slot may still be in
> + * use in the target queue thus we have to wait for a guest
> + * originated EOI
> + */
> + if (prio != MASKED && !state->old_p)
> + xive_vm_source_eoi(hw_irq, state->pt_data);
> +
> + /* Clear old_p/old_q as they are no longer relevant */
> + state->old_p = state->old_q = false;
> +
> + /* Restore guest prio (unlocks EOI) */
> + mb();
> + state->guest_priority = prio;
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(kvmppc_xive_set_mapped);
> +
> +int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
> + struct irq_desc *host_desc)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + unsigned int host_irq = irq_desc_get_irq(host_desc);
> + u16 idx;
> + u8 prio;
> + int rc;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + DBG("clr_mapped girq 0x%lx...\n", guest_irq);
> +
> + sb = kvmppc_xive_find_source(xive, guest_irq, &idx);
> + if (!sb)
> + return -EINVAL;
> + state = &sb->irq_state[idx];
> +
> + /*
> + * Mask and read state of IRQ. We need to know if its P bit
> + * is set as that means it's potentially already using a
> + * queue entry in the target
> + */
> + prio = xive_lock_and_mask(xive, sb, state);
> + DBG(" old IRQ prio %02x P:%d Q:%d\n", prio, state->old_p, state->old_q);
> +
> + /*
> + * If old_p is set, the interrupt is pending, we switch it to
> + * PQ=11. This will force a resend in the host so the interrupt
> + * isn't lost to whatver host driver may pick it up
> + */
> + if (state->old_p)
> + xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_11);
> +
> + /* Relase the passed-through interrupt to the host */
^^^^^^ Release
> + rc = irq_set_vcpu_affinity(host_irq, NULL);
> + if (rc) {
> + pr_err("Failed to clr VCPU affinity for irq %d\n", host_irq);
> + return rc;
> + }
> +
> + /* Forget about the IRQ */
> + state->pt_number = 0;
> + state->pt_data = NULL;
> +
> + /* Reconfigure the IPI */
> + xive_native_configure_irq(state->ipi_number,
> + xive->vp_base + state->act_server,
> + state->act_priority, state->number);
> +
> + /*
> + * If old_p is set (we have a queue entry potentially
> + * occupied) or the interrupt is masked, we set the IPI
> + * to PQ=10 state. Otherwise we just re-enable it (PQ=00).
> + */
> + if (prio == MASKED || state->old_p)
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_10);
> + else
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_00);
> +
> + /* Restore guest prio (unlocks EOI) */
> + mb();
> + state->guest_priority = prio;
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
> +
> +static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct kvm *kvm = vcpu->kvm;
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + int i, j;
> +
> + for (i = 0; i <= xive->max_sbid; i++) {
> + struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
> +
> + if (!sb)
> + continue;
> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
> +
> + if (!state->valid)
> + continue;
> + if (state->act_priority == MASKED)
> + continue;
> + if (state->act_server != xc->server_num)
> + continue;
> +
> + /* Clean it up */
> + arch_spin_lock(&sb->lock);
> + state->act_priority = MASKED;
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> + if (state->pt_number) {
> + xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->pt_number, 0, MASKED, 0);
> + }
> + arch_spin_unlock(&sb->lock);
> + }
> + }
> +}
> +
> +void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct kvmppc_xive *xive = xc->xive;
> + int i;
> +
> + DBG("cleanup_vcpu(cpu=%d)\n", xc->server_num);
> +
> + /* Ensure no interrupt is still routed to that VP */
> + xc->valid = false;
> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
> +
> + /* Mask the VP IPI */
> + xive_vm_esb_load(&xc->vp_ipi_data, XIVE_ESB_SET_PQ_01);
> +
> + /* Disable the VP */
> + xive_native_disable_vp(xc->vp_id);
> +
> + /* Free the queues & associated interrupts */
> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> + struct xive_q *q = &xc->queues[i];
> +
> + /* Free the escalation irq */
> + if (xc->esc_virq[i]) {
> + free_irq(xc->esc_virq[i], vcpu);
> + irq_dispose_mapping(xc->esc_virq[i]);
> + kfree(xc->esc_virq_names[i]);
> + }
> + /* Free the queue */
> + xive_native_disable_queue(xc->vp_id, q, i);
> + if (q->qpage) {
> + free_pages((unsigned long)q->qpage,
> + xive->q_alloc_order);
> + q->qpage = NULL;
> + }
> + }
> +
> + /* Free the IPI */
> + if (xc->vp_ipi) {
> + xive_cleanup_irq_data(&xc->vp_ipi_data);
> + xive_native_free_irq(xc->vp_ipi);
> + }
> + /* Free the VP */
> + kfree(xc);
> +}
> +
> +int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu)
> +{
> + struct kvmppc_xive *xive = dev->private;
> + struct kvmppc_xive_vcpu *xc;
> + int i, r = -EBUSY;
> +
> + DBG("connect_vcpu(cpu=%d)\n", cpu);
> +
> + if (dev->ops != &kvm_xive_ops) {
> + DBG("Wrong ops !\n");
> + return -EPERM;
> + }
> + if (xive->kvm != vcpu->kvm)
> + return -EPERM;
> + if (vcpu->arch.irq_type)
> + return -EBUSY;
> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
> + DBG("Duplicate !\n");
> + return -EEXIST;
> + }
> + if (cpu >= KVM_MAX_VCPUS) {
> + DBG("Out of bounds !\n");
> + return -EINVAL;
> + }
> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
> + if (!xc)
> + return -ENOMEM;
> +
> + /* We need to synchronize with queue provisioning */
> + mutex_lock(&vcpu->kvm->lock);
> + vcpu->arch.xive_vcpu = xc;
> + xc->xive = xive;
> + xc->vcpu = vcpu;
> + xc->server_num = cpu;
> + xc->vp_id = xive->vp_base + cpu;
> + xc->mfrr = 0xff;
> + xc->valid = true;
> +
> + r = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
> + if (r)
> + goto bail;
> +
> + /* Configure VCPU fields for use by assembly push/pull */
> + vcpu->arch.xive_saved_state.qw = cpu_to_be64(0xff000000);
> + vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
> +
> + /* Allocate IPI */
> + xc->vp_ipi = xive_native_alloc_irq();
> + if (!xc->vp_ipi) {
> + r = -EIO;
> + goto bail;
> + }
> + DBG(" IPI=0x%x\n", xc->vp_ipi);
> +
> + r = xive_native_populate_irq_data(xc->vp_ipi, &xc->vp_ipi_data);
> + if (r)
> + goto bail;
> +
> + /*
> + * Initialize queues. Initially we set them all for no queueing
> + * and we enable escalation for queue 0 only which we'll use for
> + * our mfrr change notifications. If the VCPU is hot-plugged, we
> + * do handle provisioning however.
> + */
> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> + struct xive_q *q = &xc->queues[i];
> +
> + /* Is queue already enabled ? Provision it */
> + if (xive->qmap & (1 << i)) {
> + r = xive_provision_queue(vcpu, i);
> + if (r == 0)
> + xive_attach_escalation(vcpu, i);
> + if (r)
> + goto bail;
> + } else {
> + r = xive_native_configure_queue(xc->vp_id,
> + q, i, NULL, 0, true);
> + if (r) {
> + pr_err("XIVE-KVM: Failed to configure queue %d"
> + " for VCPU %d\n",
> + i, cpu);
> + goto bail;
> + }
> + }
> + }
> +
> + /* If not done above, attach priority 0 escalation */
> + r = xive_attach_escalation(vcpu, 0);
> + if (r)
> + goto bail;
> +
> + /* Enable the VP */
> + r = xive_native_enable_vp(xc->vp_id);
> + if (r)
> + goto bail;
> +
> + /* Route the IPI */
> + r = xive_native_configure_irq(xc->vp_ipi, xc->vp_id, 0, XICS_IPI);
> + if (!r)
> + xive_vm_esb_load(&xc->vp_ipi_data, XIVE_ESB_SET_PQ_00);
> +
> + bail:
> + mutex_unlock(&vcpu->kvm->lock);
> + if (r) {
> + kvmppc_xive_cleanup_vcpu(vcpu);
> + return r;
> + }
> +
> + vcpu->arch.irq_type = KVMPPC_IRQ_XICS;
> + return 0;
> +}
> +
> +/*
> + * Scanning of queues before/after migration save
> + */
> +static void xive_pre_save_set_queued(struct kvmppc_xive *xive, u32 irq)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u16 idx;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return;
> +
> + state = &sb->irq_state[idx];
> +
> + /* Some sanity checking */
> + if (!state->valid) {
> + pr_err("XIVE/XIVE: invalid irq 0x%x in cpu queue!\n", irq);
> + return;
> + }
> +
> + /*
> + * If the interrupt is in a queue it should have P set.
> + * We warn so that gets reported. A backtrace isn't useful
> + * so no need to use a WARN_ON.
> + */
> + if (!state->saved_p)
> + pr_err("KVM/XIVE: Interrupt 0x%x is marked in a queue"
> + " but P not set !\n", irq);
> +
> + /* Set flag */
> + state->in_queue = true;
> +}
> +
> +static void xive_pre_scan_mask_irq(struct kvmppc_xive *xive,
> + struct kvmppc_xive_src_block *sb,
> + u32 irq)
> +{
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[irq];
> +
> + if (!state->valid)
> + return;
> +
> + /* Mask and save state, this will also sync HW queues */
> + state->saved_scan_prio = xive_lock_and_mask(xive, sb, state);
> +
> + /* Transfer P and Q */
> + state->saved_p = state->old_p;
> + state->saved_q = state->old_q;
> +
> + /* Unlock */
> + arch_spin_unlock(&sb->lock);
> +}
> +
> +static void xive_pre_scan_unmask_irq(struct kvmppc_xive *xive,
I think a better name would be "xive_pre_save_unmask", since this is
actually called after the scan.
> + struct kvmppc_xive_src_block *sb,
> + u32 irq)
> +{
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[irq];
> +
> + if (!state->valid)
> + return;
> +
> + /*
> + * Lock / exclude EOI (not technically necessary if the
> + * guest isn't running concurrently. If this becomes a
> + * performance issue we can probably remove the lock.
> + */
> + xive_lock_for_unmask(sb, state);
> +
> + /* Restore mask/prio if it wasn't masked */
> + if (state->saved_scan_prio != MASKED)
> + xive_finish_unmask(xive, sb, state, state->saved_scan_prio);
> +
> + /* Unlock */
> + arch_spin_unlock(&sb->lock);
> +}
> +
> +static void xive_pre_save_queue(struct kvmppc_xive *xive, struct xive_q *q)
> +{
> + u32 idx = q->idx;
> + u32 toggle = q->toggle;
> + u32 irq;
> +
> + do {
> + irq = __xive_read_eq(q->qpage, q->msk, &idx, &toggle);
> + if (irq > XICS_IPI)
> + xive_pre_save_set_queued(xive, irq);
> + } while(irq);
> +}
> +
> +static void xive_pre_save_scan(struct kvmppc_xive *xive)
> +{
> + struct kvm_vcpu *vcpu = NULL;
> + int i, j;
> +
> + /*
> + * See comment in xive_get_source() about how this
> + * work. Collect a stable state for all interrupts
> + */
> + for (i = 0; i <= xive->max_sbid; i++) {
> + struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
> + if (!sb)
> + continue;
> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++)
> + xive_pre_scan_mask_irq(xive, sb, j);
> + }
> +
> + /* Then scan the queues and update the "in_queue" flag */
> + kvm_for_each_vcpu(i, vcpu, xive->kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + if (!xc)
> + continue;
> + for (j = 0; j < KVMPPC_XIVE_Q_COUNT; j++) {
> + if (xc->queues[i].qpage)
> + xive_pre_save_queue(xive, &xc->queues[i]);
> + }
> + }
> +
> + /* Finally restore interrupt states */
> + for (i = 0; i <= xive->max_sbid; i++) {
> + struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
> + if (!sb)
> + continue;
> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++)
> + xive_pre_scan_unmask_irq(xive, sb, j);
> + }
> +}
> +
> +static void xive_post_save_scan(struct kvmppc_xive *xive)
> +{
> + u32 i, j;
> +
> + /* Clear all the in_queue flags */
> + for (i = 0; i <= xive->max_sbid; i++) {
> + struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
> + if (!sb)
> + continue;
> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++)
> + sb->irq_state[j].in_queue = false;
> + }
> +
> + /* Next get_source() will do a new scan */
> + xive->saved_src_count = 0;
> +}
> +
> +/*
> + * This returns the source configuration and state to user space.
> + */
> +static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u64 __user *ubufp = (u64 __user *) addr;
> + u64 val, prio;
> + u16 idx;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return -ENOENT;
> +
> + state = &sb->irq_state[idx];
> +
> + if (!state->valid)
> + return -ENOENT;
> +
> + DBG("get_source(%ld)...\n", irq);
> +
> + /*
> + * So to properly save the state into something that looks like a
> + * XICS migration stream we cannot treat interrupts individually.
> + *
> + * We need, instead, mask them all (& save their previous PQ state)
> + * to get a stable state in the HW, then sync them to ensure that
> + * any interrupt that had already fired hits its queue, and finally
> + * scan all the queues to collect which interrupts are still present
> + * in the queues, so we can set the "pending" flag on them and
> + * they can be resent on restore.
> + *
> + * So we do it all when the "first" interrupt gets saved, all the
> + * state is collected at that point, the rest of xive_get_source()
> + * will merely collect and convert that state to the expected
> + * userspace bit mask.
> + */
> + if (xive->saved_src_count == 0)
> + xive_pre_save_scan(xive);
> + xive->saved_src_count++;
> +
> + /* Convert saved state into something compatible with xics */
> + val = state->guest_server;
> + prio = state->saved_scan_prio;
> +
> + if (prio == MASKED) {
> + val |= KVM_XICS_MASKED;
> + prio = state->saved_priority;
> + }
> + val |= prio << KVM_XICS_PRIORITY_SHIFT;
> + if (state->lsi) {
> + val |= KVM_XICS_LEVEL_SENSITIVE;
> + if (state->saved_p)
> + val |= KVM_XICS_PENDING;
> + } else {
> + if (state->saved_p)
> + val |= KVM_XICS_PRESENTED;
> +
> + if (state->saved_q)
> + val |= KVM_XICS_QUEUED;
> +
> + /*
> + * We mark it pending (which will attempt a re-delivery)
> + * if we are in a queue *or* we were masked and had
> + * Q set which is equivalent to the XICS "masked pending"
> + * state
> + */
> + if (state->in_queue || (prio == MASKED && state->saved_q))
> + val |= KVM_XICS_PENDING;
> + }
> +
> + /*
> + * If that was the last interrupt saved, reset the
> + * in_queue flags
> + */
> + if (xive->saved_src_count == xive->src_count)
> + xive_post_save_scan(xive);
> +
> + /* Copy the result to userspace */
> + if (put_user(val, ubufp))
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> +static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> + int irq)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvmppc_xive_src_block *sb;
> + int i, bid;
> +
> + bid = irq >> KVMPPC_XICS_ICS_SHIFT;
> +
> + mutex_lock(&kvm->lock);
> +
> + /* block already exists - somebody else got here first */
> + if (xive->src_blocks[bid])
> + goto out;
> +
> + /* Create the ICS */
> + sb = kzalloc(sizeof(*sb), GFP_KERNEL);
> + if (!sb)
> + goto out;
> +
> + sb->id = bid;
> +
> + for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
> + sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
> + sb->irq_state[i].guest_priority = MASKED;
> + sb->irq_state[i].saved_priority = MASKED;
> + sb->irq_state[i].act_priority = MASKED;
> + }
> + smp_wmb();
> + xive->src_blocks[bid] = sb;
> +
> + if (bid > xive->max_sbid)
> + xive->max_sbid = bid;
> +
> + out:
> + mutex_unlock(&kvm->lock);
> + return xive->src_blocks[bid];
> +}
> +
> +static bool xive_check_delayed_irq(struct kvmppc_xive *xive, u32 irq)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu = NULL;
> + int i;
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> +
> + if (!xc)
> + continue;
> +
> + if (xc->delayed_irq == irq) {
> + xc->delayed_irq = 0;
> + xive->delayed_irqs--;
> + return true;
> + }
> + }
> + return false;
> +}
> +
> +static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u64 __user *ubufp = (u64 __user *) addr;
> + u16 idx;
> + u64 val;
> + u8 act_prio, guest_prio;
> + u32 server;
> + int rc = 0;
> +
> + if (irq < KVMPPC_XICS_FIRST_IRQ || irq >= KVMPPC_XICS_NR_IRQS)
> + return -ENOENT;
> +
> + DBG("set_source(irq=0x%lx)\n", irq);
> +
> + /* Find the source */
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb) {
> + DBG("No source, creating source block...\n");
> + sb = xive_create_src_block(xive, irq);
> + if (!sb) {
> + DBG("Failed to create block...\n");
> + return -ENOMEM;
> + }
> + }
> + state = &sb->irq_state[idx];
> +
> + /* Read user passed data */
> + if (get_user(val, ubufp)) {
> + DBG("fault getting user info !\n");
> + return -EFAULT;
> + }
> +
> + server = val & KVM_XICS_DESTINATION_MASK;
> + guest_prio = val >> KVM_XICS_PRIORITY_SHIFT;
> +
> + DBG(" val=0x016%llx (server=0x%x, guest_prio=%d)\n",
> + val, server, guest_prio);
> + /*
> + * If the source doesn't already have an IPI, allocate
> + * one and get the corresponding data
> + */
> + if (!state->ipi_number) {
> + state->ipi_number = xive_native_alloc_irq();
> + if (state->ipi_number == 0) {
> + DBG("Failed to allocate IPI !\n");
> + return -ENOMEM;
> + }
> + xive_native_populate_irq_data(state->ipi_number, &state->ipi_data);
> + DBG(" src_ipi=0x%x\n", state->ipi_number);
> + }
> +
> + /*
> + * We use lock_and_mask() to set us in the right masked
> + * state. We will override that state from the saved state
> + * further down, but this will handle the cases of interrupts
> + * that need FW masking. We set the initial guest_priority to
> + * 0 before calling it to ensure it actually performs the masking.
> + */
> + state->guest_priority = 0;
> + xive_lock_and_mask(xive, sb, state);
> +
> + /*
> + * Now, we select a target if we have one. If we don't we
> + * leave the interrupt untargetted. It means that an interrupt
> + * can become "untargetted" accross migration if it was masked
> + * by set_xive() but there is little we can do about it.
> + */
> +
> + /* First convert prio and mark interrupt as untargetted */
> + act_prio = xive_prio_from_guest(guest_prio);
> + state->act_priority = MASKED;
> + state->guest_server = server;
> +
> + /*
> + * We need to drop the lock due to the mutex below. Hopefully
> + * nothing is touching that interrupt yet since it hasn't been
> + * advertized to a running guest yet
> + */
> + arch_spin_unlock(&sb->lock);
> +
> + /* If we have a priority target the interrupt */
> + if (act_prio != MASKED) {
> + /* First, check provisioning of queues */
> + mutex_lock(&xive->kvm->lock);
> + rc = xive_check_provisioning(xive->kvm, act_prio);
> + mutex_unlock(&xive->kvm->lock);
> +
> + /* Target interrupt */
> + if (rc == 0)
> + rc = xive_target_interrupt(xive->kvm, state,
> + server, act_prio);
> + /*
> + * If provisioning or targetting failed, leave it
> + * alone and masked. It will remain disabled until
> + * the guest re-targets it.
> + */
> + }
> +
> + /*
> + * Find out if this was a delayed irq stashed in an ICP,
> + * in which case, treat it as pending
> + */
> + if (xive->delayed_irqs && xive_check_delayed_irq(xive, irq)) {
> + val |= KVM_XICS_PENDING;
> + DBG(" Found delayed ! forcing PENDING !\n");
> + }
> +
> + /* Cleanup the SW state */
> + state->old_p = false;
> + state->old_q = false;
> + state->lsi = false;
> + state->asserted = false;
> +
> + /* Restore LSI state */
> + if (val & KVM_XICS_LEVEL_SENSITIVE) {
> + state->lsi = true;
> + if (val & KVM_XICS_PENDING)
> + state->asserted = true;
> + DBG(" LSI ! Asserted=%d\n", state->asserted);
> + }
> +
> + /*
> + * Restore P and Q. If the interrupt was pending, we
> + * force both P and Q, which will trigger a resend.
> + *
> + * That means that a guest that had both an interrupt
> + * pending (queued) and Q set will restore with only
> + * one instance of that interrupt instead of 2, but that
> + * is perfectly fine as coalescing interrupts that haven't
> + * been presented yet is always allowed.
> + */
> + if (val & KVM_XICS_PRESENTED || val & KVM_XICS_PENDING)
> + state->old_p = true;
> + if (val & KVM_XICS_QUEUED || val & KVM_XICS_PENDING)
> + state->old_q = true;
> +
> + DBG(" P=%d, Q=%d\n", state->old_p, state->old_q);
> +
> + /*
> + * If the interrupt was unmasked, update guest priority and
> + * perform the appropriate state transition and do a
> + * re-trigger if necessary.
> + */
> + if (val & KVM_XICS_MASKED) {
> + DBG(" masked, saving prio\n");
> + state->guest_priority = MASKED;
> + state->saved_priority = guest_prio;
> + } else {
> + DBG(" unmasked, restoring to prio %d\n", guest_prio);
> + xive_finish_unmask(xive, sb, state, guest_prio);
> + state->saved_priority = guest_prio;
> + }
> +
> + /* Increment the number of valid sources and mark this one valid */
> + if (!state->valid)
> + xive->src_count++;
> + state->valid = true;
> +
> + return 0;
> +}
> +
> +int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq, int level,
> + bool line_status)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u16 idx;
> +
> + if (!xive)
> + return -ENODEV;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb)
> + return -EINVAL;
> +
> + /* Perform locklessly .... (we need to do some RCUisms here...) */
> + state = &sb->irq_state[idx];
> + if (!state->valid)
> + return -EINVAL;
> +
> + /* We don't allow a trigger on a passed-through interrupt */
> + if (state->pt_number)
> + return -EINVAL;
> +
> + if ((level == 1 && state->lsi) || level == KVM_INTERRUPT_SET_LEVEL)
> + state->asserted = 1;
> + else if (level == 0 || level == KVM_INTERRUPT_UNSET) {
> + state->asserted = 0;
> + return 0;
> + }
> +
> + /* Trigger the IPI */
> + xive_irq_trigger(&state->ipi_data);
> +
> + return 0;
> +}
> +
> +static int xive_set_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
> +{
> + struct kvmppc_xive *xive = dev->private;
> +
> + /* We honor the existing XICS ioctl */
> + switch (attr->group) {
> + case KVM_DEV_XICS_GRP_SOURCES:
> + return xive_set_source(xive, attr->attr, attr->addr);
> + }
> + return -ENXIO;
> +}
> +
> +static int xive_get_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
> +{
> + struct kvmppc_xive *xive = dev->private;
> +
> + /* We honor the existing XICS ioctl */
> + switch (attr->group) {
> + case KVM_DEV_XICS_GRP_SOURCES:
> + return xive_get_source(xive, attr->attr, attr->addr);
> + }
> + return -ENXIO;
> +}
> +
> +static int xive_has_attr(struct kvm_device *dev, struct kvm_device_attr *attr)
> +{
> + /* We honor the same limits as XICS, at least for now */
> + switch (attr->group) {
> + case KVM_DEV_XICS_GRP_SOURCES:
> + if (attr->attr >= KVMPPC_XICS_FIRST_IRQ &&
> + attr->attr < KVMPPC_XICS_NR_IRQS)
> + return 0;
> + break;
> + }
> + return -ENXIO;
> +}
> +
> +static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> +{
> + xive_vm_esb_load(xd, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(hw_num, 0, MASKED, 0);
> + xive_cleanup_irq_data(xd);
> +}
> +
> +static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> +{
> + int i;
> +
> + for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
> +
> + if (!state->valid)
> + continue;
> +
> + kvmppc_xive_cleanup_irq(state->ipi_number, &state->ipi_data);
> + xive_native_free_irq(state->ipi_number);
> +
> + /* Pass-through, cleanup too */
> + if (state->pt_number)
> + kvmppc_xive_cleanup_irq(state->pt_number, state->pt_data);
> +
> + state->valid = false;
> + }
> +}
> +
> +static void kvmppc_xive_free(struct kvm_device *dev)
> +{
> + struct kvmppc_xive *xive = dev->private;
> + struct kvm *kvm = xive->kvm;
> + int i;
> +
> + debugfs_remove(xive->dentry);
> +
> + if (kvm)
> + kvm->arch.xive = NULL;
> +
> + /* Mask and free interrupts */
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_free_sources(xive->src_blocks[i]);
> + kfree(xive->src_blocks[i]);
> + xive->src_blocks[i] = NULL;
> + }
> +
> + if (xive->vp_base != XIVE_INVALID_VP)
> + xive_native_free_vp_block(xive->vp_base);
> +
> +
> + kfree(xive);
> + kfree(dev);
> +}
> +
> +static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
> +{
> + struct kvmppc_xive *xive;
> + struct kvm *kvm = dev->kvm;
> + int ret = 0;
> +
> + DBG("Creating xive for partition\n");
> +
> + xive = kzalloc(sizeof(*xive), GFP_KERNEL);
> + if (!xive)
> + return -ENOMEM;
> +
> + dev->private = xive;
> + xive->dev = dev;
> + xive->kvm = kvm;
> +
> + /* Already there ? */
> + if (kvm->arch.xive)
> + ret = -EEXIST;
> + else
> + kvm->arch.xive = xive;
> +
> + /* We use the default queue size set by the host */
> + xive->q_order = xive_native_default_eq_shift();
> + if (xive->q_order < PAGE_SHIFT)
> + xive->q_alloc_order = 0;
> + else
> + xive->q_alloc_order = xive->q_order - PAGE_SHIFT;
> +
> + /* Allocate a bunch of VPs */
> + xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
> + DBG("VP_Base=%x\n", xive->vp_base);
> + if (xive->vp_base == XIVE_INVALID_VP)
> + ret = -ENOMEM;
> +
> + if (ret) {
> + kfree(xive);
> + return ret;
> + }
> +
> + return 0;
> +}
> +
> +
> +static int xive_debug_show(struct seq_file *m, void *private)
> +{
> + struct kvmppc_xive *xive = m->private;
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + u64 t_rm_h_xirr = 0;
> + u64 t_rm_h_ipoll = 0;
> + u64 t_rm_h_cppr = 0;
> + u64 t_rm_h_eoi = 0;
> + u64 t_rm_h_ipi = 0;
> + u64 t_vm_h_xirr = 0;
> + u64 t_vm_h_ipoll = 0;
> + u64 t_vm_h_cppr = 0;
> + u64 t_vm_h_eoi = 0;
> + u64 t_vm_h_ipi = 0;
> + unsigned int i;
> +
> + if (!kvm)
> + return 0;
> +
> + seq_printf(m, "=========\nVCPU state\n=========\n");
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> +
> + if (!xc)
> + continue;
> +
> + seq_printf(m, "cpu server %#x CPPR:%#x HWCPPR:%#x"
> + " MFRR:%#x PEND:%#x h_xirr: R=%lld V=%lld\n",
> + xc->server_num, xc->cppr, xc->hw_cppr,
> + xc->mfrr, xc->pending,
> + xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
> + t_rm_h_xirr += xc->stat_rm_h_xirr;
> + t_rm_h_ipoll += xc->stat_rm_h_ipoll;
> + t_rm_h_cppr += xc->stat_rm_h_cppr;
> + t_rm_h_eoi += xc->stat_rm_h_eoi;
> + t_rm_h_ipi += xc->stat_rm_h_ipi;
> + t_vm_h_xirr += xc->stat_vm_h_xirr;
> + t_vm_h_ipoll += xc->stat_vm_h_ipoll;
> + t_vm_h_cppr += xc->stat_vm_h_cppr;
> + t_vm_h_eoi += xc->stat_vm_h_eoi;
> + t_vm_h_ipi += xc->stat_vm_h_ipi;
> + }
> +
> + seq_printf(m, "Hcalls totals\n");
> + seq_printf(m, " H_XIRR R=%10lld V=%10lld\n", t_rm_h_xirr, t_vm_h_xirr);
> + seq_printf(m, " H_IPOLL R=%10lld V=%10lld\n", t_rm_h_ipoll, t_vm_h_ipoll);
> + seq_printf(m, " H_CPPR R=%10lld V=%10lld\n", t_rm_h_cppr, t_vm_h_cppr);
> + seq_printf(m, " H_EOI R=%10lld V=%10lld\n", t_rm_h_eoi, t_vm_h_eoi);
> + seq_printf(m, " H_IPI R=%10lld V=%10lld\n", t_rm_h_ipi, t_vm_h_ipi);
> +
> + return 0;
> +}
> +
> +static int xive_debug_open(struct inode *inode, struct file *file)
> +{
> + return single_open(file, xive_debug_show, inode->i_private);
> +}
> +
> +static const struct file_operations xive_debug_fops = {
> + .open = xive_debug_open,
> + .read = seq_read,
> + .llseek = seq_lseek,
> + .release = single_release,
> +};
> +
> +static void xive_debugfs_init(struct kvmppc_xive *xive)
> +{
> + char *name;
> +
> + name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
> + if (!name) {
> + pr_err("%s: no memory for name\n", __func__);
> + return;
> + }
> +
> + xive->dentry = debugfs_create_file(name, S_IRUGO, powerpc_debugfs_root,
> + xive, &xive_debug_fops);
> +
> + pr_debug("%s: created %s\n", __func__, name);
> + kfree(name);
> +}
> +
> +static void kvmppc_xive_init(struct kvm_device *dev)
> +{
> + struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
> +
> + /* Register some debug interfaces */
> + xive_debugfs_init(xive);
> +}
> +
> +struct kvm_device_ops kvm_xive_ops = {
> + .name = "kvm-xive",
> + .create = kvmppc_xive_create,
> + .init = kvmppc_xive_init,
> + .destroy = kvmppc_xive_free,
> + .set_attr = xive_set_attr,
> + .get_attr = xive_get_attr,
> + .has_attr = xive_has_attr,
> +};
> +
> +void kvmppc_xive_init_module(void)
> +{
> + __xive_vm_h_xirr = xive_vm_h_xirr;
> + __xive_vm_h_ipoll = xive_vm_h_ipoll;
> + __xive_vm_h_ipi = xive_vm_h_ipi;
> + __xive_vm_h_cppr = xive_vm_h_cppr;
> + __xive_vm_h_eoi = xive_vm_h_eoi;
> +}
> +
> +void kvmppc_xive_exit_module(void)
> +{
> + __xive_vm_h_xirr = NULL;
> + __xive_vm_h_ipoll = NULL;
> + __xive_vm_h_ipi = NULL;
> + __xive_vm_h_cppr = NULL;
> + __xive_vm_h_eoi = NULL;
> +}
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> new file mode 100644
> index 0000000..2b7fdbd
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -0,0 +1,251 @@
> +/*
> + * Copyright 2017 Benjamin Herrenschmidt, IBM Corporation
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation.
> + */
> +
> +#ifndef _KVM_PPC_BOOK3S_XIVE_H
> +#define _KVM_PPC_BOOK3S_XIVE_H
> +
> +#include "book3s_xics.h"
> +
> +/* State for one guest irq source.
> + *
> + * For each guest source we allocate a HW interrupt in the XIVE
> + * which we use for all SW triggers. It will be unused for
> + * pass-through but it's easier to keep around as the same
> + * guest interrupt can alternatively be emulated or pass-through
> + * if a physical device is hot unplugged and replaced with an
> + * emulated one.
> + *
> + * This state structure is very similar to the XICS one with
> + * additional XIVE specific tracking.
> + */
> +struct kvmppc_xive_irq_state {
> + bool valid; /* Interrupt entry is valid */
> +
> + u32 number; /* Guest IRQ number */
> + u32 ipi_number; /* XIVE IPI HW number */
> + struct xive_irq_data ipi_data; /* XIVE IPI associated data */
> + u32 pt_number; /* XIVE Pass-through number if any */
> + struct xive_irq_data *pt_data; /* XIVE Pass-through associated data */
> +
> + /* Targetting as set by guest */
> + u32 guest_server; /* Current guest selected target */
> + u8 guest_priority; /* Guest set priority */
> + u8 saved_priority; /* Saved priority when masking */
> +
> + /* Actual targetting */
> + u32 act_server; /* Actual server */
> + u8 act_priority; /* Actual priority */
> +
> + /* Various state bits */
> + bool in_eoi; /* Synchronize with H_EOI */
> + bool old_p; /* P bit state when masking */
> + bool old_q; /* Q bit state when masking */
> + bool lsi; /* level-sensitive interrupt */
> + bool asserted; /* Only for emulated LSI: current state */
> +
> + /* Saved for migration state */
> + bool in_queue;
> + bool saved_p;
> + bool saved_q;
> + u8 saved_scan_prio;
> +};
> +
> +/* Select the "right" interrupt (IPI vs. passthrough) */
> +static inline void kvmppc_xive_select_irq(struct kvmppc_xive_irq_state *state,
> + u32 *out_hw_irq,
> + struct xive_irq_data **out_xd)
> +{
> + if (state->pt_number) {
> + if (out_hw_irq)
> + *out_hw_irq = state->pt_number;
> + if (out_xd)
> + *out_xd = state->pt_data;
> + } else {
> + if (out_hw_irq)
> + *out_hw_irq = state->ipi_number;
> + if (out_xd)
> + *out_xd = &state->ipi_data;
> + }
> +}
> +
> +/* This corresponds to an "ICS" in XICS terminology, we use it
> + * as a mean to break up source information into multiple structures
> + */
> +struct kvmppc_xive_src_block {
> + arch_spinlock_t lock;
> + u16 id;
> + struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
> +};
> +
> +
> +struct kvmppc_xive {
> + struct kvm *kvm;
> + struct kvm_device *dev;
> + struct dentry *dentry;
> +
> + /* VP block associated with the VM */
> + u32 vp_base;
> +
> + /* Blocks of sources */
> + struct kvmppc_xive_src_block *src_blocks[KVMPPC_XICS_MAX_ICS_ID + 1];
> + u32 max_sbid;
> +
> + /*
> + * For state save, we lazily scan the queues on the first interrupt
> + * being migrated. We don't have a clean way to reset that flags
> + * so we keep track of the number of valid sources and how many of
> + * them were migrated so we can reset when all of them have been
> + * processed.
> + */
> + u32 src_count;
> + u32 saved_src_count;
> +
> + /*
> + * Some irqs are delayed on restore until the source is created,
> + * keep track here of how many of them
> + */
> + u32 delayed_irqs;
> +
> + /* Which queues (priorities) are in use by the guest */
> + u8 qmap;
> +
> + /* Queue orders */
> + u32 q_order;
> + u32 q_alloc_order;
> +
> +};
> +
> +#define KVMPPC_XIVE_Q_COUNT 8
> +
> +struct kvmppc_xive_vcpu {
> + struct kvmppc_xive *xive;
> + struct kvm_vcpu *vcpu;
> + bool valid;
> +
> + /* Server number. This is the HW CPU ID from a guest perspective */
> + u32 server_num;
> +
> + /* HW VP corresponding to this VCPU. This is the base of the VP
> + * block plus the server number
> + */
> + u32 vp_id;
> + u32 vp_chip_id;
> + u32 vp_cam;
> +
> + /* IPI used for sending ... IPIs */
> + u32 vp_ipi;
> + struct xive_irq_data vp_ipi_data;
> +
> + /* Local emulation state */
> + uint8_t cppr; /* guest CPPR */
> + uint8_t hw_cppr;/* Hardware CPPR */
> + uint8_t mfrr;
> + uint8_t pending;
> +
> + /* Each VP has 8 queues though we only provision some */
> + struct xive_q queues[KVMPPC_XIVE_Q_COUNT];
> + u32 esc_virq[KVMPPC_XIVE_Q_COUNT];
> + char *esc_virq_names[KVMPPC_XIVE_Q_COUNT];
> +
> + /* Stash a delayed irq on restore from migration (see set_icp) */
> + u32 delayed_irq;
> +
> + /* Stats */
> + u64 stat_rm_h_xirr;
> + u64 stat_rm_h_ipoll;
> + u64 stat_rm_h_cppr;
> + u64 stat_rm_h_eoi;
> + u64 stat_rm_h_ipi;
> + u64 stat_vm_h_xirr;
> + u64 stat_vm_h_ipoll;
> + u64 stat_vm_h_cppr;
> + u64 stat_vm_h_eoi;
> + u64 stat_vm_h_ipi;
> +};
> +
> +static inline struct kvm_vcpu *kvmppc_xive_find_server(struct kvm *kvm, u32 nr)
> +{
> + struct kvm_vcpu *vcpu = NULL;
> + int i;
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + if (vcpu->arch.xive_vcpu && nr == vcpu->arch.xive_vcpu->server_num)
> + return vcpu;
> + }
> + return NULL;
> +}
> +
> +static inline struct kvmppc_xive_src_block *kvmppc_xive_find_source(struct kvmppc_xive *xive,
> + u32 irq, u16 *source)
> +{
> + u32 bid = irq >> KVMPPC_XICS_ICS_SHIFT;
> + u16 src = irq & KVMPPC_XICS_SRC_MASK;
> +
> + if (source)
> + *source = src;
> + if (bid > KVMPPC_XICS_MAX_ICS_ID)
> + return NULL;
> + return xive->src_blocks[bid];
> +}
> +
> +/*
> + * Mapping between guest priorities and host priorities
> + * is as follow.
> + *
> + * Guest request for 0...6 are honored. Guest request for anything
> + * higher results in a priority of 7 being applied.
> + *
> + * However, when XIRR is returned via H_XIRR, 7 is translated to 0xb
> + * in order to match AIX expectations
> + *
> + * Similar mapping is done for CPPR values
> + */
> +static inline u8 xive_prio_from_guest(u8 prio)
> +{
> + if (prio == 0xff || prio < 8)
> + return prio;
> + return 7;
> +}
> +
> +static inline u8 xive_prio_to_guest(u8 prio)
> +{
> + if (prio == 0xff || prio < 7)
> + return prio;
> + return 0xb;
> +}
> +
> +static inline u32 __xive_read_eq(__be32 *qpage, u32 msk, u32 *idx, u32 *toggle)
> +{
> + u32 cur;
> +
> + if (!qpage)
> + return 0;
> + cur = be32_to_cpup(qpage + *idx);
> + if ((cur >> 31) == *toggle)
> + return 0;
> + *idx = (*idx + 1) & msk;
> + if (*idx == 0)
> + (*toggle) ^= 1;
> + return cur & 0x7fffffff;
> +}
> +
> +extern unsigned long xive_rm_h_xirr(struct kvm_vcpu *vcpu);
> +extern unsigned long xive_rm_h_ipoll(struct kvm_vcpu *vcpu, unsigned long server);
> +extern int xive_rm_h_ipi(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr);
> +extern int xive_rm_h_cppr(struct kvm_vcpu *vcpu, unsigned long cppr);
> +extern int xive_rm_h_eoi(struct kvm_vcpu *vcpu, unsigned long xirr);
> +
> +extern unsigned long (*__xive_vm_h_xirr)(struct kvm_vcpu *vcpu);
> +extern unsigned long (*__xive_vm_h_ipoll)(struct kvm_vcpu *vcpu, unsigned long server);
> +extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr);
> +extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
> +extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> +
> +#endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive_template.c b/arch/powerpc/kvm/book3s_xive_template.c
> new file mode 100644
> index 0000000..b28c264
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_xive_template.c
> @@ -0,0 +1,490 @@
> +/*
> + * Copyright 2017 Benjamin Herrenschmidt, IBM Corporation
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License, version 2, as
> + * published by the Free Software Foundation.
> + */
> +
> +/* File to be included by other .c files */
> +
> +#define XGLUE(a,b) a##b
> +#define GLUE(a,b) XGLUE(a,b)
> +
> +static void GLUE(X_PFX,ack_pending)(struct kvmppc_xive_vcpu *xc)
> +{
> + u8 cppr;
> + u16 ack;
> +
> + /* XXX DD1 bug workaround: Check PIPR vs. CPPR first ! */
> +
> + /* Perform the acknowledge OS to register cycle. */
> + ack = be16_to_cpu(__x_readw(__x_tm_area + TM_SPC_ACK_OS_REG));
> +
> + /* Synchronize subsequent queue accesses */
> + mb();
> +
> + /* XXX Check grouping level */
> +
> + /* Anything ? */
> + if (!((ack >> 8) & TM_QW1_NSR_EO))
> + return;
> +
> + /* Grab CPPR of the most favored pending interrupt */
> + cppr = ack & 0xff;
> + if (cppr < 8)
> + xc->pending |= 1 << cppr;
> +
> +#ifdef XIVE_RUNTIME_CHECKS
> + /* Check consistency */
> + if (cppr >= xc->hw_cppr)
> + pr_warn("KVM-XIVE: CPU %d odd ack CPPR, got %d at %d\n",
> + smp_processor_id(), cppr, xc->hw_cppr);
> +#endif
> +
> + /* Update our image of the HW CPPR. We don't yet modify
> + * xc->cppr, this will be done as we scan for interrupts
> + * in the queues.
> + */
> + xc->hw_cppr = cppr;
> +}
> +
> +static u8 GLUE(X_PFX,esb_load)(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val =__x_readq(__x_eoi_page(xd) + offset);
> +#ifdef __LITTLE_ENDIAN__
> + val >>= 64-8;
> +#endif
> + return (u8)val;
> +}
> +
> +
> +static void GLUE(X_PFX,source_eoi)(u32 hw_irq, struct xive_irq_data *xd)
> +{
> + /* If the XIVE supports the new "store EOI facility, use it */
> + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> + __x_writeq(0, __x_eoi_page(xd));
> + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> + opal_int_eoi(hw_irq);
> + } else {
> + uint64_t eoi_val;
> +
> + /* Otherwise for EOI, we use the special MMIO that does
> + * a clear of both P and Q and returns the old Q.
> + *
> + * This allows us to then do a re-trigger if Q was set
> + * rather than synthetizing an interrupt in software
> + */
> + eoi_val = GLUE(X_PFX,esb_load)(xd, XIVE_ESB_SET_PQ_00);
> + if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
> + return;
> +
> + /* Re-trigger */
> + if (__x_trig_page(xd))
> + __x_writeq(0, __x_trig_page(xd));
> + }
> +
> +}
> +
> +enum {
> + scan_fetch,
> + scan_poll,
> + scan_eoi,
> +};
> +
> +static u32 GLUE(X_PFX,scan_interrupts)(struct kvmppc_xive_vcpu *xc,
> + u8 pending, int scan_type)
> +{
> + u32 hirq = 0;
> + u8 prio = 0xff;
> +
> + /* Find highest pending priority */
> + while ((xc->mfrr != 0xff || pending != 0) && hirq == 0) {
> + struct xive_q *q;
> + u32 idx, toggle;
> + __be32 *qpage;
> +
> + /*
> + * If pending is 0 this will return 0xff which is what
> + * we want
> + */
> + prio = ffs(pending) - 1;
> +
> + /*
> + * If the most favoured prio we found pending is less
> + * favored (or equal) than a pending IPI, we return
> + * the IPI instead.
> + *
> + * Note: If pending was 0 and mfrr is 0xff, we will
> + * not spurriously take an IPI because mfrr cannot
> + * then be smaller than cppr.
> + */
> + if (prio >= xc->mfrr && xc->mfrr < xc->cppr) {
> + prio = xc->mfrr;
> + hirq = XICS_IPI;
> + break;
> + }
> +
> + /* Don't scan past the guest cppr */
> + if (prio >= xc->cppr || prio > 7)
> + break;
> +
> + /* Grab queue and pointers */
> + q = &xc->queues[prio];
> + idx = q->idx;
> + toggle = q->toggle;
> +
> + /*
> + * Snapshot the queue page. The test further down for EOI
> + * must use the same "copy" that was used by __xive_read_eq
> + * since qpage can be set concurrently and we don't want
> + * to miss an EOI.
> + */
> + qpage = READ_ONCE(q->qpage);
> +
> + skip_ipi:
> + /* Try to fetch from the queue. Will return 0 for a
> + * non-queueing priority (ie, qpage = 0)
> + */
> + hirq = __xive_read_eq(qpage, q->msk, &idx, &toggle);
> +
> + /*
> + * If this was a signal for an MFFR change done by
> + * H_IPI we skip it. Additionally, if we were fetching
> + * we EOI it now, thus re-enabling reception of a new
> + * such signal.
> + *
> + * We also need to do that if prio is 0 and we had no
> + * page for the queue. In this case, we have non-queued
> + * IPI that needs to be EOId.
> + *
> + * This is safe because if we have another pending MFRR
> + * change that wasn't observed above, the Q bit will have
> + * been set and another occurrence of the IPI will trigger.
> + */
> + if (hirq == XICS_IPI || (prio == 0 && !qpage)) {
> + if (scan_type == scan_fetch)
> + GLUE(X_PFX,source_eoi)(xc->vp_ipi,
> + &xc->vp_ipi_data);
> + /* Loop back on same queue with updated idx/toggle */
> +#ifdef XIVE_RUNTIME_CHECKS
> + WARN_ON(hirq && hirq != XICS_IPI);
> +#endif
> + if (hirq)
> + goto skip_ipi;
> + }
> +
> + /* If fetching, update queue pointers */
> + if (scan_type == scan_fetch) {
> + q->idx = idx;
> + q->toggle = toggle;
> + }
> +
> + /* Something found, stop searching */
> + if (hirq)
> + break;
> +
> + /* Clear the pending bit on the now empty queue */
> + pending &= ~(1 << prio);
> +
> + /*
> + * Check if the queue count needs adjusting due to
> + * interrupts being moved away.
> + */
> + if (atomic_read(&q->pending_count)) {
> + int p = atomic_xchg(&q->pending_count, 0);
> + if (p) {
> +#ifdef XIVE_RUNTIME_CHECKS
> + WARN_ON(p > atomic_read(&q->count));
> +#endif
> + atomic_sub(p, &q->count);
> + }
> + }
> + }
> +
> + /* If we are just taking a "peek", do nothing else */
> + if (scan_type == scan_poll)
> + return hirq;
> +
> + /* Update the pending bits */
> + xc->pending = pending;
> +
> + /* If this is an EOI that's it, no CPPR adjustment done here,
> + * all we needed was cleanup the stale pending bits and check
> + * if there's anything left.
> + */
> + if (scan_type == scan_eoi)
> + return hirq;
> +
> + /* If we found an interrupt, adjust what the guest CPPR should
> + * be as if we had just fetched that interrupt from HW
> + */
> + if (hirq)
> + xc->cppr = prio;
> + /*
> + * If it was an IPI the HW CPPR might have been lowered too much
> + * as the HW interrupt we use for IPIs is routed to priority 0.
> + *
> + * We re-sync it here.
> + */
> + if (xc->cppr != xc->hw_cppr) {
> + xc->hw_cppr = xc->cppr;
> + __x_writeb(xc->cppr, __x_tm_area + TM_QW1_OS + TM_CPPR);
> + }
> +
> + return hirq;
> +}
> +
> +X_STATIC unsigned long GLUE(X_PFX,h_xirr)(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + u8 old_cppr;
> + u32 hirq;
> +
> + DBG("H_XIRR\n");
> +
> + xc->GLUE(X_STAT_PFX,h_xirr)++;
> +
> + /* First collect pending bits from HW */
> + GLUE(X_PFX,ack_pending)(xc);
> +
> + /* Cleanup the old-style bits if needed (they may have been
> + * set by pull or an escalation interrupts)
> + */
> + if (test_bit(BOOK3S_IRQPRIO_EXTERNAL, &vcpu->arch.pending_exceptions))
> + clear_bit(BOOK3S_IRQPRIO_EXTERNAL_LEVEL,
> + &vcpu->arch.pending_exceptions);
> +
> + DBG(" new pending=0x%02x hw_cppr=%d cppr=%d\n",
> + xc->pending, xc->hw_cppr, xc->cppr);
> +
> + /* Grab previous CPPR and reverse map it */
> + old_cppr = xive_prio_to_guest(xc->cppr);
> +
> + /* Scan for actual interrupts */
> + hirq = GLUE(X_PFX,scan_interrupts)(xc, xc->pending, scan_fetch);
> +
> + DBG(" got hirq=0x%x hw_cppr=%d cppr=%d\n",
> + hirq, xc->hw_cppr, xc->cppr);
> +
> +#ifdef XIVE_RUNTIME_CHECKS
> + /* That should never hit */
> + if (hirq & 0xff000000)
> + pr_warn("XIVE: Weird guest interrupt number 0x%08x\n", hirq);
> +#endif
> +
> + /*
> + * XXX We could check if the interrupt is masked here and
> + * filter it. If we chose to do so, we would need to do:
> + *
> + * if (masked) {
> + * lock();
> + * if (masked) {
> + * old_Q = true;
> + * hirq = 0;
> + * }
> + * unlock();
> + * }
> + */
> +
> + /* Return interrupt and old CPPR in GPR4 */
> + vcpu->arch.gpr[4] = hirq | (old_cppr << 24);
> +
> + return H_SUCCESS;
> +}
> +
> +X_STATIC unsigned long GLUE(X_PFX,h_ipoll)(struct kvm_vcpu *vcpu, unsigned long server)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + u8 pending = xc->pending;
> + u32 hirq;
> + u8 pipr;
> +
> + DBG("H_IPOLL(server=%ld)\n", server);
> +
> + xc->GLUE(X_STAT_PFX,h_ipoll)++;
> +
> + /* Grab the target VCPU if not the current one */
> + if (xc->server_num != server) {
> + vcpu = kvmppc_xive_find_server(vcpu->kvm, server);
> + if (!vcpu)
> + return H_PARAMETER;
> + xc = vcpu->arch.xive_vcpu;
> +
> + /* Scan all priorities */
> + pending = 0xff;
> + } else {
> + /* Grab pending interrupt if any */
> + pipr = __x_readb(__x_tm_area + TM_QW1_OS + TM_PIPR);
> + if (pipr < 8)
> + pending |= 1 << pipr;
> + }
> +
> + hirq = GLUE(X_PFX,scan_interrupts)(xc, pending, scan_poll);
> +
> + /* Return interrupt and old CPPR in GPR4 */
> + vcpu->arch.gpr[4] = hirq | (xc->cppr << 24);
> +
> + return H_SUCCESS;
> +}
> +
> +static void GLUE(X_PFX,push_pending_to_hw)(struct kvmppc_xive_vcpu *xc)
> +{
> + u8 pending, prio;
> +
> + pending = xc->pending;
> + if (xc->mfrr != 0xff) {
> + if (xc->mfrr < 8)
> + pending |= 1 << xc->mfrr;
> + else
> + pending |= 0x80;
> + }
> + if (!pending)
> + return;
> + prio = ffs(pending) - 1;
> +
> + __x_writeb(prio, __x_tm_area + TM_SPC_SET_OS_PENDING);
> +}
> +
> +X_STATIC int GLUE(X_PFX,h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + u8 old_cppr;
> +
> + DBG("H_CPPR(cppr=%ld)\n", cppr);
> +
> + xc->GLUE(X_STAT_PFX,h_cppr)++;
> +
> + /* Map CPPR */
> + cppr = xive_prio_from_guest(cppr);
> +
> + /* Remember old and update SW state */
> + old_cppr = xc->cppr;
> + xc->cppr = cppr;
> +
> + /*
> + * We are masking less, we need to look for pending things
> + * to deliver and set VP pending bits accordingly to trigger
> + * a new interrupt otherwise we might miss MFRR changes for
> + * which we have optimized out sending an IPI signal.
> + */
> + if (cppr > old_cppr)
> + GLUE(X_PFX,push_pending_to_hw)(xc);
> +
> + /* Apply new CPPR */
> + xc->hw_cppr = cppr;
> + __x_writeb(cppr, __x_tm_area + TM_QW1_OS + TM_CPPR);
> +
> + return H_SUCCESS;
> +}
> +
> +X_STATIC int GLUE(X_PFX,h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr)
> +{
> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct xive_irq_data *xd;
> + u8 new_cppr = xirr >> 24;
> + u32 irq = xirr & 0x00ffffff, hw_num;
> + u16 src;
> + int rc = 0;
> +
> + DBG("H_EOI(xirr=%08lx)\n", xirr);
> +
> + xc->GLUE(X_STAT_PFX,h_eoi)++;
> +
> + xc->cppr = xive_prio_from_guest(new_cppr);
> +
> + /*
> + * IPIs are synthetized from MFRR and thus don't need
> + * any special EOI handling. The underlying interrupt
> + * used to signal MFRR changes is EOId when fetched from
> + * the queue.
> + */
> + if (irq == XICS_IPI || irq == 0)
> + goto bail;
> +
> + /* Find interrupt source */
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb) {
> + DBG(" source not found !\n");
> + rc = H_PARAMETER;
> + goto bail;
> + }
> + state = &sb->irq_state[src];
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> +
> + state->in_eoi = true;
> + mb();
> +
> + again:
> + if (state->guest_priority == MASKED) {
> + arch_spin_lock(&sb->lock);
> + if (state->guest_priority != MASKED) {
> + arch_spin_unlock(&sb->lock);
> + goto again;
> + }
> + DBG(" EOI on saved P...\n");
> +
> + /* Clear old_p, that will cause unmask to perform an EOI */
> + state->old_p = false;
> +
> + arch_spin_unlock(&sb->lock);
> + } else {
> + DBG(" EOI on source...\n");
> +
> + /* Perform EOI on the source */
> + GLUE(X_PFX,source_eoi)(hw_num, xd);
> +
> + /* If it's an emulated LSI, check level and resend */
> + if (state->lsi && state->asserted)
> + __x_writeq(0, __x_trig_page(xd));
> +
> + }
> +
> + mb();
> + state->in_eoi = false;
> + bail:
> +
> + /* Re-evaluate pending IRQs and update HW */
> + GLUE(X_PFX,scan_interrupts)(xc, xc->pending, scan_eoi);
> + GLUE(X_PFX,push_pending_to_hw)(xc);
> + DBG(" after scan pending=%02x\n", xc->pending);
> +
> + /* Apply new CPPR */
> + xc->hw_cppr = xc->cppr;
> + __x_writeb(xc->cppr, __x_tm_area + TM_QW1_OS + TM_CPPR);
> +
> + return rc;
> +}
> +
> +X_STATIC int GLUE(X_PFX,h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
> + unsigned long mfrr)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> +
> + DBG("H_IPI(server=%08lx,mfrr=%ld)\n", server, mfrr);
> +
> + xc->GLUE(X_STAT_PFX,h_ipi)++;
> +
> + /* Find target */
> + vcpu = kvmppc_xive_find_server(vcpu->kvm, server);
> + if (!vcpu)
> + return H_PARAMETER;
> + xc = vcpu->arch.xive_vcpu;
> +
> + /* Locklessly write over MFRR */
> + xc->mfrr = mfrr;
> +
> + /* Shoot the IPI if most favored than target cppr */
> + if (mfrr < xc->cppr)
> + __x_writeq(0, __x_trig_page(&xc->vp_ipi_data));
> +
> + return H_SUCCESS;
> +}
> diff --git a/arch/powerpc/kvm/irq.h b/arch/powerpc/kvm/irq.h
> index 5a9a10b..3f1be85 100644
> --- a/arch/powerpc/kvm/irq.h
> +++ b/arch/powerpc/kvm/irq.h
> @@ -12,6 +12,7 @@ static inline int irqchip_in_kernel(struct kvm *kvm)
> #endif
> #ifdef CONFIG_KVM_XICS
> ret = ret || (kvm->arch.xics != NULL);
> + ret = ret || (kvm->arch.xive != NULL);
> #endif
> smp_rmb();
> return ret;
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 95c91a9..de79bd72 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -37,6 +37,8 @@
> #include <asm/cputhreads.h>
> #include <asm/irqflags.h>
> #include <asm/iommu.h>
> +#include <asm/xive.h>
> +
> #include "timing.h"
> #include "irq.h"
> #include "../mm/mmu_decl.h"
> @@ -699,7 +701,10 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
> kvmppc_mpic_disconnect_vcpu(vcpu->arch.mpic, vcpu);
> break;
> case KVMPPC_IRQ_XICS:
> - kvmppc_xics_free_icp(vcpu);
> + if (xive_enabled())
> + kvmppc_xive_cleanup_vcpu(vcpu);
> + else
> + kvmppc_xics_free_icp(vcpu);
> break;
> }
>
> @@ -1219,8 +1224,12 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>
> r = -EPERM;
> dev = kvm_device_from_filp(f.file);
> - if (dev)
> - r = kvmppc_xics_connect_vcpu(dev, vcpu, cap->args[1]);
> + if (dev) {
> + if (xive_enabled())
> + r = kvmppc_xive_connect_vcpu(dev, vcpu, cap->args[1]);
> + else
> + r = kvmppc_xics_connect_vcpu(dev, vcpu, cap->args[1]);
> + }
>
> fdput(f);
> break;
> @@ -1244,7 +1253,7 @@ bool kvm_arch_intc_initialized(struct kvm *kvm)
> return true;
> #endif
> #ifdef CONFIG_KVM_XICS
> - if (kvm->arch.xics)
> + if (kvm->arch.xics || kvm->arch.xive)
> return true;
> #endif
> return false;
> diff --git a/arch/powerpc/platforms/powernv/opal.c b/arch/powerpc/platforms/powernv/opal.c
> index e0f856b..d71cd77 100644
> --- a/arch/powerpc/platforms/powernv/opal.c
> +++ b/arch/powerpc/platforms/powernv/opal.c
> @@ -890,3 +890,4 @@ EXPORT_SYMBOL_GPL(opal_leds_set_ind);
> EXPORT_SYMBOL_GPL(opal_write_oppanel_async);
> /* Export this for KVM */
> EXPORT_SYMBOL_GPL(opal_int_set_mfrr);
> +EXPORT_SYMBOL_GPL(opal_int_eoi);
> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
> index 96037e0..6429cd3 100644
> --- a/arch/powerpc/sysdev/xive/common.c
> +++ b/arch/powerpc/sysdev/xive/common.c
> @@ -45,12 +45,14 @@
> #endif
>
> bool __xive_enabled;
> +EXPORT_SYMBOL_GPL(__xive_enabled);
> bool xive_cmdline_disabled;
>
> /* We use only one priority for now */
> static u8 xive_irq_priority;
>
> void __iomem *xive_tm_area;
> +EXPORT_SYMBOL_GPL(xive_tm_area);
> u32 xive_tm_offset;
> static const struct xive_ops *xive_ops;
> static struct irq_domain *xive_irq_domain;
> @@ -304,7 +306,7 @@ static void xive_irq_eoi(struct irq_data *d)
> DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
> d->irq, irqd_to_hwirq(d), xc->pending_prio);
>
> - if (!irqd_irq_disabled(d))
> + if (!irqd_irq_disabled(d) && !irqd_is_forwarded_to_vcpu(d))
> xive_do_source_eoi(irqd_to_hwirq(d), xd);
>
> /*
> @@ -579,9 +581,10 @@ static int xive_irq_set_affinity(struct irq_data *d,
> * Only configure the irq if it's not currently passed-through to
> * a KVM guest
> */
> - rc = xive_ops->configure_irq(hw_irq,
> - get_hard_smp_processor_id(target),
> - xive_irq_priority, d->irq);
> + if (!irqd_is_forwarded_to_vcpu(d))
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> if (rc < 0) {
> pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
> return rc;
> @@ -661,6 +664,123 @@ static int xive_irq_retrigger(struct irq_data *d)
> return 1;
> }
>
> +static int xive_irq_set_vcpu_affinity(struct irq_data *d, void *state)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + int rc;
> + u8 pq;
> +
> + /*
> + * We only support this on interrupts that do not require
> + * firmware calls for masking and unmasking
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW)
> + return -EIO;
> +
> + /*
> + * This is called by KVM with state non-NULL for enabling
> + * pass-through or NULL for disabling it
> + */
> + if (state) {
> + irqd_set_forwarded_to_vcpu(d);
> +
> + /* Set it to PQ=10 state to prevent further sends */
> + pq = xive_poke_esb(xd, 0xe00);
Use XIVE_ESB_SET_PQ_xx constants in these xive_poke_esb() calls (as
you have done elsewhere).
> +
> + /* No target ? nothing to do */
> + if (xd->target == XIVE_INVALID_TARGET) {
> + /*
> + * An untargetted interrupt should have been
> + * also masked at the source
> + */
> + WARN_ON(pq & 2);
> +
> + return 0;
> + }
> +
> + /*
> + * If P was set, adjust state to PQ=11 to indicate
> + * that a resend is needed for the interrupt to reach
> + * the guest. Also remember the value of P.
> + *
> + * This also tells us that it's in flight to a host queue
> + * or has already been fetched but hasn't been EOIed yet
> + * by the host. This it's potentially using up a host
> + * queue slot. This is important to know because as long
> + * as this is the case, we must not hard-unmask it when
> + * "returning" that interrupt to the host.
> + *
> + * This saved_p is cleared by the host EOI, when we know
> + * for sure the queue slot is no longer in use.
> + */
> + if (pq & 2) {
> + pq = xive_poke_esb(xd, 0xf00);
> + xd->saved_p = true;
> +
> + /*
> + * Sync the XIVE source HW to ensure the interrupt
> + * has gone through the EAS before we change its
> + * target to the guest. That should guarantee us
> + * that we *will* eventually get an EOI for it on
> + * the host. Otherwise there would be a small window
> + * for P to be seen here but the interrupt going
> + * to the guest queue.
> + */
> + if (xive_ops->sync_source)
> + xive_ops->sync_source(hw_irq);
> + } else
> + xd->saved_p = false;
> + } else {
> + irqd_clr_forwarded_to_vcpu(d);
> +
> + /* No host target ? hard mask and return */
> + if (xd->target == XIVE_INVALID_TARGET) {
> + xive_do_source_set_mask(xd, true);
> + return 0;
> + }
> +
> + /*
> + * Sync the XIVE source HW to ensure the interrupt
> + * has gone through the EAS before we change its
> + * target to the host.
> + */
> + if (xive_ops->sync_source)
> + xive_ops->sync_source(hw_irq);
> +
> + /*
> + * By convention we are called with the interrupt in
> + * a PQ=10 or PQ=11 state, ie, it won't fire and will
> + * have latched in Q whether there's a pending HW
> + * interrupt or not.
> + *
> + * First reconfigure the target.
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + xive_irq_priority, d->irq);
> + if (rc)
> + return rc;
> +
> + /*
> + * Then if saved_p is not set, effectively re-enable the
> + * interrupt with an EOI. If it is set, we know there is
> + * still a message in a host queue somewhere that will be
> + * EOId eventually.
> + *
> + * Note: We don't check irqd_irq_disabled(). Effectively,
> + * we *will* let the irq get through even if masked if the
> + * HW is still firing it in order to deal with the whole
> + * saved_p business properly. If the interrupt triggers
> + * while masked, the generic code will re-mask it anyway.
> + */
> + if (!xd->saved_p)
> + xive_do_source_eoi(hw_irq, xd);
> +
> + }
> + return 0;
> +}
> +
> static struct irq_chip xive_irq_chip = {
> .name = "XIVE-IRQ",
> .irq_startup = xive_irq_startup,
> @@ -671,12 +791,14 @@ static struct irq_chip xive_irq_chip = {
> .irq_set_affinity = xive_irq_set_affinity,
> .irq_set_type = xive_irq_set_type,
> .irq_retrigger = xive_irq_retrigger,
> + .irq_set_vcpu_affinity = xive_irq_set_vcpu_affinity,
> };
>
> bool is_xive_irq(struct irq_chip *chip)
> {
> return chip == &xive_irq_chip;
> }
> +EXPORT_SYMBOL_GPL(is_xive_irq);
>
> void xive_cleanup_irq_data(struct xive_irq_data *xd)
> {
> @@ -691,6 +813,7 @@ void xive_cleanup_irq_data(struct xive_irq_data *xd)
> xd->trig_mmio = NULL;
> }
> }
> +EXPORT_SYMBOL_GPL(xive_cleanup_irq_data);
>
> static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
> {
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> index 26cc6bf..0130af8 100644
> --- a/arch/powerpc/sysdev/xive/native.c
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -27,6 +27,7 @@
> #include <asm/errno.h>
> #include <asm/xive.h>
> #include <asm/opal.h>
> +#include <asm/kvm_ppc.h>
>
> #include "xive-regs.h"
> #include "xive-internal.h"
> @@ -98,6 +99,7 @@ int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
> }
> return 0;
> }
> +EXPORT_SYMBOL_GPL(xive_native_populate_irq_data);
>
> int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
> {
> @@ -111,6 +113,8 @@ int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
> }
> return rc == 0 ? 0 : -ENXIO;
> }
> +EXPORT_SYMBOL_GPL(xive_native_configure_irq);
> +
>
> /* This can be called multiple time to change a queue configuration */
> int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> @@ -187,6 +191,7 @@ int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> fail:
> return rc;
> }
> +EXPORT_SYMBOL_GPL(xive_native_configure_queue);
>
> static void __xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
> {
> @@ -211,6 +216,7 @@ void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio)
> iounmap(q->eoi_mmio);
> q->eoi_mmio = NULL;
> }
> +EXPORT_SYMBOL_GPL(xive_native_disable_queue);
>
> static int xive_native_setup_queue(unsigned int cpu, struct xive_cpu *xc, u8 prio)
> {
> @@ -297,6 +303,7 @@ u32 xive_native_alloc_irq(void)
> return 0;
> return rc;
> }
> +EXPORT_SYMBOL_GPL(xive_native_alloc_irq);
>
> void xive_native_free_irq(u32 irq)
> {
> @@ -307,6 +314,7 @@ void xive_native_free_irq(u32 irq)
> msleep(1);
> }
> }
> +EXPORT_SYMBOL_GPL(xive_native_free_irq);
>
> static void xive_native_put_ipi(unsigned int cpu, struct xive_cpu *xc)
> {
> @@ -406,10 +414,11 @@ static void xive_native_teardown_cpu(unsigned int cpu, struct xive_cpu *xc)
> }
> }
>
> -static void xive_native_sync_source(u32 hw_irq)
> +void xive_native_sync_source(u32 hw_irq)
> {
> opal_xive_sync(XIVE_SYNC_EAS, hw_irq);
> }
> +EXPORT_SYMBOL_GPL(xive_native_sync_source);
>
> static const struct xive_ops xive_native_ops = {
> .populate_irq_data = xive_native_populate_irq_data,
> @@ -468,10 +477,38 @@ static bool xive_parse_provisioning(struct device_node *np)
> return true;
> }
>
> +static void xive_native_setup_pools(void)
> +{
> + u32 max_pir = 0;
> + unsigned int cpu;
> +
> + /*
> + * The HW won't let us enable OS VPs for KVM is we don't
> + * have enabled pool VPs so let's do that. First we find
> + * out our highest HW CPU ID
> + */
> + for_each_possible_cpu(cpu) {
> + u32 hw_id = get_hard_smp_processor_id(cpu);
> + if (hw_id > max_pir)
> + max_pir = hw_id;
> + }
> +
> + /* Allocate a pool big enough */
> + pr_debug("XIVE: Allocating VP block for pool size %d\n",
> + max_pir + 1);
> + xive_pool_vps = xive_native_alloc_vp_block(max_pir + 1);
> + if (WARN_ON(xive_pool_vps == XIVE_INVALID_VP))
> + pr_err("XIVE: No pool VPsvp KVM might not function\n");
> +
> + pr_debug("XIVE: Pool VPs allocated at 0x%x for max_pir 0x%x\n",
> + xive_pool_vps, max_pir);
> +}
> +
> u32 xive_native_default_eq_shift(void)
> {
> return xive_queue_shift;
> }
> +EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
>
> bool xive_native_init(void)
> {
> @@ -481,7 +518,7 @@ bool xive_native_init(void)
> struct property *prop;
> u8 max_prio = 7;
> const __be32 *p;
> - u32 val;
> + u32 val, cpu;
> s64 rc;
>
> if (xive_cmdline_disabled)
> @@ -517,6 +554,10 @@ bool xive_native_init(void)
> break;
> }
>
> + /* Configure TM areas for KVM */
> + for_each_possible_cpu(cpu)
> + kvmppc_set_xive_tm_area(cpu, r.start, tm_area);
> +
> /* Grab size of provisionning pages */
> xive_parse_provisioning(np);
>
> @@ -528,6 +569,9 @@ bool xive_native_init(void)
> return false;
> }
>
> + /* Setup some dummy HV pool VPs */
> + xive_native_setup_pools();
> +
> /* Initialize XIVE core with our backend */
> if (!xive_core_init(&xive_native_ops, tm_area, TM_QW3_HV_PHYS,
> max_prio)) {
> @@ -602,3 +646,47 @@ void xive_native_free_vp_block(u32 vp_base)
> pr_warn("XIVE: OPAL error %lld freeing VP block\n", rc);
> }
> EXPORT_SYMBOL_GPL(xive_native_free_vp_block);
> +
> +int xive_native_enable_vp(u32 vp_id)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_set_vp_info(vp_id, OPAL_XIVE_VP_ENABLED, 0);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + return rc ? -EIO : 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_enable_vp);
> +
> +int xive_native_disable_vp(u32 vp_id)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_set_vp_info(vp_id, 0, 0);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + return rc ? -EIO : 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_disable_vp);
> +
> +int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id)
> +{
> + __be64 vp_cam_be;
> + __be32 vp_chip_id_be;
> + s64 rc;
> +
> + rc = opal_xive_get_vp_info(vp_id, NULL, &vp_cam_be, NULL, &vp_chip_id_be);
> + if (rc)
> + return -EIO;
> + *out_cam_id = be64_to_cpu(vp_cam_be) & 0xffffffffu;
> + *out_chip_id = be32_to_cpu(vp_chip_id_be);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_vp_info);
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index 2c14ad9..d1a6e55 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1165,7 +1165,6 @@ int kvm_register_device_ops(struct kvm_device_ops *ops, u32 type);
> void kvm_unregister_device_ops(u32 type);
>
> extern struct kvm_device_ops kvm_mpic_ops;
> -extern struct kvm_device_ops kvm_xics_ops;
> extern struct kvm_device_ops kvm_arm_vgic_v2_ops;
> extern struct kvm_device_ops kvm_arm_vgic_v3_ops;
>
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index a17d787..1b0da57 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2839,10 +2839,6 @@ static struct kvm_device_ops *kvm_device_ops_table[KVM_DEV_TYPE_MAX] = {
> [KVM_DEV_TYPE_FSL_MPIC_20] = &kvm_mpic_ops,
> [KVM_DEV_TYPE_FSL_MPIC_42] = &kvm_mpic_ops,
> #endif
> -
> -#ifdef CONFIG_KVM_XICS
> - [KVM_DEV_TYPE_XICS] = &kvm_xics_ops,
> -#endif
> };
>
> int kvm_register_device_ops(struct kvm_device_ops *ops, u32 type)
> --
> 2.9.3
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [01/12] powerpc: Disable HFSCR:TM if TM not supported
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-03-31 12:35 ` Michael Ellerman
-1 siblings, 0 replies; 38+ messages in thread
From: Michael Ellerman @ 2017-03-31 12:35 UTC (permalink / raw)
To: Benjamin Herrenschmidt, linuxppc-dev, kvm-ppc
On Mon, 2017-03-20 at 06:49:03 UTC, Benjamin Herrenschmidt wrote:
> Otherwise KVM guests might mess with it even when told not
> to causing bad thing interrupts in the host
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Applied to powerpc fixes, thanks.
https://git.kernel.org/powerpc/c/7ed23e1bae8bf7e37fd555066550a0
cheers
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [01/12] powerpc: Disable HFSCR:TM if TM not supported
@ 2017-03-31 12:35 ` Michael Ellerman
0 siblings, 0 replies; 38+ messages in thread
From: Michael Ellerman @ 2017-03-31 12:35 UTC (permalink / raw)
To: Benjamin Herrenschmidt, linuxppc-dev, kvm-ppc
On Mon, 2017-03-20 at 06:49:03 UTC, Benjamin Herrenschmidt wrote:
> Otherwise KVM guests might mess with it even when told not
> to causing bad thing interrupts in the host
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Applied to powerpc fixes, thanks.
https://git.kernel.org/powerpc/c/7ed23e1bae8bf7e37fd555066550a0
cheers
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 12/12] powerpc/kvm: Native usage of the XIVE interrupt controller
2017-03-28 5:26 ` Paul Mackerras
@ 2017-04-03 2:25 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-04-03 2:25 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc
On Tue, 2017-03-28 at 16:26 +1100, Paul Mackerras wrote:
>
> > --- a/arch/powerpc/include/asm/kvm_book3s_asm.h
> > +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
> > @@ -111,6 +111,8 @@ struct kvmppc_host_state {
> > struct kvm_vcpu *kvm_vcpu;
> > struct kvmppc_vcore *kvm_vcore;
> > void __iomem *xics_phys;
> > + void __iomem *xive_tm_area_phys;
> > + void __iomem *xive_tm_area_virt;
>
> Does this cause the paca to become a cacheline larger? (Not that
> there is much alternative to having these fields.)
It does, though as you said, there's little I can do here.
.../...
> >
> > +/* QW0 and QW1 of a context */
> > +union xive_qw01 {
> > + struct {
> > + u8 nsr;
> > + u8 cppr;
> > + u8 ipb;
> > + u8 lsmfb;
> > + u8 ack;
> > + u8 inc;
> > + u8 age;
> > + u8 pipr;
> > + };
> > + __be64 qw;
> > +};
>
> This is slightly confusing because a "QW" (quadword) would normally
> be 128 bits, but this union is 64 bits.
It's me being wrong. It's not QW0 and QW1, it's word 0 and 1 of the QW.
Word 2 is used for setting up the CAM and Word 3 is unused. I'll fixup
the naming.
> >
> > +extern int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32
> > server,
> > + u32 priority);
> > +extern int kvmppc_xive_get_xive(struct kvm *kvm, u32 irq, u32
> > *server,
> > + u32 *priority);
>
> Might be worth a comment here to explain that the first xive is
> eXternal Interrupt Virtualization Engine and the second xive is
> eXternal Interrupt Vector Entry.
Haha, indeed ;-) I'll add something.
> >
> > +static inline void kvmppc_set_xive_tm_area_phys(int cpu, unsigned
> > long addr)
> > +{}
>
> Shouldn't this be kvmppc_set_xive_tm_area to match the other
> definition?
Yup. Bit-rot from earlier versions of the patch that only had "phys"
(real mode only).
> > --- a/arch/powerpc/include/asm/xive.h
> > +++ b/arch/powerpc/include/asm/xive.h
> > @@ -55,7 +55,8 @@ struct xive_q {
> > #define XIVE_ESB_SET_PQ_01 0xd00
> > #define XIVE_ESB_SET_PQ_10 0xe00
> > #define XIVE_ESB_SET_PQ_11 0xf00
> > -#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> > +#define XIVE_ESB_SOFT_MASK XIVE_ESB_SET_PQ_10
> > +#define XIVE_ESB_HARD_MASK XIVE_ESB_SET_PQ_01
>
> What's the difference between a "soft" mask and a "hard" mask?
I'll document, though I may not use the "aliases" anymore if it's
just confusing. (Basically soft mask will remember in Q if
something happens while masked, hard mask will not).
> >
> > - kvmppc_xics_set_mapped(kvm, guest_gsi, desc-
> > >irq_data.hwirq);
> > + if (xive_enabled())
> > + rc = kvmppc_xive_set_mapped(kvm, guest_gsi, desc);
> > + else
> > + kvmppc_xics_set_mapped(kvm, guest_gsi, desc-
> > >irq_data.hwirq);
> > + printk("set mapped for IRQ %d -> %d returned %d\n",
> > + host_irq, guest_gsi, rc);
>
> This seems like a debugging thing that should be removed or turned
> into a DBG().
Yup, forgot about it.
@@ -398,6 +422,9 @@ static long kvmppc_read_one_intr(bool *again)
> > u8 host_ipi;
> > int64_t rc;
> >
> > + if (xive_enabled())
> > + return 1;
>
> Why not do this in kvmppc_read_intr() rather than here?
Dunno, probably missed that loop. I'll change it
> > paca */
> > +#ifdef CONFIG_KVM_XICS
> > + /* We are exiting, pull the VP from the XIVE */
> > + lwz r0, VCPU_XIVE_PUSHED(r9)
> > + cmpwi cr0, r0, 0
> > + beq 1f
> > + li r7, TM_SPC_PULL_OS_CTX
> > + li r6, TM_QW1_OS
> > + mfmsr r0
> > + andi. r0, r0, MSR_IR /* in real
> > mode? */
> > + beq 2f
> > + ld r10, HSTATE_XIVE_TM_AREA_VIRT(r13)
> > + cmpldi cr0, r10, 0
> > + beq 1f
> > + lwzx r11, r7, r10
> > + eieio
> > + ldx r11, r6, r10
>
> I assume you meant to do these two loads into the same target
> register, but I don't know why, so a comment would be useful.
Right. We don't care about the result of the first one. It's
the special side-effect load to perform the pull. It doesn't
return useful info (the spec isn't clear there, so I should
document it). Once we have pulled, the TM OS area is frozen
so I can do a 64-bit load to get W0 and W1 & back them up.
> > + b 3f
> > +2: ld r10, HSTATE_XIVE_TM_AREA_PHYS(r13)
> > + cmpldi cr0, r10, 0
> > + beq 1f
> > + lwzcix r11, r7, r10
> > + eieio
> > + ldcix r11, r6, r10
> > +3: std r11, VCPU_XIVE_SAVED_STATE(r9)
> > + /* Fixup some of the state for the next load */
> > + li r10, 0
> > + li r0, 0xff
> > + stw r10, VCPU_XIVE_PUSHED(r9)
> > + stb r10, (VCPU_XIVE_SAVED_STATE+3)(r9)
> > + stb r0, (VCPU_XIVE_SAVED_STATE+4)(r9)
> > +1:
> > +#endif /* CONFIG_KVM_XICS */
> > /* Save more register state */
> > mfdar r6
> > mfdsisr r7
> > @@ -2035,7 +2086,7 @@ hcall_real_table:
> > .long DOTSYM(kvmppc_rm_h_eoi) - hcall_real_table
> > .long DOTSYM(kvmppc_rm_h_cppr) - hcall_real_table
> > .long DOTSYM(kvmppc_rm_h_ipi) - hcall_real_table
> > - .long 0 /* 0x70 - H_IPOLL */
> > + .long DOTSYM(kvmppc_rm_h_ipoll) - hcall_real_table
> > .long DOTSYM(kvmppc_rm_h_xirr) - hcall_real_table
> > #else
> > .long 0 /* 0x64 - H_EOI */
> > @@ -2205,7 +2256,11 @@ hcall_real_table:
> > .long 0 /* 0x2f0 */
> > .long 0 /* 0x2f4 */
> > .long 0 /* 0x2f8 */
> > - .long 0 /* 0x2fc */
> > +#ifdef CONFIG_KVM_XICS
> > + .long DOTSYM(kvmppc_rm_h_xirr_x) - hcall_real_table
> > +#else
> > + .long 0 /* 0x2fc - H_XIRR_X*/
> > +#endif
> > .long DOTSYM(kvmppc_h_random) - hcall_real_table
> > .globl hcall_real_table_end
> > hcall_real_table_end:
> > @@ -2980,6 +3035,7 @@ kvmppc_fix_pmao:
> > isync
> > blr
> >
> > +
>
> Gratuitous extra blank line.
Isn't it pretty ? :-)
> >
> > + /* Allocate the queue and retrieve infos on current node
> > for now */
> > + qpage = (__be32 *)__get_free_pages(GFP_KERNEL, xive-
> > >q_alloc_order);
>
> Possibly q_page_order would be a better name than q_alloc_order.
Maybe ... I'll add a comment too.
The rest of your comments don't need a reply :)
I'll respin.
Cheers,
Ben.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 12/12] powerpc/kvm: Native usage of the XIVE interrupt controller
@ 2017-04-03 2:25 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-04-03 2:25 UTC (permalink / raw)
To: Paul Mackerras; +Cc: linuxppc-dev, kvm-ppc
On Tue, 2017-03-28 at 16:26 +1100, Paul Mackerras wrote:
>
> > --- a/arch/powerpc/include/asm/kvm_book3s_asm.h
> > +++ b/arch/powerpc/include/asm/kvm_book3s_asm.h
> > @@ -111,6 +111,8 @@ struct kvmppc_host_state {
> > struct kvm_vcpu *kvm_vcpu;
> > struct kvmppc_vcore *kvm_vcore;
> > void __iomem *xics_phys;
> > + void __iomem *xive_tm_area_phys;
> > + void __iomem *xive_tm_area_virt;
>
> Does this cause the paca to become a cacheline larger? (Not that
> there is much alternative to having these fields.)
It does, though as you said, there's little I can do here.
.../...
> >
> > +/* QW0 and QW1 of a context */
> > +union xive_qw01 {
> > + struct {
> > + u8 nsr;
> > + u8 cppr;
> > + u8 ipb;
> > + u8 lsmfb;
> > + u8 ack;
> > + u8 inc;
> > + u8 age;
> > + u8 pipr;
> > + };
> > + __be64 qw;
> > +};
>
> This is slightly confusing because a "QW" (quadword) would normally
> be 128 bits, but this union is 64 bits.
It's me being wrong. It's not QW0 and QW1, it's word 0 and 1 of the QW.
Word 2 is used for setting up the CAM and Word 3 is unused. I'll fixup
the naming.
> >
> > +extern int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32
> > server,
> > + u32 priority);
> > +extern int kvmppc_xive_get_xive(struct kvm *kvm, u32 irq, u32
> > *server,
> > + u32 *priority);
>
> Might be worth a comment here to explain that the first xive is
> eXternal Interrupt Virtualization Engine and the second xive is
> eXternal Interrupt Vector Entry.
Haha, indeed ;-) I'll add something.
> >
> > +static inline void kvmppc_set_xive_tm_area_phys(int cpu, unsigned
> > long addr)
> > +{}
>
> Shouldn't this be kvmppc_set_xive_tm_area to match the other
> definition?
Yup. Bit-rot from earlier versions of the patch that only had "phys"
(real mode only).
> > --- a/arch/powerpc/include/asm/xive.h
> > +++ b/arch/powerpc/include/asm/xive.h
> > @@ -55,7 +55,8 @@ struct xive_q {
> > #define XIVE_ESB_SET_PQ_01 0xd00
> > #define XIVE_ESB_SET_PQ_10 0xe00
> > #define XIVE_ESB_SET_PQ_11 0xf00
> > -#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> > +#define XIVE_ESB_SOFT_MASK XIVE_ESB_SET_PQ_10
> > +#define XIVE_ESB_HARD_MASK XIVE_ESB_SET_PQ_01
>
> What's the difference between a "soft" mask and a "hard" mask?
I'll document, though I may not use the "aliases" anymore if it's
just confusing. (Basically soft mask will remember in Q if
something happens while masked, hard mask will not).
> >
> > - kvmppc_xics_set_mapped(kvm, guest_gsi, desc-
> > >irq_data.hwirq);
> > + if (xive_enabled())
> > + rc = kvmppc_xive_set_mapped(kvm, guest_gsi, desc);
> > + else
> > + kvmppc_xics_set_mapped(kvm, guest_gsi, desc-
> > >irq_data.hwirq);
> > + printk("set mapped for IRQ %d -> %d returned %d\n",
> > + host_irq, guest_gsi, rc);
>
> This seems like a debugging thing that should be removed or turned
> into a DBG().
Yup, forgot about it.
@@ -398,6 +422,9 @@ static long kvmppc_read_one_intr(bool *again)
> > u8 host_ipi;
> > int64_t rc;
> >
> > + if (xive_enabled())
> > + return 1;
>
> Why not do this in kvmppc_read_intr() rather than here?
Dunno, probably missed that loop. I'll change it
> > paca */
> > +#ifdef CONFIG_KVM_XICS
> > + /* We are exiting, pull the VP from the XIVE */
> > + lwz r0, VCPU_XIVE_PUSHED(r9)
> > + cmpwi cr0, r0, 0
> > + beq 1f
> > + li r7, TM_SPC_PULL_OS_CTX
> > + li r6, TM_QW1_OS
> > + mfmsr r0
> > + andi. r0, r0, MSR_IR /* in real
> > mode? */
> > + beq 2f
> > + ld r10, HSTATE_XIVE_TM_AREA_VIRT(r13)
> > + cmpldi cr0, r10, 0
> > + beq 1f
> > + lwzx r11, r7, r10
> > + eieio
> > + ldx r11, r6, r10
>
> I assume you meant to do these two loads into the same target
> register, but I don't know why, so a comment would be useful.
Right. We don't care about the result of the first one. It's
the special side-effect load to perform the pull. It doesn't
return useful info (the spec isn't clear there, so I should
document it). Once we have pulled, the TM OS area is frozen
so I can do a 64-bit load to get W0 and W1 & back them up.
> > + b 3f
> > +2: ld r10, HSTATE_XIVE_TM_AREA_PHYS(r13)
> > + cmpldi cr0, r10, 0
> > + beq 1f
> > + lwzcix r11, r7, r10
> > + eieio
> > + ldcix r11, r6, r10
> > +3: std r11, VCPU_XIVE_SAVED_STATE(r9)
> > + /* Fixup some of the state for the next load */
> > + li r10, 0
> > + li r0, 0xff
> > + stw r10, VCPU_XIVE_PUSHED(r9)
> > + stb r10, (VCPU_XIVE_SAVED_STATE+3)(r9)
> > + stb r0, (VCPU_XIVE_SAVED_STATE+4)(r9)
> > +1:
> > +#endif /* CONFIG_KVM_XICS */
> > /* Save more register state */
> > mfdar r6
> > mfdsisr r7
> > @@ -2035,7 +2086,7 @@ hcall_real_table:
> > .long DOTSYM(kvmppc_rm_h_eoi) - hcall_real_table
> > .long DOTSYM(kvmppc_rm_h_cppr) - hcall_real_table
> > .long DOTSYM(kvmppc_rm_h_ipi) - hcall_real_table
> > - .long 0 /* 0x70 - H_IPOLL */
> > + .long DOTSYM(kvmppc_rm_h_ipoll) - hcall_real_table
> > .long DOTSYM(kvmppc_rm_h_xirr) - hcall_real_table
> > #else
> > .long 0 /* 0x64 - H_EOI */
> > @@ -2205,7 +2256,11 @@ hcall_real_table:
> > .long 0 /* 0x2f0 */
> > .long 0 /* 0x2f4 */
> > .long 0 /* 0x2f8 */
> > - .long 0 /* 0x2fc */
> > +#ifdef CONFIG_KVM_XICS
> > + .long DOTSYM(kvmppc_rm_h_xirr_x) - hcall_real_table
> > +#else
> > + .long 0 /* 0x2fc - H_XIRR_X*/
> > +#endif
> > .long DOTSYM(kvmppc_h_random) - hcall_real_table
> > .globl hcall_real_table_end
> > hcall_real_table_end:
> > @@ -2980,6 +3035,7 @@ kvmppc_fix_pmao:
> > isync
> > blr
> >
> > +
>
> Gratuitous extra blank line.
Isn't it pretty ? :-)
> >
> > + /* Allocate the queue and retrieve infos on current node
> > for now */
> > + qpage = (__be32 *)__get_free_pages(GFP_KERNEL, xive-
> > >q_alloc_order);
>
> Possibly q_page_order would be a better name than q_alloc_order.
Maybe ... I'll add a comment too.
The rest of your comments don't need a reply :)
I'll respin.
Cheers,
Ben.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 02/12] powerpc: Sync opal-api.h
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-04-04 12:20 ` Michael Ellerman
-1 siblings, 0 replies; 38+ messages in thread
From: Michael Ellerman @ 2017-04-04 12:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt, linuxppc-dev, kvm-ppc
Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
...
Give me some change log !
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
> arch/powerpc/include/asm/opal-api.h | 302 ++++++++++++++++++++-----
It looks like you've just copied it over in its entirety, including lots
of unused cruft.
Please just give me the XIVE bits you need.
cheers
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 02/12] powerpc: Sync opal-api.h
@ 2017-04-04 12:20 ` Michael Ellerman
0 siblings, 0 replies; 38+ messages in thread
From: Michael Ellerman @ 2017-04-04 12:20 UTC (permalink / raw)
To: Benjamin Herrenschmidt, linuxppc-dev, kvm-ppc
Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
...
Give me some change log !
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
> arch/powerpc/include/asm/opal-api.h | 302 ++++++++++++++++++++-----
It looks like you've just copied it over in its entirety, including lots
of unused cruft.
Please just give me the XIVE bits you need.
cheers
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
2017-03-20 6:49 ` Benjamin Herrenschmidt
@ 2017-04-04 13:03 ` Michael Ellerman
-1 siblings, 0 replies; 38+ messages in thread
From: Michael Ellerman @ 2017-04-04 13:03 UTC (permalink / raw)
To: Benjamin Herrenschmidt, linuxppc-dev, kvm-ppc
Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
> The XIVE interrupt controller is the new interrupt controller
> found in POWER9. It supports advanced virtualization capabilities
> among other things.
>
> Currently we use a set of firmware calls that simulate the old
> "XICS" interrupt controller but this is fairly inefficient.
>
> This adds the framework for using XIVE along with a native
> backend which OPAL for configuration. Later, a backend allowing
^
calls?
> the use in a KVM or PowerVM guest will also be provided.
>
> This disables some fast path for interrupts in KVM when XIVE is
> enabled as these rely on the firmware emulation code which is no
> longer available when the XIVE is used natively by Linux.
>
> A latter patch will make KVM also directly exploit the XIVE, thus
> recovering the lost performance (and more).
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
> arch/powerpc/include/asm/xive.h | 116 +++
> arch/powerpc/include/asm/xmon.h | 2 +
> arch/powerpc/platforms/powernv/Kconfig | 2 +
> arch/powerpc/platforms/powernv/setup.c | 15 +-
> arch/powerpc/platforms/powernv/smp.c | 39 +-
> arch/powerpc/sysdev/Kconfig | 1 +
> arch/powerpc/sysdev/Makefile | 1 +
> arch/powerpc/sysdev/xive/Kconfig | 7 +
> arch/powerpc/sysdev/xive/Makefile | 4 +
> arch/powerpc/sysdev/xive/common.c | 1175 ++++++++++++++++++++++++++++++
> arch/powerpc/sysdev/xive/native.c | 604 +++++++++++++++
> arch/powerpc/sysdev/xive/xive-internal.h | 51 ++
> arch/powerpc/sysdev/xive/xive-regs.h | 88 +++
> arch/powerpc/xmon/xmon.c | 93 ++-
> 14 files changed, 2186 insertions(+), 12 deletions(-)
I'm not going to review this in one go, given it's 10:30pm already.
So just a few things that hit me straight away.
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> new file mode 100644
> index 0000000..b1604b73
> --- /dev/null
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -0,0 +1,116 @@
Copyright missing.
> +#ifndef _ASM_POWERPC_XIVE_H
> +#define _ASM_POWERPC_XIVE_H
> +
> +#define XIVE_INVALID_VP 0xffffffff
> +
> +#ifdef CONFIG_PPC_XIVE
> +
> +extern void __iomem *xive_tm_area;
I think Paul already commented on "tm" being an overly used acronym.
> +extern u32 xive_tm_offset;
> +
> +/*
> + * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
> + * have it stored in the xive_cpu structure. We also cache
> + * for normal interrupts the current target CPU.
> + */
> +struct xive_irq_data {
> + /* Setup by backend */
> + u64 flags;
> +#define XIVE_IRQ_FLAG_STORE_EOI 0x01
> +#define XIVE_IRQ_FLAG_LSI 0x02
> +#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
> +#define XIVE_IRQ_FLAG_MASK_FW 0x08
> +#define XIVE_IRQ_FLAG_EOI_FW 0x10
I don't love that style, prefer them just prior to the struct.
> + u64 eoi_page;
> + void __iomem *eoi_mmio;
> + u64 trig_page;
> + void __iomem *trig_mmio;
> + u32 esb_shift;
> + int src_chip;
Why not space out the members like you do in xive_q below, I think that
looks better given you have the long __iomem lines.
> +
> + /* Setup/used by frontend */
> + int target;
> + bool saved_p;
> +};
> +#define XIVE_INVALID_CHIP_ID -1
> +
> +/* A queue tracking structure in a CPU */
> +struct xive_q {
> + __be32 *qpage;
> + u32 msk;
> + u32 idx;
> + u32 toggle;
> + u64 eoi_phys;
> + void __iomem *eoi_mmio;
> + u32 esc_irq;
> + atomic_t count;
> + atomic_t pending_count;
> +};
> +
> +/*
> + * "magic" ESB MMIO offsets
What's an ESB?
> + */
> +#define XIVE_ESB_GET 0x800
> +#define XIVE_ESB_SET_PQ_00 0xc00
> +#define XIVE_ESB_SET_PQ_01 0xd00
> +#define XIVE_ESB_SET_PQ_10 0xe00
> +#define XIVE_ESB_SET_PQ_11 0xf00
> +#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> +
> +extern bool __xive_enabled;
> +
> +static inline bool xive_enabled(void) { return __xive_enabled; }
> +
> +extern bool xive_native_init(void);
> +extern void xive_smp_probe(void);
> +extern int xive_smp_prepare_cpu(unsigned int cpu);
> +extern void xive_smp_setup_cpu(void);
> +extern void xive_smp_disable_cpu(void);
> +extern void xive_kexec_teardown_cpu(int secondary);
> +extern void xive_shutdown(void);
> +extern void xive_flush_interrupt(void);
> +
> +/* xmon hook */
> +extern void xmon_xive_do_dump(int cpu);
> +
> +/* APIs used by KVM */
> +extern u32 xive_native_default_eq_shift(void);
> +extern u32 xive_native_alloc_vp_block(u32 max_vcpus);
> +extern void xive_native_free_vp_block(u32 vp_base);
> +extern int xive_native_populate_irq_data(u32 hw_irq,
> + struct xive_irq_data *data);
> +extern void xive_cleanup_irq_data(struct xive_irq_data *xd);
> +extern u32 xive_native_alloc_irq(void);
> +extern void xive_native_free_irq(u32 irq);
> +extern int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
> +
> +extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate);
> +extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
> +
> +extern bool __xive_irq_trigger(struct xive_irq_data *xd);
> +extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
> +extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
> +
> +extern bool is_xive_irq(struct irq_chip *chip);
> +
> +#else
> +
> +static inline bool xive_enabled(void) { return false; }
> +
> +static inline bool xive_native_init(void) { return false; }
> +static inline void xive_smp_probe(void) { }
> +extern inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
> +static inline void xive_smp_setup_cpu(void) { }
> +static inline void xive_smp_disable_cpu(void) { }
> +static inline void xive_kexec_teardown_cpu(int secondary) { }
> +static inline void xive_shutdown(void) { }
> +static inline void xive_flush_interrupt(void) { }
> +
> +static inline u32 xive_native_alloc_vp_block(u32 max_vcpus)
> + { return XIVE_INVALID_VP; }
> +static inline void xive_native_free_vp_block(u32 vp_base) { }
> +
> +#endif
> +
> +#endif /* _ASM_POWERPC_XIVE_H */
> diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h
> index 5eb8e59..eb42a0c 100644
> --- a/arch/powerpc/include/asm/xmon.h
> +++ b/arch/powerpc/include/asm/xmon.h
> @@ -29,5 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { };
> extern int cpus_are_in_xmon(void);
> #endif
>
> +extern void xmon_printf(const char *format, ...);
> +
> #endif /* __KERNEL __ */
> #endif /* __ASM_POWERPC_XMON_H */
> diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
> index 3a07e4d..81ee2ed 100644
> --- a/arch/powerpc/platforms/powernv/Kconfig
> +++ b/arch/powerpc/platforms/powernv/Kconfig
> @@ -4,6 +4,8 @@ config PPC_POWERNV
> select PPC_NATIVE
> select PPC_XICS
> select PPC_ICP_NATIVE
> + select PPC_XIVE
> + select PPC_XIVE_NATIVE
> select PPC_P7_NAP
> select PCI
> select PCI_MSI
> diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
> index d50c7d9..adceac9 100644
> --- a/arch/powerpc/platforms/powernv/setup.c
> +++ b/arch/powerpc/platforms/powernv/setup.c
> @@ -32,6 +32,7 @@
> #include <asm/machdep.h>
> #include <asm/firmware.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/kexec.h>
> #include <asm/smp.h>
> @@ -76,7 +77,9 @@ static void __init pnv_init(void)
>
> static void __init pnv_init_IRQ(void)
> {
> - xics_init();
> + /* Try using a XIVE if available, otherwise use a XICS */
> + if (!xive_native_init())
> + xics_init();
>
> WARN_ON(!ppc_md.get_irq);
> }
> @@ -218,10 +221,12 @@ static void pnv_kexec_wait_secondaries_down(void)
>
> static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> {
> - xics_kexec_teardown_cpu(secondary);
> + if (xive_enabled())
> + xive_kexec_teardown_cpu(secondary);
> + else
> + xics_kexec_teardown_cpu(secondary);
>
> /* On OPAL, we return all CPUs to firmware */
> -
> if (!firmware_has_feature(FW_FEATURE_OPAL))
> return;
>
> @@ -237,6 +242,10 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> /* Primary waits for the secondaries to have reached OPAL */
> pnv_kexec_wait_secondaries_down();
>
> + /* Switch XIVE back to emulation mode */
> + if (xive_enabled())
> + xive_shutdown();
> +
> /*
> * We might be running as little-endian - now that interrupts
> * are disabled, reset the HILE bit to big-endian so we don't
> diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
> index 8b67e1e..f571955 100644
> --- a/arch/powerpc/platforms/powernv/smp.c
> +++ b/arch/powerpc/platforms/powernv/smp.c
> @@ -29,6 +29,7 @@
> #include <asm/vdso_datapage.h>
> #include <asm/cputhreads.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/runlatch.h>
> #include <asm/code-patching.h>
> @@ -47,7 +48,9 @@
>
> static void pnv_smp_setup_cpu(int cpu)
> {
> - if (cpu != boot_cpuid)
> + if (xive_enabled())
> + xive_smp_setup_cpu();
> + else if (cpu != boot_cpuid)
> xics_setup_cpu();
>
> #ifdef CONFIG_PPC_DOORBELL
> @@ -132,7 +135,10 @@ static int pnv_smp_cpu_disable(void)
> vdso_data->processorCount--;
> if (cpu == boot_cpuid)
> boot_cpuid = cpumask_any(cpu_online_mask);
> - xics_migrate_irqs_away();
> + if (xive_enabled())
> + xive_smp_disable_cpu();
> + else
> + xics_migrate_irqs_away();
> return 0;
> }
>
> @@ -213,9 +219,12 @@ static void pnv_smp_cpu_kill_self(void)
> if (((srr1 & wmask) == SRR1_WAKEEE) ||
> ((srr1 & wmask) == SRR1_WAKEHVI) ||
> (local_paca->irq_happened & PACA_IRQ_EE)) {
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - icp_opal_flush_interrupt();
> - else
> + if (cpu_has_feature(CPU_FTR_ARCH_300)) {
> + if (xive_enabled())
> + xive_flush_interrupt();
> + else
> + icp_opal_flush_interrupt();
> + } else
> icp_native_flush_interrupt();
> } else if ((srr1 & wmask) == SRR1_WAKEHDBELL) {
> unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
> @@ -252,10 +261,26 @@ static int pnv_cpu_bootable(unsigned int nr)
> return smp_generic_cpu_bootable(nr);
> }
>
> +static int pnv_smp_prepare_cpu(int cpu)
> +{
> + if (xive_enabled())
> + return xive_smp_prepare_cpu(cpu);
> + return 0;
> +}
> +
> +static void __init pnv_smp_probe(void)
> +{
> + if (xive_enabled())
> + xive_smp_probe();
> + else
> + xics_smp_probe();
> +}
> +
> static struct smp_ops_t pnv_smp_ops = {
> .message_pass = smp_muxed_ipi_message_pass,
> - .cause_ipi = NULL, /* Filled at runtime by xics_smp_probe() */
> - .probe = xics_smp_probe,
> + .cause_ipi = NULL, /* Filled at runtime by xi{cs,ve}_smp_probe() */
> + .probe = pnv_smp_probe,
> + .prepare_cpu = pnv_smp_prepare_cpu,
> .kick_cpu = pnv_smp_kick_cpu,
> .setup_cpu = pnv_smp_setup_cpu,
> .cpu_bootable = pnv_cpu_bootable,
> diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig
> index 52dc165..caf882e 100644
> --- a/arch/powerpc/sysdev/Kconfig
> +++ b/arch/powerpc/sysdev/Kconfig
> @@ -28,6 +28,7 @@ config PPC_MSI_BITMAP
> default y if PPC_POWERNV
>
> source "arch/powerpc/sysdev/xics/Kconfig"
> +source "arch/powerpc/sysdev/xive/Kconfig"
>
> config PPC_SCOM
> bool
> diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile
> index a254824..c0ae11d 100644
> --- a/arch/powerpc/sysdev/Makefile
> +++ b/arch/powerpc/sysdev/Makefile
> @@ -71,5 +71,6 @@ obj-$(CONFIG_PPC_EARLY_DEBUG_MEMCONS) += udbg_memcons.o
> subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
>
> obj-$(CONFIG_PPC_XICS) += xics/
> +obj-$(CONFIG_PPC_XIVE) += xive/
>
> obj-$(CONFIG_GE_FPGA) += ge/
> diff --git a/arch/powerpc/sysdev/xive/Kconfig b/arch/powerpc/sysdev/xive/Kconfig
> new file mode 100644
> index 0000000..c8816c8
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Kconfig
> @@ -0,0 +1,7 @@
> +config PPC_XIVE
> + def_bool n
> + select PPC_SMP_MUXED_IPI
> + select HARDIRQS_SW_RESEND
> +
> +config PPC_XIVE_NATIVE
> + def_bool n
> diff --git a/arch/powerpc/sysdev/xive/Makefile b/arch/powerpc/sysdev/xive/Makefile
> new file mode 100644
> index 0000000..3fab303
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Makefile
> @@ -0,0 +1,4 @@
> +subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
> +
> +obj-y += common.o
> +obj-$(CONFIG_PPC_XIVE_NATIVE) += native.o
> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
> new file mode 100644
> index 0000000..96037e0
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/common.c
> @@ -0,0 +1,1175 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
If here you put:
#define pr_fmt(fmt) "xive: " fmt
Then you can drop the prefix from every pr_xxx() in the whole file.
> +#include <linux/types.h>
> +#include <linux/threads.h>
> +#include <linux/kernel.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
Unused?
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
Unused?
> +#include <linux/init.h>
> +#include <linux/cpu.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/msi.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/machdep.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/xmon.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#undef DEBUG_FLUSH
> +#undef DEBUG_ALL
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +#ifdef DEBUG_ALL
> +#define DBG_VERBOSE(fmt...) pr_devel("XIVE: " fmt)
> +#else
> +#define DBG_VERBOSE(fmt...) do { } while(0)
> +#endif
> +
> +bool __xive_enabled;
> +bool xive_cmdline_disabled;
> +
> +/* We use only one priority for now */
> +static u8 xive_irq_priority;
> +
> +void __iomem *xive_tm_area;
> +u32 xive_tm_offset;
> +static const struct xive_ops *xive_ops;
> +static struct irq_domain *xive_irq_domain;
> +
> +/* The IPIs all use the same logical irq number */
> +static u32 xive_ipi_irq;
> +
> +/* Xive state for each CPU */
> +static DEFINE_PER_CPU(struct xive_cpu *, xive_cpu);
> +
> +/*
> + * A "disabled" interrupt should never fire, to catch problems
> + * we set its logical number to this
> + */
> +#define XIVE_BAD_IRQ 0x7fffffff
Can it be anything? How about 0x7fbadbad ?
> +#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
> +
> +/* An invalid CPU target */
> +#define XIVE_INVALID_TARGET (-1)
> +
> +static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
Can it have a doc comment? And tell me what an EQ is?
> +{
> + u32 cur;
> +
> + if (!q->qpage)
> + return 0;
A newline or ..
> + cur = be32_to_cpup(q->qpage + q->idx);
> + if ((cur >> 31) == q->toggle)
> + return 0;
.. two wouldn't hurt here.
> + if (!just_peek) {
> + q->idx = (q->idx + 1) & q->msk;
> + if (q->idx == 0)
> + q->toggle ^= 1;
> + }
> + return cur & 0x7fffffff;
Is that XIVE_BAD_IRQ ?
> +}
> +
> +static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
> +{
> + u32 hirq = 0;
Is that a hwirq or something different?
> + u8 prio;
> +
> + /* Find highest pending priority */
> + while (xc->pending_prio != 0) {
> + struct xive_q *q;
> +
> + prio = ffs(xc->pending_prio) - 1;
> + DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
> +
> + /* Try to fetch */
> + hirq = xive_read_eq(&xc->queue[prio], prio, just_peek);
> +
> + /* Found something ? That's it */
> + if (hirq)
> + break;
> +
> + /* Clear pending bits */
> + xc->pending_prio &= ~(1 << prio);
> +
> + /*
> + * Check if the queue count needs adjusting due to
> + * interrupts being moved away.
> + */
> + q = &xc->queue[prio];
> + if (atomic_read(&q->pending_count)) {
> + int p = atomic_xchg(&q->pending_count, 0);
> + if (p) {
> + WARN_ON(p > atomic_read(&q->count));
> + atomic_sub(p, &q->count);
I am not sure what's going on there.
> + }
> + }
> + }
> +
> + /* If nothing was found, set CPPR to 0xff */
Would be nice to spell out CPPR somewhere.
> + if (hirq == 0)
> + prio = 0xff;
> +
> + /* Update HW CPPR to match if necessary */
> + if (prio != xc->cppr) {
> + DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n", prio);
> + xc->cppr = prio;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, prio);
What's the out_8() doing? I was expecting it to use xc, or something per-cpu.
> + }
> +
> + return hirq;
> +}
> +
> +#ifdef CONFIG_XMON
> +static void xive_dump_eq(const char *name, struct xive_q *q)
> +{
> + u32 i0, i1, idx;
> +
> + if (!q->qpage)
> + return;
> + idx = q->idx;
> + i0 = be32_to_cpup(q->qpage + idx);
> + idx = (idx + 1) & q->msk;
> + i1 = be32_to_cpup(q->qpage + idx);
> + xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
> + q->toggle, i0, i1);
> +}
> +
> +void xmon_xive_do_dump(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_irq_data *xd;
> + uint64_t val, offset;
u64 ?
> +
> + xmon_printf("XIVE state for CPU %d:\n", cpu);
> + xmon_printf(" pp=%02x cppr=%02x\n", xc->pending_prio, xc->cppr);
> + xive_dump_eq("IRQ", &xc->queue[xive_irq_priority]);
> + xd = &xc->ipi_data;
> + offset = 0x800;
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> + val = in_be64(xd->eoi_mmio + offset);
> + xmon_printf(" IPI state: %x:%c%c\n", xc->hw_ipi,
> + val & 2 ? 'P' : 'p',
> + val & 1 ? 'Q' : 'q');
> +}
> +#endif /* CONFIG_XMON */
> +
> +static void xive_update_pending_irqs(struct xive_cpu *xc)
> +{
> + u8 he, cppr;
> + u16 ack;
> +
> + /* Perform the acknowledge hypervisor to register cycle */
> + ack = be16_to_cpu(__raw_readw(xive_tm_area + TM_SPC_ACK_HV_REG));
> +
> + /* Synchronize subsequent queue accesses */
> + mb();
> +
> + DBG_VERBOSE("CPU %d get_irq, ack=%04x\n", smp_processor_id(), ack);
> +
> + /* Check the HE field */
> + cppr = ack & 0xff;
> + he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8));
> + switch(he) {
> + case TM_QW3_NSR_HE_NONE:
> + break;
> + case TM_QW3_NSR_HE_PHYS:
> + if (cppr == 0xff)
> + return;
> + xc->pending_prio |= 1 << cppr;
> + if (cppr >= xc->cppr)
> + pr_err("XIVE: CPU %d odd ack CPPR, got %d at %d\n",
> + smp_processor_id(), cppr, xc->cppr);
> + xc->cppr = cppr;
> + break;
> + case TM_QW3_NSR_HE_POOL:
> + case TM_QW3_NSR_HE_LSI:
> + pr_err("XIVE: CPU %d got unexpected interrupt type HE=%d\n",
> + smp_processor_id(), he);
> + return;
> + }
> +}
> +
> +static unsigned int xive_get_irq(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + u32 hirq;
> +
> + /*
> + * This can be called either as a result of a HW interrupt or
> + * as a "replay" because EOI decided there was still something
> + * in one of the queues.
> + *
> + * First we perform an ACK cycle in order to update our mask
> + * of pending priorities. This will also have the effect of
> + * updating the CPPR to the most favored pending interrupts.
> + *
> + * In the future, if we have a way to differenciate a first
> + * entry (on HW interrupt) from a replay triggered by EOI,
> + * we could skip this on replays unless we soft-mask tells us
> + * that a new HW interrupt occurred.
> + */
> + xive_update_pending_irqs(xc);
> +
> + DBG_VERBOSE("get_irq: pending=%02x\n", xc->pending_prio);
> +
> + hirq = xive_scan_interrupts(xc, false);
> +
> + DBG_VERBOSE("get_irq: got irq 0x%x, new pending=0x%02x\n",
> + hirq, xc->pending_prio);
> +
> + /* Return pending interrupt if any */
> + if (hirq == XIVE_BAD_IRQ)
> + return 0;
> + return hirq;
> +}
> +
> +
> +static void xive_do_queue_eoi(struct xive_cpu *xc)
> +{
> + if (xive_scan_interrupts(xc, true) != 0) {
> + DBG_VERBOSE("eoi: pending=0x%02x\n", xc->pending_prio);
> + force_external_irq_replay();
> + }
> +}
> +
> +static u8 xive_poke_esb(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = in_be64(xd->eoi_mmio + offset);
> +
> + return (u8)val;
> +}
> +
> +static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd)
> +{
> + /* If the XIVE supports the new "store EOI facility, use it */
> + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> + out_be64(xd->eoi_mmio, 0);
> + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> + if (WARN_ON_ONCE(!xive_ops->eoi))
> + return;
> + xive_ops->eoi(hw_irq);
> + } else {
> + uint8_t eoi_val;
u8?
> +
> + /*
> + * Otherwise for EOI, we use the special MMIO that does
> + * a clear of both P and Q and returns the old Q.
> + *
> + * This allows us to then do a re-trigger if Q was set
> + * rather than synthetizing an interrupt in software
> + */
> + eoi_val = xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> + DBG_VERBOSE("eoi_val=%x\n", offset, eoi_val);
> +
> + if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
> + return;
> +
> + /* Re-trigger */
> + if (xd->trig_mmio)
> + out_be64(xd->trig_mmio, 0);
> + }
> +
> +}
> +
> +static void xive_irq_eoi(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
> + d->irq, irqd_to_hwirq(d), xc->pending_prio);
> +
> + if (!irqd_irq_disabled(d))
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> +
> + /*
> + * Clear saved_p to indicate that it's no longer occupying
> + * a queue slot on the target queue
> + */
> + xd->saved_p = false;
> +
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_do_source_set_mask(struct xive_irq_data *xd,
> + bool masked)
> +{
> + if (masked)
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_01);
> + else
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> +}
> +
> +static bool xive_try_pick_target(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> + int max;
> +
> + /* Calculate max number of interrupts in that queue.
> + *
> + * We leave a gap of 1 just in case...
> + */
> + max = (q->msk + 1) - 1;
> + return !!atomic_add_unless(&q->count, 1, max);
> +}
> +
> +static void xive_dec_target_count(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> +
> + if (WARN_ON(cpu < 0))
> + return;
> +
> + /*
> + * We increment the "pending count" which will be used
> + * to decrement the target queue count whenever it's next
> + * processed and found empty. This ensure that we don't
> + * decrement while we still have the interrupt there
> + * occupying a slot.
> + */
> + atomic_inc(&q->pending_count);
> +}
> +
> +static int xive_find_target_in_mask(const struct cpumask *mask,
> + unsigned int fuzz)
> +{
> + int cpu, first, num, i;
> +
> + /* Pick up a starting point CPU in the mask based on fuzz */
> + num = cpumask_weight(mask);
> + first = (fuzz++) % num;
> +
> + /* Locate it */
> + cpu = cpumask_first(mask);
> + for (i = 0; i < first; i++)
> + cpu = cpumask_next(cpu, mask);
> + first = cpu;
> +
> + /*
> + * Now go through the entire mask until we find a valid
> + * target.
> + */
> + for (;;) {
> + /*
> + * We re-check online as the fallback case passes us
> + * an untested affinity mask
> + */
> + if (cpu_online(cpu) && xive_try_pick_target(cpu))
> + return cpu;
> + cpu = cpumask_next(cpu, mask);
> + if (cpu == first)
> + break;
> + }
> + return -1;
> +}
> +
> +static int xive_pick_irq_target(struct irq_data *d,
> + const struct cpumask *affinity)
> +{
> + static unsigned int fuzz;
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + cpumask_var_t mask;
> + int cpu = -1;
> +
> + /*
> + * Pick a target CPU for an interrupt. This is done at
> + * startup or if the affinity is changed in a way that
> + * invalidates the current target.
> + */
> +
> + /* If we have chip IDs, first we try to build a mask of
> + * CPUs matching ther CPU and find a target in there
> + */
> + if (xd->src_chip != XIVE_INVALID_CHIP_ID &&
> + zalloc_cpumask_var(&mask, GFP_ATOMIC)) {
> + /* Build a mask of matching chip IDs */
> + for_each_cpu_and(cpu, affinity, cpu_online_mask) {
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + if (xc->chip_id == xd->src_chip)
> + cpumask_set_cpu(cpu, mask);
> + }
> + /* Try to find a target */
> + if (!cpumask_empty(mask))
> + cpu = xive_find_target_in_mask(mask, fuzz++);
> + free_cpumask_var(mask);
> + if (cpu >= 0)
> + return cpu;
> + fuzz--;
> + }
> +
> + /* No chip IDs, fallback to using the affinity mask */
> + return xive_find_target_in_mask(affinity, fuzz++);
> +}
> +
> +static unsigned int xive_irq_startup(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + int target, rc;
> +
> + DBG("xive_irq_startup: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> +#ifdef CONFIG_PCI_MSI
> + /*
> + * The generic MSI code returns with the interrupt disabled on the
> + * card, using the MSI mask bits. Firmware doesn't appear to unmask
> + * at that level, so we do it here by hand.
> + */
> + if (irq_data_get_msi_desc(d))
> + pci_msi_unmask_irq(d);
> +#endif
> +
> + /* Pick a target */
> + target = xive_pick_irq_target(d, irq_data_get_affinity_mask(d));
> + if (target == XIVE_INVALID_TARGET) {
> + /* Try again breaking affinity */
> + target = xive_pick_irq_target(d, cpu_online_mask);
> + if (target == XIVE_INVALID_TARGET)
> + return -ENXIO;
> + pr_warn("XIVE: irq %d started with broken affinity\n",
> + d->irq);
> + }
> + xd->target = target;
> +
> + /*
> + * Configure the logical number to be the Linux IRQ number
> + * and set the target queue
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc)
> + return rc;
> +
> + /* Unmask the ESB */
> + xive_do_source_set_mask(xd, false);
> +
> + return 0;
> +}
> +
> +static void xive_irq_shutdown(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + DBG("xive_irq_shutdown: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> + if (WARN_ON(xd->target == XIVE_INVALID_TARGET))
> + return;
> +
> + /* Mask the interrupt at the source */
> + xive_do_source_set_mask(xd, true);
> +
> + /* Mask the interrupt in HW in the IVT/EAS */
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, hw_irq);
> +
> + xive_dec_target_count(xd->target);
> + xd->target = XIVE_INVALID_TARGET;
> +}
> +
> +static void xive_irq_unmask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_unmask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call FW to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + xive_irq_priority, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, false);
> +}
> +
> +static void xive_irq_mask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_mask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call OPAL to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, true);
> +}
> +
> +static int xive_irq_set_affinity(struct irq_data *d,
> + const struct cpumask *cpumask,
> + bool force)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + u32 target, old_target;
> + int rc = 0;
> +
> + DBG("xive_irq_set_affinity: irq %d\n", d->irq);
> +
> + /* Is this valid ? */
> + if (cpumask_any_and(cpumask, cpu_online_mask) >= nr_cpu_ids)
> + return -EINVAL;
> +
> + /* If existing target is already in the new mask, and is
> + * online then do nothing.
> + */
> + if (cpu_online(xd->target) &&
> + cpumask_test_cpu(xd->target, cpumask))
> + return IRQ_SET_MASK_OK;
> +
> + /* Pick a new target */
> + target = xive_pick_irq_target(d, cpumask);
> +
> + /* No target found */
> + if (target == XIVE_INVALID_TARGET)
> + return -ENXIO;
> +
> + old_target = xd->target;
> +
> + /*
> + * Only configure the irq if it's not currently passed-through to
> + * a KVM guest
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
> + return rc;
> + }
> +
> + DBG(" target: 0x%x\n", target);
> + xd->target = target;
> +
> + /* Give up previous target */
> + if (old_target != XIVE_INVALID_TARGET)
> + xive_dec_target_count(old_target);
> +
> + return IRQ_SET_MASK_OK;
> +}
> +
> +static int xive_irq_set_type(struct irq_data *d, unsigned int flow_type)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /*
> + * We only support these. This has really no effect other than setting
> + * the corresponding descriptor bits mind you but those will in turn
> + * affect the resend function when re-enabling an edge interrupt.
> + *
> + * Set set the default to edge as explained in map().
> + */
> + if (flow_type == IRQ_TYPE_DEFAULT || flow_type == IRQ_TYPE_NONE)
> + flow_type = IRQ_TYPE_EDGE_RISING;
> +
> + if (flow_type != IRQ_TYPE_EDGE_RISING &&
> + flow_type != IRQ_TYPE_LEVEL_LOW)
> + return -EINVAL;
> +
> + irqd_set_trigger_type(d, flow_type);
> +
> + /*
> + * Double check it matches what the FW thinks
> + *
> + * NOTE: We don't know yet if the PAPR interface will provide
> + * the LSI vs MSI information appart from the device-tree so
> + * this check might have to move into an optional backend call
> + * that is specific to the native backend
> + */
> + if ((flow_type == IRQ_TYPE_LEVEL_LOW) !=
> + !!(xd->flags & XIVE_IRQ_FLAG_LSI))
> + pr_warn("XIVE: Interrupt %d (HW 0x%x) type mismatch,"
> + " Linux says %s, FW says %s\n",
> + d->irq, (u32)irqd_to_hwirq(d),
> + (flow_type == IRQ_TYPE_LEVEL_LOW) ? "Level" : "Edge",
> + (xd->flags & XIVE_IRQ_FLAG_LSI) ? "Level" : "Edge");
> +
> + return IRQ_SET_MASK_OK_NOCOPY;
> +}
> +
> +static int xive_irq_retrigger(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /* This should be only for MSIs */
> + if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
> + return 0;
> +
> + /*
> + * To perform a retrigger, we first set the PQ bits to
> + * 11, then perform an EOI.
> + */
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_11);
> +
> + /*
> + * Note: We pass "0" to the hw_irq argument in order to
> + * avoid calling into the backend EOI code which we don't
> + * want to do in the case of a re-trigger. Backends typically
> + * only do EOI for LSIs anyway.
> + */
> + xive_do_source_eoi(0, xd);
> +
> + return 1;
> +}
> +
> +static struct irq_chip xive_irq_chip = {
> + .name = "XIVE-IRQ",
> + .irq_startup = xive_irq_startup,
> + .irq_shutdown = xive_irq_shutdown,
> + .irq_eoi = xive_irq_eoi,
> + .irq_mask = xive_irq_mask,
> + .irq_unmask = xive_irq_unmask,
> + .irq_set_affinity = xive_irq_set_affinity,
> + .irq_set_type = xive_irq_set_type,
> + .irq_retrigger = xive_irq_retrigger,
> +};
> +
> +bool is_xive_irq(struct irq_chip *chip)
> +{
> + return chip == &xive_irq_chip;
> +}
> +
> +void xive_cleanup_irq_data(struct xive_irq_data *xd)
> +{
> + if (xd->eoi_mmio) {
> + iounmap(xd->eoi_mmio);
> + if (xd->eoi_mmio == xd->trig_mmio)
> + xd->trig_mmio = NULL;
> + xd->eoi_mmio = NULL;
> + }
> + if (xd->trig_mmio) {
> + iounmap(xd->trig_mmio);
> + xd->trig_mmio = NULL;
> + }
> +}
> +
> +static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
> +{
> + struct xive_irq_data *xd;
> + int rc;
> +
> + xd = kzalloc(sizeof(struct xive_irq_data), GFP_KERNEL);
> + if (!xd)
> + return -ENOMEM;
> + rc = xive_ops->populate_irq_data(hw, xd);
> + if (rc) {
> + kfree(xd);
> + return rc;
> + }
> + xd->target = XIVE_INVALID_TARGET;
> + irq_set_handler_data(virq, xd);
> +
> + return 0;
> +}
> +
> +static void xive_irq_free_data(unsigned int virq)
> +{
> + struct xive_irq_data *xd = irq_get_handler_data(virq);
> +
> + if (!xd)
> + return;
> + irq_set_handler_data(virq, NULL);
> + xive_cleanup_irq_data(xd);
> + kfree(xd);
> +}
> +
> +#ifdef CONFIG_SMP
> +
> +static void xive_cause_ipi(int cpu, unsigned long msg)
> +{
> + struct xive_cpu *xc;
> + struct xive_irq_data *xd;
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + DBG_VERBOSE("IPI msg#%ld CPU %d -> %d (HW IRQ 0x%x)\n",
> + msg, smp_processor_id(), cpu, xc->hw_ipi);
> +
> + xd = &xc->ipi_data;
> + if (WARN_ON(!xd->trig_mmio))
> + return;
> + out_be64(xd->trig_mmio, 0);
> +}
> +
> +static irqreturn_t xive_muxed_ipi_action(int irq, void *dev_id)
> +{
> + return smp_ipi_demux();
> +}
> +
> +static void xive_ipi_eoi(struct irq_data *d)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Handle possible race with unplug and drop stale IPIs */
> + if (!xc)
> + return;
> + xive_do_source_eoi(xc->hw_ipi, &xc->ipi_data);
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_ipi_unmask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static void xive_ipi_mask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static struct irq_chip xive_ipi_chip = {
> + .name = "XIVE-IPI",
> + .irq_eoi = xive_ipi_eoi,
> + .irq_mask = xive_ipi_mask,
> + .irq_unmask = xive_ipi_unmask,
> +};
> +
> +static void __init xive_request_ipi(void)
> +{
> + unsigned int virq;
> +
> + /* Initialize it */
> + virq = irq_create_mapping(xive_irq_domain, 0);
> + xive_ipi_irq = virq;
> +
> + BUG_ON(request_irq(virq, xive_muxed_ipi_action,
> + IRQF_PERCPU | IRQF_NO_THREAD, "IPI", NULL));
> +}
> +
> +static int xive_setup_cpu_ipi(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> + int rc;
> +
> + pr_debug("XIVE: Setting up IPI for CPU %d\n", cpu);
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + /* Check if we are already setup */
> + if (xc->hw_ipi != 0)
> + return 0;
> +
> + /* Grab an IPI from the backend, this will populate xc->hw_ipi */
> + if (xive_ops->get_ipi(cpu, xc))
> + return -EIO;
> +
> + /* Populate the IRQ data in the xive_cpu structure and
> + * configure the HW / enable the IPIs
> + */
> + rc = xive_ops->populate_irq_data(xc->hw_ipi, &xc->ipi_data);
> + if (rc) {
> + pr_err("XIVE: Failed to populate IPI data on CPU %d\n", cpu);
> + return -EIO;
> + }
> + rc = xive_ops->configure_irq(xc->hw_ipi,
> + get_hard_smp_processor_id(cpu),
> + xive_irq_priority, xive_ipi_irq);
> + if (rc) {
> + pr_err("XIVE: Failed to map IPI CPU %d\n", cpu);
> + return -EIO;
> + }
> + DBG("XIVE: CPU %d HW IPI %x, virq %d, trig_mmio=%p\n", cpu,
> + xc->hw_ipi, xive_ipi_irq, xc->ipi_data.trig_mmio);
> +
> + /* Unmask it */
> + xive_do_source_set_mask(&xc->ipi_data, false);
> +
> + return 0;
> +}
> +
> +static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + /* Disable the IPI and free the IRQ data */
> +
> + /* Already cleaned up ? */
> + if (xc->hw_ipi == 0)
> + return;
> +
> + /* Mask the IPI */
> + xive_do_source_set_mask(&xc->ipi_data, true);
> +
> + /*
> + * Note: We don't call xive_cleanup_irq_data() to free
> + * the mappings as this is called from an IPI on kexec
> + * which is not a safe environment to call iounmap()
> + */
> +
> + /* Deconfigure/mask in the backend */
> + xive_ops->configure_irq(xc->hw_ipi, hard_smp_processor_id(),
> + 0xff, xive_ipi_irq);
> +
> + /* Free the IPIs in the backend */
> + xive_ops->put_ipi(cpu, xc);
> +}
> +
> +void __init xive_smp_probe(void)
> +{
> + smp_ops->cause_ipi = xive_cause_ipi;
> +
> + /* Register the IPI */
> + xive_request_ipi();
> +
> + /* Allocate and setup IPI for the boot CPU */
> + xive_setup_cpu_ipi(smp_processor_id());
> +}
> +
> +#endif /* CONFIG_SMP */
> +
> +static int xive_irq_domain_map(struct irq_domain *h, unsigned int virq,
> + irq_hw_number_t hw)
> +{
> + int rc;
> +
> + /*
> + * Mark interrupts as edge sensitive by default so that resend
> + * actually works. Will fix that up below if needed.
> + */
> + irq_clear_status_flags(virq, IRQ_LEVEL);
> +
> + /* IPIs are special and come up with HW number 0 */
> + if (hw == 0) {
> + /*
> + * IPIs are marked per-cpu. We use separate HW interrupts under
> + * the hood but associated with the same "linux" interrupt
> + */
> + irq_set_chip_and_handler(virq, &xive_ipi_chip,
> + handle_percpu_irq);
> + return 0;
> + }
> +
> + rc = xive_irq_alloc_data(virq, hw);
> + if (rc)
> + return rc;
> +
> + irq_set_chip_and_handler(virq, &xive_irq_chip, handle_fasteoi_irq);
> +
> + return 0;
> +}
> +
> +static void xive_irq_domain_unmap(struct irq_domain *d, unsigned int virq)
> +{
> + struct irq_data *data = irq_get_irq_data(virq);
> + unsigned int hw_irq;
> +
> + if (!data)
> + return;
> + hw_irq = (unsigned int)irqd_to_hwirq(data);
> + if (hw_irq)
> + xive_irq_free_data(virq);
> +}
> +
> +static int xive_irq_domain_xlate(struct irq_domain *h, struct device_node *ct,
> + const u32 *intspec, unsigned int intsize,
> + irq_hw_number_t *out_hwirq, unsigned int *out_flags)
> +
> +{
> + *out_hwirq = intspec[0];
> +
> + /*
> + * If intsize is at least 2, we look for the type in the second cell,
> + * we assume the LSB indicates a level interrupt.
> + */
> + if (intsize > 1) {
> + if (intspec[1] & 1)
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> + else
> + *out_flags = IRQ_TYPE_EDGE_RISING;
> + } else
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> +
> + return 0;
> +}
> +
> +static int xive_irq_domain_match(struct irq_domain *h, struct device_node *node,
> + enum irq_domain_bus_token bus_token)
> +{
> + return xive_ops->match(node);
> +}
> +
> +static const struct irq_domain_ops xive_irq_domain_ops = {
> + .match = xive_irq_domain_match,
> + .map = xive_irq_domain_map,
> + .unmap = xive_irq_domain_unmap,
> + .xlate = xive_irq_domain_xlate,
> +};
> +
> +static void __init xive_init_host(void)
> +{
> + xive_irq_domain = irq_domain_add_nomap(NULL, XIVE_MAX_IRQ,
> + &xive_irq_domain_ops, NULL);
> + BUG_ON(xive_irq_domain == NULL);
> + irq_set_default_host(xive_irq_domain);
> +}
> +
> +static void xive_cleanup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + if (xc->queue[xive_irq_priority].qpage)
> + xive_ops->cleanup_queue(cpu, xc, xive_irq_priority);
> +}
> +
> +static int xive_setup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + int rc = 0;
> +
> + /* We setup 1 queues for now with a 64k page */
> + if (!xc->queue[xive_irq_priority].qpage)
> + rc = xive_ops->setup_queue(cpu, xc, xive_irq_priority);
> +
> + return rc;
> +}
> +
> +static int xive_prepare_cpu(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> +
> + xc = per_cpu(xive_cpu, cpu);
> + if (!xc) {
> + struct device_node *np;
> +
> + xc = kzalloc_node(sizeof(struct xive_cpu),
> + GFP_KERNEL, cpu_to_node(cpu));
> + if (!xc)
> + return -ENOMEM;
> + np = of_get_cpu_node(cpu, NULL);
> + if (np)
> + xc->chip_id = of_get_ibm_chip_id(np);
> + of_node_put(np);
> +
> + per_cpu(xive_cpu, cpu) = xc;
> + }
> +
> + /* Setup EQs if not already */
> + return xive_setup_cpu_queues(cpu, xc);
> +}
> +
> +static void xive_setup_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Debug: Dump the TM state */
> + DBG("CPU %d [HW 0x%02x] VT=%02x\n",
> + smp_processor_id(), hard_smp_processor_id(),
> + in_8(xive_tm_area + xive_tm_offset + TM_WORD2));
> +
> + /* The backend might have additional things to do */
> + if (xive_ops->setup_cpu)
> + xive_ops->setup_cpu(smp_processor_id(), xc);
> +
> + /* Set CPPR to 0xff to enable flow of interrupts */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +#ifdef CONFIG_SMP
> +void xive_smp_setup_cpu(void)
> +{
> + DBG("XIVE: SMP setup CPU %d\n", smp_processor_id());
> +
> + /* This will have already been done on the boot CPU */
> + if (smp_processor_id() != boot_cpuid)
> + xive_setup_cpu();
> +
> +}
> +
> +int xive_smp_prepare_cpu(unsigned int cpu)
> +{
> + int rc;
> +
> + /* Allocate per-CPU data and queues */
> + rc = xive_prepare_cpu(cpu);
> + if (rc)
> + return rc;
> +
> + /* Allocate and setup IPI for the new CPU */
> + return xive_setup_cpu_ipi(cpu);
> +}
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +static void xive_flush_cpu_queue(unsigned int cpu, struct xive_cpu *xc)
> +{
> + u32 irq;
> +
> + /* We assume local irqs are disabled */
> + WARN_ON(!irqs_disabled());
> +
> + /* Check what's already in the CPU queue */
> + while ((irq = xive_scan_interrupts(xc, false)) != 0) {
> + /*
> + * We need to re-route that interrupt to its new distination.
> + * First get and lock the descriptor
> + */
> + struct irq_desc *desc = irq_to_desc(irq);
> + struct irq_data *d = irq_desc_get_irq_data(desc);
> + struct xive_irq_data *xd;
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + /*
> + * Ignore anything that isn't a XIVE irq and ignore
> + * IPIs, so can just be dropped.
> + */
> + if (d->domain != xive_irq_domain || hw_irq == 0)
> + continue;
> +#ifdef DEBUG_FLUSH
> + pr_info("CPU %d: Got irq %d while offline, re-routing...\n",
> + cpu, irq);
> +#endif
> + raw_spin_lock(&desc->lock);
> + xd = irq_desc_get_handler_data(desc);
> +
> + /* For LSIs, we EOI, this will cause a resend if it's
> + * still asserted. Otherwise do an MSI retrigger
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_LSI)
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> + else
> + xive_irq_retrigger(d);
> + raw_spin_unlock(&desc->lock);
> + }
> +}
> +
> +void xive_smp_disable_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Migrate interrupts away from the CPU */
> + irq_migrate_all_off_this_cpu();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Flush everything still in the queue */
> + xive_flush_cpu_queue(cpu, xc);
> +
> + /* Re-enable CPPR */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +void xive_flush_interrupt(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Called if an interrupt occurs while the CPU is hot unplugged */
> + xive_flush_cpu_queue(cpu, xc);
> +}
> +
> +#endif /* CONFIG_HOTPLUG_CPU */
> +
> +#endif /* CONFIG_SMP */
> +
> +void xive_kexec_teardown_cpu(int secondary)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Backend cleanup if any */
> + if (xive_ops->teardown_cpu)
> + xive_ops->teardown_cpu(cpu, xc);
> +
> + /* Get rid of IPI */
> + xive_cleanup_cpu_ipi(cpu, xc);
> +
> + /* Disable and free the queues */
> + xive_cleanup_cpu_queues(cpu, xc);
> +}
> +
> +void xive_shutdown(void)
> +{
> + xive_ops->shutdown();
> +}
> +
> +bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
> + u8 max_prio)
> +{
> + xive_tm_area = area;
> + xive_tm_offset = offset;
> + xive_ops = ops;
> + xive_irq_priority = max_prio;
> +
> + ppc_md.get_irq = xive_get_irq;
> + __xive_enabled = true;
> +
> + DBG("Initializing host..\n");
> + xive_init_host();
> +
> + DBG("Initializing boot CPU..\n");
> +
> + /* Allocate per-CPU data and queues */
> + xive_prepare_cpu(smp_processor_id());
> +
> + /* Get ready for interrupts */
> + xive_setup_cpu();
> +
> + pr_info("XIVE: Interrupt handling intialized with %s backend\n",
> + xive_ops->name);
> + pr_info("XIVE: Using priority %d for all interrupts\n", max_prio);
> +
> + return true;
> +}
> +
> +static int __init xive_off(char *arg)
> +{
> + xive_cmdline_disabled = true;
> + return 0;
> +}
> +__setup("xive=off", xive_off);
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> new file mode 100644
> index 0000000..26cc6bf
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -0,0 +1,604 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +#include <linux/types.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
Unused?
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
Unused?
> +#include <linux/init.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/cpumask.h>
> +#include <linux/mm.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/opal.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +/* Enable this for using queue MMIO page for EOI. We don't currently
> + * use it as we always notify
> + */
> +#undef USE_QUEUE_MMIO
Dead code? Or we want to keep it?
> +static u32 xive_provision_size;
> +static u32 *xive_provision_chips;
> +static u32 xive_provision_chip_count;
> +static u32 xive_queue_shift;
> +static u32 xive_pool_vps = XIVE_INVALID_VP;
> +static struct kmem_cache *xive_provision_cache;
> +
> +int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
> +{
> + __be64 flags, eoi_page, trig_page;
> + __be32 esb_shift, src_chip;
> + u64 opal_flags;
> + s64 rc;
> +
> + memset(data, 0, sizeof(*data));
> +
> + rc = opal_xive_get_irq_info(hw_irq, &flags, &eoi_page, &trig_page,
> + &esb_shift, &src_chip);
> + if (rc) {
> + pr_err("XIVE: opal_xive_get_irq_info(0x%x) returned %lld\n",
> + hw_irq, rc);
> + return -EINVAL;
> + }
> +
> + opal_flags = be64_to_cpu(flags);
> + if (opal_flags & OPAL_XIVE_IRQ_STORE_EOI)
> + data->flags |= XIVE_IRQ_FLAG_STORE_EOI;
> + if (opal_flags & OPAL_XIVE_IRQ_LSI)
> + data->flags |= XIVE_IRQ_FLAG_LSI;
> + if (opal_flags & OPAL_XIVE_IRQ_SHIFT_BUG)
> + data->flags |= XIVE_IRQ_FLAG_SHIFT_BUG;
> + if (opal_flags & OPAL_XIVE_IRQ_MASK_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_MASK_FW;
> + if (opal_flags & OPAL_XIVE_IRQ_EOI_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_EOI_FW;
> + data->eoi_page = be64_to_cpu(eoi_page);
> + data->trig_page = be64_to_cpu(trig_page);
> + data->esb_shift = be32_to_cpu(esb_shift);
> + data->src_chip = be32_to_cpu(src_chip);
> +
> + data->eoi_mmio = ioremap(data->eoi_page, 1u << data->esb_shift);
> + if (!data->eoi_mmio) {
> + pr_err("XIVE: Failed to map EOI page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> +
> + if (!data->trig_page)
> + return 0;
> + if (data->trig_page == data->eoi_page) {
> + data->trig_mmio = data->eoi_mmio;
> + return 0;
> + }
> +
> + data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
> + if (!data->trig_mmio) {
> + pr_err("XIVE: Failed to map trigger page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> + return 0;
> +}
> +
> +int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_set_irq_config(hw_irq, target, prio, sw_irq);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + return rc == 0 ? 0 : -ENXIO;
> +}
> +
> +/* This can be called multiple time to change a queue configuration */
> +int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate)
> +{
> + s64 rc = 0;
> + __be64 qeoi_page_be;
> + __be32 esc_irq_be;
> + u64 flags, qpage_phys;
> +
> + /* If there's an actual queue page, clean it */
> + if (order) {
> + BUG_ON(!qpage);
Can't we just return an error?
> + qpage_phys = __pa(qpage);
> + } else
> + qpage_phys = 0;
> +
> + /* Initialize the rest of the fields */
> + q->msk = order ? ((1u << (order - 2)) - 1) : 0;
> + q->idx = 0;
> + q->toggle = 0;
> +
> + rc = opal_xive_get_queue_info(vp_id, prio, NULL, NULL,
> + &qeoi_page_be,
> + &esc_irq_be,
> + NULL);
> + if (rc) {
> + pr_err("XIVE: Error %lld getting queue info prio %d\n",
> + rc, prio);
> + rc = -EIO;
> + goto fail;
> + }
> + q->eoi_phys = be64_to_cpu(qeoi_page_be);
> +
> +#ifdef USE_QUEUE_MMIO
> + if (!q->eoi_mmio)
> + q->eoi_mmio = ioremap(q->eoi_phys, PAGE_SIZE);
> + if (!q->eoi_mmio) {
> + pr_err("XIVE: Failed to map queue MMIO prio %d CPU %d\n",
> + rc, prio, cpu);
> + rc = -ENOMEM;
> + goto fail;
> + }
> +#endif /* USE_QUEUE_MMIO */
> +
> +
...
> +static bool xive_parse_provisioning(struct device_node *np)
> +{
> + int rc;
> +
> + if (of_property_read_u32(np, "ibm,xive-provision-page-size",
> + &xive_provision_size) < 0)
> + return true;
> + rc = of_property_count_elems_of_size(np, "ibm,xive-provision-chips", 4);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d getting provision chips array\n", rc);
> + return false;
> + }
> + xive_provision_chip_count = rc;
> + if (rc == 0)
> + return true;
> +
> + xive_provision_chips = kzalloc(4 * xive_provision_chip_count,
> + GFP_KERNEL);
> + BUG_ON(!xive_provision_chips);
return false?
> +
> + rc = of_property_read_u32_array(np, "ibm,xive-provision-chips",
> + xive_provision_chips,
> + xive_provision_chip_count);
...
> diff --git a/arch/powerpc/sysdev/xive/xive-internal.h b/arch/powerpc/sysdev/xive/xive-internal.h
> new file mode 100644
> index 0000000..e736fc5
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-internal.h
> @@ -0,0 +1,51 @@
Copyright missing.
> +#ifndef __XIVE_INTERNAL_H
> +#define __XIVE_INTERNAL_H
...
> diff --git a/arch/powerpc/sysdev/xive/xive-regs.h b/arch/powerpc/sysdev/xive/xive-regs.h
> new file mode 100644
> index 0000000..f1edb23
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-regs.h
> @@ -0,0 +1,88 @@
Copyright missing.
> +#ifndef __XIVE_REGS_H__
> +#define __XIVE_REGS_H__
...
> diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> index 16321ad..c71e919 100644
> --- a/arch/powerpc/xmon/xmon.c
> +++ b/arch/powerpc/xmon/xmon.c
...
> +
> +static void dump_one_xive_irq(uint32_t num)
u32?
> +{
> + int64_t rc;
> + __be64 vp;
> + uint8_t prio;
u8?
zzzzz ...
cheers
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
@ 2017-04-04 13:03 ` Michael Ellerman
0 siblings, 0 replies; 38+ messages in thread
From: Michael Ellerman @ 2017-04-04 13:03 UTC (permalink / raw)
To: Benjamin Herrenschmidt, linuxppc-dev, kvm-ppc
Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
> The XIVE interrupt controller is the new interrupt controller
> found in POWER9. It supports advanced virtualization capabilities
> among other things.
>
> Currently we use a set of firmware calls that simulate the old
> "XICS" interrupt controller but this is fairly inefficient.
>
> This adds the framework for using XIVE along with a native
> backend which OPAL for configuration. Later, a backend allowing
^
calls?
> the use in a KVM or PowerVM guest will also be provided.
>
> This disables some fast path for interrupts in KVM when XIVE is
> enabled as these rely on the firmware emulation code which is no
> longer available when the XIVE is used natively by Linux.
>
> A latter patch will make KVM also directly exploit the XIVE, thus
> recovering the lost performance (and more).
>
> Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> ---
> arch/powerpc/include/asm/xive.h | 116 +++
> arch/powerpc/include/asm/xmon.h | 2 +
> arch/powerpc/platforms/powernv/Kconfig | 2 +
> arch/powerpc/platforms/powernv/setup.c | 15 +-
> arch/powerpc/platforms/powernv/smp.c | 39 +-
> arch/powerpc/sysdev/Kconfig | 1 +
> arch/powerpc/sysdev/Makefile | 1 +
> arch/powerpc/sysdev/xive/Kconfig | 7 +
> arch/powerpc/sysdev/xive/Makefile | 4 +
> arch/powerpc/sysdev/xive/common.c | 1175 ++++++++++++++++++++++++++++++
> arch/powerpc/sysdev/xive/native.c | 604 +++++++++++++++
> arch/powerpc/sysdev/xive/xive-internal.h | 51 ++
> arch/powerpc/sysdev/xive/xive-regs.h | 88 +++
> arch/powerpc/xmon/xmon.c | 93 ++-
> 14 files changed, 2186 insertions(+), 12 deletions(-)
I'm not going to review this in one go, given it's 10:30pm already.
So just a few things that hit me straight away.
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> new file mode 100644
> index 0000000..b1604b73
> --- /dev/null
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -0,0 +1,116 @@
Copyright missing.
> +#ifndef _ASM_POWERPC_XIVE_H
> +#define _ASM_POWERPC_XIVE_H
> +
> +#define XIVE_INVALID_VP 0xffffffff
> +
> +#ifdef CONFIG_PPC_XIVE
> +
> +extern void __iomem *xive_tm_area;
I think Paul already commented on "tm" being an overly used acronym.
> +extern u32 xive_tm_offset;
> +
> +/*
> + * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
> + * have it stored in the xive_cpu structure. We also cache
> + * for normal interrupts the current target CPU.
> + */
> +struct xive_irq_data {
> + /* Setup by backend */
> + u64 flags;
> +#define XIVE_IRQ_FLAG_STORE_EOI 0x01
> +#define XIVE_IRQ_FLAG_LSI 0x02
> +#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
> +#define XIVE_IRQ_FLAG_MASK_FW 0x08
> +#define XIVE_IRQ_FLAG_EOI_FW 0x10
I don't love that style, prefer them just prior to the struct.
> + u64 eoi_page;
> + void __iomem *eoi_mmio;
> + u64 trig_page;
> + void __iomem *trig_mmio;
> + u32 esb_shift;
> + int src_chip;
Why not space out the members like you do in xive_q below, I think that
looks better given you have the long __iomem lines.
> +
> + /* Setup/used by frontend */
> + int target;
> + bool saved_p;
> +};
> +#define XIVE_INVALID_CHIP_ID -1
> +
> +/* A queue tracking structure in a CPU */
> +struct xive_q {
> + __be32 *qpage;
> + u32 msk;
> + u32 idx;
> + u32 toggle;
> + u64 eoi_phys;
> + void __iomem *eoi_mmio;
> + u32 esc_irq;
> + atomic_t count;
> + atomic_t pending_count;
> +};
> +
> +/*
> + * "magic" ESB MMIO offsets
What's an ESB?
> + */
> +#define XIVE_ESB_GET 0x800
> +#define XIVE_ESB_SET_PQ_00 0xc00
> +#define XIVE_ESB_SET_PQ_01 0xd00
> +#define XIVE_ESB_SET_PQ_10 0xe00
> +#define XIVE_ESB_SET_PQ_11 0xf00
> +#define XIVE_ESB_MASK XIVE_ESB_SET_PQ_01
> +
> +extern bool __xive_enabled;
> +
> +static inline bool xive_enabled(void) { return __xive_enabled; }
> +
> +extern bool xive_native_init(void);
> +extern void xive_smp_probe(void);
> +extern int xive_smp_prepare_cpu(unsigned int cpu);
> +extern void xive_smp_setup_cpu(void);
> +extern void xive_smp_disable_cpu(void);
> +extern void xive_kexec_teardown_cpu(int secondary);
> +extern void xive_shutdown(void);
> +extern void xive_flush_interrupt(void);
> +
> +/* xmon hook */
> +extern void xmon_xive_do_dump(int cpu);
> +
> +/* APIs used by KVM */
> +extern u32 xive_native_default_eq_shift(void);
> +extern u32 xive_native_alloc_vp_block(u32 max_vcpus);
> +extern void xive_native_free_vp_block(u32 vp_base);
> +extern int xive_native_populate_irq_data(u32 hw_irq,
> + struct xive_irq_data *data);
> +extern void xive_cleanup_irq_data(struct xive_irq_data *xd);
> +extern u32 xive_native_alloc_irq(void);
> +extern void xive_native_free_irq(u32 irq);
> +extern int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq);
> +
> +extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate);
> +extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
> +
> +extern bool __xive_irq_trigger(struct xive_irq_data *xd);
> +extern bool __xive_irq_retrigger(struct xive_irq_data *xd);
> +extern void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd);
> +
> +extern bool is_xive_irq(struct irq_chip *chip);
> +
> +#else
> +
> +static inline bool xive_enabled(void) { return false; }
> +
> +static inline bool xive_native_init(void) { return false; }
> +static inline void xive_smp_probe(void) { }
> +extern inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
> +static inline void xive_smp_setup_cpu(void) { }
> +static inline void xive_smp_disable_cpu(void) { }
> +static inline void xive_kexec_teardown_cpu(int secondary) { }
> +static inline void xive_shutdown(void) { }
> +static inline void xive_flush_interrupt(void) { }
> +
> +static inline u32 xive_native_alloc_vp_block(u32 max_vcpus)
> + { return XIVE_INVALID_VP; }
> +static inline void xive_native_free_vp_block(u32 vp_base) { }
> +
> +#endif
> +
> +#endif /* _ASM_POWERPC_XIVE_H */
> diff --git a/arch/powerpc/include/asm/xmon.h b/arch/powerpc/include/asm/xmon.h
> index 5eb8e59..eb42a0c 100644
> --- a/arch/powerpc/include/asm/xmon.h
> +++ b/arch/powerpc/include/asm/xmon.h
> @@ -29,5 +29,7 @@ static inline void xmon_register_spus(struct list_head *list) { };
> extern int cpus_are_in_xmon(void);
> #endif
>
> +extern void xmon_printf(const char *format, ...);
> +
> #endif /* __KERNEL __ */
> #endif /* __ASM_POWERPC_XMON_H */
> diff --git a/arch/powerpc/platforms/powernv/Kconfig b/arch/powerpc/platforms/powernv/Kconfig
> index 3a07e4d..81ee2ed 100644
> --- a/arch/powerpc/platforms/powernv/Kconfig
> +++ b/arch/powerpc/platforms/powernv/Kconfig
> @@ -4,6 +4,8 @@ config PPC_POWERNV
> select PPC_NATIVE
> select PPC_XICS
> select PPC_ICP_NATIVE
> + select PPC_XIVE
> + select PPC_XIVE_NATIVE
> select PPC_P7_NAP
> select PCI
> select PCI_MSI
> diff --git a/arch/powerpc/platforms/powernv/setup.c b/arch/powerpc/platforms/powernv/setup.c
> index d50c7d9..adceac9 100644
> --- a/arch/powerpc/platforms/powernv/setup.c
> +++ b/arch/powerpc/platforms/powernv/setup.c
> @@ -32,6 +32,7 @@
> #include <asm/machdep.h>
> #include <asm/firmware.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/kexec.h>
> #include <asm/smp.h>
> @@ -76,7 +77,9 @@ static void __init pnv_init(void)
>
> static void __init pnv_init_IRQ(void)
> {
> - xics_init();
> + /* Try using a XIVE if available, otherwise use a XICS */
> + if (!xive_native_init())
> + xics_init();
>
> WARN_ON(!ppc_md.get_irq);
> }
> @@ -218,10 +221,12 @@ static void pnv_kexec_wait_secondaries_down(void)
>
> static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> {
> - xics_kexec_teardown_cpu(secondary);
> + if (xive_enabled())
> + xive_kexec_teardown_cpu(secondary);
> + else
> + xics_kexec_teardown_cpu(secondary);
>
> /* On OPAL, we return all CPUs to firmware */
> -
> if (!firmware_has_feature(FW_FEATURE_OPAL))
> return;
>
> @@ -237,6 +242,10 @@ static void pnv_kexec_cpu_down(int crash_shutdown, int secondary)
> /* Primary waits for the secondaries to have reached OPAL */
> pnv_kexec_wait_secondaries_down();
>
> + /* Switch XIVE back to emulation mode */
> + if (xive_enabled())
> + xive_shutdown();
> +
> /*
> * We might be running as little-endian - now that interrupts
> * are disabled, reset the HILE bit to big-endian so we don't
> diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
> index 8b67e1e..f571955 100644
> --- a/arch/powerpc/platforms/powernv/smp.c
> +++ b/arch/powerpc/platforms/powernv/smp.c
> @@ -29,6 +29,7 @@
> #include <asm/vdso_datapage.h>
> #include <asm/cputhreads.h>
> #include <asm/xics.h>
> +#include <asm/xive.h>
> #include <asm/opal.h>
> #include <asm/runlatch.h>
> #include <asm/code-patching.h>
> @@ -47,7 +48,9 @@
>
> static void pnv_smp_setup_cpu(int cpu)
> {
> - if (cpu != boot_cpuid)
> + if (xive_enabled())
> + xive_smp_setup_cpu();
> + else if (cpu != boot_cpuid)
> xics_setup_cpu();
>
> #ifdef CONFIG_PPC_DOORBELL
> @@ -132,7 +135,10 @@ static int pnv_smp_cpu_disable(void)
> vdso_data->processorCount--;
> if (cpu = boot_cpuid)
> boot_cpuid = cpumask_any(cpu_online_mask);
> - xics_migrate_irqs_away();
> + if (xive_enabled())
> + xive_smp_disable_cpu();
> + else
> + xics_migrate_irqs_away();
> return 0;
> }
>
> @@ -213,9 +219,12 @@ static void pnv_smp_cpu_kill_self(void)
> if (((srr1 & wmask) = SRR1_WAKEEE) ||
> ((srr1 & wmask) = SRR1_WAKEHVI) ||
> (local_paca->irq_happened & PACA_IRQ_EE)) {
> - if (cpu_has_feature(CPU_FTR_ARCH_300))
> - icp_opal_flush_interrupt();
> - else
> + if (cpu_has_feature(CPU_FTR_ARCH_300)) {
> + if (xive_enabled())
> + xive_flush_interrupt();
> + else
> + icp_opal_flush_interrupt();
> + } else
> icp_native_flush_interrupt();
> } else if ((srr1 & wmask) = SRR1_WAKEHDBELL) {
> unsigned long msg = PPC_DBELL_TYPE(PPC_DBELL_SERVER);
> @@ -252,10 +261,26 @@ static int pnv_cpu_bootable(unsigned int nr)
> return smp_generic_cpu_bootable(nr);
> }
>
> +static int pnv_smp_prepare_cpu(int cpu)
> +{
> + if (xive_enabled())
> + return xive_smp_prepare_cpu(cpu);
> + return 0;
> +}
> +
> +static void __init pnv_smp_probe(void)
> +{
> + if (xive_enabled())
> + xive_smp_probe();
> + else
> + xics_smp_probe();
> +}
> +
> static struct smp_ops_t pnv_smp_ops = {
> .message_pass = smp_muxed_ipi_message_pass,
> - .cause_ipi = NULL, /* Filled at runtime by xics_smp_probe() */
> - .probe = xics_smp_probe,
> + .cause_ipi = NULL, /* Filled at runtime by xi{cs,ve}_smp_probe() */
> + .probe = pnv_smp_probe,
> + .prepare_cpu = pnv_smp_prepare_cpu,
> .kick_cpu = pnv_smp_kick_cpu,
> .setup_cpu = pnv_smp_setup_cpu,
> .cpu_bootable = pnv_cpu_bootable,
> diff --git a/arch/powerpc/sysdev/Kconfig b/arch/powerpc/sysdev/Kconfig
> index 52dc165..caf882e 100644
> --- a/arch/powerpc/sysdev/Kconfig
> +++ b/arch/powerpc/sysdev/Kconfig
> @@ -28,6 +28,7 @@ config PPC_MSI_BITMAP
> default y if PPC_POWERNV
>
> source "arch/powerpc/sysdev/xics/Kconfig"
> +source "arch/powerpc/sysdev/xive/Kconfig"
>
> config PPC_SCOM
> bool
> diff --git a/arch/powerpc/sysdev/Makefile b/arch/powerpc/sysdev/Makefile
> index a254824..c0ae11d 100644
> --- a/arch/powerpc/sysdev/Makefile
> +++ b/arch/powerpc/sysdev/Makefile
> @@ -71,5 +71,6 @@ obj-$(CONFIG_PPC_EARLY_DEBUG_MEMCONS) += udbg_memcons.o
> subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
>
> obj-$(CONFIG_PPC_XICS) += xics/
> +obj-$(CONFIG_PPC_XIVE) += xive/
>
> obj-$(CONFIG_GE_FPGA) += ge/
> diff --git a/arch/powerpc/sysdev/xive/Kconfig b/arch/powerpc/sysdev/xive/Kconfig
> new file mode 100644
> index 0000000..c8816c8
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Kconfig
> @@ -0,0 +1,7 @@
> +config PPC_XIVE
> + def_bool n
> + select PPC_SMP_MUXED_IPI
> + select HARDIRQS_SW_RESEND
> +
> +config PPC_XIVE_NATIVE
> + def_bool n
> diff --git a/arch/powerpc/sysdev/xive/Makefile b/arch/powerpc/sysdev/xive/Makefile
> new file mode 100644
> index 0000000..3fab303
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/Makefile
> @@ -0,0 +1,4 @@
> +subdir-ccflags-$(CONFIG_PPC_WERROR) := -Werror
> +
> +obj-y += common.o
> +obj-$(CONFIG_PPC_XIVE_NATIVE) += native.o
> diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
> new file mode 100644
> index 0000000..96037e0
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/common.c
> @@ -0,0 +1,1175 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
If here you put:
#define pr_fmt(fmt) "xive: " fmt
Then you can drop the prefix from every pr_xxx() in the whole file.
> +#include <linux/types.h>
> +#include <linux/threads.h>
> +#include <linux/kernel.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
Unused?
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
Unused?
> +#include <linux/init.h>
> +#include <linux/cpu.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/msi.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/machdep.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/xmon.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#undef DEBUG_FLUSH
> +#undef DEBUG_ALL
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +#ifdef DEBUG_ALL
> +#define DBG_VERBOSE(fmt...) pr_devel("XIVE: " fmt)
> +#else
> +#define DBG_VERBOSE(fmt...) do { } while(0)
> +#endif
> +
> +bool __xive_enabled;
> +bool xive_cmdline_disabled;
> +
> +/* We use only one priority for now */
> +static u8 xive_irq_priority;
> +
> +void __iomem *xive_tm_area;
> +u32 xive_tm_offset;
> +static const struct xive_ops *xive_ops;
> +static struct irq_domain *xive_irq_domain;
> +
> +/* The IPIs all use the same logical irq number */
> +static u32 xive_ipi_irq;
> +
> +/* Xive state for each CPU */
> +static DEFINE_PER_CPU(struct xive_cpu *, xive_cpu);
> +
> +/*
> + * A "disabled" interrupt should never fire, to catch problems
> + * we set its logical number to this
> + */
> +#define XIVE_BAD_IRQ 0x7fffffff
Can it be anything? How about 0x7fbadbad ?
> +#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
> +
> +/* An invalid CPU target */
> +#define XIVE_INVALID_TARGET (-1)
> +
> +static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
Can it have a doc comment? And tell me what an EQ is?
> +{
> + u32 cur;
> +
> + if (!q->qpage)
> + return 0;
A newline or ..
> + cur = be32_to_cpup(q->qpage + q->idx);
> + if ((cur >> 31) = q->toggle)
> + return 0;
.. two wouldn't hurt here.
> + if (!just_peek) {
> + q->idx = (q->idx + 1) & q->msk;
> + if (q->idx = 0)
> + q->toggle ^= 1;
> + }
> + return cur & 0x7fffffff;
Is that XIVE_BAD_IRQ ?
> +}
> +
> +static u32 xive_scan_interrupts(struct xive_cpu *xc, bool just_peek)
> +{
> + u32 hirq = 0;
Is that a hwirq or something different?
> + u8 prio;
> +
> + /* Find highest pending priority */
> + while (xc->pending_prio != 0) {
> + struct xive_q *q;
> +
> + prio = ffs(xc->pending_prio) - 1;
> + DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
> +
> + /* Try to fetch */
> + hirq = xive_read_eq(&xc->queue[prio], prio, just_peek);
> +
> + /* Found something ? That's it */
> + if (hirq)
> + break;
> +
> + /* Clear pending bits */
> + xc->pending_prio &= ~(1 << prio);
> +
> + /*
> + * Check if the queue count needs adjusting due to
> + * interrupts being moved away.
> + */
> + q = &xc->queue[prio];
> + if (atomic_read(&q->pending_count)) {
> + int p = atomic_xchg(&q->pending_count, 0);
> + if (p) {
> + WARN_ON(p > atomic_read(&q->count));
> + atomic_sub(p, &q->count);
I am not sure what's going on there.
> + }
> + }
> + }
> +
> + /* If nothing was found, set CPPR to 0xff */
Would be nice to spell out CPPR somewhere.
> + if (hirq = 0)
> + prio = 0xff;
> +
> + /* Update HW CPPR to match if necessary */
> + if (prio != xc->cppr) {
> + DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n", prio);
> + xc->cppr = prio;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, prio);
What's the out_8() doing? I was expecting it to use xc, or something per-cpu.
> + }
> +
> + return hirq;
> +}
> +
> +#ifdef CONFIG_XMON
> +static void xive_dump_eq(const char *name, struct xive_q *q)
> +{
> + u32 i0, i1, idx;
> +
> + if (!q->qpage)
> + return;
> + idx = q->idx;
> + i0 = be32_to_cpup(q->qpage + idx);
> + idx = (idx + 1) & q->msk;
> + i1 = be32_to_cpup(q->qpage + idx);
> + xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
> + q->toggle, i0, i1);
> +}
> +
> +void xmon_xive_do_dump(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_irq_data *xd;
> + uint64_t val, offset;
u64 ?
> +
> + xmon_printf("XIVE state for CPU %d:\n", cpu);
> + xmon_printf(" pp=%02x cppr=%02x\n", xc->pending_prio, xc->cppr);
> + xive_dump_eq("IRQ", &xc->queue[xive_irq_priority]);
> + xd = &xc->ipi_data;
> + offset = 0x800;
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> + val = in_be64(xd->eoi_mmio + offset);
> + xmon_printf(" IPI state: %x:%c%c\n", xc->hw_ipi,
> + val & 2 ? 'P' : 'p',
> + val & 1 ? 'Q' : 'q');
> +}
> +#endif /* CONFIG_XMON */
> +
> +static void xive_update_pending_irqs(struct xive_cpu *xc)
> +{
> + u8 he, cppr;
> + u16 ack;
> +
> + /* Perform the acknowledge hypervisor to register cycle */
> + ack = be16_to_cpu(__raw_readw(xive_tm_area + TM_SPC_ACK_HV_REG));
> +
> + /* Synchronize subsequent queue accesses */
> + mb();
> +
> + DBG_VERBOSE("CPU %d get_irq, ack=%04x\n", smp_processor_id(), ack);
> +
> + /* Check the HE field */
> + cppr = ack & 0xff;
> + he = GETFIELD(TM_QW3_NSR_HE, (ack >> 8));
> + switch(he) {
> + case TM_QW3_NSR_HE_NONE:
> + break;
> + case TM_QW3_NSR_HE_PHYS:
> + if (cppr = 0xff)
> + return;
> + xc->pending_prio |= 1 << cppr;
> + if (cppr >= xc->cppr)
> + pr_err("XIVE: CPU %d odd ack CPPR, got %d at %d\n",
> + smp_processor_id(), cppr, xc->cppr);
> + xc->cppr = cppr;
> + break;
> + case TM_QW3_NSR_HE_POOL:
> + case TM_QW3_NSR_HE_LSI:
> + pr_err("XIVE: CPU %d got unexpected interrupt type HE=%d\n",
> + smp_processor_id(), he);
> + return;
> + }
> +}
> +
> +static unsigned int xive_get_irq(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + u32 hirq;
> +
> + /*
> + * This can be called either as a result of a HW interrupt or
> + * as a "replay" because EOI decided there was still something
> + * in one of the queues.
> + *
> + * First we perform an ACK cycle in order to update our mask
> + * of pending priorities. This will also have the effect of
> + * updating the CPPR to the most favored pending interrupts.
> + *
> + * In the future, if we have a way to differenciate a first
> + * entry (on HW interrupt) from a replay triggered by EOI,
> + * we could skip this on replays unless we soft-mask tells us
> + * that a new HW interrupt occurred.
> + */
> + xive_update_pending_irqs(xc);
> +
> + DBG_VERBOSE("get_irq: pending=%02x\n", xc->pending_prio);
> +
> + hirq = xive_scan_interrupts(xc, false);
> +
> + DBG_VERBOSE("get_irq: got irq 0x%x, new pending=0x%02x\n",
> + hirq, xc->pending_prio);
> +
> + /* Return pending interrupt if any */
> + if (hirq = XIVE_BAD_IRQ)
> + return 0;
> + return hirq;
> +}
> +
> +
> +static void xive_do_queue_eoi(struct xive_cpu *xc)
> +{
> + if (xive_scan_interrupts(xc, true) != 0) {
> + DBG_VERBOSE("eoi: pending=0x%02x\n", xc->pending_prio);
> + force_external_irq_replay();
> + }
> +}
> +
> +static u8 xive_poke_esb(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = in_be64(xd->eoi_mmio + offset);
> +
> + return (u8)val;
> +}
> +
> +static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data *xd)
> +{
> + /* If the XIVE supports the new "store EOI facility, use it */
> + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> + out_be64(xd->eoi_mmio, 0);
> + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> + if (WARN_ON_ONCE(!xive_ops->eoi))
> + return;
> + xive_ops->eoi(hw_irq);
> + } else {
> + uint8_t eoi_val;
u8?
> +
> + /*
> + * Otherwise for EOI, we use the special MMIO that does
> + * a clear of both P and Q and returns the old Q.
> + *
> + * This allows us to then do a re-trigger if Q was set
> + * rather than synthetizing an interrupt in software
> + */
> + eoi_val = xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> + DBG_VERBOSE("eoi_val=%x\n", offset, eoi_val);
> +
> + if ((xd->flags & XIVE_IRQ_FLAG_LSI) || !(eoi_val & 1))
> + return;
> +
> + /* Re-trigger */
> + if (xd->trig_mmio)
> + out_be64(xd->trig_mmio, 0);
> + }
> +
> +}
> +
> +static void xive_irq_eoi(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + DBG_VERBOSE("eoi_irq: irq=%d [0x%lx] pending=%02x\n",
> + d->irq, irqd_to_hwirq(d), xc->pending_prio);
> +
> + if (!irqd_irq_disabled(d))
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> +
> + /*
> + * Clear saved_p to indicate that it's no longer occupying
> + * a queue slot on the target queue
> + */
> + xd->saved_p = false;
> +
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_do_source_set_mask(struct xive_irq_data *xd,
> + bool masked)
> +{
> + if (masked)
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_01);
> + else
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_00);
> +}
> +
> +static bool xive_try_pick_target(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> + int max;
> +
> + /* Calculate max number of interrupts in that queue.
> + *
> + * We leave a gap of 1 just in case...
> + */
> + max = (q->msk + 1) - 1;
> + return !!atomic_add_unless(&q->count, 1, max);
> +}
> +
> +static void xive_dec_target_count(int cpu)
> +{
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + struct xive_q *q = &xc->queue[xive_irq_priority];
> +
> + if (WARN_ON(cpu < 0))
> + return;
> +
> + /*
> + * We increment the "pending count" which will be used
> + * to decrement the target queue count whenever it's next
> + * processed and found empty. This ensure that we don't
> + * decrement while we still have the interrupt there
> + * occupying a slot.
> + */
> + atomic_inc(&q->pending_count);
> +}
> +
> +static int xive_find_target_in_mask(const struct cpumask *mask,
> + unsigned int fuzz)
> +{
> + int cpu, first, num, i;
> +
> + /* Pick up a starting point CPU in the mask based on fuzz */
> + num = cpumask_weight(mask);
> + first = (fuzz++) % num;
> +
> + /* Locate it */
> + cpu = cpumask_first(mask);
> + for (i = 0; i < first; i++)
> + cpu = cpumask_next(cpu, mask);
> + first = cpu;
> +
> + /*
> + * Now go through the entire mask until we find a valid
> + * target.
> + */
> + for (;;) {
> + /*
> + * We re-check online as the fallback case passes us
> + * an untested affinity mask
> + */
> + if (cpu_online(cpu) && xive_try_pick_target(cpu))
> + return cpu;
> + cpu = cpumask_next(cpu, mask);
> + if (cpu = first)
> + break;
> + }
> + return -1;
> +}
> +
> +static int xive_pick_irq_target(struct irq_data *d,
> + const struct cpumask *affinity)
> +{
> + static unsigned int fuzz;
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + cpumask_var_t mask;
> + int cpu = -1;
> +
> + /*
> + * Pick a target CPU for an interrupt. This is done at
> + * startup or if the affinity is changed in a way that
> + * invalidates the current target.
> + */
> +
> + /* If we have chip IDs, first we try to build a mask of
> + * CPUs matching ther CPU and find a target in there
> + */
> + if (xd->src_chip != XIVE_INVALID_CHIP_ID &&
> + zalloc_cpumask_var(&mask, GFP_ATOMIC)) {
> + /* Build a mask of matching chip IDs */
> + for_each_cpu_and(cpu, affinity, cpu_online_mask) {
> + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> + if (xc->chip_id = xd->src_chip)
> + cpumask_set_cpu(cpu, mask);
> + }
> + /* Try to find a target */
> + if (!cpumask_empty(mask))
> + cpu = xive_find_target_in_mask(mask, fuzz++);
> + free_cpumask_var(mask);
> + if (cpu >= 0)
> + return cpu;
> + fuzz--;
> + }
> +
> + /* No chip IDs, fallback to using the affinity mask */
> + return xive_find_target_in_mask(affinity, fuzz++);
> +}
> +
> +static unsigned int xive_irq_startup(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + int target, rc;
> +
> + DBG("xive_irq_startup: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> +#ifdef CONFIG_PCI_MSI
> + /*
> + * The generic MSI code returns with the interrupt disabled on the
> + * card, using the MSI mask bits. Firmware doesn't appear to unmask
> + * at that level, so we do it here by hand.
> + */
> + if (irq_data_get_msi_desc(d))
> + pci_msi_unmask_irq(d);
> +#endif
> +
> + /* Pick a target */
> + target = xive_pick_irq_target(d, irq_data_get_affinity_mask(d));
> + if (target = XIVE_INVALID_TARGET) {
> + /* Try again breaking affinity */
> + target = xive_pick_irq_target(d, cpu_online_mask);
> + if (target = XIVE_INVALID_TARGET)
> + return -ENXIO;
> + pr_warn("XIVE: irq %d started with broken affinity\n",
> + d->irq);
> + }
> + xd->target = target;
> +
> + /*
> + * Configure the logical number to be the Linux IRQ number
> + * and set the target queue
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc)
> + return rc;
> +
> + /* Unmask the ESB */
> + xive_do_source_set_mask(xd, false);
> +
> + return 0;
> +}
> +
> +static void xive_irq_shutdown(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + DBG("xive_irq_shutdown: irq %d [0x%x] data @%p\n",
> + d->irq, hw_irq, d);
> +
> + if (WARN_ON(xd->target = XIVE_INVALID_TARGET))
> + return;
> +
> + /* Mask the interrupt at the source */
> + xive_do_source_set_mask(xd, true);
> +
> + /* Mask the interrupt in HW in the IVT/EAS */
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, hw_irq);
> +
> + xive_dec_target_count(xd->target);
> + xd->target = XIVE_INVALID_TARGET;
> +}
> +
> +static void xive_irq_unmask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_unmask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call FW to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + xive_irq_priority, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, false);
> +}
> +
> +static void xive_irq_mask(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + DBG("xive_irq_mask: irq %d data @%p\n", d->irq, xd);
> +
> + /*
> + * This is a workaround for PCI LSI problems on P9, for
> + * these, we call OPAL to set the mask. The problems might
> + * be fixed by P9 DD2.0, if that is the case, we will make
> + * this a DD1 workaround only
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_MASK_FW) {
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(xd->target),
> + 0xff, d->irq);
> + return;
> + }
> +
> + xive_do_source_set_mask(xd, true);
> +}
> +
> +static int xive_irq_set_affinity(struct irq_data *d,
> + const struct cpumask *cpumask,
> + bool force)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> + u32 target, old_target;
> + int rc = 0;
> +
> + DBG("xive_irq_set_affinity: irq %d\n", d->irq);
> +
> + /* Is this valid ? */
> + if (cpumask_any_and(cpumask, cpu_online_mask) >= nr_cpu_ids)
> + return -EINVAL;
> +
> + /* If existing target is already in the new mask, and is
> + * online then do nothing.
> + */
> + if (cpu_online(xd->target) &&
> + cpumask_test_cpu(xd->target, cpumask))
> + return IRQ_SET_MASK_OK;
> +
> + /* Pick a new target */
> + target = xive_pick_irq_target(d, cpumask);
> +
> + /* No target found */
> + if (target = XIVE_INVALID_TARGET)
> + return -ENXIO;
> +
> + old_target = xd->target;
> +
> + /*
> + * Only configure the irq if it's not currently passed-through to
> + * a KVM guest
> + */
> + rc = xive_ops->configure_irq(hw_irq,
> + get_hard_smp_processor_id(target),
> + xive_irq_priority, d->irq);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d reconfiguring irq %d\n", rc, d->irq);
> + return rc;
> + }
> +
> + DBG(" target: 0x%x\n", target);
> + xd->target = target;
> +
> + /* Give up previous target */
> + if (old_target != XIVE_INVALID_TARGET)
> + xive_dec_target_count(old_target);
> +
> + return IRQ_SET_MASK_OK;
> +}
> +
> +static int xive_irq_set_type(struct irq_data *d, unsigned int flow_type)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /*
> + * We only support these. This has really no effect other than setting
> + * the corresponding descriptor bits mind you but those will in turn
> + * affect the resend function when re-enabling an edge interrupt.
> + *
> + * Set set the default to edge as explained in map().
> + */
> + if (flow_type = IRQ_TYPE_DEFAULT || flow_type = IRQ_TYPE_NONE)
> + flow_type = IRQ_TYPE_EDGE_RISING;
> +
> + if (flow_type != IRQ_TYPE_EDGE_RISING &&
> + flow_type != IRQ_TYPE_LEVEL_LOW)
> + return -EINVAL;
> +
> + irqd_set_trigger_type(d, flow_type);
> +
> + /*
> + * Double check it matches what the FW thinks
> + *
> + * NOTE: We don't know yet if the PAPR interface will provide
> + * the LSI vs MSI information appart from the device-tree so
> + * this check might have to move into an optional backend call
> + * that is specific to the native backend
> + */
> + if ((flow_type = IRQ_TYPE_LEVEL_LOW) !> + !!(xd->flags & XIVE_IRQ_FLAG_LSI))
> + pr_warn("XIVE: Interrupt %d (HW 0x%x) type mismatch,"
> + " Linux says %s, FW says %s\n",
> + d->irq, (u32)irqd_to_hwirq(d),
> + (flow_type = IRQ_TYPE_LEVEL_LOW) ? "Level" : "Edge",
> + (xd->flags & XIVE_IRQ_FLAG_LSI) ? "Level" : "Edge");
> +
> + return IRQ_SET_MASK_OK_NOCOPY;
> +}
> +
> +static int xive_irq_retrigger(struct irq_data *d)
> +{
> + struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> +
> + /* This should be only for MSIs */
> + if (WARN_ON(xd->flags & XIVE_IRQ_FLAG_LSI))
> + return 0;
> +
> + /*
> + * To perform a retrigger, we first set the PQ bits to
> + * 11, then perform an EOI.
> + */
> + xive_poke_esb(xd, XIVE_ESB_SET_PQ_11);
> +
> + /*
> + * Note: We pass "0" to the hw_irq argument in order to
> + * avoid calling into the backend EOI code which we don't
> + * want to do in the case of a re-trigger. Backends typically
> + * only do EOI for LSIs anyway.
> + */
> + xive_do_source_eoi(0, xd);
> +
> + return 1;
> +}
> +
> +static struct irq_chip xive_irq_chip = {
> + .name = "XIVE-IRQ",
> + .irq_startup = xive_irq_startup,
> + .irq_shutdown = xive_irq_shutdown,
> + .irq_eoi = xive_irq_eoi,
> + .irq_mask = xive_irq_mask,
> + .irq_unmask = xive_irq_unmask,
> + .irq_set_affinity = xive_irq_set_affinity,
> + .irq_set_type = xive_irq_set_type,
> + .irq_retrigger = xive_irq_retrigger,
> +};
> +
> +bool is_xive_irq(struct irq_chip *chip)
> +{
> + return chip = &xive_irq_chip;
> +}
> +
> +void xive_cleanup_irq_data(struct xive_irq_data *xd)
> +{
> + if (xd->eoi_mmio) {
> + iounmap(xd->eoi_mmio);
> + if (xd->eoi_mmio = xd->trig_mmio)
> + xd->trig_mmio = NULL;
> + xd->eoi_mmio = NULL;
> + }
> + if (xd->trig_mmio) {
> + iounmap(xd->trig_mmio);
> + xd->trig_mmio = NULL;
> + }
> +}
> +
> +static int xive_irq_alloc_data(unsigned int virq, irq_hw_number_t hw)
> +{
> + struct xive_irq_data *xd;
> + int rc;
> +
> + xd = kzalloc(sizeof(struct xive_irq_data), GFP_KERNEL);
> + if (!xd)
> + return -ENOMEM;
> + rc = xive_ops->populate_irq_data(hw, xd);
> + if (rc) {
> + kfree(xd);
> + return rc;
> + }
> + xd->target = XIVE_INVALID_TARGET;
> + irq_set_handler_data(virq, xd);
> +
> + return 0;
> +}
> +
> +static void xive_irq_free_data(unsigned int virq)
> +{
> + struct xive_irq_data *xd = irq_get_handler_data(virq);
> +
> + if (!xd)
> + return;
> + irq_set_handler_data(virq, NULL);
> + xive_cleanup_irq_data(xd);
> + kfree(xd);
> +}
> +
> +#ifdef CONFIG_SMP
> +
> +static void xive_cause_ipi(int cpu, unsigned long msg)
> +{
> + struct xive_cpu *xc;
> + struct xive_irq_data *xd;
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + DBG_VERBOSE("IPI msg#%ld CPU %d -> %d (HW IRQ 0x%x)\n",
> + msg, smp_processor_id(), cpu, xc->hw_ipi);
> +
> + xd = &xc->ipi_data;
> + if (WARN_ON(!xd->trig_mmio))
> + return;
> + out_be64(xd->trig_mmio, 0);
> +}
> +
> +static irqreturn_t xive_muxed_ipi_action(int irq, void *dev_id)
> +{
> + return smp_ipi_demux();
> +}
> +
> +static void xive_ipi_eoi(struct irq_data *d)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Handle possible race with unplug and drop stale IPIs */
> + if (!xc)
> + return;
> + xive_do_source_eoi(xc->hw_ipi, &xc->ipi_data);
> + xive_do_queue_eoi(xc);
> +}
> +
> +static void xive_ipi_unmask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static void xive_ipi_mask(struct irq_data *d)
> +{
> + /* Nothing to do, we never mask IPIs, but the callback
> + * must exist
> + */
> +}
> +
> +static struct irq_chip xive_ipi_chip = {
> + .name = "XIVE-IPI",
> + .irq_eoi = xive_ipi_eoi,
> + .irq_mask = xive_ipi_mask,
> + .irq_unmask = xive_ipi_unmask,
> +};
> +
> +static void __init xive_request_ipi(void)
> +{
> + unsigned int virq;
> +
> + /* Initialize it */
> + virq = irq_create_mapping(xive_irq_domain, 0);
> + xive_ipi_irq = virq;
> +
> + BUG_ON(request_irq(virq, xive_muxed_ipi_action,
> + IRQF_PERCPU | IRQF_NO_THREAD, "IPI", NULL));
> +}
> +
> +static int xive_setup_cpu_ipi(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> + int rc;
> +
> + pr_debug("XIVE: Setting up IPI for CPU %d\n", cpu);
> +
> + xc = per_cpu(xive_cpu, cpu);
> +
> + /* Check if we are already setup */
> + if (xc->hw_ipi != 0)
> + return 0;
> +
> + /* Grab an IPI from the backend, this will populate xc->hw_ipi */
> + if (xive_ops->get_ipi(cpu, xc))
> + return -EIO;
> +
> + /* Populate the IRQ data in the xive_cpu structure and
> + * configure the HW / enable the IPIs
> + */
> + rc = xive_ops->populate_irq_data(xc->hw_ipi, &xc->ipi_data);
> + if (rc) {
> + pr_err("XIVE: Failed to populate IPI data on CPU %d\n", cpu);
> + return -EIO;
> + }
> + rc = xive_ops->configure_irq(xc->hw_ipi,
> + get_hard_smp_processor_id(cpu),
> + xive_irq_priority, xive_ipi_irq);
> + if (rc) {
> + pr_err("XIVE: Failed to map IPI CPU %d\n", cpu);
> + return -EIO;
> + }
> + DBG("XIVE: CPU %d HW IPI %x, virq %d, trig_mmio=%p\n", cpu,
> + xc->hw_ipi, xive_ipi_irq, xc->ipi_data.trig_mmio);
> +
> + /* Unmask it */
> + xive_do_source_set_mask(&xc->ipi_data, false);
> +
> + return 0;
> +}
> +
> +static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
> +{
> + /* Disable the IPI and free the IRQ data */
> +
> + /* Already cleaned up ? */
> + if (xc->hw_ipi = 0)
> + return;
> +
> + /* Mask the IPI */
> + xive_do_source_set_mask(&xc->ipi_data, true);
> +
> + /*
> + * Note: We don't call xive_cleanup_irq_data() to free
> + * the mappings as this is called from an IPI on kexec
> + * which is not a safe environment to call iounmap()
> + */
> +
> + /* Deconfigure/mask in the backend */
> + xive_ops->configure_irq(xc->hw_ipi, hard_smp_processor_id(),
> + 0xff, xive_ipi_irq);
> +
> + /* Free the IPIs in the backend */
> + xive_ops->put_ipi(cpu, xc);
> +}
> +
> +void __init xive_smp_probe(void)
> +{
> + smp_ops->cause_ipi = xive_cause_ipi;
> +
> + /* Register the IPI */
> + xive_request_ipi();
> +
> + /* Allocate and setup IPI for the boot CPU */
> + xive_setup_cpu_ipi(smp_processor_id());
> +}
> +
> +#endif /* CONFIG_SMP */
> +
> +static int xive_irq_domain_map(struct irq_domain *h, unsigned int virq,
> + irq_hw_number_t hw)
> +{
> + int rc;
> +
> + /*
> + * Mark interrupts as edge sensitive by default so that resend
> + * actually works. Will fix that up below if needed.
> + */
> + irq_clear_status_flags(virq, IRQ_LEVEL);
> +
> + /* IPIs are special and come up with HW number 0 */
> + if (hw = 0) {
> + /*
> + * IPIs are marked per-cpu. We use separate HW interrupts under
> + * the hood but associated with the same "linux" interrupt
> + */
> + irq_set_chip_and_handler(virq, &xive_ipi_chip,
> + handle_percpu_irq);
> + return 0;
> + }
> +
> + rc = xive_irq_alloc_data(virq, hw);
> + if (rc)
> + return rc;
> +
> + irq_set_chip_and_handler(virq, &xive_irq_chip, handle_fasteoi_irq);
> +
> + return 0;
> +}
> +
> +static void xive_irq_domain_unmap(struct irq_domain *d, unsigned int virq)
> +{
> + struct irq_data *data = irq_get_irq_data(virq);
> + unsigned int hw_irq;
> +
> + if (!data)
> + return;
> + hw_irq = (unsigned int)irqd_to_hwirq(data);
> + if (hw_irq)
> + xive_irq_free_data(virq);
> +}
> +
> +static int xive_irq_domain_xlate(struct irq_domain *h, struct device_node *ct,
> + const u32 *intspec, unsigned int intsize,
> + irq_hw_number_t *out_hwirq, unsigned int *out_flags)
> +
> +{
> + *out_hwirq = intspec[0];
> +
> + /*
> + * If intsize is at least 2, we look for the type in the second cell,
> + * we assume the LSB indicates a level interrupt.
> + */
> + if (intsize > 1) {
> + if (intspec[1] & 1)
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> + else
> + *out_flags = IRQ_TYPE_EDGE_RISING;
> + } else
> + *out_flags = IRQ_TYPE_LEVEL_LOW;
> +
> + return 0;
> +}
> +
> +static int xive_irq_domain_match(struct irq_domain *h, struct device_node *node,
> + enum irq_domain_bus_token bus_token)
> +{
> + return xive_ops->match(node);
> +}
> +
> +static const struct irq_domain_ops xive_irq_domain_ops = {
> + .match = xive_irq_domain_match,
> + .map = xive_irq_domain_map,
> + .unmap = xive_irq_domain_unmap,
> + .xlate = xive_irq_domain_xlate,
> +};
> +
> +static void __init xive_init_host(void)
> +{
> + xive_irq_domain = irq_domain_add_nomap(NULL, XIVE_MAX_IRQ,
> + &xive_irq_domain_ops, NULL);
> + BUG_ON(xive_irq_domain = NULL);
> + irq_set_default_host(xive_irq_domain);
> +}
> +
> +static void xive_cleanup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + if (xc->queue[xive_irq_priority].qpage)
> + xive_ops->cleanup_queue(cpu, xc, xive_irq_priority);
> +}
> +
> +static int xive_setup_cpu_queues(unsigned int cpu, struct xive_cpu *xc)
> +{
> + int rc = 0;
> +
> + /* We setup 1 queues for now with a 64k page */
> + if (!xc->queue[xive_irq_priority].qpage)
> + rc = xive_ops->setup_queue(cpu, xc, xive_irq_priority);
> +
> + return rc;
> +}
> +
> +static int xive_prepare_cpu(unsigned int cpu)
> +{
> + struct xive_cpu *xc;
> +
> + xc = per_cpu(xive_cpu, cpu);
> + if (!xc) {
> + struct device_node *np;
> +
> + xc = kzalloc_node(sizeof(struct xive_cpu),
> + GFP_KERNEL, cpu_to_node(cpu));
> + if (!xc)
> + return -ENOMEM;
> + np = of_get_cpu_node(cpu, NULL);
> + if (np)
> + xc->chip_id = of_get_ibm_chip_id(np);
> + of_node_put(np);
> +
> + per_cpu(xive_cpu, cpu) = xc;
> + }
> +
> + /* Setup EQs if not already */
> + return xive_setup_cpu_queues(cpu, xc);
> +}
> +
> +static void xive_setup_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> +
> + /* Debug: Dump the TM state */
> + DBG("CPU %d [HW 0x%02x] VT=%02x\n",
> + smp_processor_id(), hard_smp_processor_id(),
> + in_8(xive_tm_area + xive_tm_offset + TM_WORD2));
> +
> + /* The backend might have additional things to do */
> + if (xive_ops->setup_cpu)
> + xive_ops->setup_cpu(smp_processor_id(), xc);
> +
> + /* Set CPPR to 0xff to enable flow of interrupts */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +#ifdef CONFIG_SMP
> +void xive_smp_setup_cpu(void)
> +{
> + DBG("XIVE: SMP setup CPU %d\n", smp_processor_id());
> +
> + /* This will have already been done on the boot CPU */
> + if (smp_processor_id() != boot_cpuid)
> + xive_setup_cpu();
> +
> +}
> +
> +int xive_smp_prepare_cpu(unsigned int cpu)
> +{
> + int rc;
> +
> + /* Allocate per-CPU data and queues */
> + rc = xive_prepare_cpu(cpu);
> + if (rc)
> + return rc;
> +
> + /* Allocate and setup IPI for the new CPU */
> + return xive_setup_cpu_ipi(cpu);
> +}
> +
> +#ifdef CONFIG_HOTPLUG_CPU
> +static void xive_flush_cpu_queue(unsigned int cpu, struct xive_cpu *xc)
> +{
> + u32 irq;
> +
> + /* We assume local irqs are disabled */
> + WARN_ON(!irqs_disabled());
> +
> + /* Check what's already in the CPU queue */
> + while ((irq = xive_scan_interrupts(xc, false)) != 0) {
> + /*
> + * We need to re-route that interrupt to its new distination.
> + * First get and lock the descriptor
> + */
> + struct irq_desc *desc = irq_to_desc(irq);
> + struct irq_data *d = irq_desc_get_irq_data(desc);
> + struct xive_irq_data *xd;
> + unsigned int hw_irq = (unsigned int)irqd_to_hwirq(d);
> +
> + /*
> + * Ignore anything that isn't a XIVE irq and ignore
> + * IPIs, so can just be dropped.
> + */
> + if (d->domain != xive_irq_domain || hw_irq = 0)
> + continue;
> +#ifdef DEBUG_FLUSH
> + pr_info("CPU %d: Got irq %d while offline, re-routing...\n",
> + cpu, irq);
> +#endif
> + raw_spin_lock(&desc->lock);
> + xd = irq_desc_get_handler_data(desc);
> +
> + /* For LSIs, we EOI, this will cause a resend if it's
> + * still asserted. Otherwise do an MSI retrigger
> + */
> + if (xd->flags & XIVE_IRQ_FLAG_LSI)
> + xive_do_source_eoi(irqd_to_hwirq(d), xd);
> + else
> + xive_irq_retrigger(d);
> + raw_spin_unlock(&desc->lock);
> + }
> +}
> +
> +void xive_smp_disable_cpu(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Migrate interrupts away from the CPU */
> + irq_migrate_all_off_this_cpu();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Flush everything still in the queue */
> + xive_flush_cpu_queue(cpu, xc);
> +
> + /* Re-enable CPPR */
> + xc->cppr = 0xff;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0xff);
> +}
> +
> +void xive_flush_interrupt(void)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Called if an interrupt occurs while the CPU is hot unplugged */
> + xive_flush_cpu_queue(cpu, xc);
> +}
> +
> +#endif /* CONFIG_HOTPLUG_CPU */
> +
> +#endif /* CONFIG_SMP */
> +
> +void xive_kexec_teardown_cpu(int secondary)
> +{
> + struct xive_cpu *xc = __this_cpu_read(xive_cpu);
> + unsigned int cpu = smp_processor_id();
> +
> + /* Set CPPR to 0 to disable flow of interrupts */
> + xc->cppr = 0;
> + out_8(xive_tm_area + xive_tm_offset + TM_CPPR, 0);
> +
> + /* Backend cleanup if any */
> + if (xive_ops->teardown_cpu)
> + xive_ops->teardown_cpu(cpu, xc);
> +
> + /* Get rid of IPI */
> + xive_cleanup_cpu_ipi(cpu, xc);
> +
> + /* Disable and free the queues */
> + xive_cleanup_cpu_queues(cpu, xc);
> +}
> +
> +void xive_shutdown(void)
> +{
> + xive_ops->shutdown();
> +}
> +
> +bool xive_core_init(const struct xive_ops *ops, void __iomem *area, u32 offset,
> + u8 max_prio)
> +{
> + xive_tm_area = area;
> + xive_tm_offset = offset;
> + xive_ops = ops;
> + xive_irq_priority = max_prio;
> +
> + ppc_md.get_irq = xive_get_irq;
> + __xive_enabled = true;
> +
> + DBG("Initializing host..\n");
> + xive_init_host();
> +
> + DBG("Initializing boot CPU..\n");
> +
> + /* Allocate per-CPU data and queues */
> + xive_prepare_cpu(smp_processor_id());
> +
> + /* Get ready for interrupts */
> + xive_setup_cpu();
> +
> + pr_info("XIVE: Interrupt handling intialized with %s backend\n",
> + xive_ops->name);
> + pr_info("XIVE: Using priority %d for all interrupts\n", max_prio);
> +
> + return true;
> +}
> +
> +static int __init xive_off(char *arg)
> +{
> + xive_cmdline_disabled = true;
> + return 0;
> +}
> +__setup("xive=off", xive_off);
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> new file mode 100644
> index 0000000..26cc6bf
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -0,0 +1,604 @@
> +/*
> + * Copyright 2016,2017 IBM Corporation.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License
> + * as published by the Free Software Foundation; either version
> + * 2 of the License, or (at your option) any later version.
> + */
> +#include <linux/types.h>
> +#include <linux/irq.h>
> +#include <linux/debugfs.h>
Unused?
> +#include <linux/smp.h>
> +#include <linux/interrupt.h>
> +#include <linux/seq_file.h>
Unused?
> +#include <linux/init.h>
> +#include <linux/of.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/cpumask.h>
> +#include <linux/mm.h>
> +
> +#include <asm/prom.h>
> +#include <asm/io.h>
> +#include <asm/smp.h>
> +#include <asm/irq.h>
> +#include <asm/errno.h>
> +#include <asm/xive.h>
> +#include <asm/opal.h>
> +
> +#include "xive-regs.h"
> +#include "xive-internal.h"
> +
> +#define DBG(fmt...) pr_devel("XIVE: " fmt)
> +
> +/* Enable this for using queue MMIO page for EOI. We don't currently
> + * use it as we always notify
> + */
> +#undef USE_QUEUE_MMIO
Dead code? Or we want to keep it?
> +static u32 xive_provision_size;
> +static u32 *xive_provision_chips;
> +static u32 xive_provision_chip_count;
> +static u32 xive_queue_shift;
> +static u32 xive_pool_vps = XIVE_INVALID_VP;
> +static struct kmem_cache *xive_provision_cache;
> +
> +int xive_native_populate_irq_data(u32 hw_irq, struct xive_irq_data *data)
> +{
> + __be64 flags, eoi_page, trig_page;
> + __be32 esb_shift, src_chip;
> + u64 opal_flags;
> + s64 rc;
> +
> + memset(data, 0, sizeof(*data));
> +
> + rc = opal_xive_get_irq_info(hw_irq, &flags, &eoi_page, &trig_page,
> + &esb_shift, &src_chip);
> + if (rc) {
> + pr_err("XIVE: opal_xive_get_irq_info(0x%x) returned %lld\n",
> + hw_irq, rc);
> + return -EINVAL;
> + }
> +
> + opal_flags = be64_to_cpu(flags);
> + if (opal_flags & OPAL_XIVE_IRQ_STORE_EOI)
> + data->flags |= XIVE_IRQ_FLAG_STORE_EOI;
> + if (opal_flags & OPAL_XIVE_IRQ_LSI)
> + data->flags |= XIVE_IRQ_FLAG_LSI;
> + if (opal_flags & OPAL_XIVE_IRQ_SHIFT_BUG)
> + data->flags |= XIVE_IRQ_FLAG_SHIFT_BUG;
> + if (opal_flags & OPAL_XIVE_IRQ_MASK_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_MASK_FW;
> + if (opal_flags & OPAL_XIVE_IRQ_EOI_VIA_FW)
> + data->flags |= XIVE_IRQ_FLAG_EOI_FW;
> + data->eoi_page = be64_to_cpu(eoi_page);
> + data->trig_page = be64_to_cpu(trig_page);
> + data->esb_shift = be32_to_cpu(esb_shift);
> + data->src_chip = be32_to_cpu(src_chip);
> +
> + data->eoi_mmio = ioremap(data->eoi_page, 1u << data->esb_shift);
> + if (!data->eoi_mmio) {
> + pr_err("XIVE: Failed to map EOI page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> +
> + if (!data->trig_page)
> + return 0;
> + if (data->trig_page = data->eoi_page) {
> + data->trig_mmio = data->eoi_mmio;
> + return 0;
> + }
> +
> + data->trig_mmio = ioremap(data->trig_page, 1u << data->esb_shift);
> + if (!data->trig_mmio) {
> + pr_err("XIVE: Failed to map trigger page for irq 0x%x\n", hw_irq);
> + return -ENOMEM;
> + }
> + return 0;
> +}
> +
> +int xive_native_configure_irq(u32 hw_irq, u32 target, u8 prio, u32 sw_irq)
> +{
> + s64 rc;
> +
> + for (;;) {
> + rc = opal_xive_set_irq_config(hw_irq, target, prio, sw_irq);
> + if (rc != OPAL_BUSY)
> + break;
> + msleep(1);
> + }
> + return rc = 0 ? 0 : -ENXIO;
> +}
> +
> +/* This can be called multiple time to change a queue configuration */
> +int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> + __be32 *qpage, u32 order, bool can_escalate)
> +{
> + s64 rc = 0;
> + __be64 qeoi_page_be;
> + __be32 esc_irq_be;
> + u64 flags, qpage_phys;
> +
> + /* If there's an actual queue page, clean it */
> + if (order) {
> + BUG_ON(!qpage);
Can't we just return an error?
> + qpage_phys = __pa(qpage);
> + } else
> + qpage_phys = 0;
> +
> + /* Initialize the rest of the fields */
> + q->msk = order ? ((1u << (order - 2)) - 1) : 0;
> + q->idx = 0;
> + q->toggle = 0;
> +
> + rc = opal_xive_get_queue_info(vp_id, prio, NULL, NULL,
> + &qeoi_page_be,
> + &esc_irq_be,
> + NULL);
> + if (rc) {
> + pr_err("XIVE: Error %lld getting queue info prio %d\n",
> + rc, prio);
> + rc = -EIO;
> + goto fail;
> + }
> + q->eoi_phys = be64_to_cpu(qeoi_page_be);
> +
> +#ifdef USE_QUEUE_MMIO
> + if (!q->eoi_mmio)
> + q->eoi_mmio = ioremap(q->eoi_phys, PAGE_SIZE);
> + if (!q->eoi_mmio) {
> + pr_err("XIVE: Failed to map queue MMIO prio %d CPU %d\n",
> + rc, prio, cpu);
> + rc = -ENOMEM;
> + goto fail;
> + }
> +#endif /* USE_QUEUE_MMIO */
> +
> +
...
> +static bool xive_parse_provisioning(struct device_node *np)
> +{
> + int rc;
> +
> + if (of_property_read_u32(np, "ibm,xive-provision-page-size",
> + &xive_provision_size) < 0)
> + return true;
> + rc = of_property_count_elems_of_size(np, "ibm,xive-provision-chips", 4);
> + if (rc < 0) {
> + pr_err("XIVE: Error %d getting provision chips array\n", rc);
> + return false;
> + }
> + xive_provision_chip_count = rc;
> + if (rc = 0)
> + return true;
> +
> + xive_provision_chips = kzalloc(4 * xive_provision_chip_count,
> + GFP_KERNEL);
> + BUG_ON(!xive_provision_chips);
return false?
> +
> + rc = of_property_read_u32_array(np, "ibm,xive-provision-chips",
> + xive_provision_chips,
> + xive_provision_chip_count);
...
> diff --git a/arch/powerpc/sysdev/xive/xive-internal.h b/arch/powerpc/sysdev/xive/xive-internal.h
> new file mode 100644
> index 0000000..e736fc5
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-internal.h
> @@ -0,0 +1,51 @@
Copyright missing.
> +#ifndef __XIVE_INTERNAL_H
> +#define __XIVE_INTERNAL_H
...
> diff --git a/arch/powerpc/sysdev/xive/xive-regs.h b/arch/powerpc/sysdev/xive/xive-regs.h
> new file mode 100644
> index 0000000..f1edb23
> --- /dev/null
> +++ b/arch/powerpc/sysdev/xive/xive-regs.h
> @@ -0,0 +1,88 @@
Copyright missing.
> +#ifndef __XIVE_REGS_H__
> +#define __XIVE_REGS_H__
...
> diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> index 16321ad..c71e919 100644
> --- a/arch/powerpc/xmon/xmon.c
> +++ b/arch/powerpc/xmon/xmon.c
...
> +
> +static void dump_one_xive_irq(uint32_t num)
u32?
> +{
> + int64_t rc;
> + __be64 vp;
> + uint8_t prio;
u8?
zzzzz ...
cheers
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 02/12] powerpc: Sync opal-api.h
2017-04-04 12:20 ` Michael Ellerman
@ 2017-04-04 13:47 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-04-04 13:47 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev, kvm-ppc
On Tue, 2017-04-04 at 22:20 +1000, Michael Ellerman wrote:
> Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
>
> ...
>
> Give me some change log !
Well, the subject says it all :-) Sync the API with the latest OPAL :-)
> > Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > ---
> > arch/powerpc/include/asm/opal-api.h | 302
> > ++++++++++++++++++++-----
>
> It looks like you've just copied it over in its entirety, including
> lots of unused cruft.
>
> Please just give me the XIVE bits you need.
Why ? It's a lot easier in the long run to have the file actually in
sync between the two projects no ?
Cheers,
Ben.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 02/12] powerpc: Sync opal-api.h
@ 2017-04-04 13:47 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-04-04 13:47 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev, kvm-ppc
On Tue, 2017-04-04 at 22:20 +1000, Michael Ellerman wrote:
> Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
>
> ...
>
> Give me some change log !
Well, the subject says it all :-) Sync the API with the latest OPAL :-)
> > Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
> > ---
> > arch/powerpc/include/asm/opal-api.h | 302
> > ++++++++++++++++++++-----
>
> It looks like you've just copied it over in its entirety, including
> lots of unused cruft.
>
> Please just give me the XIVE bits you need.
Why ? It's a lot easier in the long run to have the file actually in
sync between the two projects no ?
Cheers,
Ben.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
2017-04-04 13:03 ` Michael Ellerman
@ 2017-04-04 14:12 ` Benjamin Herrenschmidt
-1 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-04-04 14:12 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev, kvm-ppc
On Tue, 2017-04-04 at 23:03 +1000, Michael Ellerman wrote:
>
> > 14 files changed, 2186 insertions(+), 12 deletions(-)
>
> I'm not going to review this in one go, given it's 10:30pm already.
Well, good, I was about to send (well tomorrow morning actually) v2
hoping it was going to be final since nobody else hard reviewed it :-)
> +extern void __iomem *xive_tm_area;
>
> I think Paul already commented on "tm" being an overly used acronym.
He asked me to spell it out in a comment which I did in v2. I haven't
changed the name of the variable though which percolates through the
KVM bits etc... I could rename it (painfully) to use "tma" instead
(Thread Management Area).
> > +extern u32 xive_tm_offset;
> > +
> > +/*
> > + * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
> > + * have it stored in the xive_cpu structure. We also cache
> > + * for normal interrupts the current target CPU.
> > + */
> > +struct xive_irq_data {
> > + /* Setup by backend */
> > + u64 flags;
> > +#define XIVE_IRQ_FLAG_STORE_EOI 0x01
> > +#define XIVE_IRQ_FLAG_LSI 0x02
> > +#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
> > +#define XIVE_IRQ_FLAG_MASK_FW 0x08
> > +#define XIVE_IRQ_FLAG_EOI_FW 0x10
>
> I don't love that style, prefer them just prior to the struct.
I much prefer having the definitions next to the variable they apply
to but if you feel strongly about it, I will move them.
> > + u64 eoi_page;
> > + void __iomem *eoi_mmio;
> > + u64 trig_page;
> > + void __iomem *trig_mmio;
> > + u32 esb_shift;
> > + int src_chip;
>
> Why not space out the members like you do in xive_q below, I think
> that looks better given you have the long __iomem lines.
Ok.
> > +
> > + /* Setup/used by frontend */
> > + int target;
> > + bool saved_p;
> > +};
> > +#define XIVE_INVALID_CHIP_ID -1
> > +
> > +/* A queue tracking structure in a CPU */
> > +struct xive_q {
> > + __be32 *qpage;
> > + u32 msk;
> > + u32 idx;
> > + u32 toggle;
> > + u64 eoi_phys;
> > + void __iomem *eoi_mmio;
> > + u32 esc_irq;
> > + atomic_t count;
> > + atomic_t pending_count;
> > +};
> > +
> > +/*
> > + * "magic" ESB MMIO offsets
>
> What's an ESB?
Well, the problem here is that if I start answering that one along with
a chunk of the rest of your questions, I basically end up writing a
summary of the XIVE specification in comments, which would probably
take 2 or 3 pages ;-)
I don't know where to start there or rather how far to go. I could
spell out the acronyms but it's not necessarily that useful.
Another problem with XIVE is that everything has 2 names ! The original
design came with (rather sane) names but the "architects" later on
renamed everything into weird stuff. For example, the HW name of an
event queue descriptor is "EQD". The "architecture" name is "END"
(Event Notification Descriptor I *think*).
Sadly bcs we have docs mix & matching both, I ended up accidentally
making a bit of a mess myself though I've generally favored the HW
names (EQ vs. END, VP (Virtual Processor) vs. NVT (Notification Virtual
Target), etc...
> If here you put:
>
> #define pr_fmt(fmt) "xive: " fmt
>
> Then you can drop the prefix from every pr_xxx() in the whole file.
Yup. I live in the past obviously :-)
> > +/*
> > + * A "disabled" interrupt should never fire, to catch problems
> > + * we set its logical number to this
> > + */
> > +#define XIVE_BAD_IRQ 0x7fffffff
>
> Can it be anything? How about 0x7fbadbad ?
It can be anything as long as we never assign that number to an
interrupt. So we have to limit the IRQ numbers to that value. Talking
of which I need to make sure I enforce the limitation on the numbers
today.
> > +#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
> > +
> > +/* An invalid CPU target */
> > +#define XIVE_INVALID_TARGET (-1)
> > +
> > +static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
>
> Can it have a doc comment? And tell me what an EQ is?
I added a description in v2.
> > +{
> > + u32 cur;
> > +
> > + if (!q->qpage)
> > + return 0;
>
> A newline or ..
>
> > + cur = be32_to_cpup(q->qpage + q->idx);
> > + if ((cur >> 31) == q->toggle)
> > + return 0;
>
> .. two wouldn't hurt here.
>
> > + if (!just_peek) {
> > + q->idx = (q->idx + 1) & q->msk;
> > + if (q->idx == 0)
> > + q->toggle ^= 1;
> > + }
> > + return cur & 0x7fffffff;
>
> Is that XIVE_BAD_IRQ ?
No. This is a mask. The top bit is the toggle valid bit, we mask it out
on the way back. Will add a comment.
> > +}
> > +
> > +static u32 xive_scan_interrupts(struct xive_cpu *xc, bool
> > just_peek)
> > +{
> > + u32 hirq = 0;
>
> Is that a hwirq or something different?
not sure why I called it hirq ... it's what comes out of the queue
which is a linux irq number since we reconfigure the hw to give those
to us directly.
> > + u8 prio;
> > +
> > + /* Find highest pending priority */
> > + while (xc->pending_prio != 0) {
> > + struct xive_q *q;
> > +
> > + prio = ffs(xc->pending_prio) - 1;
> > + DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
> > +
> > + /* Try to fetch */
> > + hirq = xive_read_eq(&xc->queue[prio], prio,
> > just_peek);
> > +
> > + /* Found something ? That's it */
> > + if (hirq)
> > + break;
> > +
> > + /* Clear pending bits */
> > + xc->pending_prio &= ~(1 << prio);
> > +
> > + /*
> > + * Check if the queue count needs adjusting due to
> > + * interrupts being moved away.
> > + */
> > + q = &xc->queue[prio];
> > + if (atomic_read(&q->pending_count)) {
> > + int p = atomic_xchg(&q->pending_count, 0);
> > + if (p) {
> > + WARN_ON(p > atomic_read(&q-
> > >count));
> > + atomic_sub(p, &q->count);
>
> I am not sure what's going on there.
Black magic :-) It's documented in a comment elsewhere iirc. I could
try to add a reference to it in the comment above.
> > + }
> > + }
> > + }
> > +
> > + /* If nothing was found, set CPPR to 0xff */
>
> Would be nice to spell out CPPR somewhere.
We never did on XICS :-) Means the same thing. But yeah I can spell it
out in the first use.
> > + if (hirq == 0)
> > + prio = 0xff;
> > +
> > + /* Update HW CPPR to match if necessary */
> > + if (prio != xc->cppr) {
> > + DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n",
> > prio);
> > + xc->cppr = prio;
> > + out_8(xive_tm_area + xive_tm_offset + TM_CPPR,
> > prio);
>
> What's the out_8() doing? I was expecting it to use xc, or something
> per-cpu.
The HW makes it magically per-cpu :-) That's the whole point of the
TMA. The powerbus knows where the accesses come from and will route you
to the right "instance" magically.
It's especially important in guests because it means that you don't
have to create some kind of mapping that has to change as the guest
gets reschedule on a different HW CPU.
The guest just accesses the TMA at its fixed address and the HW sorts
it out (it knows what guest is running on what physical CPU as KVM
tells it when the guest gets context switched in).
> > + }
> > +
> > + return hirq;
> > +}
> > +
> > +#ifdef CONFIG_XMON
> > +static void xive_dump_eq(const char *name, struct xive_q *q)
> > +{
> > + u32 i0, i1, idx;
> > +
> > + if (!q->qpage)
> > + return;
> > + idx = q->idx;
> > + i0 = be32_to_cpup(q->qpage + idx);
> > + idx = (idx + 1) & q->msk;
> > + i1 = be32_to_cpup(q->qpage + idx);
> > + xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
> > + q->toggle, i0, i1);
> > +}
> > +
> > +void xmon_xive_do_dump(int cpu)
> > +{
> > + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> > + struct xive_irq_data *xd;
> > + uint64_t val, offset;
>
> u64 ?
Yeah yeah ... :)
> > +static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data
> > *xd)
> > +{
> > + /* If the XIVE supports the new "store EOI facility, use
> > it */
> > + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> > + out_be64(xd->eoi_mmio, 0);
> > + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> > + if (WARN_ON_ONCE(!xive_ops->eoi))
> > + return;
> > + xive_ops->eoi(hw_irq);
> > + } else {
> > + uint8_t eoi_val;
>
> u8?
Do we actually care ? :-)
> > +/* Enable this for using queue MMIO page for EOI. We don't
> > currently
> > + * use it as we always notify
> > + */
> > +#undef USE_QUEUE_MMIO
>
> Dead code? Or we want to keep it?
We might want to run some tests with it at some point, though I haven't
tested the code so it probably doesn't work... I'll probably remove it
for now.
> > +/* This can be called multiple time to change a queue
> > configuration */
> > +int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8
> > prio,
> > + __be32 *qpage, u32 order, bool
> > can_escalate)
> > +{
> > + s64 rc = 0;
> > + __be64 qeoi_page_be;
> > + __be32 esc_irq_be;
> > + u64 flags, qpage_phys;
> > +
> > + /* If there's an actual queue page, clean it */
> > + if (order) {
> > + BUG_ON(!qpage);
>
> Can't we just return an error?
Maybe but this should really really never happen, and if it does a
backtrace is welcome. Maybe I can use a WARN_ON instead and return an
error.
> > +
> > + xive_provision_chips = kzalloc(4 *
> > xive_provision_chip_count,
> > + GFP_KERNEL);
> > + BUG_ON(!xive_provision_chips);
>
> return false?
We are pretty stuffed if that happens. Well, ok, KVM is pretty stuffed,
the host can probably survive, but hell, if we can't allocate a few
bytes at boot time I think we have bigger worries :-)
> > +
> > + rc = of_property_read_u32_array(np, "ibm,xive-provision-
> > chips",
> > + xive_provision_chips,
> > + xive_provision_chip_count)
> > ;
>
> ...
> > diff --git a/arch/powerpc/sysdev/xive/xive-internal.h
> > b/arch/powerpc/sysdev/xive/xive-internal.h
> > new file mode 100644
> > index 0000000..e736fc5
> > --- /dev/null
> > +++ b/arch/powerpc/sysdev/xive/xive-internal.h
> > @@ -0,0 +1,51 @@
>
> Copyright missing.
>
> > +#ifndef __XIVE_INTERNAL_H
> > +#define __XIVE_INTERNAL_H
>
> ...
> > diff --git a/arch/powerpc/sysdev/xive/xive-regs.h
> > b/arch/powerpc/sysdev/xive/xive-regs.h
> > new file mode 100644
> > index 0000000..f1edb23
> > --- /dev/null
> > +++ b/arch/powerpc/sysdev/xive/xive-regs.h
> > @@ -0,0 +1,88 @@
>
> Copyright missing.
>
> > +#ifndef __XIVE_REGS_H__
> > +#define __XIVE_REGS_H__
>
> ...
> > diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> > index 16321ad..c71e919 100644
> > --- a/arch/powerpc/xmon/xmon.c
> > +++ b/arch/powerpc/xmon/xmon.c
>
> ...
> > +
> > +static void dump_one_xive_irq(uint32_t num)
>
> u32?
>
> > +{
> > + int64_t rc;
> > + __be64 vp;
> > + uint8_t prio;
>
> u8?
>
>
> zzzzz ...
rrrrzzzz...
Ben.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller
@ 2017-04-04 14:12 ` Benjamin Herrenschmidt
0 siblings, 0 replies; 38+ messages in thread
From: Benjamin Herrenschmidt @ 2017-04-04 14:12 UTC (permalink / raw)
To: Michael Ellerman, linuxppc-dev, kvm-ppc
On Tue, 2017-04-04 at 23:03 +1000, Michael Ellerman wrote:
>
> > 14 files changed, 2186 insertions(+), 12 deletions(-)
>
> I'm not going to review this in one go, given it's 10:30pm already.
Well, good, I was about to send (well tomorrow morning actually) v2
hoping it was going to be final since nobody else hard reviewed it :-)
> +extern void __iomem *xive_tm_area;
>
> I think Paul already commented on "tm" being an overly used acronym.
He asked me to spell it out in a comment which I did in v2. I haven't
changed the name of the variable though which percolates through the
KVM bits etc... I could rename it (painfully) to use "tma" instead
(Thread Management Area).
> > +extern u32 xive_tm_offset;
> > +
> > +/*
> > + * Per-irq data (irq_get_handler_data for normal IRQs), IPIs
> > + * have it stored in the xive_cpu structure. We also cache
> > + * for normal interrupts the current target CPU.
> > + */
> > +struct xive_irq_data {
> > + /* Setup by backend */
> > + u64 flags;
> > +#define XIVE_IRQ_FLAG_STORE_EOI 0x01
> > +#define XIVE_IRQ_FLAG_LSI 0x02
> > +#define XIVE_IRQ_FLAG_SHIFT_BUG 0x04
> > +#define XIVE_IRQ_FLAG_MASK_FW 0x08
> > +#define XIVE_IRQ_FLAG_EOI_FW 0x10
>
> I don't love that style, prefer them just prior to the struct.
I much prefer having the definitions next to the variable they apply
to but if you feel strongly about it, I will move them.
> > + u64 eoi_page;
> > + void __iomem *eoi_mmio;
> > + u64 trig_page;
> > + void __iomem *trig_mmio;
> > + u32 esb_shift;
> > + int src_chip;
>
> Why not space out the members like you do in xive_q below, I think
> that looks better given you have the long __iomem lines.
Ok.
> > +
> > + /* Setup/used by frontend */
> > + int target;
> > + bool saved_p;
> > +};
> > +#define XIVE_INVALID_CHIP_ID -1
> > +
> > +/* A queue tracking structure in a CPU */
> > +struct xive_q {
> > + __be32 *qpage;
> > + u32 msk;
> > + u32 idx;
> > + u32 toggle;
> > + u64 eoi_phys;
> > + void __iomem *eoi_mmio;
> > + u32 esc_irq;
> > + atomic_t count;
> > + atomic_t pending_count;
> > +};
> > +
> > +/*
> > + * "magic" ESB MMIO offsets
>
> What's an ESB?
Well, the problem here is that if I start answering that one along with
a chunk of the rest of your questions, I basically end up writing a
summary of the XIVE specification in comments, which would probably
take 2 or 3 pages ;-)
I don't know where to start there or rather how far to go. I could
spell out the acronyms but it's not necessarily that useful.
Another problem with XIVE is that everything has 2 names ! The original
design came with (rather sane) names but the "architects" later on
renamed everything into weird stuff. For example, the HW name of an
event queue descriptor is "EQD". The "architecture" name is "END"
(Event Notification Descriptor I *think*).
Sadly bcs we have docs mix & matching both, I ended up accidentally
making a bit of a mess myself though I've generally favored the HW
names (EQ vs. END, VP (Virtual Processor) vs. NVT (Notification Virtual
Target), etc...
> If here you put:
>
> #define pr_fmt(fmt) "xive: " fmt
>
> Then you can drop the prefix from every pr_xxx() in the whole file.
Yup. I live in the past obviously :-)
> > +/*
> > + * A "disabled" interrupt should never fire, to catch problems
> > + * we set its logical number to this
> > + */
> > +#define XIVE_BAD_IRQ 0x7fffffff
>
> Can it be anything? How about 0x7fbadbad ?
It can be anything as long as we never assign that number to an
interrupt. So we have to limit the IRQ numbers to that value. Talking
of which I need to make sure I enforce the limitation on the numbers
today.
> > +#define XIVE_MAX_IRQ (XIVE_BAD_IRQ - 1)
> > +
> > +/* An invalid CPU target */
> > +#define XIVE_INVALID_TARGET (-1)
> > +
> > +static u32 xive_read_eq(struct xive_q *q, u8 prio, bool just_peek)
>
> Can it have a doc comment? And tell me what an EQ is?
I added a description in v2.
> > +{
> > + u32 cur;
> > +
> > + if (!q->qpage)
> > + return 0;
>
> A newline or ..
>
> > + cur = be32_to_cpup(q->qpage + q->idx);
> > + if ((cur >> 31) = q->toggle)
> > + return 0;
>
> .. two wouldn't hurt here.
>
> > + if (!just_peek) {
> > + q->idx = (q->idx + 1) & q->msk;
> > + if (q->idx = 0)
> > + q->toggle ^= 1;
> > + }
> > + return cur & 0x7fffffff;
>
> Is that XIVE_BAD_IRQ ?
No. This is a mask. The top bit is the toggle valid bit, we mask it out
on the way back. Will add a comment.
> > +}
> > +
> > +static u32 xive_scan_interrupts(struct xive_cpu *xc, bool
> > just_peek)
> > +{
> > + u32 hirq = 0;
>
> Is that a hwirq or something different?
not sure why I called it hirq ... it's what comes out of the queue
which is a linux irq number since we reconfigure the hw to give those
to us directly.
> > + u8 prio;
> > +
> > + /* Find highest pending priority */
> > + while (xc->pending_prio != 0) {
> > + struct xive_q *q;
> > +
> > + prio = ffs(xc->pending_prio) - 1;
> > + DBG_VERBOSE("scan_irq: trying prio %d\n", prio);
> > +
> > + /* Try to fetch */
> > + hirq = xive_read_eq(&xc->queue[prio], prio,
> > just_peek);
> > +
> > + /* Found something ? That's it */
> > + if (hirq)
> > + break;
> > +
> > + /* Clear pending bits */
> > + xc->pending_prio &= ~(1 << prio);
> > +
> > + /*
> > + * Check if the queue count needs adjusting due to
> > + * interrupts being moved away.
> > + */
> > + q = &xc->queue[prio];
> > + if (atomic_read(&q->pending_count)) {
> > + int p = atomic_xchg(&q->pending_count, 0);
> > + if (p) {
> > + WARN_ON(p > atomic_read(&q-
> > >count));
> > + atomic_sub(p, &q->count);
>
> I am not sure what's going on there.
Black magic :-) It's documented in a comment elsewhere iirc. I could
try to add a reference to it in the comment above.
> > + }
> > + }
> > + }
> > +
> > + /* If nothing was found, set CPPR to 0xff */
>
> Would be nice to spell out CPPR somewhere.
We never did on XICS :-) Means the same thing. But yeah I can spell it
out in the first use.
> > + if (hirq = 0)
> > + prio = 0xff;
> > +
> > + /* Update HW CPPR to match if necessary */
> > + if (prio != xc->cppr) {
> > + DBG_VERBOSE("scan_irq: adjusting CPPR to %d\n",
> > prio);
> > + xc->cppr = prio;
> > + out_8(xive_tm_area + xive_tm_offset + TM_CPPR,
> > prio);
>
> What's the out_8() doing? I was expecting it to use xc, or something
> per-cpu.
The HW makes it magically per-cpu :-) That's the whole point of the
TMA. The powerbus knows where the accesses come from and will route you
to the right "instance" magically.
It's especially important in guests because it means that you don't
have to create some kind of mapping that has to change as the guest
gets reschedule on a different HW CPU.
The guest just accesses the TMA at its fixed address and the HW sorts
it out (it knows what guest is running on what physical CPU as KVM
tells it when the guest gets context switched in).
> > + }
> > +
> > + return hirq;
> > +}
> > +
> > +#ifdef CONFIG_XMON
> > +static void xive_dump_eq(const char *name, struct xive_q *q)
> > +{
> > + u32 i0, i1, idx;
> > +
> > + if (!q->qpage)
> > + return;
> > + idx = q->idx;
> > + i0 = be32_to_cpup(q->qpage + idx);
> > + idx = (idx + 1) & q->msk;
> > + i1 = be32_to_cpup(q->qpage + idx);
> > + xmon_printf(" %s Q T=%d %08x %08x ...\n", name,
> > + q->toggle, i0, i1);
> > +}
> > +
> > +void xmon_xive_do_dump(int cpu)
> > +{
> > + struct xive_cpu *xc = per_cpu(xive_cpu, cpu);
> > + struct xive_irq_data *xd;
> > + uint64_t val, offset;
>
> u64 ?
Yeah yeah ... :)
> > +static void xive_do_source_eoi(u32 hw_irq, struct xive_irq_data
> > *xd)
> > +{
> > + /* If the XIVE supports the new "store EOI facility, use
> > it */
> > + if (xd->flags & XIVE_IRQ_FLAG_STORE_EOI)
> > + out_be64(xd->eoi_mmio, 0);
> > + else if (hw_irq && xd->flags & XIVE_IRQ_FLAG_EOI_FW) {
> > + if (WARN_ON_ONCE(!xive_ops->eoi))
> > + return;
> > + xive_ops->eoi(hw_irq);
> > + } else {
> > + uint8_t eoi_val;
>
> u8?
Do we actually care ? :-)
> > +/* Enable this for using queue MMIO page for EOI. We don't
> > currently
> > + * use it as we always notify
> > + */
> > +#undef USE_QUEUE_MMIO
>
> Dead code? Or we want to keep it?
We might want to run some tests with it at some point, though I haven't
tested the code so it probably doesn't work... I'll probably remove it
for now.
> > +/* This can be called multiple time to change a queue
> > configuration */
> > +int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8
> > prio,
> > + __be32 *qpage, u32 order, bool
> > can_escalate)
> > +{
> > + s64 rc = 0;
> > + __be64 qeoi_page_be;
> > + __be32 esc_irq_be;
> > + u64 flags, qpage_phys;
> > +
> > + /* If there's an actual queue page, clean it */
> > + if (order) {
> > + BUG_ON(!qpage);
>
> Can't we just return an error?
Maybe but this should really really never happen, and if it does a
backtrace is welcome. Maybe I can use a WARN_ON instead and return an
error.
> > +
> > + xive_provision_chips = kzalloc(4 *
> > xive_provision_chip_count,
> > + GFP_KERNEL);
> > + BUG_ON(!xive_provision_chips);
>
> return false?
We are pretty stuffed if that happens. Well, ok, KVM is pretty stuffed,
the host can probably survive, but hell, if we can't allocate a few
bytes at boot time I think we have bigger worries :-)
> > +
> > + rc = of_property_read_u32_array(np, "ibm,xive-provision-
> > chips",
> > + xive_provision_chips,
> > + xive_provision_chip_count)
> > ;
>
> ...
> > diff --git a/arch/powerpc/sysdev/xive/xive-internal.h
> > b/arch/powerpc/sysdev/xive/xive-internal.h
> > new file mode 100644
> > index 0000000..e736fc5
> > --- /dev/null
> > +++ b/arch/powerpc/sysdev/xive/xive-internal.h
> > @@ -0,0 +1,51 @@
>
> Copyright missing.
>
> > +#ifndef __XIVE_INTERNAL_H
> > +#define __XIVE_INTERNAL_H
>
> ...
> > diff --git a/arch/powerpc/sysdev/xive/xive-regs.h
> > b/arch/powerpc/sysdev/xive/xive-regs.h
> > new file mode 100644
> > index 0000000..f1edb23
> > --- /dev/null
> > +++ b/arch/powerpc/sysdev/xive/xive-regs.h
> > @@ -0,0 +1,88 @@
>
> Copyright missing.
>
> > +#ifndef __XIVE_REGS_H__
> > +#define __XIVE_REGS_H__
>
> ...
> > diff --git a/arch/powerpc/xmon/xmon.c b/arch/powerpc/xmon/xmon.c
> > index 16321ad..c71e919 100644
> > --- a/arch/powerpc/xmon/xmon.c
> > +++ b/arch/powerpc/xmon/xmon.c
>
> ...
> > +
> > +static void dump_one_xive_irq(uint32_t num)
>
> u32?
>
> > +{
> > + int64_t rc;
> > + __be64 vp;
> > + uint8_t prio;
>
> u8?
>
>
> zzzzz ...
rrrrzzzz...
Ben.
^ permalink raw reply [flat|nested] 38+ messages in thread
end of thread, other threads:[~2017-04-04 14:12 UTC | newest]
Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-20 6:49 [PATCH 01/12] powerpc: Disable HFSCR:TM if TM not supported Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 02/12] powerpc: Sync opal-api.h Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-04-04 12:20 ` Michael Ellerman
2017-04-04 12:20 ` Michael Ellerman
2017-04-04 13:47 ` Benjamin Herrenschmidt
2017-04-04 13:47 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 03/12] powerpc: Add more PPC bit conversion macros Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 04/12] powerpc: Add optional smp_ops->prepare_cpu SMP callback Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 05/12] powerpc/smp: Remove migrate_irq() custom implementation Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 06/12] powerpc/xive: Native exploitation of the XIVE interrupt controller Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-24 5:22 ` Paul Mackerras
2017-03-24 5:22 ` Paul Mackerras
2017-04-04 13:03 ` Michael Ellerman
2017-04-04 13:03 ` Michael Ellerman
2017-04-04 14:12 ` Benjamin Herrenschmidt
2017-04-04 14:12 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 07/12] powerpc/kvm: Massage order of #include Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 08/12] powerpc/kvm: Make kvmppc_xics_create_icp static Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 09/12] powerpc/kvm: Remove obsolete kvm_vm_ioctl_xics_irq declaration Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 10/12] powerpc: Consolidate variants of real-mode MMIOs Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 11/12] powerpc: Fixup LPCR:PECE and HEIC setting on POWER9 Benjamin Herrenschmidt
2017-03-20 6:49 ` Benjamin Herrenschmidt
2017-03-20 6:49 ` [PATCH 12/12] powerpc/kvm: Native usage of the XIVE interrupt controller Benjamin Herrenschmidt
2017-03-28 5:26 ` Paul Mackerras
2017-04-03 2:25 ` Benjamin Herrenschmidt
2017-04-03 2:25 ` Benjamin Herrenschmidt
2017-03-31 12:35 ` [01/12] powerpc: Disable HFSCR:TM if TM not supported Michael Ellerman
2017-03-31 12:35 ` Michael Ellerman
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.