* [PATCH v2 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Hello,
On the POWER9 processor, the XIVE interrupt controller can control
interrupt sources using MMIOs to trigger events, to EOI or to turn off
the sources. Priority management and interrupt acknowledgment is also
controlled by MMIO in the CPU presenter subengine.
PowerNV/baremetal Linux runs natively under XIVE but sPAPR guests need
special support from the hypervisor to do the same. This is called the
XIVE native exploitation mode and today, it can be activated under the
PowerPC Hypervisor, pHyp. However, Linux/KVM lacks XIVE native support
and still offers the old interrupt mode interface using a KVM device
implementing the XICS hcalls over XIVE.
The following series is proposal to add the same support under KVM.
A new KVM device is introduced for the XIVE native exploitation
mode. It reuses most of the XICS-over-XIVE glue implementation
structures which are internal to KVM but has a completely different
interface. A set of KVM device ioctls provide support for the
hypervisor calls, all handled in QEMU, to configure the sources and
the event queues. From there, all interrupt control is transferred to
the guest which can use MMIOs.
These MMIO regions (ESB and TIMA) are exposed to guests in QEMU,
similarly to VFIO, and the associated VMAs are populated dynamically
with the appropriate pages using a fault handler. These are now
implemented using mmap()s of the KVM device fd.
Migration has its own specific needs regarding memory. The patchset
provides a specific control to quiesce XIVE before capturing the
memory. The save and restore of the internal state is based on the
same ioctls used for the hcalls.
On a POWER9 sPAPR machine, the Client Architecture Support (CAS)
negotiation process determines whether the guest operates with a
interrupt controller using the XICS legacy model, as found on POWER8,
or in XIVE exploitation mode. Which means that the KVM interrupt
device should be created at runtime, after the machine has started.
This requires extra support from KVM to destroy KVM devices. It is
introduced at the end of the patcshet as it still requires some
attention and a XIVE-only VM would not need.
This is 5.2 material. I expect a couple of respin for fixes, and the
OPAL patches have not yet been merged.
GitHub trees available here :
QEMU sPAPR:
https://github.com/legoater/qemu/commits/xive-next
Linux/KVM:
https://github.com/legoater/linux/commits/xive-5.0
OPAL:
https://github.com/legoater/skiboot/commits/xive
Thanks,
C.
Changes since v1:
- Better documentation (was missing)
- Nested support. XIVE not advertised on non PowerNV platforms. This
is a good way to test the fallback on QEMU emulated devices.
- ESB and TIMA special mapping done using the KVM device fd
- All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
- Reworked of the KVM device ioctl controls to support hcalls and
migration needs to capture/save states
- Merged the control syncing XIVE and marking the EQ page dirty
- Fixed passthrough support using the KVM device file address_space
to clear the ESB pages from the mapping
- Misc enhancements and fixes
Cédric Le Goater (16):
powerpc/xive: add OPAL extensions for the XIVE native exploitation
support
KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a new capability
KVM_CAP_PPC_IRQ_XIVE
KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
KVM: PPC: Book3S HV: XIVE: add a control to configure a source
KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
KVM: PPC: Book3S HV: XIVE: add a global reset control
KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
KVM: introduce a 'mmap' method for KVM devices
KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
KVM: PPC: Book3S HV: XIVE: add passthrough support
KVM: introduce a KVM_DESTROY_DEVICE ioctl
KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
arch/powerpc/include/asm/kvm_host.h | 2 +
arch/powerpc/include/asm/kvm_ppc.h | 32 +
arch/powerpc/include/asm/opal-api.h | 11 +-
arch/powerpc/include/asm/opal.h | 7 +
arch/powerpc/include/asm/xive.h | 17 +
arch/powerpc/include/uapi/asm/kvm.h | 48 +
arch/powerpc/kvm/book3s_xive.h | 31 +
include/linux/kvm_host.h | 1 +
include/uapi/linux/kvm.h | 10 +
arch/powerpc/kvm/book3s.c | 31 +-
arch/powerpc/kvm/book3s_xics.c | 19 +
arch/powerpc/kvm/book3s_xive.c | 149 ++-
arch/powerpc/kvm/book3s_xive_native.c | 1171 +++++++++++++++++
arch/powerpc/kvm/powerpc.c | 33 +
arch/powerpc/sysdev/xive/native.c | 110 ++
virt/kvm/kvm_main.c | 49 +
Documentation/virtual/kvm/api.txt | 28 +
Documentation/virtual/kvm/devices/xive.txt | 190 +++
arch/powerpc/kvm/Makefile | 2 +-
.../powerpc/platforms/powernv/opal-wrappers.S | 3 +
20 files changed, 1896 insertions(+), 48 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
--
2.20.1
^ permalink raw reply [flat|nested] 142+ messages in thread* [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The support for XIVE native exploitation mode in Linux/KVM needs a
couple more OPAL calls to configure the sPAPR guest and to get/set the
state of the XIVE internal structures.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/opal-api.h | 11 ++-
arch/powerpc/include/asm/opal.h | 7 ++
arch/powerpc/include/asm/xive.h | 14 +++
arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
.../powerpc/platforms/powernv/opal-wrappers.S | 3 +
5 files changed, 130 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 870fb7b239ea..cdfc54f78101 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -186,8 +186,8 @@
#define OPAL_XIVE_FREE_IRQ 140
#define OPAL_XIVE_SYNC 141
#define OPAL_XIVE_DUMP 142
-#define OPAL_XIVE_RESERVED3 143
-#define OPAL_XIVE_RESERVED4 144
+#define OPAL_XIVE_GET_QUEUE_STATE 143
+#define OPAL_XIVE_SET_QUEUE_STATE 144
#define OPAL_SIGNAL_SYSTEM_RESET 145
#define OPAL_NPU_INIT_CONTEXT 146
#define OPAL_NPU_DESTROY_CONTEXT 147
@@ -209,8 +209,11 @@
#define OPAL_SENSOR_GROUP_ENABLE 163
#define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
#define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
-#define OPAL_NX_COPROC_INIT 167
-#define OPAL_LAST 167
+#define OPAL_HANDLE_HMI2 166
+#define OPAL_NX_COPROC_INIT 167
+#define OPAL_NPU_SET_RELAXED_ORDER 168
+#define OPAL_NPU_GET_RELAXED_ORDER 169
+#define OPAL_XIVE_GET_VP_STATE 170
#define QUIESCE_HOLD 1 /* Spin all calls at entry */
#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index a55b01c90bb1..4e978d4dea5c 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
int64_t opal_xive_free_irq(uint32_t girq);
int64_t opal_xive_sync(uint32_t type, uint32_t id);
int64_t opal_xive_dump(uint32_t type, uint32_t id);
+int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
+ __be32 *out_qtoggle,
+ __be32 *out_qindex);
+int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
+ uint32_t qtoggle,
+ uint32_t qindex);
+int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
uint64_t desc, uint16_t pe_number);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 3c704f5dd3ae..b579a943407b 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
extern void xive_native_sync_source(u32 hw_irq);
+extern void xive_native_sync_queue(u32 hw_irq);
extern bool is_xive_irq(struct irq_chip *chip);
extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
extern int xive_native_disable_vp(u32 vp_id);
extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
extern bool xive_native_has_single_escalation(void);
+extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags);
+
+extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
+ u32 *qindex);
+extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
+ u32 qindex);
+extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
+
#else
static inline bool xive_enabled(void) { return false; }
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 1ca127d052a6..0c037e933e55 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
}
EXPORT_SYMBOL_GPL(xive_native_sync_source);
+void xive_native_sync_queue(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
+}
+EXPORT_SYMBOL_GPL(xive_native_sync_queue);
+
static const struct xive_ops xive_native_ops = {
.populate_irq_data = xive_native_populate_irq_data,
.configure_irq = xive_native_configure_irq,
@@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
return xive_has_single_esc;
}
EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
+
+int xive_native_get_queue_info(u32 vp_id, u32 prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags)
+{
+ __be64 qpage;
+ __be64 qsize;
+ __be64 qeoi_page;
+ __be32 escalate_irq;
+ __be64 qflags;
+ s64 rc;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc) {
+ pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (out_qpage)
+ *out_qpage = be64_to_cpu(qpage);
+ if (out_qsize)
+ *out_qsize = be32_to_cpu(qsize);
+ if (out_qeoi_page)
+ *out_qeoi_page = be64_to_cpu(qeoi_page);
+ if (out_escalate_irq)
+ *out_escalate_irq = be32_to_cpu(escalate_irq);
+ if (out_qflags)
+ *out_qflags = be64_to_cpu(qflags);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
+
+int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
+{
+ __be32 opal_qtoggle;
+ __be32 opal_qindex;
+ s64 rc;
+
+ rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
+ &opal_qindex);
+ if (rc) {
+ pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (qtoggle)
+ *qtoggle = be32_to_cpu(opal_qtoggle);
+ if (qindex)
+ *qindex = be32_to_cpu(opal_qindex);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
+
+int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
+{
+ s64 rc;
+
+ rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
+ if (rc) {
+ pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
+
+int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
+{
+ __be64 state;
+ s64 rc;
+
+ rc = opal_xive_get_vp_state(vp_id, &state);
+ if (rc) {
+ pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
+ vp_id, rc);
+ return -EIO;
+ }
+
+ if (out_state)
+ *out_state = be64_to_cpu(state);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index f4875fe3f8ff..3179953d6b56 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -309,6 +309,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
+OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
+OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
+OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The support for XIVE native exploitation mode in Linux/KVM needs a
couple more OPAL calls to configure the sPAPR guest and to get/set the
state of the XIVE internal structures.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/opal-api.h | 11 ++-
arch/powerpc/include/asm/opal.h | 7 ++
arch/powerpc/include/asm/xive.h | 14 +++
arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
.../powerpc/platforms/powernv/opal-wrappers.S | 3 +
5 files changed, 130 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 870fb7b239ea..cdfc54f78101 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -186,8 +186,8 @@
#define OPAL_XIVE_FREE_IRQ 140
#define OPAL_XIVE_SYNC 141
#define OPAL_XIVE_DUMP 142
-#define OPAL_XIVE_RESERVED3 143
-#define OPAL_XIVE_RESERVED4 144
+#define OPAL_XIVE_GET_QUEUE_STATE 143
+#define OPAL_XIVE_SET_QUEUE_STATE 144
#define OPAL_SIGNAL_SYSTEM_RESET 145
#define OPAL_NPU_INIT_CONTEXT 146
#define OPAL_NPU_DESTROY_CONTEXT 147
@@ -209,8 +209,11 @@
#define OPAL_SENSOR_GROUP_ENABLE 163
#define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
#define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
-#define OPAL_NX_COPROC_INIT 167
-#define OPAL_LAST 167
+#define OPAL_HANDLE_HMI2 166
+#define OPAL_NX_COPROC_INIT 167
+#define OPAL_NPU_SET_RELAXED_ORDER 168
+#define OPAL_NPU_GET_RELAXED_ORDER 169
+#define OPAL_XIVE_GET_VP_STATE 170
#define QUIESCE_HOLD 1 /* Spin all calls at entry */
#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index a55b01c90bb1..4e978d4dea5c 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
int64_t opal_xive_free_irq(uint32_t girq);
int64_t opal_xive_sync(uint32_t type, uint32_t id);
int64_t opal_xive_dump(uint32_t type, uint32_t id);
+int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
+ __be32 *out_qtoggle,
+ __be32 *out_qindex);
+int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
+ uint32_t qtoggle,
+ uint32_t qindex);
+int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
uint64_t desc, uint16_t pe_number);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 3c704f5dd3ae..b579a943407b 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
extern void xive_native_sync_source(u32 hw_irq);
+extern void xive_native_sync_queue(u32 hw_irq);
extern bool is_xive_irq(struct irq_chip *chip);
extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
extern int xive_native_disable_vp(u32 vp_id);
extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
extern bool xive_native_has_single_escalation(void);
+extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags);
+
+extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
+ u32 *qindex);
+extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
+ u32 qindex);
+extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
+
#else
static inline bool xive_enabled(void) { return false; }
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 1ca127d052a6..0c037e933e55 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
}
EXPORT_SYMBOL_GPL(xive_native_sync_source);
+void xive_native_sync_queue(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
+}
+EXPORT_SYMBOL_GPL(xive_native_sync_queue);
+
static const struct xive_ops xive_native_ops = {
.populate_irq_data = xive_native_populate_irq_data,
.configure_irq = xive_native_configure_irq,
@@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
return xive_has_single_esc;
}
EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
+
+int xive_native_get_queue_info(u32 vp_id, u32 prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags)
+{
+ __be64 qpage;
+ __be64 qsize;
+ __be64 qeoi_page;
+ __be32 escalate_irq;
+ __be64 qflags;
+ s64 rc;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc) {
+ pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (out_qpage)
+ *out_qpage = be64_to_cpu(qpage);
+ if (out_qsize)
+ *out_qsize = be32_to_cpu(qsize);
+ if (out_qeoi_page)
+ *out_qeoi_page = be64_to_cpu(qeoi_page);
+ if (out_escalate_irq)
+ *out_escalate_irq = be32_to_cpu(escalate_irq);
+ if (out_qflags)
+ *out_qflags = be64_to_cpu(qflags);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
+
+int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
+{
+ __be32 opal_qtoggle;
+ __be32 opal_qindex;
+ s64 rc;
+
+ rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
+ &opal_qindex);
+ if (rc) {
+ pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (qtoggle)
+ *qtoggle = be32_to_cpu(opal_qtoggle);
+ if (qindex)
+ *qindex = be32_to_cpu(opal_qindex);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
+
+int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
+{
+ s64 rc;
+
+ rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
+ if (rc) {
+ pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
+
+int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
+{
+ __be64 state;
+ s64 rc;
+
+ rc = opal_xive_get_vp_state(vp_id, &state);
+ if (rc) {
+ pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
+ vp_id, rc);
+ return -EIO;
+ }
+
+ if (out_state)
+ *out_state = be64_to_cpu(state);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
index f4875fe3f8ff..3179953d6b56 100644
--- a/arch/powerpc/platforms/powernv/opal-wrappers.S
+++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
@@ -309,6 +309,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
+OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
+OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
+OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-24 23:42 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-24 23:42 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 8614 bytes --]
On Fri, Feb 22, 2019 at 12:28:25PM +0100, Cédric Le Goater wrote:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to configure the sPAPR guest and to get/set the
> state of the XIVE internal structures.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/asm/opal-api.h | 11 ++-
> arch/powerpc/include/asm/opal.h | 7 ++
> arch/powerpc/include/asm/xive.h | 14 +++
> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
> 5 files changed, 130 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
> index 870fb7b239ea..cdfc54f78101 100644
> --- a/arch/powerpc/include/asm/opal-api.h
> +++ b/arch/powerpc/include/asm/opal-api.h
> @@ -186,8 +186,8 @@
> #define OPAL_XIVE_FREE_IRQ 140
> #define OPAL_XIVE_SYNC 141
> #define OPAL_XIVE_DUMP 142
> -#define OPAL_XIVE_RESERVED3 143
> -#define OPAL_XIVE_RESERVED4 144
> +#define OPAL_XIVE_GET_QUEUE_STATE 143
> +#define OPAL_XIVE_SET_QUEUE_STATE 144
> #define OPAL_SIGNAL_SYSTEM_RESET 145
> #define OPAL_NPU_INIT_CONTEXT 146
> #define OPAL_NPU_DESTROY_CONTEXT 147
> @@ -209,8 +209,11 @@
> #define OPAL_SENSOR_GROUP_ENABLE 163
> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
> -#define OPAL_NX_COPROC_INIT 167
> -#define OPAL_LAST 167
> +#define OPAL_HANDLE_HMI2 166
> +#define OPAL_NX_COPROC_INIT 167
> +#define OPAL_NPU_SET_RELAXED_ORDER 168
> +#define OPAL_NPU_GET_RELAXED_ORDER 169
> +#define OPAL_XIVE_GET_VP_STATE 170
>
> #define QUIESCE_HOLD 1 /* Spin all calls at entry */
> #define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
> index a55b01c90bb1..4e978d4dea5c 100644
> --- a/arch/powerpc/include/asm/opal.h
> +++ b/arch/powerpc/include/asm/opal.h
> @@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
> int64_t opal_xive_free_irq(uint32_t girq);
> int64_t opal_xive_sync(uint32_t type, uint32_t id);
> int64_t opal_xive_dump(uint32_t type, uint32_t id);
> +int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
> + __be32 *out_qtoggle,
> + __be32 *out_qindex);
> +int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
> + uint32_t qtoggle,
> + uint32_t qindex);
> +int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
> int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
> uint64_t desc, uint16_t pe_number);
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index 3c704f5dd3ae..b579a943407b 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
>
> extern void xive_native_sync_source(u32 hw_irq);
> +extern void xive_native_sync_queue(u32 hw_irq);
> extern bool is_xive_irq(struct irq_chip *chip);
> extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
> extern int xive_native_disable_vp(u32 vp_id);
> extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
> extern bool xive_native_has_single_escalation(void);
>
> +extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
> + u64 *out_qpage,
> + u64 *out_qsize,
> + u64 *out_qeoi_page,
> + u32 *out_escalate_irq,
> + u64 *out_qflags);
> +
> +extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
> + u32 *qindex);
> +extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
> + u32 qindex);
> +extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
> +
> #else
>
> static inline bool xive_enabled(void) { return false; }
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> index 1ca127d052a6..0c037e933e55 100644
> --- a/arch/powerpc/sysdev/xive/native.c
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
> }
> EXPORT_SYMBOL_GPL(xive_native_sync_source);
>
> +void xive_native_sync_queue(u32 hw_irq)
> +{
> + opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
> +}
> +EXPORT_SYMBOL_GPL(xive_native_sync_queue);
> +
> static const struct xive_ops xive_native_ops = {
> .populate_irq_data = xive_native_populate_irq_data,
> .configure_irq = xive_native_configure_irq,
> @@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
> return xive_has_single_esc;
> }
> EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
> +
> +int xive_native_get_queue_info(u32 vp_id, u32 prio,
> + u64 *out_qpage,
> + u64 *out_qsize,
> + u64 *out_qeoi_page,
> + u32 *out_escalate_irq,
> + u64 *out_qflags)
> +{
> + __be64 qpage;
> + __be64 qsize;
> + __be64 qeoi_page;
> + __be32 escalate_irq;
> + __be64 qflags;
> + s64 rc;
> +
> + rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc) {
> + pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
> + vp_id, prio, rc);
> + return -EIO;
> + }
> +
> + if (out_qpage)
> + *out_qpage = be64_to_cpu(qpage);
> + if (out_qsize)
> + *out_qsize = be32_to_cpu(qsize);
> + if (out_qeoi_page)
> + *out_qeoi_page = be64_to_cpu(qeoi_page);
> + if (out_escalate_irq)
> + *out_escalate_irq = be32_to_cpu(escalate_irq);
> + if (out_qflags)
> + *out_qflags = be64_to_cpu(qflags);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
> +
> +int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
> +{
> + __be32 opal_qtoggle;
> + __be32 opal_qindex;
> + s64 rc;
> +
> + rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
> + &opal_qindex);
> + if (rc) {
> + pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
> + vp_id, prio, rc);
> + return -EIO;
> + }
> +
> + if (qtoggle)
> + *qtoggle = be32_to_cpu(opal_qtoggle);
> + if (qindex)
> + *qindex = be32_to_cpu(opal_qindex);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
> +
> +int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
> +{
> + s64 rc;
> +
> + rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
> + if (rc) {
> + pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
> + vp_id, prio, rc);
> + return -EIO;
> + }
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
> +
> +int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
> +{
> + __be64 state;
> + s64 rc;
> +
> + rc = opal_xive_get_vp_state(vp_id, &state);
> + if (rc) {
> + pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
> + vp_id, rc);
> + return -EIO;
> + }
> +
> + if (out_state)
> + *out_state = be64_to_cpu(state);
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
> diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
> index f4875fe3f8ff..3179953d6b56 100644
> --- a/arch/powerpc/platforms/powernv/opal-wrappers.S
> +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
> @@ -309,6 +309,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
> OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
> OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
> OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
> +OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
> +OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
> +OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
> OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
> OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
> OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-02-24 23:42 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-24 23:42 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 8614 bytes --]
On Fri, Feb 22, 2019 at 12:28:25PM +0100, Cédric Le Goater wrote:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to configure the sPAPR guest and to get/set the
> state of the XIVE internal structures.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/asm/opal-api.h | 11 ++-
> arch/powerpc/include/asm/opal.h | 7 ++
> arch/powerpc/include/asm/xive.h | 14 +++
> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
> 5 files changed, 130 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
> index 870fb7b239ea..cdfc54f78101 100644
> --- a/arch/powerpc/include/asm/opal-api.h
> +++ b/arch/powerpc/include/asm/opal-api.h
> @@ -186,8 +186,8 @@
> #define OPAL_XIVE_FREE_IRQ 140
> #define OPAL_XIVE_SYNC 141
> #define OPAL_XIVE_DUMP 142
> -#define OPAL_XIVE_RESERVED3 143
> -#define OPAL_XIVE_RESERVED4 144
> +#define OPAL_XIVE_GET_QUEUE_STATE 143
> +#define OPAL_XIVE_SET_QUEUE_STATE 144
> #define OPAL_SIGNAL_SYSTEM_RESET 145
> #define OPAL_NPU_INIT_CONTEXT 146
> #define OPAL_NPU_DESTROY_CONTEXT 147
> @@ -209,8 +209,11 @@
> #define OPAL_SENSOR_GROUP_ENABLE 163
> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
> -#define OPAL_NX_COPROC_INIT 167
> -#define OPAL_LAST 167
> +#define OPAL_HANDLE_HMI2 166
> +#define OPAL_NX_COPROC_INIT 167
> +#define OPAL_NPU_SET_RELAXED_ORDER 168
> +#define OPAL_NPU_GET_RELAXED_ORDER 169
> +#define OPAL_XIVE_GET_VP_STATE 170
>
> #define QUIESCE_HOLD 1 /* Spin all calls at entry */
> #define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
> diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
> index a55b01c90bb1..4e978d4dea5c 100644
> --- a/arch/powerpc/include/asm/opal.h
> +++ b/arch/powerpc/include/asm/opal.h
> @@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
> int64_t opal_xive_free_irq(uint32_t girq);
> int64_t opal_xive_sync(uint32_t type, uint32_t id);
> int64_t opal_xive_dump(uint32_t type, uint32_t id);
> +int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
> + __be32 *out_qtoggle,
> + __be32 *out_qindex);
> +int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
> + uint32_t qtoggle,
> + uint32_t qindex);
> +int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
> int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
> uint64_t desc, uint16_t pe_number);
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index 3c704f5dd3ae..b579a943407b 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
> extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
>
> extern void xive_native_sync_source(u32 hw_irq);
> +extern void xive_native_sync_queue(u32 hw_irq);
> extern bool is_xive_irq(struct irq_chip *chip);
> extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
> extern int xive_native_disable_vp(u32 vp_id);
> extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
> extern bool xive_native_has_single_escalation(void);
>
> +extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
> + u64 *out_qpage,
> + u64 *out_qsize,
> + u64 *out_qeoi_page,
> + u32 *out_escalate_irq,
> + u64 *out_qflags);
> +
> +extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
> + u32 *qindex);
> +extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
> + u32 qindex);
> +extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
> +
> #else
>
> static inline bool xive_enabled(void) { return false; }
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> index 1ca127d052a6..0c037e933e55 100644
> --- a/arch/powerpc/sysdev/xive/native.c
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
> }
> EXPORT_SYMBOL_GPL(xive_native_sync_source);
>
> +void xive_native_sync_queue(u32 hw_irq)
> +{
> + opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
> +}
> +EXPORT_SYMBOL_GPL(xive_native_sync_queue);
> +
> static const struct xive_ops xive_native_ops = {
> .populate_irq_data = xive_native_populate_irq_data,
> .configure_irq = xive_native_configure_irq,
> @@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
> return xive_has_single_esc;
> }
> EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
> +
> +int xive_native_get_queue_info(u32 vp_id, u32 prio,
> + u64 *out_qpage,
> + u64 *out_qsize,
> + u64 *out_qeoi_page,
> + u32 *out_escalate_irq,
> + u64 *out_qflags)
> +{
> + __be64 qpage;
> + __be64 qsize;
> + __be64 qeoi_page;
> + __be32 escalate_irq;
> + __be64 qflags;
> + s64 rc;
> +
> + rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc) {
> + pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
> + vp_id, prio, rc);
> + return -EIO;
> + }
> +
> + if (out_qpage)
> + *out_qpage = be64_to_cpu(qpage);
> + if (out_qsize)
> + *out_qsize = be32_to_cpu(qsize);
> + if (out_qeoi_page)
> + *out_qeoi_page = be64_to_cpu(qeoi_page);
> + if (out_escalate_irq)
> + *out_escalate_irq = be32_to_cpu(escalate_irq);
> + if (out_qflags)
> + *out_qflags = be64_to_cpu(qflags);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
> +
> +int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
> +{
> + __be32 opal_qtoggle;
> + __be32 opal_qindex;
> + s64 rc;
> +
> + rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
> + &opal_qindex);
> + if (rc) {
> + pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
> + vp_id, prio, rc);
> + return -EIO;
> + }
> +
> + if (qtoggle)
> + *qtoggle = be32_to_cpu(opal_qtoggle);
> + if (qindex)
> + *qindex = be32_to_cpu(opal_qindex);
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
> +
> +int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
> +{
> + s64 rc;
> +
> + rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
> + if (rc) {
> + pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
> + vp_id, prio, rc);
> + return -EIO;
> + }
> +
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
> +
> +int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
> +{
> + __be64 state;
> + s64 rc;
> +
> + rc = opal_xive_get_vp_state(vp_id, &state);
> + if (rc) {
> + pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
> + vp_id, rc);
> + return -EIO;
> + }
> +
> + if (out_state)
> + *out_state = be64_to_cpu(state);
> + return 0;
> +}
> +EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
> diff --git a/arch/powerpc/platforms/powernv/opal-wrappers.S b/arch/powerpc/platforms/powernv/opal-wrappers.S
> index f4875fe3f8ff..3179953d6b56 100644
> --- a/arch/powerpc/platforms/powernv/opal-wrappers.S
> +++ b/arch/powerpc/platforms/powernv/opal-wrappers.S
> @@ -309,6 +309,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
> OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
> OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
> OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
> +OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
> +OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
> +OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
> OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
> OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
> OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 3:50 ` Michael Ellerman
-1 siblings, 0 replies; 142+ messages in thread
From: Michael Ellerman @ 2019-02-25 3:50 UTC (permalink / raw)
To: Cédric Le Goater, kvm-ppc, Stewart Smith
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Cédric Le Goater <clg@kaod.org> writes:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to configure the sPAPR guest and to get/set the
> state of the XIVE internal structures.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/opal-api.h | 11 ++-
> arch/powerpc/include/asm/opal.h | 7 ++
> arch/powerpc/include/asm/xive.h | 14 +++
> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
> 5 files changed, 130 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
> index 870fb7b239ea..cdfc54f78101 100644
> --- a/arch/powerpc/include/asm/opal-api.h
> +++ b/arch/powerpc/include/asm/opal-api.h
> @@ -186,8 +186,8 @@
> #define OPAL_XIVE_FREE_IRQ 140
> #define OPAL_XIVE_SYNC 141
> #define OPAL_XIVE_DUMP 142
> -#define OPAL_XIVE_RESERVED3 143
> -#define OPAL_XIVE_RESERVED4 144
> +#define OPAL_XIVE_GET_QUEUE_STATE 143
> +#define OPAL_XIVE_SET_QUEUE_STATE 144
> #define OPAL_SIGNAL_SYSTEM_RESET 145
> #define OPAL_NPU_INIT_CONTEXT 146
> #define OPAL_NPU_DESTROY_CONTEXT 147
> @@ -209,8 +209,11 @@
> #define OPAL_SENSOR_GROUP_ENABLE 163
> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
> -#define OPAL_NX_COPROC_INIT 167
> -#define OPAL_LAST 167
> +#define OPAL_HANDLE_HMI2 166
> +#define OPAL_NX_COPROC_INIT 167
> +#define OPAL_NPU_SET_RELAXED_ORDER 168
> +#define OPAL_NPU_GET_RELAXED_ORDER 169
> +#define OPAL_XIVE_GET_VP_STATE 170
You should only be defining the calls you need, leaving gaps for other
things, and you need to retain OPAL_LAST. So it should look more like:
-#define OPAL_LAST 167
+#define OPAL_XIVE_GET_VP_STATE 170
+#define OPAL_LAST 170
Also I can't merge this until it's merged into skiboot.
cheers
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-02-25 3:50 ` Michael Ellerman
0 siblings, 0 replies; 142+ messages in thread
From: Michael Ellerman @ 2019-02-25 3:50 UTC (permalink / raw)
To: Cédric Le Goater, kvm-ppc, Stewart Smith
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Cédric Le Goater <clg@kaod.org> writes:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to configure the sPAPR guest and to get/set the
> state of the XIVE internal structures.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/opal-api.h | 11 ++-
> arch/powerpc/include/asm/opal.h | 7 ++
> arch/powerpc/include/asm/xive.h | 14 +++
> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
> 5 files changed, 130 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
> index 870fb7b239ea..cdfc54f78101 100644
> --- a/arch/powerpc/include/asm/opal-api.h
> +++ b/arch/powerpc/include/asm/opal-api.h
> @@ -186,8 +186,8 @@
> #define OPAL_XIVE_FREE_IRQ 140
> #define OPAL_XIVE_SYNC 141
> #define OPAL_XIVE_DUMP 142
> -#define OPAL_XIVE_RESERVED3 143
> -#define OPAL_XIVE_RESERVED4 144
> +#define OPAL_XIVE_GET_QUEUE_STATE 143
> +#define OPAL_XIVE_SET_QUEUE_STATE 144
> #define OPAL_SIGNAL_SYSTEM_RESET 145
> #define OPAL_NPU_INIT_CONTEXT 146
> #define OPAL_NPU_DESTROY_CONTEXT 147
> @@ -209,8 +209,11 @@
> #define OPAL_SENSOR_GROUP_ENABLE 163
> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
> -#define OPAL_NX_COPROC_INIT 167
> -#define OPAL_LAST 167
> +#define OPAL_HANDLE_HMI2 166
> +#define OPAL_NX_COPROC_INIT 167
> +#define OPAL_NPU_SET_RELAXED_ORDER 168
> +#define OPAL_NPU_GET_RELAXED_ORDER 169
> +#define OPAL_XIVE_GET_VP_STATE 170
You should only be defining the calls you need, leaving gaps for other
things, and you need to retain OPAL_LAST. So it should look more like:
-#define OPAL_LAST 167
+#define OPAL_XIVE_GET_VP_STATE 170
+#define OPAL_LAST 170
Also I can't merge this until it's merged into skiboot.
cheers
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-02-25 3:50 ` Michael Ellerman
@ 2019-02-25 10:11 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-25 10:11 UTC (permalink / raw)
To: Michael Ellerman, kvm-ppc, Stewart Smith
Cc: Paul Mackerras, linuxppc-dev, kvm, David Gibson
On 2/25/19 4:50 AM, Michael Ellerman wrote:
> Cédric Le Goater <clg@kaod.org> writes:
>
>> The support for XIVE native exploitation mode in Linux/KVM needs a
>> couple more OPAL calls to configure the sPAPR guest and to get/set the
>> state of the XIVE internal structures.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/opal-api.h | 11 ++-
>> arch/powerpc/include/asm/opal.h | 7 ++
>> arch/powerpc/include/asm/xive.h | 14 +++
>> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
>> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
>> 5 files changed, 130 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
>> index 870fb7b239ea..cdfc54f78101 100644
>> --- a/arch/powerpc/include/asm/opal-api.h
>> +++ b/arch/powerpc/include/asm/opal-api.h
>> @@ -186,8 +186,8 @@
>> #define OPAL_XIVE_FREE_IRQ 140
>> #define OPAL_XIVE_SYNC 141
>> #define OPAL_XIVE_DUMP 142
>> -#define OPAL_XIVE_RESERVED3 143
>> -#define OPAL_XIVE_RESERVED4 144
>> +#define OPAL_XIVE_GET_QUEUE_STATE 143
>> +#define OPAL_XIVE_SET_QUEUE_STATE 144
>> #define OPAL_SIGNAL_SYSTEM_RESET 145
>> #define OPAL_NPU_INIT_CONTEXT 146
>> #define OPAL_NPU_DESTROY_CONTEXT 147
>> @@ -209,8 +209,11 @@
>> #define OPAL_SENSOR_GROUP_ENABLE 163
>> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
>> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
>> -#define OPAL_NX_COPROC_INIT 167
>> -#define OPAL_LAST 167
>> +#define OPAL_HANDLE_HMI2 166
>> +#define OPAL_NX_COPROC_INIT 167
>> +#define OPAL_NPU_SET_RELAXED_ORDER 168
>> +#define OPAL_NPU_GET_RELAXED_ORDER 169
>> +#define OPAL_XIVE_GET_VP_STATE 170
>
> You should only be defining the calls you need, leaving gaps for other
> things, and you need to retain OPAL_LAST. So it should look more like:
>
> -#define OPAL_LAST 167
> +#define OPAL_XIVE_GET_VP_STATE 170
> +#define OPAL_LAST 170
>
>
> Also I can't merge this until it's merged into skiboot.
OK. Let's start with skiboot.
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-02-25 10:11 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-25 10:11 UTC (permalink / raw)
To: Michael Ellerman, kvm-ppc, Stewart Smith
Cc: Paul Mackerras, linuxppc-dev, kvm, David Gibson
On 2/25/19 4:50 AM, Michael Ellerman wrote:
> Cédric Le Goater <clg@kaod.org> writes:
>
>> The support for XIVE native exploitation mode in Linux/KVM needs a
>> couple more OPAL calls to configure the sPAPR guest and to get/set the
>> state of the XIVE internal structures.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/opal-api.h | 11 ++-
>> arch/powerpc/include/asm/opal.h | 7 ++
>> arch/powerpc/include/asm/xive.h | 14 +++
>> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
>> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
>> 5 files changed, 130 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
>> index 870fb7b239ea..cdfc54f78101 100644
>> --- a/arch/powerpc/include/asm/opal-api.h
>> +++ b/arch/powerpc/include/asm/opal-api.h
>> @@ -186,8 +186,8 @@
>> #define OPAL_XIVE_FREE_IRQ 140
>> #define OPAL_XIVE_SYNC 141
>> #define OPAL_XIVE_DUMP 142
>> -#define OPAL_XIVE_RESERVED3 143
>> -#define OPAL_XIVE_RESERVED4 144
>> +#define OPAL_XIVE_GET_QUEUE_STATE 143
>> +#define OPAL_XIVE_SET_QUEUE_STATE 144
>> #define OPAL_SIGNAL_SYSTEM_RESET 145
>> #define OPAL_NPU_INIT_CONTEXT 146
>> #define OPAL_NPU_DESTROY_CONTEXT 147
>> @@ -209,8 +209,11 @@
>> #define OPAL_SENSOR_GROUP_ENABLE 163
>> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
>> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
>> -#define OPAL_NX_COPROC_INIT 167
>> -#define OPAL_LAST 167
>> +#define OPAL_HANDLE_HMI2 166
>> +#define OPAL_NX_COPROC_INIT 167
>> +#define OPAL_NPU_SET_RELAXED_ORDER 168
>> +#define OPAL_NPU_GET_RELAXED_ORDER 169
>> +#define OPAL_XIVE_GET_VP_STATE 170
>
> You should only be defining the calls you need, leaving gaps for other
> things, and you need to retain OPAL_LAST. So it should look more like:
>
> -#define OPAL_LAST 167
> +#define OPAL_XIVE_GET_VP_STATE 170
> +#define OPAL_LAST 170
>
>
> Also I can't merge this until it's merged into skiboot.
OK. Let's start with skiboot.
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-02-25 10:11 ` Cédric Le Goater
@ 2019-02-26 4:21 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-26 4:21 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm, kvm-ppc, Paul Mackerras, Stewart Smith, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2680 bytes --]
On Mon, Feb 25, 2019 at 11:11:58AM +0100, Cédric Le Goater wrote:
> On 2/25/19 4:50 AM, Michael Ellerman wrote:
> > Cédric Le Goater <clg@kaod.org> writes:
> >
> >> The support for XIVE native exploitation mode in Linux/KVM needs a
> >> couple more OPAL calls to configure the sPAPR guest and to get/set the
> >> state of the XIVE internal structures.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/asm/opal-api.h | 11 ++-
> >> arch/powerpc/include/asm/opal.h | 7 ++
> >> arch/powerpc/include/asm/xive.h | 14 +++
> >> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
> >> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
> >> 5 files changed, 130 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
> >> index 870fb7b239ea..cdfc54f78101 100644
> >> --- a/arch/powerpc/include/asm/opal-api.h
> >> +++ b/arch/powerpc/include/asm/opal-api.h
> >> @@ -186,8 +186,8 @@
> >> #define OPAL_XIVE_FREE_IRQ 140
> >> #define OPAL_XIVE_SYNC 141
> >> #define OPAL_XIVE_DUMP 142
> >> -#define OPAL_XIVE_RESERVED3 143
> >> -#define OPAL_XIVE_RESERVED4 144
> >> +#define OPAL_XIVE_GET_QUEUE_STATE 143
> >> +#define OPAL_XIVE_SET_QUEUE_STATE 144
> >> #define OPAL_SIGNAL_SYSTEM_RESET 145
> >> #define OPAL_NPU_INIT_CONTEXT 146
> >> #define OPAL_NPU_DESTROY_CONTEXT 147
> >> @@ -209,8 +209,11 @@
> >> #define OPAL_SENSOR_GROUP_ENABLE 163
> >> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
> >> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
> >> -#define OPAL_NX_COPROC_INIT 167
> >> -#define OPAL_LAST 167
> >> +#define OPAL_HANDLE_HMI2 166
> >> +#define OPAL_NX_COPROC_INIT 167
> >> +#define OPAL_NPU_SET_RELAXED_ORDER 168
> >> +#define OPAL_NPU_GET_RELAXED_ORDER 169
> >> +#define OPAL_XIVE_GET_VP_STATE 170
> >
> > You should only be defining the calls you need, leaving gaps for other
> > things, and you need to retain OPAL_LAST. So it should look more like:
> >
> > -#define OPAL_LAST 167
> > +#define OPAL_XIVE_GET_VP_STATE 170
> > +#define OPAL_LAST 170
> >
> >
> > Also I can't merge this until it's merged into skiboot.
>
> OK. Let's start with skiboot.
Yeah.. where are we at with skiboot in general. We can't test this
downstream until we have a released skiboot with the necessary
support.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-02-26 4:21 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-26 4:21 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm, kvm-ppc, Paul Mackerras, Stewart Smith, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2680 bytes --]
On Mon, Feb 25, 2019 at 11:11:58AM +0100, Cédric Le Goater wrote:
> On 2/25/19 4:50 AM, Michael Ellerman wrote:
> > Cédric Le Goater <clg@kaod.org> writes:
> >
> >> The support for XIVE native exploitation mode in Linux/KVM needs a
> >> couple more OPAL calls to configure the sPAPR guest and to get/set the
> >> state of the XIVE internal structures.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/asm/opal-api.h | 11 ++-
> >> arch/powerpc/include/asm/opal.h | 7 ++
> >> arch/powerpc/include/asm/xive.h | 14 +++
> >> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
> >> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
> >> 5 files changed, 130 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
> >> index 870fb7b239ea..cdfc54f78101 100644
> >> --- a/arch/powerpc/include/asm/opal-api.h
> >> +++ b/arch/powerpc/include/asm/opal-api.h
> >> @@ -186,8 +186,8 @@
> >> #define OPAL_XIVE_FREE_IRQ 140
> >> #define OPAL_XIVE_SYNC 141
> >> #define OPAL_XIVE_DUMP 142
> >> -#define OPAL_XIVE_RESERVED3 143
> >> -#define OPAL_XIVE_RESERVED4 144
> >> +#define OPAL_XIVE_GET_QUEUE_STATE 143
> >> +#define OPAL_XIVE_SET_QUEUE_STATE 144
> >> #define OPAL_SIGNAL_SYSTEM_RESET 145
> >> #define OPAL_NPU_INIT_CONTEXT 146
> >> #define OPAL_NPU_DESTROY_CONTEXT 147
> >> @@ -209,8 +209,11 @@
> >> #define OPAL_SENSOR_GROUP_ENABLE 163
> >> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
> >> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
> >> -#define OPAL_NX_COPROC_INIT 167
> >> -#define OPAL_LAST 167
> >> +#define OPAL_HANDLE_HMI2 166
> >> +#define OPAL_NX_COPROC_INIT 167
> >> +#define OPAL_NPU_SET_RELAXED_ORDER 168
> >> +#define OPAL_NPU_GET_RELAXED_ORDER 169
> >> +#define OPAL_XIVE_GET_VP_STATE 170
> >
> > You should only be defining the calls you need, leaving gaps for other
> > things, and you need to retain OPAL_LAST. So it should look more like:
> >
> > -#define OPAL_LAST 167
> > +#define OPAL_XIVE_GET_VP_STATE 170
> > +#define OPAL_LAST 170
> >
> >
> > Also I can't merge this until it's merged into skiboot.
>
> OK. Let's start with skiboot.
Yeah.. where are we at with skiboot in general. We can't test this
downstream until we have a released skiboot with the necessary
support.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-02-26 4:21 ` David Gibson
@ 2019-03-12 18:25 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 18:25 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, Stewart Smith, linuxppc-dev
On 2/26/19 5:21 AM, David Gibson wrote:
> On Mon, Feb 25, 2019 at 11:11:58AM +0100, Cédric Le Goater wrote:
>> On 2/25/19 4:50 AM, Michael Ellerman wrote:
>>> Cédric Le Goater <clg@kaod.org> writes:
>>>
>>>> The support for XIVE native exploitation mode in Linux/KVM needs a
>>>> couple more OPAL calls to configure the sPAPR guest and to get/set the
>>>> state of the XIVE internal structures.
>>>>
>>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>>> ---
>>>> arch/powerpc/include/asm/opal-api.h | 11 ++-
>>>> arch/powerpc/include/asm/opal.h | 7 ++
>>>> arch/powerpc/include/asm/xive.h | 14 +++
>>>> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
>>>> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
>>>> 5 files changed, 130 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
>>>> index 870fb7b239ea..cdfc54f78101 100644
>>>> --- a/arch/powerpc/include/asm/opal-api.h
>>>> +++ b/arch/powerpc/include/asm/opal-api.h
>>>> @@ -186,8 +186,8 @@
>>>> #define OPAL_XIVE_FREE_IRQ 140
>>>> #define OPAL_XIVE_SYNC 141
>>>> #define OPAL_XIVE_DUMP 142
>>>> -#define OPAL_XIVE_RESERVED3 143
>>>> -#define OPAL_XIVE_RESERVED4 144
>>>> +#define OPAL_XIVE_GET_QUEUE_STATE 143
>>>> +#define OPAL_XIVE_SET_QUEUE_STATE 144
>>>> #define OPAL_SIGNAL_SYSTEM_RESET 145
>>>> #define OPAL_NPU_INIT_CONTEXT 146
>>>> #define OPAL_NPU_DESTROY_CONTEXT 147
>>>> @@ -209,8 +209,11 @@
>>>> #define OPAL_SENSOR_GROUP_ENABLE 163
>>>> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
>>>> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
>>>> -#define OPAL_NX_COPROC_INIT 167
>>>> -#define OPAL_LAST 167
>>>> +#define OPAL_HANDLE_HMI2 166
>>>> +#define OPAL_NX_COPROC_INIT 167
>>>> +#define OPAL_NPU_SET_RELAXED_ORDER 168
>>>> +#define OPAL_NPU_GET_RELAXED_ORDER 169
>>>> +#define OPAL_XIVE_GET_VP_STATE 170
>>>
>>> You should only be defining the calls you need, leaving gaps for other
>>> things, and you need to retain OPAL_LAST. So it should look more like:
>>>
>>> -#define OPAL_LAST 167
>>> +#define OPAL_XIVE_GET_VP_STATE 170
>>> +#define OPAL_LAST 170
>>>
>>>
>>> Also I can't merge this until it's merged into skiboot.
>>
>> OK. Let's start with skiboot.
>
> Yeah.. where are we at with skiboot in general. We can't test this
> downstream until we have a released skiboot with the necessary
> support.
If we add a flag to remove the OPAL call when setting the EQ, you could
test, without migration though.
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-03-12 18:25 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 18:25 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, Stewart Smith, linuxppc-dev
On 2/26/19 5:21 AM, David Gibson wrote:
> On Mon, Feb 25, 2019 at 11:11:58AM +0100, Cédric Le Goater wrote:
>> On 2/25/19 4:50 AM, Michael Ellerman wrote:
>>> Cédric Le Goater <clg@kaod.org> writes:
>>>
>>>> The support for XIVE native exploitation mode in Linux/KVM needs a
>>>> couple more OPAL calls to configure the sPAPR guest and to get/set the
>>>> state of the XIVE internal structures.
>>>>
>>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>>> ---
>>>> arch/powerpc/include/asm/opal-api.h | 11 ++-
>>>> arch/powerpc/include/asm/opal.h | 7 ++
>>>> arch/powerpc/include/asm/xive.h | 14 +++
>>>> arch/powerpc/sysdev/xive/native.c | 99 +++++++++++++++++++
>>>> .../powerpc/platforms/powernv/opal-wrappers.S | 3 +
>>>> 5 files changed, 130 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
>>>> index 870fb7b239ea..cdfc54f78101 100644
>>>> --- a/arch/powerpc/include/asm/opal-api.h
>>>> +++ b/arch/powerpc/include/asm/opal-api.h
>>>> @@ -186,8 +186,8 @@
>>>> #define OPAL_XIVE_FREE_IRQ 140
>>>> #define OPAL_XIVE_SYNC 141
>>>> #define OPAL_XIVE_DUMP 142
>>>> -#define OPAL_XIVE_RESERVED3 143
>>>> -#define OPAL_XIVE_RESERVED4 144
>>>> +#define OPAL_XIVE_GET_QUEUE_STATE 143
>>>> +#define OPAL_XIVE_SET_QUEUE_STATE 144
>>>> #define OPAL_SIGNAL_SYSTEM_RESET 145
>>>> #define OPAL_NPU_INIT_CONTEXT 146
>>>> #define OPAL_NPU_DESTROY_CONTEXT 147
>>>> @@ -209,8 +209,11 @@
>>>> #define OPAL_SENSOR_GROUP_ENABLE 163
>>>> #define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
>>>> #define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
>>>> -#define OPAL_NX_COPROC_INIT 167
>>>> -#define OPAL_LAST 167
>>>> +#define OPAL_HANDLE_HMI2 166
>>>> +#define OPAL_NX_COPROC_INIT 167
>>>> +#define OPAL_NPU_SET_RELAXED_ORDER 168
>>>> +#define OPAL_NPU_GET_RELAXED_ORDER 169
>>>> +#define OPAL_XIVE_GET_VP_STATE 170
>>>
>>> You should only be defining the calls you need, leaving gaps for other
>>> things, and you need to retain OPAL_LAST. So it should look more like:
>>>
>>> -#define OPAL_LAST 167
>>> +#define OPAL_XIVE_GET_VP_STATE 170
>>> +#define OPAL_LAST 170
>>>
>>>
>>> Also I can't merge this until it's merged into skiboot.
>>
>> OK. Let's start with skiboot.
>
> Yeah.. where are we at with skiboot in general. We can't test this
> downstream until we have a released skiboot with the necessary
> support.
If we add a flag to remove the OPAL call when setting the EQ, you could
test, without migration though.
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This is the basic framework for the new KVM device supporting the XIVE
native exploitation mode. The user interface exposes a new KVM device
to be created by QEMU when running on a L0 hypervisor only. Support
for nested guests is not available yet.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 8 +
arch/powerpc/include/uapi/asm/kvm.h | 3 +
include/uapi/linux/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 7 +-
arch/powerpc/kvm/book3s_xive_native.c | 191 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 ++
arch/powerpc/kvm/Makefile | 2 +-
8 files changed, 231 insertions(+), 2 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 091430339db1..9f75a75a07f2 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -220,6 +220,7 @@ extern struct kvm_device_ops kvm_xics_ops;
struct kvmppc_xive;
struct kvmppc_xive_vcpu;
extern struct kvm_device_ops kvm_xive_ops;
+extern struct kvm_device_ops kvm_xive_native_ops;
struct kvmppc_passthru_irqmap;
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index b3bf4f61b30c..4b72ddde7dc1 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -593,6 +593,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+
+extern void kvmppc_xive_native_init_module(void);
+extern void kvmppc_xive_native_exit_module(void);
+
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
u32 priority) { return -1; }
@@ -616,6 +620,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+
+static inline void kvmppc_xive_native_init_module(void) { }
+static inline void kvmppc_xive_native_exit_module(void) { }
+
#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_PPC_POWERNV
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 8c876c166ef2..b002c0c67787 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -675,4 +675,7 @@ struct kvm_ppc_cpu_char {
#define KVM_XICS_PRESENTED (1ULL << 43)
#define KVM_XICS_QUEUED (1ULL << 44)
+/* POWER9 XIVE Native Interrupt Controller */
+#define KVM_DEV_XIVE_GRP_CTRL 1
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6d4ea4b6c922..e6368163d3a0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1211,6 +1211,8 @@ enum kvm_device_type {
#define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
KVM_DEV_TYPE_ARM_VGIC_ITS,
#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
+ KVM_DEV_TYPE_XIVE,
+#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
KVM_DEV_TYPE_MAX,
};
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 601c094f15ab..96d43f091255 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -1040,6 +1040,9 @@ static int kvmppc_book3s_init(void)
if (xics_on_xive()) {
kvmppc_xive_init_module();
kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
+ kvmppc_xive_native_init_module();
+ kvm_register_device_ops(&kvm_xive_native_ops,
+ KVM_DEV_TYPE_XIVE);
} else
#endif
kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
@@ -1050,8 +1053,10 @@ static int kvmppc_book3s_init(void)
static void kvmppc_book3s_exit(void)
{
#ifdef CONFIG_KVM_XICS
- if (xics_on_xive())
+ if (xics_on_xive()) {
kvmppc_xive_exit_module();
+ kvmppc_xive_native_exit_module();
+ }
#endif
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
kvmppc_book3s_exit_pr();
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
new file mode 100644
index 000000000000..e475ce83ad14
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -0,0 +1,191 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2017-2019, IBM Corporation.
+ */
+
+#define pr_fmt(fmt) "xive-kvm: " fmt
+
+#include <linux/anon_inodes.h>
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/percpu.h>
+#include <linux/cpumask.h>
+#include <asm/uaccess.h>
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xics.h>
+#include <asm/xive.h>
+#include <asm/xive-regs.h>
+#include <asm/debug.h>
+#include <asm/debugfs.h>
+#include <asm/time.h>
+#include <asm/opal.h>
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+#include "book3s_xive.h"
+
+static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static void kvmppc_xive_native_free(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+
+ debugfs_remove(xive->dentry);
+
+ pr_devel("Destroying xive native device\n");
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ kfree(xive);
+ kfree(dev);
+}
+
+static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
+{
+ struct kvmppc_xive *xive;
+ struct kvm *kvm = dev->kvm;
+ int ret = 0;
+
+ pr_devel("Creating xive native device\n");
+
+ if (kvm->arch.xive)
+ return -EEXIST;
+
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ if (!xive)
+ return -ENOMEM;
+
+ dev->private = xive;
+ xive->dev = dev;
+ xive->kvm = kvm;
+ kvm->arch.xive = xive;
+
+ /* We use the default queue size set by the host */
+ xive->q_order = xive_native_default_eq_shift();
+ if (xive->q_order < PAGE_SHIFT)
+ xive->q_page_order = 0;
+ else
+ xive->q_page_order = xive->q_order - PAGE_SHIFT;
+
+ /*
+ * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
+ * a default. Getting the max number of CPUs the VM was
+ * configured with would improve our usage of the XIVE VP space.
+ */
+ xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
+ pr_devel("VP_Base=%x\n", xive->vp_base);
+
+ if (xive->vp_base == XIVE_INVALID_VP)
+ ret = -ENOMEM;
+
+ xive->single_escalation = xive_native_has_single_escalation();
+
+ if (ret)
+ kfree(xive);
+
+ return ret;
+}
+
+static int xive_native_debug_show(struct seq_file *m, void *private)
+{
+ struct kvmppc_xive *xive = m->private;
+ struct kvm *kvm = xive->kvm;
+
+ if (!kvm)
+ return 0;
+
+ return 0;
+}
+
+static int xive_native_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, xive_native_debug_show, inode->i_private);
+}
+
+static const struct file_operations xive_native_debug_fops = {
+ .open = xive_native_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void xive_native_debugfs_init(struct kvmppc_xive *xive)
+{
+ char *name;
+
+ name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
+ if (!name) {
+ pr_err("%s: no memory for name\n", __func__);
+ return;
+ }
+
+ xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
+ xive, &xive_native_debug_fops);
+
+ pr_debug("%s: created %s\n", __func__, name);
+ kfree(name);
+}
+
+static void kvmppc_xive_native_init(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
+
+ /* Register some debug interfaces */
+ xive_native_debugfs_init(xive);
+}
+
+struct kvm_device_ops kvm_xive_native_ops = {
+ .name = "kvm-xive-native",
+ .create = kvmppc_xive_native_create,
+ .init = kvmppc_xive_native_init,
+ .destroy = kvmppc_xive_native_free,
+ .set_attr = kvmppc_xive_native_set_attr,
+ .get_attr = kvmppc_xive_native_get_attr,
+ .has_attr = kvmppc_xive_native_has_attr,
+};
+
+void kvmppc_xive_native_init_module(void)
+{
+ ;
+}
+
+void kvmppc_xive_native_exit_module(void)
+{
+ ;
+}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
new file mode 100644
index 000000000000..fdbd2ff92a88
--- /dev/null
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -0,0 +1,19 @@
+POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
+==========================================================
+
+Device types supported:
+ KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
+
+This device acts as a VM interrupt controller. It provides the KVM
+interface to configure the interrupt sources of a VM in the underlying
+POWER9 XIVE interrupt controller.
+
+Only one XIVE instance may be instantiated. A guest XIVE device
+requires a POWER9 host and the guest OS should have support for the
+XIVE native exploitation interrupt mode. If not, it should run using
+the legacy interrupt mode, referred as XICS (POWER7/8).
+
+* Groups:
+
+ 1. KVM_DEV_XIVE_GRP_CTRL
+ Provides global controls on the device
diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
index 64f1135e7732..806cbe488410 100644
--- a/arch/powerpc/kvm/Makefile
+++ b/arch/powerpc/kvm/Makefile
@@ -99,7 +99,7 @@ endif
kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
book3s_xics.o
-kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
+kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
kvm-book3s_64-module-objs := \
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This is the basic framework for the new KVM device supporting the XIVE
native exploitation mode. The user interface exposes a new KVM device
to be created by QEMU when running on a L0 hypervisor only. Support
for nested guests is not available yet.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 8 +
arch/powerpc/include/uapi/asm/kvm.h | 3 +
include/uapi/linux/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 7 +-
arch/powerpc/kvm/book3s_xive_native.c | 191 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 ++
arch/powerpc/kvm/Makefile | 2 +-
8 files changed, 231 insertions(+), 2 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 091430339db1..9f75a75a07f2 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -220,6 +220,7 @@ extern struct kvm_device_ops kvm_xics_ops;
struct kvmppc_xive;
struct kvmppc_xive_vcpu;
extern struct kvm_device_ops kvm_xive_ops;
+extern struct kvm_device_ops kvm_xive_native_ops;
struct kvmppc_passthru_irqmap;
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index b3bf4f61b30c..4b72ddde7dc1 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -593,6 +593,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+
+extern void kvmppc_xive_native_init_module(void);
+extern void kvmppc_xive_native_exit_module(void);
+
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
u32 priority) { return -1; }
@@ -616,6 +620,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+
+static inline void kvmppc_xive_native_init_module(void) { }
+static inline void kvmppc_xive_native_exit_module(void) { }
+
#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_PPC_POWERNV
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 8c876c166ef2..b002c0c67787 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -675,4 +675,7 @@ struct kvm_ppc_cpu_char {
#define KVM_XICS_PRESENTED (1ULL << 43)
#define KVM_XICS_QUEUED (1ULL << 44)
+/* POWER9 XIVE Native Interrupt Controller */
+#define KVM_DEV_XIVE_GRP_CTRL 1
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6d4ea4b6c922..e6368163d3a0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1211,6 +1211,8 @@ enum kvm_device_type {
#define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
KVM_DEV_TYPE_ARM_VGIC_ITS,
#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
+ KVM_DEV_TYPE_XIVE,
+#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
KVM_DEV_TYPE_MAX,
};
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 601c094f15ab..96d43f091255 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -1040,6 +1040,9 @@ static int kvmppc_book3s_init(void)
if (xics_on_xive()) {
kvmppc_xive_init_module();
kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
+ kvmppc_xive_native_init_module();
+ kvm_register_device_ops(&kvm_xive_native_ops,
+ KVM_DEV_TYPE_XIVE);
} else
#endif
kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
@@ -1050,8 +1053,10 @@ static int kvmppc_book3s_init(void)
static void kvmppc_book3s_exit(void)
{
#ifdef CONFIG_KVM_XICS
- if (xics_on_xive())
+ if (xics_on_xive()) {
kvmppc_xive_exit_module();
+ kvmppc_xive_native_exit_module();
+ }
#endif
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
kvmppc_book3s_exit_pr();
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
new file mode 100644
index 000000000000..e475ce83ad14
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -0,0 +1,191 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2017-2019, IBM Corporation.
+ */
+
+#define pr_fmt(fmt) "xive-kvm: " fmt
+
+#include <linux/anon_inodes.h>
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <linux/percpu.h>
+#include <linux/cpumask.h>
+#include <asm/uaccess.h>
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xics.h>
+#include <asm/xive.h>
+#include <asm/xive-regs.h>
+#include <asm/debug.h>
+#include <asm/debugfs.h>
+#include <asm/time.h>
+#include <asm/opal.h>
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+#include "book3s_xive.h"
+
+static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static void kvmppc_xive_native_free(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+
+ debugfs_remove(xive->dentry);
+
+ pr_devel("Destroying xive native device\n");
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ kfree(xive);
+ kfree(dev);
+}
+
+static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
+{
+ struct kvmppc_xive *xive;
+ struct kvm *kvm = dev->kvm;
+ int ret = 0;
+
+ pr_devel("Creating xive native device\n");
+
+ if (kvm->arch.xive)
+ return -EEXIST;
+
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ if (!xive)
+ return -ENOMEM;
+
+ dev->private = xive;
+ xive->dev = dev;
+ xive->kvm = kvm;
+ kvm->arch.xive = xive;
+
+ /* We use the default queue size set by the host */
+ xive->q_order = xive_native_default_eq_shift();
+ if (xive->q_order < PAGE_SHIFT)
+ xive->q_page_order = 0;
+ else
+ xive->q_page_order = xive->q_order - PAGE_SHIFT;
+
+ /*
+ * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
+ * a default. Getting the max number of CPUs the VM was
+ * configured with would improve our usage of the XIVE VP space.
+ */
+ xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
+ pr_devel("VP_Base=%x\n", xive->vp_base);
+
+ if (xive->vp_base = XIVE_INVALID_VP)
+ ret = -ENOMEM;
+
+ xive->single_escalation = xive_native_has_single_escalation();
+
+ if (ret)
+ kfree(xive);
+
+ return ret;
+}
+
+static int xive_native_debug_show(struct seq_file *m, void *private)
+{
+ struct kvmppc_xive *xive = m->private;
+ struct kvm *kvm = xive->kvm;
+
+ if (!kvm)
+ return 0;
+
+ return 0;
+}
+
+static int xive_native_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, xive_native_debug_show, inode->i_private);
+}
+
+static const struct file_operations xive_native_debug_fops = {
+ .open = xive_native_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void xive_native_debugfs_init(struct kvmppc_xive *xive)
+{
+ char *name;
+
+ name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
+ if (!name) {
+ pr_err("%s: no memory for name\n", __func__);
+ return;
+ }
+
+ xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
+ xive, &xive_native_debug_fops);
+
+ pr_debug("%s: created %s\n", __func__, name);
+ kfree(name);
+}
+
+static void kvmppc_xive_native_init(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
+
+ /* Register some debug interfaces */
+ xive_native_debugfs_init(xive);
+}
+
+struct kvm_device_ops kvm_xive_native_ops = {
+ .name = "kvm-xive-native",
+ .create = kvmppc_xive_native_create,
+ .init = kvmppc_xive_native_init,
+ .destroy = kvmppc_xive_native_free,
+ .set_attr = kvmppc_xive_native_set_attr,
+ .get_attr = kvmppc_xive_native_get_attr,
+ .has_attr = kvmppc_xive_native_has_attr,
+};
+
+void kvmppc_xive_native_init_module(void)
+{
+ ;
+}
+
+void kvmppc_xive_native_exit_module(void)
+{
+ ;
+}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
new file mode 100644
index 000000000000..fdbd2ff92a88
--- /dev/null
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -0,0 +1,19 @@
+POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
+=============================
+
+Device types supported:
+ KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
+
+This device acts as a VM interrupt controller. It provides the KVM
+interface to configure the interrupt sources of a VM in the underlying
+POWER9 XIVE interrupt controller.
+
+Only one XIVE instance may be instantiated. A guest XIVE device
+requires a POWER9 host and the guest OS should have support for the
+XIVE native exploitation interrupt mode. If not, it should run using
+the legacy interrupt mode, referred as XICS (POWER7/8).
+
+* Groups:
+
+ 1. KVM_DEV_XIVE_GRP_CTRL
+ Provides global controls on the device
diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
index 64f1135e7732..806cbe488410 100644
--- a/arch/powerpc/kvm/Makefile
+++ b/arch/powerpc/kvm/Makefile
@@ -99,7 +99,7 @@ endif
kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
book3s_xics.o
-kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
+kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
kvm-book3s_64-module-objs := \
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 0:08 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 0:08 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 11820 bytes --]
On Fri, Feb 22, 2019 at 12:28:26PM +0100, Cédric Le Goater wrote:
> This is the basic framework for the new KVM device supporting the XIVE
> native exploitation mode. The user interface exposes a new KVM device
> to be created by QEMU when running on a L0 hypervisor only. Support
> for nested guests is not available yet.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/kvm_host.h | 1 +
> arch/powerpc/include/asm/kvm_ppc.h | 8 +
> arch/powerpc/include/uapi/asm/kvm.h | 3 +
> include/uapi/linux/kvm.h | 2 +
> arch/powerpc/kvm/book3s.c | 7 +-
> arch/powerpc/kvm/book3s_xive_native.c | 191 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 19 ++
> arch/powerpc/kvm/Makefile | 2 +-
> 8 files changed, 231 insertions(+), 2 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 091430339db1..9f75a75a07f2 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -220,6 +220,7 @@ extern struct kvm_device_ops kvm_xics_ops;
> struct kvmppc_xive;
> struct kvmppc_xive_vcpu;
> extern struct kvm_device_ops kvm_xive_ops;
> +extern struct kvm_device_ops kvm_xive_native_ops;
>
> struct kvmppc_passthru_irqmap;
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index b3bf4f61b30c..4b72ddde7dc1 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -593,6 +593,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
> extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status);
> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
> +
> +extern void kvmppc_xive_native_init_module(void);
> +extern void kvmppc_xive_native_exit_module(void);
> +
> #else
> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> u32 priority) { return -1; }
> @@ -616,6 +620,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
> static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status) { return -ENODEV; }
> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
> +
> +static inline void kvmppc_xive_native_init_module(void) { }
> +static inline void kvmppc_xive_native_exit_module(void) { }
> +
> #endif /* CONFIG_KVM_XIVE */
>
> #ifdef CONFIG_PPC_POWERNV
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 8c876c166ef2..b002c0c67787 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -675,4 +675,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_XICS_PRESENTED (1ULL << 43)
> #define KVM_XICS_QUEUED (1ULL << 44)
>
> +/* POWER9 XIVE Native Interrupt Controller */
> +#define KVM_DEV_XIVE_GRP_CTRL 1
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 6d4ea4b6c922..e6368163d3a0 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1211,6 +1211,8 @@ enum kvm_device_type {
> #define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
> KVM_DEV_TYPE_ARM_VGIC_ITS,
> #define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
> + KVM_DEV_TYPE_XIVE,
> +#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
> KVM_DEV_TYPE_MAX,
> };
>
> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> index 601c094f15ab..96d43f091255 100644
> --- a/arch/powerpc/kvm/book3s.c
> +++ b/arch/powerpc/kvm/book3s.c
> @@ -1040,6 +1040,9 @@ static int kvmppc_book3s_init(void)
> if (xics_on_xive()) {
> kvmppc_xive_init_module();
> kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
> + kvmppc_xive_native_init_module();
> + kvm_register_device_ops(&kvm_xive_native_ops,
> + KVM_DEV_TYPE_XIVE);
> } else
> #endif
> kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
> @@ -1050,8 +1053,10 @@ static int kvmppc_book3s_init(void)
> static void kvmppc_book3s_exit(void)
> {
> #ifdef CONFIG_KVM_XICS
> - if (xics_on_xive())
> + if (xics_on_xive()) {
> kvmppc_xive_exit_module();
> + kvmppc_xive_native_exit_module();
> + }
> #endif
> #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
> kvmppc_book3s_exit_pr();
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> new file mode 100644
> index 000000000000..e475ce83ad14
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -0,0 +1,191 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2017-2019, IBM Corporation.
> + */
> +
> +#define pr_fmt(fmt) "xive-kvm: " fmt
> +
> +#include <linux/anon_inodes.h>
> +#include <linux/kernel.h>
> +#include <linux/kvm_host.h>
> +#include <linux/err.h>
> +#include <linux/gfp.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/percpu.h>
> +#include <linux/cpumask.h>
> +#include <asm/uaccess.h>
> +#include <asm/kvm_book3s.h>
> +#include <asm/kvm_ppc.h>
> +#include <asm/hvcall.h>
> +#include <asm/xics.h>
> +#include <asm/xive.h>
> +#include <asm/xive-regs.h>
> +#include <asm/debug.h>
> +#include <asm/debugfs.h>
> +#include <asm/time.h>
> +#include <asm/opal.h>
> +
> +#include <linux/debugfs.h>
> +#include <linux/seq_file.h>
> +
> +#include "book3s_xive.h"
> +
> +static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> + struct kvm_device_attr *attr)
> +{
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_CTRL:
> + break;
> + }
> + return -ENXIO;
> +}
> +
> +static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> + struct kvm_device_attr *attr)
> +{
> + return -ENXIO;
> +}
> +
> +static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> + struct kvm_device_attr *attr)
> +{
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_CTRL:
> + break;
> + }
> + return -ENXIO;
> +}
> +
> +static void kvmppc_xive_native_free(struct kvm_device *dev)
> +{
> + struct kvmppc_xive *xive = dev->private;
> + struct kvm *kvm = xive->kvm;
> +
> + debugfs_remove(xive->dentry);
> +
> + pr_devel("Destroying xive native device\n");
> +
> + if (kvm)
> + kvm->arch.xive = NULL;
> +
> + if (xive->vp_base != XIVE_INVALID_VP)
> + xive_native_free_vp_block(xive->vp_base);
> +
> + kfree(xive);
> + kfree(dev);
> +}
> +
> +static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> +{
> + struct kvmppc_xive *xive;
> + struct kvm *kvm = dev->kvm;
> + int ret = 0;
> +
> + pr_devel("Creating xive native device\n");
> +
> + if (kvm->arch.xive)
> + return -EEXIST;
> +
> + xive = kzalloc(sizeof(*xive), GFP_KERNEL);
> + if (!xive)
> + return -ENOMEM;
> +
> + dev->private = xive;
> + xive->dev = dev;
> + xive->kvm = kvm;
> + kvm->arch.xive = xive;
> +
> + /* We use the default queue size set by the host */
IIUC the queue is examined directly by the guest, so the guest must
know its size. In which case letting the host decide the size would
be a problem for migration.
> + xive->q_order = xive_native_default_eq_shift();
> + if (xive->q_order < PAGE_SHIFT)
> + xive->q_page_order = 0;
> + else
> + xive->q_page_order = xive->q_order - PAGE_SHIFT;
> +
> + /*
> + * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
> + * a default. Getting the max number of CPUs the VM was
> + * configured with would improve our usage of the XIVE VP space.
> + */
> + xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
> + pr_devel("VP_Base=%x\n", xive->vp_base);
> +
> + if (xive->vp_base == XIVE_INVALID_VP)
> + ret = -ENOMEM;
> +
> + xive->single_escalation = xive_native_has_single_escalation();
> +
> + if (ret)
> + kfree(xive);
> +
> + return ret;
> +}
> +
> +static int xive_native_debug_show(struct seq_file *m, void *private)
> +{
> + struct kvmppc_xive *xive = m->private;
> + struct kvm *kvm = xive->kvm;
> +
> + if (!kvm)
> + return 0;
> +
> + return 0;
> +}
> +
> +static int xive_native_debug_open(struct inode *inode, struct file *file)
> +{
> + return single_open(file, xive_native_debug_show, inode->i_private);
> +}
> +
> +static const struct file_operations xive_native_debug_fops = {
> + .open = xive_native_debug_open,
> + .read = seq_read,
> + .llseek = seq_lseek,
> + .release = single_release,
> +};
> +
> +static void xive_native_debugfs_init(struct kvmppc_xive *xive)
> +{
> + char *name;
> +
> + name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
> + if (!name) {
> + pr_err("%s: no memory for name\n", __func__);
> + return;
> + }
> +
> + xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
> + xive, &xive_native_debug_fops);
> +
> + pr_debug("%s: created %s\n", __func__, name);
> + kfree(name);
> +}
> +
> +static void kvmppc_xive_native_init(struct kvm_device *dev)
> +{
> + struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
> +
> + /* Register some debug interfaces */
> + xive_native_debugfs_init(xive);
> +}
> +
> +struct kvm_device_ops kvm_xive_native_ops = {
> + .name = "kvm-xive-native",
> + .create = kvmppc_xive_native_create,
> + .init = kvmppc_xive_native_init,
> + .destroy = kvmppc_xive_native_free,
> + .set_attr = kvmppc_xive_native_set_attr,
> + .get_attr = kvmppc_xive_native_get_attr,
> + .has_attr = kvmppc_xive_native_has_attr,
> +};
> +
> +void kvmppc_xive_native_init_module(void)
> +{
> + ;
> +}
> +
> +void kvmppc_xive_native_exit_module(void)
> +{
> + ;
> +}
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> new file mode 100644
> index 000000000000..fdbd2ff92a88
> --- /dev/null
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -0,0 +1,19 @@
> +POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
> +==========================================================
> +
> +Device types supported:
> + KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
> +
> +This device acts as a VM interrupt controller. It provides the KVM
> +interface to configure the interrupt sources of a VM in the underlying
> +POWER9 XIVE interrupt controller.
> +
> +Only one XIVE instance may be instantiated. A guest XIVE device
> +requires a POWER9 host and the guest OS should have support for the
> +XIVE native exploitation interrupt mode. If not, it should run using
> +the legacy interrupt mode, referred as XICS (POWER7/8).
> +
> +* Groups:
> +
> + 1. KVM_DEV_XIVE_GRP_CTRL
> + Provides global controls on the device
> diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
> index 64f1135e7732..806cbe488410 100644
> --- a/arch/powerpc/kvm/Makefile
> +++ b/arch/powerpc/kvm/Makefile
> @@ -99,7 +99,7 @@ endif
> kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
> book3s_xics.o
>
> -kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
> +kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
> kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
>
> kvm-book3s_64-module-objs := \
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
@ 2019-02-25 0:08 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 0:08 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 11820 bytes --]
On Fri, Feb 22, 2019 at 12:28:26PM +0100, Cédric Le Goater wrote:
> This is the basic framework for the new KVM device supporting the XIVE
> native exploitation mode. The user interface exposes a new KVM device
> to be created by QEMU when running on a L0 hypervisor only. Support
> for nested guests is not available yet.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/kvm_host.h | 1 +
> arch/powerpc/include/asm/kvm_ppc.h | 8 +
> arch/powerpc/include/uapi/asm/kvm.h | 3 +
> include/uapi/linux/kvm.h | 2 +
> arch/powerpc/kvm/book3s.c | 7 +-
> arch/powerpc/kvm/book3s_xive_native.c | 191 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 19 ++
> arch/powerpc/kvm/Makefile | 2 +-
> 8 files changed, 231 insertions(+), 2 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 091430339db1..9f75a75a07f2 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -220,6 +220,7 @@ extern struct kvm_device_ops kvm_xics_ops;
> struct kvmppc_xive;
> struct kvmppc_xive_vcpu;
> extern struct kvm_device_ops kvm_xive_ops;
> +extern struct kvm_device_ops kvm_xive_native_ops;
>
> struct kvmppc_passthru_irqmap;
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index b3bf4f61b30c..4b72ddde7dc1 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -593,6 +593,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
> extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status);
> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
> +
> +extern void kvmppc_xive_native_init_module(void);
> +extern void kvmppc_xive_native_exit_module(void);
> +
> #else
> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> u32 priority) { return -1; }
> @@ -616,6 +620,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
> static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status) { return -ENODEV; }
> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
> +
> +static inline void kvmppc_xive_native_init_module(void) { }
> +static inline void kvmppc_xive_native_exit_module(void) { }
> +
> #endif /* CONFIG_KVM_XIVE */
>
> #ifdef CONFIG_PPC_POWERNV
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 8c876c166ef2..b002c0c67787 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -675,4 +675,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_XICS_PRESENTED (1ULL << 43)
> #define KVM_XICS_QUEUED (1ULL << 44)
>
> +/* POWER9 XIVE Native Interrupt Controller */
> +#define KVM_DEV_XIVE_GRP_CTRL 1
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 6d4ea4b6c922..e6368163d3a0 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1211,6 +1211,8 @@ enum kvm_device_type {
> #define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
> KVM_DEV_TYPE_ARM_VGIC_ITS,
> #define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
> + KVM_DEV_TYPE_XIVE,
> +#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
> KVM_DEV_TYPE_MAX,
> };
>
> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> index 601c094f15ab..96d43f091255 100644
> --- a/arch/powerpc/kvm/book3s.c
> +++ b/arch/powerpc/kvm/book3s.c
> @@ -1040,6 +1040,9 @@ static int kvmppc_book3s_init(void)
> if (xics_on_xive()) {
> kvmppc_xive_init_module();
> kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
> + kvmppc_xive_native_init_module();
> + kvm_register_device_ops(&kvm_xive_native_ops,
> + KVM_DEV_TYPE_XIVE);
> } else
> #endif
> kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
> @@ -1050,8 +1053,10 @@ static int kvmppc_book3s_init(void)
> static void kvmppc_book3s_exit(void)
> {
> #ifdef CONFIG_KVM_XICS
> - if (xics_on_xive())
> + if (xics_on_xive()) {
> kvmppc_xive_exit_module();
> + kvmppc_xive_native_exit_module();
> + }
> #endif
> #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
> kvmppc_book3s_exit_pr();
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> new file mode 100644
> index 000000000000..e475ce83ad14
> --- /dev/null
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -0,0 +1,191 @@
> +// SPDX-License-Identifier: GPL-2.0
> +/*
> + * Copyright (c) 2017-2019, IBM Corporation.
> + */
> +
> +#define pr_fmt(fmt) "xive-kvm: " fmt
> +
> +#include <linux/anon_inodes.h>
> +#include <linux/kernel.h>
> +#include <linux/kvm_host.h>
> +#include <linux/err.h>
> +#include <linux/gfp.h>
> +#include <linux/spinlock.h>
> +#include <linux/delay.h>
> +#include <linux/percpu.h>
> +#include <linux/cpumask.h>
> +#include <asm/uaccess.h>
> +#include <asm/kvm_book3s.h>
> +#include <asm/kvm_ppc.h>
> +#include <asm/hvcall.h>
> +#include <asm/xics.h>
> +#include <asm/xive.h>
> +#include <asm/xive-regs.h>
> +#include <asm/debug.h>
> +#include <asm/debugfs.h>
> +#include <asm/time.h>
> +#include <asm/opal.h>
> +
> +#include <linux/debugfs.h>
> +#include <linux/seq_file.h>
> +
> +#include "book3s_xive.h"
> +
> +static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> + struct kvm_device_attr *attr)
> +{
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_CTRL:
> + break;
> + }
> + return -ENXIO;
> +}
> +
> +static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> + struct kvm_device_attr *attr)
> +{
> + return -ENXIO;
> +}
> +
> +static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> + struct kvm_device_attr *attr)
> +{
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_CTRL:
> + break;
> + }
> + return -ENXIO;
> +}
> +
> +static void kvmppc_xive_native_free(struct kvm_device *dev)
> +{
> + struct kvmppc_xive *xive = dev->private;
> + struct kvm *kvm = xive->kvm;
> +
> + debugfs_remove(xive->dentry);
> +
> + pr_devel("Destroying xive native device\n");
> +
> + if (kvm)
> + kvm->arch.xive = NULL;
> +
> + if (xive->vp_base != XIVE_INVALID_VP)
> + xive_native_free_vp_block(xive->vp_base);
> +
> + kfree(xive);
> + kfree(dev);
> +}
> +
> +static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> +{
> + struct kvmppc_xive *xive;
> + struct kvm *kvm = dev->kvm;
> + int ret = 0;
> +
> + pr_devel("Creating xive native device\n");
> +
> + if (kvm->arch.xive)
> + return -EEXIST;
> +
> + xive = kzalloc(sizeof(*xive), GFP_KERNEL);
> + if (!xive)
> + return -ENOMEM;
> +
> + dev->private = xive;
> + xive->dev = dev;
> + xive->kvm = kvm;
> + kvm->arch.xive = xive;
> +
> + /* We use the default queue size set by the host */
IIUC the queue is examined directly by the guest, so the guest must
know its size. In which case letting the host decide the size would
be a problem for migration.
> + xive->q_order = xive_native_default_eq_shift();
> + if (xive->q_order < PAGE_SHIFT)
> + xive->q_page_order = 0;
> + else
> + xive->q_page_order = xive->q_order - PAGE_SHIFT;
> +
> + /*
> + * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
> + * a default. Getting the max number of CPUs the VM was
> + * configured with would improve our usage of the XIVE VP space.
> + */
> + xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
> + pr_devel("VP_Base=%x\n", xive->vp_base);
> +
> + if (xive->vp_base == XIVE_INVALID_VP)
> + ret = -ENOMEM;
> +
> + xive->single_escalation = xive_native_has_single_escalation();
> +
> + if (ret)
> + kfree(xive);
> +
> + return ret;
> +}
> +
> +static int xive_native_debug_show(struct seq_file *m, void *private)
> +{
> + struct kvmppc_xive *xive = m->private;
> + struct kvm *kvm = xive->kvm;
> +
> + if (!kvm)
> + return 0;
> +
> + return 0;
> +}
> +
> +static int xive_native_debug_open(struct inode *inode, struct file *file)
> +{
> + return single_open(file, xive_native_debug_show, inode->i_private);
> +}
> +
> +static const struct file_operations xive_native_debug_fops = {
> + .open = xive_native_debug_open,
> + .read = seq_read,
> + .llseek = seq_lseek,
> + .release = single_release,
> +};
> +
> +static void xive_native_debugfs_init(struct kvmppc_xive *xive)
> +{
> + char *name;
> +
> + name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
> + if (!name) {
> + pr_err("%s: no memory for name\n", __func__);
> + return;
> + }
> +
> + xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
> + xive, &xive_native_debug_fops);
> +
> + pr_debug("%s: created %s\n", __func__, name);
> + kfree(name);
> +}
> +
> +static void kvmppc_xive_native_init(struct kvm_device *dev)
> +{
> + struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
> +
> + /* Register some debug interfaces */
> + xive_native_debugfs_init(xive);
> +}
> +
> +struct kvm_device_ops kvm_xive_native_ops = {
> + .name = "kvm-xive-native",
> + .create = kvmppc_xive_native_create,
> + .init = kvmppc_xive_native_init,
> + .destroy = kvmppc_xive_native_free,
> + .set_attr = kvmppc_xive_native_set_attr,
> + .get_attr = kvmppc_xive_native_get_attr,
> + .has_attr = kvmppc_xive_native_has_attr,
> +};
> +
> +void kvmppc_xive_native_init_module(void)
> +{
> + ;
> +}
> +
> +void kvmppc_xive_native_exit_module(void)
> +{
> + ;
> +}
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> new file mode 100644
> index 000000000000..fdbd2ff92a88
> --- /dev/null
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -0,0 +1,19 @@
> +POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
> +==========================================================
> +
> +Device types supported:
> + KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
> +
> +This device acts as a VM interrupt controller. It provides the KVM
> +interface to configure the interrupt sources of a VM in the underlying
> +POWER9 XIVE interrupt controller.
> +
> +Only one XIVE instance may be instantiated. A guest XIVE device
> +requires a POWER9 host and the guest OS should have support for the
> +XIVE native exploitation interrupt mode. If not, it should run using
> +the legacy interrupt mode, referred as XICS (POWER7/8).
> +
> +* Groups:
> +
> + 1. KVM_DEV_XIVE_GRP_CTRL
> + Provides global controls on the device
> diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
> index 64f1135e7732..806cbe488410 100644
> --- a/arch/powerpc/kvm/Makefile
> +++ b/arch/powerpc/kvm/Makefile
> @@ -99,7 +99,7 @@ endif
> kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
> book3s_xics.o
>
> -kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
> +kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
> kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
>
> kvm-book3s_64-module-objs := \
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
2019-02-25 0:08 ` David Gibson
@ 2019-03-12 11:14 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 11:14 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 1:08 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:26PM +0100, Cédric Le Goater wrote:
>> This is the basic framework for the new KVM device supporting the XIVE
>> native exploitation mode. The user interface exposes a new KVM device
>> to be created by QEMU when running on a L0 hypervisor only. Support
>> for nested guests is not available yet.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/kvm_host.h | 1 +
>> arch/powerpc/include/asm/kvm_ppc.h | 8 +
>> arch/powerpc/include/uapi/asm/kvm.h | 3 +
>> include/uapi/linux/kvm.h | 2 +
>> arch/powerpc/kvm/book3s.c | 7 +-
>> arch/powerpc/kvm/book3s_xive_native.c | 191 +++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 19 ++
>> arch/powerpc/kvm/Makefile | 2 +-
>> 8 files changed, 231 insertions(+), 2 deletions(-)
>> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
>> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>>
>> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
>> index 091430339db1..9f75a75a07f2 100644
>> --- a/arch/powerpc/include/asm/kvm_host.h
>> +++ b/arch/powerpc/include/asm/kvm_host.h
>> @@ -220,6 +220,7 @@ extern struct kvm_device_ops kvm_xics_ops;
>> struct kvmppc_xive;
>> struct kvmppc_xive_vcpu;
>> extern struct kvm_device_ops kvm_xive_ops;
>> +extern struct kvm_device_ops kvm_xive_native_ops;
>>
>> struct kvmppc_passthru_irqmap;
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index b3bf4f61b30c..4b72ddde7dc1 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -593,6 +593,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
>> extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status);
>> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>> +
>> +extern void kvmppc_xive_native_init_module(void);
>> +extern void kvmppc_xive_native_exit_module(void);
>> +
>> #else
>> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
>> u32 priority) { return -1; }
>> @@ -616,6 +620,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
>> static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status) { return -ENODEV; }
>> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>> +
>> +static inline void kvmppc_xive_native_init_module(void) { }
>> +static inline void kvmppc_xive_native_exit_module(void) { }
>> +
>> #endif /* CONFIG_KVM_XIVE */
>>
>> #ifdef CONFIG_PPC_POWERNV
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index 8c876c166ef2..b002c0c67787 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -675,4 +675,7 @@ struct kvm_ppc_cpu_char {
>> #define KVM_XICS_PRESENTED (1ULL << 43)
>> #define KVM_XICS_QUEUED (1ULL << 44)
>>
>> +/* POWER9 XIVE Native Interrupt Controller */
>> +#define KVM_DEV_XIVE_GRP_CTRL 1
>> +
>> #endif /* __LINUX_KVM_POWERPC_H */
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 6d4ea4b6c922..e6368163d3a0 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1211,6 +1211,8 @@ enum kvm_device_type {
>> #define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
>> KVM_DEV_TYPE_ARM_VGIC_ITS,
>> #define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
>> + KVM_DEV_TYPE_XIVE,
>> +#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
>> KVM_DEV_TYPE_MAX,
>> };
>>
>> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
>> index 601c094f15ab..96d43f091255 100644
>> --- a/arch/powerpc/kvm/book3s.c
>> +++ b/arch/powerpc/kvm/book3s.c
>> @@ -1040,6 +1040,9 @@ static int kvmppc_book3s_init(void)
>> if (xics_on_xive()) {
>> kvmppc_xive_init_module();
>> kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
>> + kvmppc_xive_native_init_module();
>> + kvm_register_device_ops(&kvm_xive_native_ops,
>> + KVM_DEV_TYPE_XIVE);
>> } else
>> #endif
>> kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
>> @@ -1050,8 +1053,10 @@ static int kvmppc_book3s_init(void)
>> static void kvmppc_book3s_exit(void)
>> {
>> #ifdef CONFIG_KVM_XICS
>> - if (xics_on_xive())
>> + if (xics_on_xive()) {
>> kvmppc_xive_exit_module();
>> + kvmppc_xive_native_exit_module();
>> + }
>> #endif
>> #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
>> kvmppc_book3s_exit_pr();
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> new file mode 100644
>> index 000000000000..e475ce83ad14
>> --- /dev/null
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -0,0 +1,191 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2017-2019, IBM Corporation.
>> + */
>> +
>> +#define pr_fmt(fmt) "xive-kvm: " fmt
>> +
>> +#include <linux/anon_inodes.h>
>> +#include <linux/kernel.h>
>> +#include <linux/kvm_host.h>
>> +#include <linux/err.h>
>> +#include <linux/gfp.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/delay.h>
>> +#include <linux/percpu.h>
>> +#include <linux/cpumask.h>
>> +#include <asm/uaccess.h>
>> +#include <asm/kvm_book3s.h>
>> +#include <asm/kvm_ppc.h>
>> +#include <asm/hvcall.h>
>> +#include <asm/xics.h>
>> +#include <asm/xive.h>
>> +#include <asm/xive-regs.h>
>> +#include <asm/debug.h>
>> +#include <asm/debugfs.h>
>> +#include <asm/time.h>
>> +#include <asm/opal.h>
>> +
>> +#include <linux/debugfs.h>
>> +#include <linux/seq_file.h>
>> +
>> +#include "book3s_xive.h"
>> +
>> +static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> + struct kvm_device_attr *attr)
>> +{
>> + switch (attr->group) {
>> + case KVM_DEV_XIVE_GRP_CTRL:
>> + break;
>> + }
>> + return -ENXIO;
>> +}
>> +
>> +static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
>> + struct kvm_device_attr *attr)
>> +{
>> + return -ENXIO;
>> +}
>> +
>> +static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> + struct kvm_device_attr *attr)
>> +{
>> + switch (attr->group) {
>> + case KVM_DEV_XIVE_GRP_CTRL:
>> + break;
>> + }
>> + return -ENXIO;
>> +}
>> +
>> +static void kvmppc_xive_native_free(struct kvm_device *dev)
>> +{
>> + struct kvmppc_xive *xive = dev->private;
>> + struct kvm *kvm = xive->kvm;
>> +
>> + debugfs_remove(xive->dentry);
>> +
>> + pr_devel("Destroying xive native device\n");
>> +
>> + if (kvm)
>> + kvm->arch.xive = NULL;
>> +
>> + if (xive->vp_base != XIVE_INVALID_VP)
>> + xive_native_free_vp_block(xive->vp_base);
>> +
>> + kfree(xive);
>> + kfree(dev);
>> +}
>> +
>> +static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
>> +{
>> + struct kvmppc_xive *xive;
>> + struct kvm *kvm = dev->kvm;
>> + int ret = 0;
>> +
>> + pr_devel("Creating xive native device\n");
>> +
>> + if (kvm->arch.xive)
>> + return -EEXIST;
>> +
>> + xive = kzalloc(sizeof(*xive), GFP_KERNEL);
>> + if (!xive)
>> + return -ENOMEM;
>> +
>> + dev->private = xive;
>> + xive->dev = dev;
>> + xive->kvm = kvm;
>> + kvm->arch.xive = xive;
>> +
>> + /* We use the default queue size set by the host */
>
> IIUC the queue is examined directly by the guest, so the guest must
> know its size. In which case letting the host decide the size would
> be a problem for migration.
yes. This is a left over from the XICS-over-XIVE KVM device. I will
remove the code, we don't use it.
Thanks,
C.
>> + xive->q_order = xive_native_default_eq_shift();
>> + if (xive->q_order < PAGE_SHIFT)
>> + xive->q_page_order = 0;
>> + else
>> + xive->q_page_order = xive->q_order - PAGE_SHIFT;
>> +
>> + /*
>> + * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
>> + * a default. Getting the max number of CPUs the VM was
>> + * configured with would improve our usage of the XIVE VP space.
>> + */
>> + xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
>> + pr_devel("VP_Base=%x\n", xive->vp_base);
>> +
>> + if (xive->vp_base == XIVE_INVALID_VP)
>> + ret = -ENOMEM;
>> +
>> + xive->single_escalation = xive_native_has_single_escalation();
>> +
>> + if (ret)
>> + kfree(xive);
>> +
>> + return ret;
>> +}
>> +
>> +static int xive_native_debug_show(struct seq_file *m, void *private)
>> +{
>> + struct kvmppc_xive *xive = m->private;
>> + struct kvm *kvm = xive->kvm;
>> +
>> + if (!kvm)
>> + return 0;
>> +
>> + return 0;
>> +}
>> +
>> +static int xive_native_debug_open(struct inode *inode, struct file *file)
>> +{
>> + return single_open(file, xive_native_debug_show, inode->i_private);
>> +}
>> +
>> +static const struct file_operations xive_native_debug_fops = {
>> + .open = xive_native_debug_open,
>> + .read = seq_read,
>> + .llseek = seq_lseek,
>> + .release = single_release,
>> +};
>> +
>> +static void xive_native_debugfs_init(struct kvmppc_xive *xive)
>> +{
>> + char *name;
>> +
>> + name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
>> + if (!name) {
>> + pr_err("%s: no memory for name\n", __func__);
>> + return;
>> + }
>> +
>> + xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
>> + xive, &xive_native_debug_fops);
>> +
>> + pr_debug("%s: created %s\n", __func__, name);
>> + kfree(name);
>> +}
>> +
>> +static void kvmppc_xive_native_init(struct kvm_device *dev)
>> +{
>> + struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
>> +
>> + /* Register some debug interfaces */
>> + xive_native_debugfs_init(xive);
>> +}
>> +
>> +struct kvm_device_ops kvm_xive_native_ops = {
>> + .name = "kvm-xive-native",
>> + .create = kvmppc_xive_native_create,
>> + .init = kvmppc_xive_native_init,
>> + .destroy = kvmppc_xive_native_free,
>> + .set_attr = kvmppc_xive_native_set_attr,
>> + .get_attr = kvmppc_xive_native_get_attr,
>> + .has_attr = kvmppc_xive_native_has_attr,
>> +};
>> +
>> +void kvmppc_xive_native_init_module(void)
>> +{
>> + ;
>> +}
>> +
>> +void kvmppc_xive_native_exit_module(void)
>> +{
>> + ;
>> +}
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> new file mode 100644
>> index 000000000000..fdbd2ff92a88
>> --- /dev/null
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -0,0 +1,19 @@
>> +POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
>> +==========================================================
>> +
>> +Device types supported:
>> + KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
>> +
>> +This device acts as a VM interrupt controller. It provides the KVM
>> +interface to configure the interrupt sources of a VM in the underlying
>> +POWER9 XIVE interrupt controller.
>> +
>> +Only one XIVE instance may be instantiated. A guest XIVE device
>> +requires a POWER9 host and the guest OS should have support for the
>> +XIVE native exploitation interrupt mode. If not, it should run using
>> +the legacy interrupt mode, referred as XICS (POWER7/8).
>> +
>> +* Groups:
>> +
>> + 1. KVM_DEV_XIVE_GRP_CTRL
>> + Provides global controls on the device
>> diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
>> index 64f1135e7732..806cbe488410 100644
>> --- a/arch/powerpc/kvm/Makefile
>> +++ b/arch/powerpc/kvm/Makefile
>> @@ -99,7 +99,7 @@ endif
>> kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
>> book3s_xics.o
>>
>> -kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
>> +kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
>> kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
>>
>> kvm-book3s_64-module-objs := \
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
@ 2019-03-12 11:14 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 11:14 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 1:08 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:26PM +0100, Cédric Le Goater wrote:
>> This is the basic framework for the new KVM device supporting the XIVE
>> native exploitation mode. The user interface exposes a new KVM device
>> to be created by QEMU when running on a L0 hypervisor only. Support
>> for nested guests is not available yet.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/kvm_host.h | 1 +
>> arch/powerpc/include/asm/kvm_ppc.h | 8 +
>> arch/powerpc/include/uapi/asm/kvm.h | 3 +
>> include/uapi/linux/kvm.h | 2 +
>> arch/powerpc/kvm/book3s.c | 7 +-
>> arch/powerpc/kvm/book3s_xive_native.c | 191 +++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 19 ++
>> arch/powerpc/kvm/Makefile | 2 +-
>> 8 files changed, 231 insertions(+), 2 deletions(-)
>> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
>> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>>
>> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
>> index 091430339db1..9f75a75a07f2 100644
>> --- a/arch/powerpc/include/asm/kvm_host.h
>> +++ b/arch/powerpc/include/asm/kvm_host.h
>> @@ -220,6 +220,7 @@ extern struct kvm_device_ops kvm_xics_ops;
>> struct kvmppc_xive;
>> struct kvmppc_xive_vcpu;
>> extern struct kvm_device_ops kvm_xive_ops;
>> +extern struct kvm_device_ops kvm_xive_native_ops;
>>
>> struct kvmppc_passthru_irqmap;
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index b3bf4f61b30c..4b72ddde7dc1 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -593,6 +593,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
>> extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status);
>> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>> +
>> +extern void kvmppc_xive_native_init_module(void);
>> +extern void kvmppc_xive_native_exit_module(void);
>> +
>> #else
>> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
>> u32 priority) { return -1; }
>> @@ -616,6 +620,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
>> static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status) { return -ENODEV; }
>> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>> +
>> +static inline void kvmppc_xive_native_init_module(void) { }
>> +static inline void kvmppc_xive_native_exit_module(void) { }
>> +
>> #endif /* CONFIG_KVM_XIVE */
>>
>> #ifdef CONFIG_PPC_POWERNV
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index 8c876c166ef2..b002c0c67787 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -675,4 +675,7 @@ struct kvm_ppc_cpu_char {
>> #define KVM_XICS_PRESENTED (1ULL << 43)
>> #define KVM_XICS_QUEUED (1ULL << 44)
>>
>> +/* POWER9 XIVE Native Interrupt Controller */
>> +#define KVM_DEV_XIVE_GRP_CTRL 1
>> +
>> #endif /* __LINUX_KVM_POWERPC_H */
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 6d4ea4b6c922..e6368163d3a0 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1211,6 +1211,8 @@ enum kvm_device_type {
>> #define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
>> KVM_DEV_TYPE_ARM_VGIC_ITS,
>> #define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
>> + KVM_DEV_TYPE_XIVE,
>> +#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
>> KVM_DEV_TYPE_MAX,
>> };
>>
>> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
>> index 601c094f15ab..96d43f091255 100644
>> --- a/arch/powerpc/kvm/book3s.c
>> +++ b/arch/powerpc/kvm/book3s.c
>> @@ -1040,6 +1040,9 @@ static int kvmppc_book3s_init(void)
>> if (xics_on_xive()) {
>> kvmppc_xive_init_module();
>> kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
>> + kvmppc_xive_native_init_module();
>> + kvm_register_device_ops(&kvm_xive_native_ops,
>> + KVM_DEV_TYPE_XIVE);
>> } else
>> #endif
>> kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
>> @@ -1050,8 +1053,10 @@ static int kvmppc_book3s_init(void)
>> static void kvmppc_book3s_exit(void)
>> {
>> #ifdef CONFIG_KVM_XICS
>> - if (xics_on_xive())
>> + if (xics_on_xive()) {
>> kvmppc_xive_exit_module();
>> + kvmppc_xive_native_exit_module();
>> + }
>> #endif
>> #ifdef CONFIG_KVM_BOOK3S_32_HANDLER
>> kvmppc_book3s_exit_pr();
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> new file mode 100644
>> index 000000000000..e475ce83ad14
>> --- /dev/null
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -0,0 +1,191 @@
>> +// SPDX-License-Identifier: GPL-2.0
>> +/*
>> + * Copyright (c) 2017-2019, IBM Corporation.
>> + */
>> +
>> +#define pr_fmt(fmt) "xive-kvm: " fmt
>> +
>> +#include <linux/anon_inodes.h>
>> +#include <linux/kernel.h>
>> +#include <linux/kvm_host.h>
>> +#include <linux/err.h>
>> +#include <linux/gfp.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/delay.h>
>> +#include <linux/percpu.h>
>> +#include <linux/cpumask.h>
>> +#include <asm/uaccess.h>
>> +#include <asm/kvm_book3s.h>
>> +#include <asm/kvm_ppc.h>
>> +#include <asm/hvcall.h>
>> +#include <asm/xics.h>
>> +#include <asm/xive.h>
>> +#include <asm/xive-regs.h>
>> +#include <asm/debug.h>
>> +#include <asm/debugfs.h>
>> +#include <asm/time.h>
>> +#include <asm/opal.h>
>> +
>> +#include <linux/debugfs.h>
>> +#include <linux/seq_file.h>
>> +
>> +#include "book3s_xive.h"
>> +
>> +static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> + struct kvm_device_attr *attr)
>> +{
>> + switch (attr->group) {
>> + case KVM_DEV_XIVE_GRP_CTRL:
>> + break;
>> + }
>> + return -ENXIO;
>> +}
>> +
>> +static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
>> + struct kvm_device_attr *attr)
>> +{
>> + return -ENXIO;
>> +}
>> +
>> +static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> + struct kvm_device_attr *attr)
>> +{
>> + switch (attr->group) {
>> + case KVM_DEV_XIVE_GRP_CTRL:
>> + break;
>> + }
>> + return -ENXIO;
>> +}
>> +
>> +static void kvmppc_xive_native_free(struct kvm_device *dev)
>> +{
>> + struct kvmppc_xive *xive = dev->private;
>> + struct kvm *kvm = xive->kvm;
>> +
>> + debugfs_remove(xive->dentry);
>> +
>> + pr_devel("Destroying xive native device\n");
>> +
>> + if (kvm)
>> + kvm->arch.xive = NULL;
>> +
>> + if (xive->vp_base != XIVE_INVALID_VP)
>> + xive_native_free_vp_block(xive->vp_base);
>> +
>> + kfree(xive);
>> + kfree(dev);
>> +}
>> +
>> +static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
>> +{
>> + struct kvmppc_xive *xive;
>> + struct kvm *kvm = dev->kvm;
>> + int ret = 0;
>> +
>> + pr_devel("Creating xive native device\n");
>> +
>> + if (kvm->arch.xive)
>> + return -EEXIST;
>> +
>> + xive = kzalloc(sizeof(*xive), GFP_KERNEL);
>> + if (!xive)
>> + return -ENOMEM;
>> +
>> + dev->private = xive;
>> + xive->dev = dev;
>> + xive->kvm = kvm;
>> + kvm->arch.xive = xive;
>> +
>> + /* We use the default queue size set by the host */
>
> IIUC the queue is examined directly by the guest, so the guest must
> know its size. In which case letting the host decide the size would
> be a problem for migration.
yes. This is a left over from the XICS-over-XIVE KVM device. I will
remove the code, we don't use it.
Thanks,
C.
>> + xive->q_order = xive_native_default_eq_shift();
>> + if (xive->q_order < PAGE_SHIFT)
>> + xive->q_page_order = 0;
>> + else
>> + xive->q_page_order = xive->q_order - PAGE_SHIFT;
>> +
>> + /*
>> + * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
>> + * a default. Getting the max number of CPUs the VM was
>> + * configured with would improve our usage of the XIVE VP space.
>> + */
>> + xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
>> + pr_devel("VP_Base=%x\n", xive->vp_base);
>> +
>> + if (xive->vp_base = XIVE_INVALID_VP)
>> + ret = -ENOMEM;
>> +
>> + xive->single_escalation = xive_native_has_single_escalation();
>> +
>> + if (ret)
>> + kfree(xive);
>> +
>> + return ret;
>> +}
>> +
>> +static int xive_native_debug_show(struct seq_file *m, void *private)
>> +{
>> + struct kvmppc_xive *xive = m->private;
>> + struct kvm *kvm = xive->kvm;
>> +
>> + if (!kvm)
>> + return 0;
>> +
>> + return 0;
>> +}
>> +
>> +static int xive_native_debug_open(struct inode *inode, struct file *file)
>> +{
>> + return single_open(file, xive_native_debug_show, inode->i_private);
>> +}
>> +
>> +static const struct file_operations xive_native_debug_fops = {
>> + .open = xive_native_debug_open,
>> + .read = seq_read,
>> + .llseek = seq_lseek,
>> + .release = single_release,
>> +};
>> +
>> +static void xive_native_debugfs_init(struct kvmppc_xive *xive)
>> +{
>> + char *name;
>> +
>> + name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
>> + if (!name) {
>> + pr_err("%s: no memory for name\n", __func__);
>> + return;
>> + }
>> +
>> + xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
>> + xive, &xive_native_debug_fops);
>> +
>> + pr_debug("%s: created %s\n", __func__, name);
>> + kfree(name);
>> +}
>> +
>> +static void kvmppc_xive_native_init(struct kvm_device *dev)
>> +{
>> + struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
>> +
>> + /* Register some debug interfaces */
>> + xive_native_debugfs_init(xive);
>> +}
>> +
>> +struct kvm_device_ops kvm_xive_native_ops = {
>> + .name = "kvm-xive-native",
>> + .create = kvmppc_xive_native_create,
>> + .init = kvmppc_xive_native_init,
>> + .destroy = kvmppc_xive_native_free,
>> + .set_attr = kvmppc_xive_native_set_attr,
>> + .get_attr = kvmppc_xive_native_get_attr,
>> + .has_attr = kvmppc_xive_native_has_attr,
>> +};
>> +
>> +void kvmppc_xive_native_init_module(void)
>> +{
>> + ;
>> +}
>> +
>> +void kvmppc_xive_native_exit_module(void)
>> +{
>> + ;
>> +}
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> new file mode 100644
>> index 000000000000..fdbd2ff92a88
>> --- /dev/null
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -0,0 +1,19 @@
>> +POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
>> +=============================
>> +
>> +Device types supported:
>> + KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
>> +
>> +This device acts as a VM interrupt controller. It provides the KVM
>> +interface to configure the interrupt sources of a VM in the underlying
>> +POWER9 XIVE interrupt controller.
>> +
>> +Only one XIVE instance may be instantiated. A guest XIVE device
>> +requires a POWER9 host and the guest OS should have support for the
>> +XIVE native exploitation interrupt mode. If not, it should run using
>> +the legacy interrupt mode, referred as XICS (POWER7/8).
>> +
>> +* Groups:
>> +
>> + 1. KVM_DEV_XIVE_GRP_CTRL
>> + Provides global controls on the device
>> diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
>> index 64f1135e7732..806cbe488410 100644
>> --- a/arch/powerpc/kvm/Makefile
>> +++ b/arch/powerpc/kvm/Makefile
>> @@ -99,7 +99,7 @@ endif
>> kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
>> book3s_xics.o
>>
>> -kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
>> +kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
>> kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
>>
>> kvm-book3s_64-module-objs := \
>
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The user interface exposes a new capability to let QEMU connect the
vCPU to the XIVE KVM device if required. The capability is only
advertised on a PowerNV Hypervisor as support for nested guests
(pseries KVM Hypervisor) is not yet available.
Internally, the interface to the new KVM device is protected with a
new interrupt mode: KVMPPC_IRQ_XIVE.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 13 +++
arch/powerpc/kvm/book3s_xive.h | 6 ++
include/uapi/linux/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 67 +++++++-----
arch/powerpc/kvm/book3s_xive_native.c | 144 ++++++++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 33 ++++++
Documentation/virtual/kvm/api.txt | 9 ++
8 files changed, 246 insertions(+), 28 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 9f75a75a07f2..eb8581be0ee8 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -448,6 +448,7 @@ struct kvmppc_passthru_irqmap {
#define KVMPPC_IRQ_DEFAULT 0
#define KVMPPC_IRQ_MPIC 1
#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
+#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
#define MMIO_HPTE_CACHE_SIZE 4
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 4b72ddde7dc1..1e61877fe147 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -594,6 +594,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+{
+ return vcpu->arch.irq_type == KVMPPC_IRQ_XIVE;
+}
+
+extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu);
+extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
@@ -621,6 +629,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+ { return 0; }
+static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
+static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index a08ae6fd4c51..bcb1bbcf0359 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -248,5 +248,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+/*
+ * Common Xive routines for XICS-over-XIVE and XIVE native
+ */
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e6368163d3a0..52bf74a1616e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_ARM_VM_IPA_SIZE 165
#define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
#define KVM_CAP_HYPERV_CPUID 167
+#define KVM_CAP_PPC_IRQ_XIVE 168
#ifdef KVM_CAP_IRQ_ROUTING
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index f78d002f0fe0..d1cc18a5b1c4 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1049,7 +1049,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
}
EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
-static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct kvm *kvm = vcpu->kvm;
@@ -1883,6 +1883,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
return 0;
}
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int i;
+
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+ u32 i0, i1, idx;
+
+ if (!q->qpage && !xc->esc_virq[i])
+ continue;
+
+ seq_printf(m, " [q%d]: ", i);
+
+ if (q->qpage) {
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
+ i0, i1);
+ }
+ if (xc->esc_virq[i]) {
+ struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
+ struct xive_irq_data *xd =
+ irq_data_get_irq_handler_data(d);
+ u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
+
+ seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
+ (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
+ (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
+ xc->esc_virq[i], pq, xd->eoi_page);
+ seq_puts(m, "\n");
+ }
+ }
+ return 0;
+}
static int xive_debug_show(struct seq_file *m, void *private)
{
@@ -1908,7 +1945,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
kvm_for_each_vcpu(i, vcpu, kvm) {
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- unsigned int i;
if (!xc)
continue;
@@ -1918,33 +1954,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
xc->server_num, xc->cppr, xc->hw_cppr,
xc->mfrr, xc->pending,
xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
- for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
- struct xive_q *q = &xc->queues[i];
- u32 i0, i1, idx;
- if (!q->qpage && !xc->esc_virq[i])
- continue;
-
- seq_printf(m, " [q%d]: ", i);
-
- if (q->qpage) {
- idx = q->idx;
- i0 = be32_to_cpup(q->qpage + idx);
- idx = (idx + 1) & q->msk;
- i1 = be32_to_cpup(q->qpage + idx);
- seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
- }
- if (xc->esc_virq[i]) {
- struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
- struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
- u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
- seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
- (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
- (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
- xc->esc_virq[i], pq, xd->eoi_page);
- seq_printf(m, "\n");
- }
- }
+ kvmppc_xive_debug_show_queues(m, vcpu);
t_rm_h_xirr += xc->stat_rm_h_xirr;
t_rm_h_ipoll += xc->stat_rm_h_ipoll;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index e475ce83ad14..1f3da47a4a6a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -31,6 +31,128 @@
#include "book3s_xive.h"
+static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q = &xc->queues[prio];
+
+ xive_native_disable_queue(xc->vp_id, q, prio);
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+}
+
+void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ int i;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
+ pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
+
+ /* Ensure no interrupt is still routed to that VP */
+ xc->valid = false;
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ /* Disable the VP */
+ xive_native_disable_vp(xc->vp_id);
+
+ /* Free the queues & associated interrupts */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ /* Free the escalation irq */
+ if (xc->esc_virq[i]) {
+ free_irq(xc->esc_virq[i], vcpu);
+ irq_dispose_mapping(xc->esc_virq[i]);
+ kfree(xc->esc_virq_names[i]);
+ xc->esc_virq[i] = 0;
+ }
+
+ /* Free the queue */
+ kvmppc_xive_native_cleanup_queue(vcpu, i);
+ }
+
+ /* Free the VP */
+ kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
+}
+
+int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_vcpu *xc;
+ int rc;
+
+ pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
+
+ if (dev->ops != &kvm_xive_native_ops) {
+ pr_devel("Wrong ops !\n");
+ return -EPERM;
+ }
+ if (xive->kvm != vcpu->kvm)
+ return -EPERM;
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
+ return -EBUSY;
+ if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
+ pr_devel("Duplicate !\n");
+ return -EEXIST;
+ }
+ if (cpu >= KVM_MAX_VCPUS) {
+ pr_devel("Out of bounds !\n");
+ return -EINVAL;
+ }
+ xc = kzalloc(sizeof(*xc), GFP_KERNEL);
+ if (!xc)
+ return -ENOMEM;
+
+ mutex_lock(&vcpu->kvm->lock);
+ vcpu->arch.xive_vcpu = xc;
+ xc->xive = xive;
+ xc->vcpu = vcpu;
+ xc->server_num = cpu;
+ xc->vp_id = xive->vp_base + cpu;
+ xc->valid = true;
+
+ rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
+ if (rc) {
+ pr_err("Failed to get VP info from OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /*
+ * Enable the VP first as the single escalation mode will
+ * affect escalation interrupts numbering
+ */
+ rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
+ if (rc) {
+ pr_err("Failed to enable VP in OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /* Configure VCPU fields for use by assembly push/pull */
+ vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
+ vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
+
+ /* TODO: initialize queues ? */
+
+bail:
+ vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+ mutex_unlock(&vcpu->kvm->lock);
+ if (rc)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -126,10 +248,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
if (!kvm)
return 0;
+ seq_puts(m, "=========\nVCPU state\n=========\n");
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
+ xc->server_num,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word);
+
+ kvmppc_xive_debug_show_queues(m, vcpu);
+ }
+
return 0;
}
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 8c69af10f91d..a38a643a24dd 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_PPC_GET_CPU_CHAR:
r = 1;
break;
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE:
+ /* only for PowerNV */
+ r = !!cpu_has_feature(CPU_FTR_HVMODE);
+ break;
+#endif
case KVM_CAP_PPC_ALLOC_HTAB:
r = hv_enabled;
@@ -753,6 +759,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
else
kvmppc_xics_free_icp(vcpu);
break;
+ case KVMPPC_IRQ_XIVE:
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ break;
}
kvmppc_core_vcpu_free(vcpu);
@@ -1941,6 +1950,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
break;
}
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE: {
+ struct fd f;
+ struct kvm_device *dev;
+
+ r = -EBADF;
+ f = fdget(cap->args[0]);
+ if (!f.file)
+ break;
+
+ r = -ENXIO;
+ if (!xive_enabled())
+ break;
+
+ r = -EPERM;
+ dev = kvm_device_from_filp(f.file);
+ if (dev)
+ r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
+ cap->args[1]);
+
+ fdput(f);
+ break;
+ }
+#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
case KVM_CAP_PPC_FWNMI:
r = -EINVAL;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 356156f5c52d..1db1435769b4 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -4458,6 +4458,15 @@ struct kvm_sync_regs {
struct kvm_vcpu_events events;
};
+6.75 KVM_CAP_PPC_IRQ_XIVE
+
+Architectures: ppc
+Target: vcpu
+Parameters: args[0] is the XIVE device fd
+ args[1] is the XIVE CPU number (server ID) for this vcpu
+
+This capability connects the vcpu to an in-kernel XIVE device.
+
7. Capabilities that can be enabled on VMs
------------------------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The user interface exposes a new capability to let QEMU connect the
vCPU to the XIVE KVM device if required. The capability is only
advertised on a PowerNV Hypervisor as support for nested guests
(pseries KVM Hypervisor) is not yet available.
Internally, the interface to the new KVM device is protected with a
new interrupt mode: KVMPPC_IRQ_XIVE.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 13 +++
arch/powerpc/kvm/book3s_xive.h | 6 ++
include/uapi/linux/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 67 +++++++-----
arch/powerpc/kvm/book3s_xive_native.c | 144 ++++++++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 33 ++++++
Documentation/virtual/kvm/api.txt | 9 ++
8 files changed, 246 insertions(+), 28 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 9f75a75a07f2..eb8581be0ee8 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -448,6 +448,7 @@ struct kvmppc_passthru_irqmap {
#define KVMPPC_IRQ_DEFAULT 0
#define KVMPPC_IRQ_MPIC 1
#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
+#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
#define MMIO_HPTE_CACHE_SIZE 4
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 4b72ddde7dc1..1e61877fe147 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -594,6 +594,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+{
+ return vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+}
+
+extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu);
+extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
@@ -621,6 +629,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+ { return 0; }
+static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
+static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index a08ae6fd4c51..bcb1bbcf0359 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -248,5 +248,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+/*
+ * Common Xive routines for XICS-over-XIVE and XIVE native
+ */
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e6368163d3a0..52bf74a1616e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_ARM_VM_IPA_SIZE 165
#define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
#define KVM_CAP_HYPERV_CPUID 167
+#define KVM_CAP_PPC_IRQ_XIVE 168
#ifdef KVM_CAP_IRQ_ROUTING
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index f78d002f0fe0..d1cc18a5b1c4 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1049,7 +1049,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
}
EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
-static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct kvm *kvm = vcpu->kvm;
@@ -1883,6 +1883,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
return 0;
}
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int i;
+
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+ u32 i0, i1, idx;
+
+ if (!q->qpage && !xc->esc_virq[i])
+ continue;
+
+ seq_printf(m, " [q%d]: ", i);
+
+ if (q->qpage) {
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
+ i0, i1);
+ }
+ if (xc->esc_virq[i]) {
+ struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
+ struct xive_irq_data *xd + irq_data_get_irq_handler_data(d);
+ u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
+
+ seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
+ (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
+ (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
+ xc->esc_virq[i], pq, xd->eoi_page);
+ seq_puts(m, "\n");
+ }
+ }
+ return 0;
+}
static int xive_debug_show(struct seq_file *m, void *private)
{
@@ -1908,7 +1945,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
kvm_for_each_vcpu(i, vcpu, kvm) {
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- unsigned int i;
if (!xc)
continue;
@@ -1918,33 +1954,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
xc->server_num, xc->cppr, xc->hw_cppr,
xc->mfrr, xc->pending,
xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
- for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
- struct xive_q *q = &xc->queues[i];
- u32 i0, i1, idx;
- if (!q->qpage && !xc->esc_virq[i])
- continue;
-
- seq_printf(m, " [q%d]: ", i);
-
- if (q->qpage) {
- idx = q->idx;
- i0 = be32_to_cpup(q->qpage + idx);
- idx = (idx + 1) & q->msk;
- i1 = be32_to_cpup(q->qpage + idx);
- seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
- }
- if (xc->esc_virq[i]) {
- struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
- struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
- u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
- seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
- (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
- (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
- xc->esc_virq[i], pq, xd->eoi_page);
- seq_printf(m, "\n");
- }
- }
+ kvmppc_xive_debug_show_queues(m, vcpu);
t_rm_h_xirr += xc->stat_rm_h_xirr;
t_rm_h_ipoll += xc->stat_rm_h_ipoll;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index e475ce83ad14..1f3da47a4a6a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -31,6 +31,128 @@
#include "book3s_xive.h"
+static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q = &xc->queues[prio];
+
+ xive_native_disable_queue(xc->vp_id, q, prio);
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+}
+
+void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ int i;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
+ pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
+
+ /* Ensure no interrupt is still routed to that VP */
+ xc->valid = false;
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ /* Disable the VP */
+ xive_native_disable_vp(xc->vp_id);
+
+ /* Free the queues & associated interrupts */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ /* Free the escalation irq */
+ if (xc->esc_virq[i]) {
+ free_irq(xc->esc_virq[i], vcpu);
+ irq_dispose_mapping(xc->esc_virq[i]);
+ kfree(xc->esc_virq_names[i]);
+ xc->esc_virq[i] = 0;
+ }
+
+ /* Free the queue */
+ kvmppc_xive_native_cleanup_queue(vcpu, i);
+ }
+
+ /* Free the VP */
+ kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
+}
+
+int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_vcpu *xc;
+ int rc;
+
+ pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
+
+ if (dev->ops != &kvm_xive_native_ops) {
+ pr_devel("Wrong ops !\n");
+ return -EPERM;
+ }
+ if (xive->kvm != vcpu->kvm)
+ return -EPERM;
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
+ return -EBUSY;
+ if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
+ pr_devel("Duplicate !\n");
+ return -EEXIST;
+ }
+ if (cpu >= KVM_MAX_VCPUS) {
+ pr_devel("Out of bounds !\n");
+ return -EINVAL;
+ }
+ xc = kzalloc(sizeof(*xc), GFP_KERNEL);
+ if (!xc)
+ return -ENOMEM;
+
+ mutex_lock(&vcpu->kvm->lock);
+ vcpu->arch.xive_vcpu = xc;
+ xc->xive = xive;
+ xc->vcpu = vcpu;
+ xc->server_num = cpu;
+ xc->vp_id = xive->vp_base + cpu;
+ xc->valid = true;
+
+ rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
+ if (rc) {
+ pr_err("Failed to get VP info from OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /*
+ * Enable the VP first as the single escalation mode will
+ * affect escalation interrupts numbering
+ */
+ rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
+ if (rc) {
+ pr_err("Failed to enable VP in OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /* Configure VCPU fields for use by assembly push/pull */
+ vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
+ vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
+
+ /* TODO: initialize queues ? */
+
+bail:
+ vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+ mutex_unlock(&vcpu->kvm->lock);
+ if (rc)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -126,10 +248,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
if (!kvm)
return 0;
+ seq_puts(m, "=====\nVCPU state\n=====\n");
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
+ xc->server_num,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word);
+
+ kvmppc_xive_debug_show_queues(m, vcpu);
+ }
+
return 0;
}
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 8c69af10f91d..a38a643a24dd 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_PPC_GET_CPU_CHAR:
r = 1;
break;
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE:
+ /* only for PowerNV */
+ r = !!cpu_has_feature(CPU_FTR_HVMODE);
+ break;
+#endif
case KVM_CAP_PPC_ALLOC_HTAB:
r = hv_enabled;
@@ -753,6 +759,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
else
kvmppc_xics_free_icp(vcpu);
break;
+ case KVMPPC_IRQ_XIVE:
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ break;
}
kvmppc_core_vcpu_free(vcpu);
@@ -1941,6 +1950,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
break;
}
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE: {
+ struct fd f;
+ struct kvm_device *dev;
+
+ r = -EBADF;
+ f = fdget(cap->args[0]);
+ if (!f.file)
+ break;
+
+ r = -ENXIO;
+ if (!xive_enabled())
+ break;
+
+ r = -EPERM;
+ dev = kvm_device_from_filp(f.file);
+ if (dev)
+ r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
+ cap->args[1]);
+
+ fdput(f);
+ break;
+ }
+#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
case KVM_CAP_PPC_FWNMI:
r = -EINVAL;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 356156f5c52d..1db1435769b4 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -4458,6 +4458,15 @@ struct kvm_sync_regs {
struct kvm_vcpu_events events;
};
+6.75 KVM_CAP_PPC_IRQ_XIVE
+
+Architectures: ppc
+Target: vcpu
+Parameters: args[0] is the XIVE device fd
+ args[1] is the XIVE CPU number (server ID) for this vcpu
+
+This capability connects the vcpu to an in-kernel XIVE device.
+
7. Capabilities that can be enabled on VMs
------------------------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 0:35 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 0:35 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 15259 bytes --]
On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> The user interface exposes a new capability to let QEMU connect the
> vCPU to the XIVE KVM device if required. The capability is only
> advertised on a PowerNV Hypervisor as support for nested guests
> (pseries KVM Hypervisor) is not yet available.
>
> Internally, the interface to the new KVM device is protected with a
> new interrupt mode: KVMPPC_IRQ_XIVE.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/kvm_host.h | 1 +
> arch/powerpc/include/asm/kvm_ppc.h | 13 +++
> arch/powerpc/kvm/book3s_xive.h | 6 ++
> include/uapi/linux/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive.c | 67 +++++++-----
> arch/powerpc/kvm/book3s_xive_native.c | 144 ++++++++++++++++++++++++++
> arch/powerpc/kvm/powerpc.c | 33 ++++++
> Documentation/virtual/kvm/api.txt | 9 ++
> 8 files changed, 246 insertions(+), 28 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 9f75a75a07f2..eb8581be0ee8 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -448,6 +448,7 @@ struct kvmppc_passthru_irqmap {
> #define KVMPPC_IRQ_DEFAULT 0
> #define KVMPPC_IRQ_MPIC 1
> #define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
> +#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
>
> #define MMIO_HPTE_CACHE_SIZE 4
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index 4b72ddde7dc1..1e61877fe147 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -594,6 +594,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status);
> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>
> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
> +{
> + return vcpu->arch.irq_type == KVMPPC_IRQ_XIVE;
> +}
> +
> +extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu);
> +extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
> extern void kvmppc_xive_native_init_module(void);
> extern void kvmppc_xive_native_exit_module(void);
>
> @@ -621,6 +629,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
> int level, bool line_status) { return -ENODEV; }
> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>
> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
> + { return 0; }
> +static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
> +static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
> static inline void kvmppc_xive_native_init_module(void) { }
> static inline void kvmppc_xive_native_exit_module(void) { }
>
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index a08ae6fd4c51..bcb1bbcf0359 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -248,5 +248,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
> extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
> extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
>
> +/*
> + * Common Xive routines for XICS-over-XIVE and XIVE native
> + */
> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> +
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index e6368163d3a0..52bf74a1616e 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
> #define KVM_CAP_ARM_VM_IPA_SIZE 165
> #define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
> #define KVM_CAP_HYPERV_CPUID 167
> +#define KVM_CAP_PPC_IRQ_XIVE 168
>
> #ifdef KVM_CAP_IRQ_ROUTING
>
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index f78d002f0fe0..d1cc18a5b1c4 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1049,7 +1049,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
> }
> EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
>
> -static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct kvm *kvm = vcpu->kvm;
> @@ -1883,6 +1883,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
> return 0;
> }
>
> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + unsigned int i;
> +
> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> + struct xive_q *q = &xc->queues[i];
> + u32 i0, i1, idx;
> +
> + if (!q->qpage && !xc->esc_virq[i])
> + continue;
> +
> + seq_printf(m, " [q%d]: ", i);
> +
> + if (q->qpage) {
> + idx = q->idx;
> + i0 = be32_to_cpup(q->qpage + idx);
> + idx = (idx + 1) & q->msk;
> + i1 = be32_to_cpup(q->qpage + idx);
> + seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
> + i0, i1);
> + }
> + if (xc->esc_virq[i]) {
> + struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
> + struct xive_irq_data *xd =
> + irq_data_get_irq_handler_data(d);
> + u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
> +
> + seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
> + (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
> + (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
> + xc->esc_virq[i], pq, xd->eoi_page);
> + seq_puts(m, "\n");
> + }
> + }
> + return 0;
> +}
>
> static int xive_debug_show(struct seq_file *m, void *private)
> {
> @@ -1908,7 +1945,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
>
> kvm_for_each_vcpu(i, vcpu, kvm) {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - unsigned int i;
>
> if (!xc)
> continue;
> @@ -1918,33 +1954,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
> xc->server_num, xc->cppr, xc->hw_cppr,
> xc->mfrr, xc->pending,
> xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
> - for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> - struct xive_q *q = &xc->queues[i];
> - u32 i0, i1, idx;
>
> - if (!q->qpage && !xc->esc_virq[i])
> - continue;
> -
> - seq_printf(m, " [q%d]: ", i);
> -
> - if (q->qpage) {
> - idx = q->idx;
> - i0 = be32_to_cpup(q->qpage + idx);
> - idx = (idx + 1) & q->msk;
> - i1 = be32_to_cpup(q->qpage + idx);
> - seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
> - }
> - if (xc->esc_virq[i]) {
> - struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
> - struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> - u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
> - seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
> - (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
> - (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
> - xc->esc_virq[i], pq, xd->eoi_page);
> - seq_printf(m, "\n");
> - }
> - }
> + kvmppc_xive_debug_show_queues(m, vcpu);
>
> t_rm_h_xirr += xc->stat_rm_h_xirr;
> t_rm_h_ipoll += xc->stat_rm_h_ipoll;
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index e475ce83ad14..1f3da47a4a6a 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -31,6 +31,128 @@
>
> #include "book3s_xive.h"
>
> +static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct xive_q *q = &xc->queues[prio];
> +
> + xive_native_disable_queue(xc->vp_id, q, prio);
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +}
> +
> +void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + int i;
> +
> + if (!kvmppc_xive_enabled(vcpu))
> + return;
> +
> + if (!xc)
> + return;
> +
> + pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
> +
> + /* Ensure no interrupt is still routed to that VP */
> + xc->valid = false;
> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
> +
> + /* Disable the VP */
> + xive_native_disable_vp(xc->vp_id);
> +
> + /* Free the queues & associated interrupts */
> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> + /* Free the escalation irq */
> + if (xc->esc_virq[i]) {
> + free_irq(xc->esc_virq[i], vcpu);
> + irq_dispose_mapping(xc->esc_virq[i]);
> + kfree(xc->esc_virq_names[i]);
> + xc->esc_virq[i] = 0;
> + }
> +
> + /* Free the queue */
> + kvmppc_xive_native_cleanup_queue(vcpu, i);
> + }
> +
> + /* Free the VP */
> + kfree(xc);
> +
> + /* Cleanup the vcpu */
> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
> + vcpu->arch.xive_vcpu = NULL;
> +}
> +
> +int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu)
> +{
> + struct kvmppc_xive *xive = dev->private;
> + struct kvmppc_xive_vcpu *xc;
> + int rc;
> +
> + pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
> +
> + if (dev->ops != &kvm_xive_native_ops) {
> + pr_devel("Wrong ops !\n");
> + return -EPERM;
> + }
> + if (xive->kvm != vcpu->kvm)
> + return -EPERM;
> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
> + return -EBUSY;
> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
You haven't taken the kvm->lock yet, so couldn't a race mean a
duplicate server gets inserted after you make this check?
> + pr_devel("Duplicate !\n");
> + return -EEXIST;
> + }
> + if (cpu >= KVM_MAX_VCPUS) {
> + pr_devel("Out of bounds !\n");
> + return -EINVAL;
> + }
> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
> + if (!xc)
> + return -ENOMEM;
> +
> + mutex_lock(&vcpu->kvm->lock);
> + vcpu->arch.xive_vcpu = xc;
Similarly you don't verify this is NULL after taking the lock, so
couldn't another thread race and make a connect which gets clobbered
here?
> + xc->xive = xive;
> + xc->vcpu = vcpu;
> + xc->server_num = cpu;
> + xc->vp_id = xive->vp_base + cpu;
Hrm. This ties the internal VP id to the userspace chosen server
number, which isn't ideal. It puts a constraint on those server
numbers that you wouldn't otherwise have.
> + xc->valid = true;
> +
> + rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
> + if (rc) {
> + pr_err("Failed to get VP info from OPAL: %d\n", rc);
> + goto bail;
> + }
> +
> + /*
> + * Enable the VP first as the single escalation mode will
> + * affect escalation interrupts numbering
> + */
> + rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
> + if (rc) {
> + pr_err("Failed to enable VP in OPAL: %d\n", rc);
> + goto bail;
> + }
> +
> + /* Configure VCPU fields for use by assembly push/pull */
> + vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
> + vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
> +
> + /* TODO: initialize queues ? */
> +
> +bail:
> + vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
> + mutex_unlock(&vcpu->kvm->lock);
> + if (rc)
> + kvmppc_xive_native_cleanup_vcpu(vcpu);
> +
> + return rc;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -126,10 +248,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
> {
> struct kvmppc_xive *xive = m->private;
> struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + unsigned int i;
>
> if (!kvm)
> return 0;
>
> + seq_puts(m, "=========\nVCPU state\n=========\n");
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> +
> + if (!xc)
> + continue;
> +
> + seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
> + xc->server_num,
> + vcpu->arch.xive_saved_state.nsr,
> + vcpu->arch.xive_saved_state.cppr,
> + vcpu->arch.xive_saved_state.ipb,
> + vcpu->arch.xive_saved_state.pipr,
> + vcpu->arch.xive_saved_state.w01,
> + (u32) vcpu->arch.xive_cam_word);
> +
> + kvmppc_xive_debug_show_queues(m, vcpu);
> + }
> +
> return 0;
> }
>
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 8c69af10f91d..a38a643a24dd 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> case KVM_CAP_PPC_GET_CPU_CHAR:
> r = 1;
> break;
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_CAP_PPC_IRQ_XIVE:
> + /* only for PowerNV */
> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
> + break;
> +#endif
>
> case KVM_CAP_PPC_ALLOC_HTAB:
> r = hv_enabled;
> @@ -753,6 +759,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
> else
> kvmppc_xics_free_icp(vcpu);
> break;
> + case KVMPPC_IRQ_XIVE:
> + kvmppc_xive_native_cleanup_vcpu(vcpu);
> + break;
> }
>
> kvmppc_core_vcpu_free(vcpu);
> @@ -1941,6 +1950,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
> break;
> }
> #endif /* CONFIG_KVM_XICS */
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_CAP_PPC_IRQ_XIVE: {
> + struct fd f;
> + struct kvm_device *dev;
> +
> + r = -EBADF;
> + f = fdget(cap->args[0]);
> + if (!f.file)
> + break;
> +
> + r = -ENXIO;
> + if (!xive_enabled())
> + break;
> +
> + r = -EPERM;
> + dev = kvm_device_from_filp(f.file);
> + if (dev)
> + r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
> + cap->args[1]);
> +
> + fdput(f);
> + break;
> + }
> +#endif /* CONFIG_KVM_XIVE */
> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> case KVM_CAP_PPC_FWNMI:
> r = -EINVAL;
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 356156f5c52d..1db1435769b4 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -4458,6 +4458,15 @@ struct kvm_sync_regs {
> struct kvm_vcpu_events events;
> };
>
> +6.75 KVM_CAP_PPC_IRQ_XIVE
> +
> +Architectures: ppc
> +Target: vcpu
> +Parameters: args[0] is the XIVE device fd
> + args[1] is the XIVE CPU number (server ID) for this vcpu
> +
> +This capability connects the vcpu to an in-kernel XIVE device.
> +
> 7. Capabilities that can be enabled on VMs
> ------------------------------------------
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-02-25 0:35 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 0:35 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 15259 bytes --]
On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> The user interface exposes a new capability to let QEMU connect the
> vCPU to the XIVE KVM device if required. The capability is only
> advertised on a PowerNV Hypervisor as support for nested guests
> (pseries KVM Hypervisor) is not yet available.
>
> Internally, the interface to the new KVM device is protected with a
> new interrupt mode: KVMPPC_IRQ_XIVE.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/kvm_host.h | 1 +
> arch/powerpc/include/asm/kvm_ppc.h | 13 +++
> arch/powerpc/kvm/book3s_xive.h | 6 ++
> include/uapi/linux/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive.c | 67 +++++++-----
> arch/powerpc/kvm/book3s_xive_native.c | 144 ++++++++++++++++++++++++++
> arch/powerpc/kvm/powerpc.c | 33 ++++++
> Documentation/virtual/kvm/api.txt | 9 ++
> 8 files changed, 246 insertions(+), 28 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
> index 9f75a75a07f2..eb8581be0ee8 100644
> --- a/arch/powerpc/include/asm/kvm_host.h
> +++ b/arch/powerpc/include/asm/kvm_host.h
> @@ -448,6 +448,7 @@ struct kvmppc_passthru_irqmap {
> #define KVMPPC_IRQ_DEFAULT 0
> #define KVMPPC_IRQ_MPIC 1
> #define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
> +#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
>
> #define MMIO_HPTE_CACHE_SIZE 4
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index 4b72ddde7dc1..1e61877fe147 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -594,6 +594,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
> int level, bool line_status);
> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>
> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
> +{
> + return vcpu->arch.irq_type == KVMPPC_IRQ_XIVE;
> +}
> +
> +extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu);
> +extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
> extern void kvmppc_xive_native_init_module(void);
> extern void kvmppc_xive_native_exit_module(void);
>
> @@ -621,6 +629,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
> int level, bool line_status) { return -ENODEV; }
> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>
> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
> + { return 0; }
> +static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
> +static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
> static inline void kvmppc_xive_native_init_module(void) { }
> static inline void kvmppc_xive_native_exit_module(void) { }
>
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index a08ae6fd4c51..bcb1bbcf0359 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -248,5 +248,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
> extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
> extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
>
> +/*
> + * Common Xive routines for XICS-over-XIVE and XIVE native
> + */
> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> +
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index e6368163d3a0..52bf74a1616e 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
> #define KVM_CAP_ARM_VM_IPA_SIZE 165
> #define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
> #define KVM_CAP_HYPERV_CPUID 167
> +#define KVM_CAP_PPC_IRQ_XIVE 168
>
> #ifdef KVM_CAP_IRQ_ROUTING
>
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index f78d002f0fe0..d1cc18a5b1c4 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1049,7 +1049,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
> }
> EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
>
> -static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct kvm *kvm = vcpu->kvm;
> @@ -1883,6 +1883,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
> return 0;
> }
>
> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + unsigned int i;
> +
> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> + struct xive_q *q = &xc->queues[i];
> + u32 i0, i1, idx;
> +
> + if (!q->qpage && !xc->esc_virq[i])
> + continue;
> +
> + seq_printf(m, " [q%d]: ", i);
> +
> + if (q->qpage) {
> + idx = q->idx;
> + i0 = be32_to_cpup(q->qpage + idx);
> + idx = (idx + 1) & q->msk;
> + i1 = be32_to_cpup(q->qpage + idx);
> + seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
> + i0, i1);
> + }
> + if (xc->esc_virq[i]) {
> + struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
> + struct xive_irq_data *xd =
> + irq_data_get_irq_handler_data(d);
> + u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
> +
> + seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
> + (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
> + (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
> + xc->esc_virq[i], pq, xd->eoi_page);
> + seq_puts(m, "\n");
> + }
> + }
> + return 0;
> +}
>
> static int xive_debug_show(struct seq_file *m, void *private)
> {
> @@ -1908,7 +1945,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
>
> kvm_for_each_vcpu(i, vcpu, kvm) {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - unsigned int i;
>
> if (!xc)
> continue;
> @@ -1918,33 +1954,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
> xc->server_num, xc->cppr, xc->hw_cppr,
> xc->mfrr, xc->pending,
> xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
> - for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> - struct xive_q *q = &xc->queues[i];
> - u32 i0, i1, idx;
>
> - if (!q->qpage && !xc->esc_virq[i])
> - continue;
> -
> - seq_printf(m, " [q%d]: ", i);
> -
> - if (q->qpage) {
> - idx = q->idx;
> - i0 = be32_to_cpup(q->qpage + idx);
> - idx = (idx + 1) & q->msk;
> - i1 = be32_to_cpup(q->qpage + idx);
> - seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
> - }
> - if (xc->esc_virq[i]) {
> - struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
> - struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
> - u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
> - seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
> - (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
> - (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
> - xc->esc_virq[i], pq, xd->eoi_page);
> - seq_printf(m, "\n");
> - }
> - }
> + kvmppc_xive_debug_show_queues(m, vcpu);
>
> t_rm_h_xirr += xc->stat_rm_h_xirr;
> t_rm_h_ipoll += xc->stat_rm_h_ipoll;
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index e475ce83ad14..1f3da47a4a6a 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -31,6 +31,128 @@
>
> #include "book3s_xive.h"
>
> +static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct xive_q *q = &xc->queues[prio];
> +
> + xive_native_disable_queue(xc->vp_id, q, prio);
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +}
> +
> +void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + int i;
> +
> + if (!kvmppc_xive_enabled(vcpu))
> + return;
> +
> + if (!xc)
> + return;
> +
> + pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
> +
> + /* Ensure no interrupt is still routed to that VP */
> + xc->valid = false;
> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
> +
> + /* Disable the VP */
> + xive_native_disable_vp(xc->vp_id);
> +
> + /* Free the queues & associated interrupts */
> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
> + /* Free the escalation irq */
> + if (xc->esc_virq[i]) {
> + free_irq(xc->esc_virq[i], vcpu);
> + irq_dispose_mapping(xc->esc_virq[i]);
> + kfree(xc->esc_virq_names[i]);
> + xc->esc_virq[i] = 0;
> + }
> +
> + /* Free the queue */
> + kvmppc_xive_native_cleanup_queue(vcpu, i);
> + }
> +
> + /* Free the VP */
> + kfree(xc);
> +
> + /* Cleanup the vcpu */
> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
> + vcpu->arch.xive_vcpu = NULL;
> +}
> +
> +int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> + struct kvm_vcpu *vcpu, u32 cpu)
> +{
> + struct kvmppc_xive *xive = dev->private;
> + struct kvmppc_xive_vcpu *xc;
> + int rc;
> +
> + pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
> +
> + if (dev->ops != &kvm_xive_native_ops) {
> + pr_devel("Wrong ops !\n");
> + return -EPERM;
> + }
> + if (xive->kvm != vcpu->kvm)
> + return -EPERM;
> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
> + return -EBUSY;
> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
You haven't taken the kvm->lock yet, so couldn't a race mean a
duplicate server gets inserted after you make this check?
> + pr_devel("Duplicate !\n");
> + return -EEXIST;
> + }
> + if (cpu >= KVM_MAX_VCPUS) {
> + pr_devel("Out of bounds !\n");
> + return -EINVAL;
> + }
> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
> + if (!xc)
> + return -ENOMEM;
> +
> + mutex_lock(&vcpu->kvm->lock);
> + vcpu->arch.xive_vcpu = xc;
Similarly you don't verify this is NULL after taking the lock, so
couldn't another thread race and make a connect which gets clobbered
here?
> + xc->xive = xive;
> + xc->vcpu = vcpu;
> + xc->server_num = cpu;
> + xc->vp_id = xive->vp_base + cpu;
Hrm. This ties the internal VP id to the userspace chosen server
number, which isn't ideal. It puts a constraint on those server
numbers that you wouldn't otherwise have.
> + xc->valid = true;
> +
> + rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
> + if (rc) {
> + pr_err("Failed to get VP info from OPAL: %d\n", rc);
> + goto bail;
> + }
> +
> + /*
> + * Enable the VP first as the single escalation mode will
> + * affect escalation interrupts numbering
> + */
> + rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
> + if (rc) {
> + pr_err("Failed to enable VP in OPAL: %d\n", rc);
> + goto bail;
> + }
> +
> + /* Configure VCPU fields for use by assembly push/pull */
> + vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
> + vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
> +
> + /* TODO: initialize queues ? */
> +
> +bail:
> + vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
> + mutex_unlock(&vcpu->kvm->lock);
> + if (rc)
> + kvmppc_xive_native_cleanup_vcpu(vcpu);
> +
> + return rc;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -126,10 +248,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
> {
> struct kvmppc_xive *xive = m->private;
> struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + unsigned int i;
>
> if (!kvm)
> return 0;
>
> + seq_puts(m, "=========\nVCPU state\n=========\n");
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> +
> + if (!xc)
> + continue;
> +
> + seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
> + xc->server_num,
> + vcpu->arch.xive_saved_state.nsr,
> + vcpu->arch.xive_saved_state.cppr,
> + vcpu->arch.xive_saved_state.ipb,
> + vcpu->arch.xive_saved_state.pipr,
> + vcpu->arch.xive_saved_state.w01,
> + (u32) vcpu->arch.xive_cam_word);
> +
> + kvmppc_xive_debug_show_queues(m, vcpu);
> + }
> +
> return 0;
> }
>
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 8c69af10f91d..a38a643a24dd 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> case KVM_CAP_PPC_GET_CPU_CHAR:
> r = 1;
> break;
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_CAP_PPC_IRQ_XIVE:
> + /* only for PowerNV */
> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
> + break;
> +#endif
>
> case KVM_CAP_PPC_ALLOC_HTAB:
> r = hv_enabled;
> @@ -753,6 +759,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
> else
> kvmppc_xics_free_icp(vcpu);
> break;
> + case KVMPPC_IRQ_XIVE:
> + kvmppc_xive_native_cleanup_vcpu(vcpu);
> + break;
> }
>
> kvmppc_core_vcpu_free(vcpu);
> @@ -1941,6 +1950,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
> break;
> }
> #endif /* CONFIG_KVM_XICS */
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_CAP_PPC_IRQ_XIVE: {
> + struct fd f;
> + struct kvm_device *dev;
> +
> + r = -EBADF;
> + f = fdget(cap->args[0]);
> + if (!f.file)
> + break;
> +
> + r = -ENXIO;
> + if (!xive_enabled())
> + break;
> +
> + r = -EPERM;
> + dev = kvm_device_from_filp(f.file);
> + if (dev)
> + r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
> + cap->args[1]);
> +
> + fdput(f);
> + break;
> + }
> +#endif /* CONFIG_KVM_XIVE */
> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
> case KVM_CAP_PPC_FWNMI:
> r = -EINVAL;
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 356156f5c52d..1db1435769b4 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -4458,6 +4458,15 @@ struct kvm_sync_regs {
> struct kvm_vcpu_events events;
> };
>
> +6.75 KVM_CAP_PPC_IRQ_XIVE
> +
> +Architectures: ppc
> +Target: vcpu
> +Parameters: args[0] is the XIVE device fd
> + args[1] is the XIVE CPU number (server ID) for this vcpu
> +
> +This capability connects the vcpu to an in-kernel XIVE device.
> +
> 7. Capabilities that can be enabled on VMs
> ------------------------------------------
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-25 0:35 ` David Gibson
@ 2019-02-25 4:59 ` Paul Mackerras
-1 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-25 4:59 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Cédric Le Goater, linuxppc-dev
On Mon, Feb 25, 2019 at 11:35:27AM +1100, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> > + xc->xive = xive;
> > + xc->vcpu = vcpu;
> > + xc->server_num = cpu;
> > + xc->vp_id = xive->vp_base + cpu;
>
> Hrm. This ties the internal VP id to the userspace chosen server
> number, which isn't ideal. It puts a constraint on those server
> numbers that you wouldn't otherwise have.
We should probably do the same as the xics-on-xive code, which is to
put the server number through kvmppc_pack_vcpu_id(), which is a
folding function that maps the QEMU vcpu id (which is the server
number) down to the range 0..KVM_MAX_VCPUS-1, and works for the
allocation patterns used in the various vSMT modes.
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-02-25 4:59 ` Paul Mackerras
0 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-25 4:59 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Cédric Le Goater, linuxppc-dev
On Mon, Feb 25, 2019 at 11:35:27AM +1100, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> > + xc->xive = xive;
> > + xc->vcpu = vcpu;
> > + xc->server_num = cpu;
> > + xc->vp_id = xive->vp_base + cpu;
>
> Hrm. This ties the internal VP id to the userspace chosen server
> number, which isn't ideal. It puts a constraint on those server
> numbers that you wouldn't otherwise have.
We should probably do the same as the xics-on-xive code, which is to
put the server number through kvmppc_pack_vcpu_id(), which is a
folding function that maps the QEMU vcpu id (which is the server
number) down to the range 0..KVM_MAX_VCPUS-1, and works for the
allocation patterns used in the various vSMT modes.
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-25 4:59 ` Paul Mackerras
@ 2019-03-12 14:10 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 14:10 UTC (permalink / raw)
To: Paul Mackerras, David Gibson; +Cc: linuxppc-dev, kvm, kvm-ppc
On 2/25/19 5:59 AM, Paul Mackerras wrote:
> On Mon, Feb 25, 2019 at 11:35:27AM +1100, David Gibson wrote:
>> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
>>> + xc->xive = xive;
>>> + xc->vcpu = vcpu;
>>> + xc->server_num = cpu;
>>> + xc->vp_id = xive->vp_base + cpu;
>>
>> Hrm. This ties the internal VP id to the userspace chosen server
>> number, which isn't ideal. It puts a constraint on those server
>> numbers that you wouldn't otherwise have.
>
> We should probably do the same as the xics-on-xive code, which is to
> put the server number through kvmppc_pack_vcpu_id(), which is a
> folding function that maps the QEMU vcpu id (which is the server
> number) down to the range 0..KVM_MAX_VCPUS-1, and works for the
> allocation patterns used in the various vSMT modes.
yes. I will see how it goes.
Thanks,
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-03-12 14:10 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 14:10 UTC (permalink / raw)
To: Paul Mackerras, David Gibson; +Cc: linuxppc-dev, kvm, kvm-ppc
On 2/25/19 5:59 AM, Paul Mackerras wrote:
> On Mon, Feb 25, 2019 at 11:35:27AM +1100, David Gibson wrote:
>> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
>>> + xc->xive = xive;
>>> + xc->vcpu = vcpu;
>>> + xc->server_num = cpu;
>>> + xc->vp_id = xive->vp_base + cpu;
>>
>> Hrm. This ties the internal VP id to the userspace chosen server
>> number, which isn't ideal. It puts a constraint on those server
>> numbers that you wouldn't otherwise have.
>
> We should probably do the same as the xics-on-xive code, which is to
> put the server number through kvmppc_pack_vcpu_id(), which is a
> folding function that maps the QEMU vcpu id (which is the server
> number) down to the range 0..KVM_MAX_VCPUS-1, and works for the
> allocation patterns used in the various vSMT modes.
yes. I will see how it goes.
Thanks,
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-25 0:35 ` David Gibson
@ 2019-03-12 14:03 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 14:03 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 1:35 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
>> The user interface exposes a new capability to let QEMU connect the
>> vCPU to the XIVE KVM device if required. The capability is only
>> advertised on a PowerNV Hypervisor as support for nested guests
>> (pseries KVM Hypervisor) is not yet available.
>>
>> Internally, the interface to the new KVM device is protected with a
>> new interrupt mode: KVMPPC_IRQ_XIVE.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/kvm_host.h | 1 +
>> arch/powerpc/include/asm/kvm_ppc.h | 13 +++
>> arch/powerpc/kvm/book3s_xive.h | 6 ++
>> include/uapi/linux/kvm.h | 1 +
>> arch/powerpc/kvm/book3s_xive.c | 67 +++++++-----
>> arch/powerpc/kvm/book3s_xive_native.c | 144 ++++++++++++++++++++++++++
>> arch/powerpc/kvm/powerpc.c | 33 ++++++
>> Documentation/virtual/kvm/api.txt | 9 ++
>> 8 files changed, 246 insertions(+), 28 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
>> index 9f75a75a07f2..eb8581be0ee8 100644
>> --- a/arch/powerpc/include/asm/kvm_host.h
>> +++ b/arch/powerpc/include/asm/kvm_host.h
>> @@ -448,6 +448,7 @@ struct kvmppc_passthru_irqmap {
>> #define KVMPPC_IRQ_DEFAULT 0
>> #define KVMPPC_IRQ_MPIC 1
>> #define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
>> +#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
>>
>> #define MMIO_HPTE_CACHE_SIZE 4
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index 4b72ddde7dc1..1e61877fe147 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -594,6 +594,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status);
>> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>>
>> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
>> +{
>> + return vcpu->arch.irq_type == KVMPPC_IRQ_XIVE;
>> +}
>> +
>> +extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> + struct kvm_vcpu *vcpu, u32 cpu);
>> +extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
>> extern void kvmppc_xive_native_init_module(void);
>> extern void kvmppc_xive_native_exit_module(void);
>>
>> @@ -621,6 +629,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
>> int level, bool line_status) { return -ENODEV; }
>> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>>
>> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
>> + { return 0; }
>> +static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> + struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
>> +static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
>> static inline void kvmppc_xive_native_init_module(void) { }
>> static inline void kvmppc_xive_native_exit_module(void) { }
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
>> index a08ae6fd4c51..bcb1bbcf0359 100644
>> --- a/arch/powerpc/kvm/book3s_xive.h
>> +++ b/arch/powerpc/kvm/book3s_xive.h
>> @@ -248,5 +248,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
>> extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
>> extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
>>
>> +/*
>> + * Common Xive routines for XICS-over-XIVE and XIVE native
>> + */
>> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
>> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
>> +
>> #endif /* CONFIG_KVM_XICS */
>> #endif /* _KVM_PPC_BOOK3S_XICS_H */
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index e6368163d3a0..52bf74a1616e 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
>> #define KVM_CAP_ARM_VM_IPA_SIZE 165
>> #define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
>> #define KVM_CAP_HYPERV_CPUID 167
>> +#define KVM_CAP_PPC_IRQ_XIVE 168
>>
>> #ifdef KVM_CAP_IRQ_ROUTING
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index f78d002f0fe0..d1cc18a5b1c4 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -1049,7 +1049,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
>> }
>> EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
>>
>> -static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> struct kvm *kvm = vcpu->kvm;
>> @@ -1883,6 +1883,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
>> return 0;
>> }
>>
>> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + unsigned int i;
>> +
>> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
>> + struct xive_q *q = &xc->queues[i];
>> + u32 i0, i1, idx;
>> +
>> + if (!q->qpage && !xc->esc_virq[i])
>> + continue;
>> +
>> + seq_printf(m, " [q%d]: ", i);
>> +
>> + if (q->qpage) {
>> + idx = q->idx;
>> + i0 = be32_to_cpup(q->qpage + idx);
>> + idx = (idx + 1) & q->msk;
>> + i1 = be32_to_cpup(q->qpage + idx);
>> + seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
>> + i0, i1);
>> + }
>> + if (xc->esc_virq[i]) {
>> + struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
>> + struct xive_irq_data *xd =
>> + irq_data_get_irq_handler_data(d);
>> + u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
>> +
>> + seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
>> + (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
>> + (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
>> + xc->esc_virq[i], pq, xd->eoi_page);
>> + seq_puts(m, "\n");
>> + }
>> + }
>> + return 0;
>> +}
>>
>> static int xive_debug_show(struct seq_file *m, void *private)
>> {
>> @@ -1908,7 +1945,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
>>
>> kvm_for_each_vcpu(i, vcpu, kvm) {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> - unsigned int i;
>>
>> if (!xc)
>> continue;
>> @@ -1918,33 +1954,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
>> xc->server_num, xc->cppr, xc->hw_cppr,
>> xc->mfrr, xc->pending,
>> xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
>> - for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
>> - struct xive_q *q = &xc->queues[i];
>> - u32 i0, i1, idx;
>>
>> - if (!q->qpage && !xc->esc_virq[i])
>> - continue;
>> -
>> - seq_printf(m, " [q%d]: ", i);
>> -
>> - if (q->qpage) {
>> - idx = q->idx;
>> - i0 = be32_to_cpup(q->qpage + idx);
>> - idx = (idx + 1) & q->msk;
>> - i1 = be32_to_cpup(q->qpage + idx);
>> - seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
>> - }
>> - if (xc->esc_virq[i]) {
>> - struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
>> - struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>> - u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
>> - seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
>> - (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
>> - (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
>> - xc->esc_virq[i], pq, xd->eoi_page);
>> - seq_printf(m, "\n");
>> - }
>> - }
>> + kvmppc_xive_debug_show_queues(m, vcpu);
>>
>> t_rm_h_xirr += xc->stat_rm_h_xirr;
>> t_rm_h_ipoll += xc->stat_rm_h_ipoll;
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index e475ce83ad14..1f3da47a4a6a 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -31,6 +31,128 @@
>>
>> #include "book3s_xive.h"
>>
>> +static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + struct xive_q *q = &xc->queues[prio];
>> +
>> + xive_native_disable_queue(xc->vp_id, q, prio);
>> + if (q->qpage) {
>> + put_page(virt_to_page(q->qpage));
>> + q->qpage = NULL;
>> + }
>> +}
>> +
>> +void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + int i;
>> +
>> + if (!kvmppc_xive_enabled(vcpu))
>> + return;
>> +
>> + if (!xc)
>> + return;
>> +
>> + pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
>> +
>> + /* Ensure no interrupt is still routed to that VP */
>> + xc->valid = false;
>> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
>> +
>> + /* Disable the VP */
>> + xive_native_disable_vp(xc->vp_id);
>> +
>> + /* Free the queues & associated interrupts */
>> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
>> + /* Free the escalation irq */
>> + if (xc->esc_virq[i]) {
>> + free_irq(xc->esc_virq[i], vcpu);
>> + irq_dispose_mapping(xc->esc_virq[i]);
>> + kfree(xc->esc_virq_names[i]);
>> + xc->esc_virq[i] = 0;
>> + }
>> +
>> + /* Free the queue */
>> + kvmppc_xive_native_cleanup_queue(vcpu, i);
>> + }
>> +
>> + /* Free the VP */
>> + kfree(xc);
>> +
>> + /* Cleanup the vcpu */
>> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
>> + vcpu->arch.xive_vcpu = NULL;
>> +}
>> +
>> +int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> + struct kvm_vcpu *vcpu, u32 cpu)
>> +{
>> + struct kvmppc_xive *xive = dev->private;
>> + struct kvmppc_xive_vcpu *xc;
>> + int rc;
>> +
>> + pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
>> +
>> + if (dev->ops != &kvm_xive_native_ops) {
>> + pr_devel("Wrong ops !\n");
>> + return -EPERM;
>> + }
>> + if (xive->kvm != vcpu->kvm)
>> + return -EPERM;
>> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
>> + return -EBUSY;
>> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
>
> You haven't taken the kvm->lock yet, so couldn't a race mean a
> duplicate server gets inserted after you make this check?
>
>> + pr_devel("Duplicate !\n");
>> + return -EEXIST;
>> + }
>> + if (cpu >= KVM_MAX_VCPUS) {
>> + pr_devel("Out of bounds !\n");
>> + return -EINVAL;
>> + }
>> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
>> + if (!xc)
>> + return -ENOMEM;
>> +
>> + mutex_lock(&vcpu->kvm->lock);
>> + vcpu->arch.xive_vcpu = xc;
>
> Similarly you don't verify this is NULL after taking the lock, so
> couldn't another thread race and make a connect which gets clobbered
> here?
Yes. this is not very safe ... We need to clean up all the KVM device
methods doing the connection of the presenter to the vCPU AFAICT.
I will fix the XIVE native one for now.
And also, this CPU parameter is useless. There is no reason to connect
a vCPU from another vCPU.
>> + xc->xive = xive;
>> + xc->vcpu = vcpu;
>> + xc->server_num = cpu;
>> + xc->vp_id = xive->vp_base + cpu;
>
> Hrm. This ties the internal VP id to the userspace chosen server
> number, which isn't ideal. It puts a constraint on those server
> numbers that you wouldn't otherwise have.
Ah yes. I should be using the kvmppc_pack_vcpu_id() like we do for
the XICS-over-XIVE device probably. I need to check that it is correct
in this mode.
Thanks,
C.
>> + xc->valid = true;
>> +
>> + rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
>> + if (rc) {
>> + pr_err("Failed to get VP info from OPAL: %d\n", rc);
>> + goto bail;
>> + }
>> +
>> + /*
>> + * Enable the VP first as the single escalation mode will
>> + * affect escalation interrupts numbering
>> + */
>> + rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
>> + if (rc) {
>> + pr_err("Failed to enable VP in OPAL: %d\n", rc);
>> + goto bail;
>> + }
>> +
>> + /* Configure VCPU fields for use by assembly push/pull */
>> + vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
>> + vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
>> +
>> + /* TODO: initialize queues ? */
>> +
>> +bail:
>> + vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
>> + mutex_unlock(&vcpu->kvm->lock);
>> + if (rc)
>> + kvmppc_xive_native_cleanup_vcpu(vcpu);
>> +
>> + return rc;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> @@ -126,10 +248,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
>> {
>> struct kvmppc_xive *xive = m->private;
>> struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + unsigned int i;
>>
>> if (!kvm)
>> return 0;
>>
>> + seq_puts(m, "=========\nVCPU state\n=========\n");
>> +
>> + kvm_for_each_vcpu(i, vcpu, kvm) {
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> +
>> + if (!xc)
>> + continue;
>> +
>> + seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
>> + xc->server_num,
>> + vcpu->arch.xive_saved_state.nsr,
>> + vcpu->arch.xive_saved_state.cppr,
>> + vcpu->arch.xive_saved_state.ipb,
>> + vcpu->arch.xive_saved_state.pipr,
>> + vcpu->arch.xive_saved_state.w01,
>> + (u32) vcpu->arch.xive_cam_word);
>> +
>> + kvmppc_xive_debug_show_queues(m, vcpu);
>> + }
>> +
>> return 0;
>> }
>>
>> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
>> index 8c69af10f91d..a38a643a24dd 100644
>> --- a/arch/powerpc/kvm/powerpc.c
>> +++ b/arch/powerpc/kvm/powerpc.c
>> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>> case KVM_CAP_PPC_GET_CPU_CHAR:
>> r = 1;
>> break;
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_CAP_PPC_IRQ_XIVE:
>> + /* only for PowerNV */
>> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
>> + break;
>> +#endif
>>
>> case KVM_CAP_PPC_ALLOC_HTAB:
>> r = hv_enabled;
>> @@ -753,6 +759,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
>> else
>> kvmppc_xics_free_icp(vcpu);
>> break;
>> + case KVMPPC_IRQ_XIVE:
>> + kvmppc_xive_native_cleanup_vcpu(vcpu);
>> + break;
>> }
>>
>> kvmppc_core_vcpu_free(vcpu);
>> @@ -1941,6 +1950,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>> break;
>> }
>> #endif /* CONFIG_KVM_XICS */
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_CAP_PPC_IRQ_XIVE: {
>> + struct fd f;
>> + struct kvm_device *dev;
>> +
>> + r = -EBADF;
>> + f = fdget(cap->args[0]);
>> + if (!f.file)
>> + break;
>> +
>> + r = -ENXIO;
>> + if (!xive_enabled())
>> + break;
>> +
>> + r = -EPERM;
>> + dev = kvm_device_from_filp(f.file);
>> + if (dev)
>> + r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
>> + cap->args[1]);
>> +
>> + fdput(f);
>> + break;
>> + }
>> +#endif /* CONFIG_KVM_XIVE */
>> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
>> case KVM_CAP_PPC_FWNMI:
>> r = -EINVAL;
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 356156f5c52d..1db1435769b4 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -4458,6 +4458,15 @@ struct kvm_sync_regs {
>> struct kvm_vcpu_events events;
>> };
>>
>> +6.75 KVM_CAP_PPC_IRQ_XIVE
>> +
>> +Architectures: ppc
>> +Target: vcpu
>> +Parameters: args[0] is the XIVE device fd
>> + args[1] is the XIVE CPU number (server ID) for this vcpu
>> +
>> +This capability connects the vcpu to an in-kernel XIVE device.
>> +
>> 7. Capabilities that can be enabled on VMs
>> ------------------------------------------
>>
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-03-12 14:03 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 14:03 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 1:35 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
>> The user interface exposes a new capability to let QEMU connect the
>> vCPU to the XIVE KVM device if required. The capability is only
>> advertised on a PowerNV Hypervisor as support for nested guests
>> (pseries KVM Hypervisor) is not yet available.
>>
>> Internally, the interface to the new KVM device is protected with a
>> new interrupt mode: KVMPPC_IRQ_XIVE.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/kvm_host.h | 1 +
>> arch/powerpc/include/asm/kvm_ppc.h | 13 +++
>> arch/powerpc/kvm/book3s_xive.h | 6 ++
>> include/uapi/linux/kvm.h | 1 +
>> arch/powerpc/kvm/book3s_xive.c | 67 +++++++-----
>> arch/powerpc/kvm/book3s_xive_native.c | 144 ++++++++++++++++++++++++++
>> arch/powerpc/kvm/powerpc.c | 33 ++++++
>> Documentation/virtual/kvm/api.txt | 9 ++
>> 8 files changed, 246 insertions(+), 28 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
>> index 9f75a75a07f2..eb8581be0ee8 100644
>> --- a/arch/powerpc/include/asm/kvm_host.h
>> +++ b/arch/powerpc/include/asm/kvm_host.h
>> @@ -448,6 +448,7 @@ struct kvmppc_passthru_irqmap {
>> #define KVMPPC_IRQ_DEFAULT 0
>> #define KVMPPC_IRQ_MPIC 1
>> #define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
>> +#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
>>
>> #define MMIO_HPTE_CACHE_SIZE 4
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index 4b72ddde7dc1..1e61877fe147 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -594,6 +594,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
>> int level, bool line_status);
>> extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
>>
>> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
>> +{
>> + return vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
>> +}
>> +
>> +extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> + struct kvm_vcpu *vcpu, u32 cpu);
>> +extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
>> extern void kvmppc_xive_native_init_module(void);
>> extern void kvmppc_xive_native_exit_module(void);
>>
>> @@ -621,6 +629,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
>> int level, bool line_status) { return -ENODEV; }
>> static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
>>
>> +static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
>> + { return 0; }
>> +static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> + struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
>> +static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
>> static inline void kvmppc_xive_native_init_module(void) { }
>> static inline void kvmppc_xive_native_exit_module(void) { }
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
>> index a08ae6fd4c51..bcb1bbcf0359 100644
>> --- a/arch/powerpc/kvm/book3s_xive.h
>> +++ b/arch/powerpc/kvm/book3s_xive.h
>> @@ -248,5 +248,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
>> extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
>> extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
>>
>> +/*
>> + * Common Xive routines for XICS-over-XIVE and XIVE native
>> + */
>> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
>> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
>> +
>> #endif /* CONFIG_KVM_XICS */
>> #endif /* _KVM_PPC_BOOK3S_XICS_H */
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index e6368163d3a0..52bf74a1616e 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
>> #define KVM_CAP_ARM_VM_IPA_SIZE 165
>> #define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
>> #define KVM_CAP_HYPERV_CPUID 167
>> +#define KVM_CAP_PPC_IRQ_XIVE 168
>>
>> #ifdef KVM_CAP_IRQ_ROUTING
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index f78d002f0fe0..d1cc18a5b1c4 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -1049,7 +1049,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
>> }
>> EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
>>
>> -static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>> +void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> struct kvm *kvm = vcpu->kvm;
>> @@ -1883,6 +1883,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
>> return 0;
>> }
>>
>> +int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + unsigned int i;
>> +
>> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
>> + struct xive_q *q = &xc->queues[i];
>> + u32 i0, i1, idx;
>> +
>> + if (!q->qpage && !xc->esc_virq[i])
>> + continue;
>> +
>> + seq_printf(m, " [q%d]: ", i);
>> +
>> + if (q->qpage) {
>> + idx = q->idx;
>> + i0 = be32_to_cpup(q->qpage + idx);
>> + idx = (idx + 1) & q->msk;
>> + i1 = be32_to_cpup(q->qpage + idx);
>> + seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
>> + i0, i1);
>> + }
>> + if (xc->esc_virq[i]) {
>> + struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
>> + struct xive_irq_data *xd >> + irq_data_get_irq_handler_data(d);
>> + u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
>> +
>> + seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
>> + (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
>> + (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
>> + xc->esc_virq[i], pq, xd->eoi_page);
>> + seq_puts(m, "\n");
>> + }
>> + }
>> + return 0;
>> +}
>>
>> static int xive_debug_show(struct seq_file *m, void *private)
>> {
>> @@ -1908,7 +1945,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
>>
>> kvm_for_each_vcpu(i, vcpu, kvm) {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> - unsigned int i;
>>
>> if (!xc)
>> continue;
>> @@ -1918,33 +1954,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
>> xc->server_num, xc->cppr, xc->hw_cppr,
>> xc->mfrr, xc->pending,
>> xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
>> - for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
>> - struct xive_q *q = &xc->queues[i];
>> - u32 i0, i1, idx;
>>
>> - if (!q->qpage && !xc->esc_virq[i])
>> - continue;
>> -
>> - seq_printf(m, " [q%d]: ", i);
>> -
>> - if (q->qpage) {
>> - idx = q->idx;
>> - i0 = be32_to_cpup(q->qpage + idx);
>> - idx = (idx + 1) & q->msk;
>> - i1 = be32_to_cpup(q->qpage + idx);
>> - seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
>> - }
>> - if (xc->esc_virq[i]) {
>> - struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
>> - struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>> - u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
>> - seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
>> - (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
>> - (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
>> - xc->esc_virq[i], pq, xd->eoi_page);
>> - seq_printf(m, "\n");
>> - }
>> - }
>> + kvmppc_xive_debug_show_queues(m, vcpu);
>>
>> t_rm_h_xirr += xc->stat_rm_h_xirr;
>> t_rm_h_ipoll += xc->stat_rm_h_ipoll;
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index e475ce83ad14..1f3da47a4a6a 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -31,6 +31,128 @@
>>
>> #include "book3s_xive.h"
>>
>> +static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + struct xive_q *q = &xc->queues[prio];
>> +
>> + xive_native_disable_queue(xc->vp_id, q, prio);
>> + if (q->qpage) {
>> + put_page(virt_to_page(q->qpage));
>> + q->qpage = NULL;
>> + }
>> +}
>> +
>> +void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + int i;
>> +
>> + if (!kvmppc_xive_enabled(vcpu))
>> + return;
>> +
>> + if (!xc)
>> + return;
>> +
>> + pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
>> +
>> + /* Ensure no interrupt is still routed to that VP */
>> + xc->valid = false;
>> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
>> +
>> + /* Disable the VP */
>> + xive_native_disable_vp(xc->vp_id);
>> +
>> + /* Free the queues & associated interrupts */
>> + for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
>> + /* Free the escalation irq */
>> + if (xc->esc_virq[i]) {
>> + free_irq(xc->esc_virq[i], vcpu);
>> + irq_dispose_mapping(xc->esc_virq[i]);
>> + kfree(xc->esc_virq_names[i]);
>> + xc->esc_virq[i] = 0;
>> + }
>> +
>> + /* Free the queue */
>> + kvmppc_xive_native_cleanup_queue(vcpu, i);
>> + }
>> +
>> + /* Free the VP */
>> + kfree(xc);
>> +
>> + /* Cleanup the vcpu */
>> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
>> + vcpu->arch.xive_vcpu = NULL;
>> +}
>> +
>> +int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> + struct kvm_vcpu *vcpu, u32 cpu)
>> +{
>> + struct kvmppc_xive *xive = dev->private;
>> + struct kvmppc_xive_vcpu *xc;
>> + int rc;
>> +
>> + pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
>> +
>> + if (dev->ops != &kvm_xive_native_ops) {
>> + pr_devel("Wrong ops !\n");
>> + return -EPERM;
>> + }
>> + if (xive->kvm != vcpu->kvm)
>> + return -EPERM;
>> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
>> + return -EBUSY;
>> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
>
> You haven't taken the kvm->lock yet, so couldn't a race mean a
> duplicate server gets inserted after you make this check?
>
>> + pr_devel("Duplicate !\n");
>> + return -EEXIST;
>> + }
>> + if (cpu >= KVM_MAX_VCPUS) {
>> + pr_devel("Out of bounds !\n");
>> + return -EINVAL;
>> + }
>> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
>> + if (!xc)
>> + return -ENOMEM;
>> +
>> + mutex_lock(&vcpu->kvm->lock);
>> + vcpu->arch.xive_vcpu = xc;
>
> Similarly you don't verify this is NULL after taking the lock, so
> couldn't another thread race and make a connect which gets clobbered
> here?
Yes. this is not very safe ... We need to clean up all the KVM device
methods doing the connection of the presenter to the vCPU AFAICT.
I will fix the XIVE native one for now.
And also, this CPU parameter is useless. There is no reason to connect
a vCPU from another vCPU.
>> + xc->xive = xive;
>> + xc->vcpu = vcpu;
>> + xc->server_num = cpu;
>> + xc->vp_id = xive->vp_base + cpu;
>
> Hrm. This ties the internal VP id to the userspace chosen server
> number, which isn't ideal. It puts a constraint on those server
> numbers that you wouldn't otherwise have.
Ah yes. I should be using the kvmppc_pack_vcpu_id() like we do for
the XICS-over-XIVE device probably. I need to check that it is correct
in this mode.
Thanks,
C.
>> + xc->valid = true;
>> +
>> + rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
>> + if (rc) {
>> + pr_err("Failed to get VP info from OPAL: %d\n", rc);
>> + goto bail;
>> + }
>> +
>> + /*
>> + * Enable the VP first as the single escalation mode will
>> + * affect escalation interrupts numbering
>> + */
>> + rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
>> + if (rc) {
>> + pr_err("Failed to enable VP in OPAL: %d\n", rc);
>> + goto bail;
>> + }
>> +
>> + /* Configure VCPU fields for use by assembly push/pull */
>> + vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
>> + vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
>> +
>> + /* TODO: initialize queues ? */
>> +
>> +bail:
>> + vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
>> + mutex_unlock(&vcpu->kvm->lock);
>> + if (rc)
>> + kvmppc_xive_native_cleanup_vcpu(vcpu);
>> +
>> + return rc;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> @@ -126,10 +248,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
>> {
>> struct kvmppc_xive *xive = m->private;
>> struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + unsigned int i;
>>
>> if (!kvm)
>> return 0;
>>
>> + seq_puts(m, "=====\nVCPU state\n=====\n");
>> +
>> + kvm_for_each_vcpu(i, vcpu, kvm) {
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> +
>> + if (!xc)
>> + continue;
>> +
>> + seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
>> + xc->server_num,
>> + vcpu->arch.xive_saved_state.nsr,
>> + vcpu->arch.xive_saved_state.cppr,
>> + vcpu->arch.xive_saved_state.ipb,
>> + vcpu->arch.xive_saved_state.pipr,
>> + vcpu->arch.xive_saved_state.w01,
>> + (u32) vcpu->arch.xive_cam_word);
>> +
>> + kvmppc_xive_debug_show_queues(m, vcpu);
>> + }
>> +
>> return 0;
>> }
>>
>> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
>> index 8c69af10f91d..a38a643a24dd 100644
>> --- a/arch/powerpc/kvm/powerpc.c
>> +++ b/arch/powerpc/kvm/powerpc.c
>> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>> case KVM_CAP_PPC_GET_CPU_CHAR:
>> r = 1;
>> break;
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_CAP_PPC_IRQ_XIVE:
>> + /* only for PowerNV */
>> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
>> + break;
>> +#endif
>>
>> case KVM_CAP_PPC_ALLOC_HTAB:
>> r = hv_enabled;
>> @@ -753,6 +759,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
>> else
>> kvmppc_xics_free_icp(vcpu);
>> break;
>> + case KVMPPC_IRQ_XIVE:
>> + kvmppc_xive_native_cleanup_vcpu(vcpu);
>> + break;
>> }
>>
>> kvmppc_core_vcpu_free(vcpu);
>> @@ -1941,6 +1950,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
>> break;
>> }
>> #endif /* CONFIG_KVM_XICS */
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_CAP_PPC_IRQ_XIVE: {
>> + struct fd f;
>> + struct kvm_device *dev;
>> +
>> + r = -EBADF;
>> + f = fdget(cap->args[0]);
>> + if (!f.file)
>> + break;
>> +
>> + r = -ENXIO;
>> + if (!xive_enabled())
>> + break;
>> +
>> + r = -EPERM;
>> + dev = kvm_device_from_filp(f.file);
>> + if (dev)
>> + r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
>> + cap->args[1]);
>> +
>> + fdput(f);
>> + break;
>> + }
>> +#endif /* CONFIG_KVM_XIVE */
>> #ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
>> case KVM_CAP_PPC_FWNMI:
>> r = -EINVAL;
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 356156f5c52d..1db1435769b4 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -4458,6 +4458,15 @@ struct kvm_sync_regs {
>> struct kvm_vcpu_events events;
>> };
>>
>> +6.75 KVM_CAP_PPC_IRQ_XIVE
>> +
>> +Architectures: ppc
>> +Target: vcpu
>> +Parameters: args[0] is the XIVE device fd
>> + args[1] is the XIVE CPU number (server ID) for this vcpu
>> +
>> +This capability connects the vcpu to an in-kernel XIVE device.
>> +
>> 7. Capabilities that can be enabled on VMs
>> ------------------------------------------
>>
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-03-12 14:03 ` Cédric Le Goater
@ 2019-03-13 4:05 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-13 4:05 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]
On Tue, Mar 12, 2019 at 03:03:25PM +0100, Cédric Le Goater wrote:
> On 2/25/19 1:35 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
[snip]
> >> +int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> + struct kvm_vcpu *vcpu, u32 cpu)
> >> +{
> >> + struct kvmppc_xive *xive = dev->private;
> >> + struct kvmppc_xive_vcpu *xc;
> >> + int rc;
> >> +
> >> + pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
> >> +
> >> + if (dev->ops != &kvm_xive_native_ops) {
> >> + pr_devel("Wrong ops !\n");
> >> + return -EPERM;
> >> + }
> >> + if (xive->kvm != vcpu->kvm)
> >> + return -EPERM;
> >> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
> >> + return -EBUSY;
> >> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
> >
> > You haven't taken the kvm->lock yet, so couldn't a race mean a
> > duplicate server gets inserted after you make this check?
> >
> >> + pr_devel("Duplicate !\n");
> >> + return -EEXIST;
> >> + }
> >> + if (cpu >= KVM_MAX_VCPUS) {
> >> + pr_devel("Out of bounds !\n");
> >> + return -EINVAL;
> >> + }
> >> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
> >> + if (!xc)
> >> + return -ENOMEM;
> >> +
> >> + mutex_lock(&vcpu->kvm->lock);
> >> + vcpu->arch.xive_vcpu = xc;
> >
> > Similarly you don't verify this is NULL after taking the lock, so
> > couldn't another thread race and make a connect which gets clobbered
> > here?
>
> Yes. this is not very safe ... We need to clean up all the KVM device
> methods doing the connection of the presenter to the vCPU AFAICT.
> I will fix the XIVE native one for now.
>
> And also, this CPU parameter is useless. There is no reason to connect
> a vCPU from another vCPU.
Hmm.. I thought the point of the 'cpu' parameter (not a great name) is
that it lets userspace chose the guest visible irq server ID. I think
that's preferable to tying it to an existing cpu id, if possible.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-03-13 4:05 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-13 4:05 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2164 bytes --]
On Tue, Mar 12, 2019 at 03:03:25PM +0100, Cédric Le Goater wrote:
> On 2/25/19 1:35 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
[snip]
> >> +int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> + struct kvm_vcpu *vcpu, u32 cpu)
> >> +{
> >> + struct kvmppc_xive *xive = dev->private;
> >> + struct kvmppc_xive_vcpu *xc;
> >> + int rc;
> >> +
> >> + pr_devel("native_connect_vcpu(cpu=%d)\n", cpu);
> >> +
> >> + if (dev->ops != &kvm_xive_native_ops) {
> >> + pr_devel("Wrong ops !\n");
> >> + return -EPERM;
> >> + }
> >> + if (xive->kvm != vcpu->kvm)
> >> + return -EPERM;
> >> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
> >> + return -EBUSY;
> >> + if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
> >
> > You haven't taken the kvm->lock yet, so couldn't a race mean a
> > duplicate server gets inserted after you make this check?
> >
> >> + pr_devel("Duplicate !\n");
> >> + return -EEXIST;
> >> + }
> >> + if (cpu >= KVM_MAX_VCPUS) {
> >> + pr_devel("Out of bounds !\n");
> >> + return -EINVAL;
> >> + }
> >> + xc = kzalloc(sizeof(*xc), GFP_KERNEL);
> >> + if (!xc)
> >> + return -ENOMEM;
> >> +
> >> + mutex_lock(&vcpu->kvm->lock);
> >> + vcpu->arch.xive_vcpu = xc;
> >
> > Similarly you don't verify this is NULL after taking the lock, so
> > couldn't another thread race and make a connect which gets clobbered
> > here?
>
> Yes. this is not very safe ... We need to clean up all the KVM device
> methods doing the connection of the presenter to the vCPU AFAICT.
> I will fix the XIVE native one for now.
>
> And also, this CPU parameter is useless. There is no reason to connect
> a vCPU from another vCPU.
Hmm.. I thought the point of the 'cpu' parameter (not a great name) is
that it lets userspace chose the guest visible irq server ID. I think
that's preferable to tying it to an existing cpu id, if possible.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 4:35 ` Paul Mackerras
-1 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-25 4:35 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> The user interface exposes a new capability to let QEMU connect the
> vCPU to the XIVE KVM device if required. The capability is only
> advertised on a PowerNV Hypervisor as support for nested guests
> (pseries KVM Hypervisor) is not yet available.
If a bisection happened to land on this commit, we would have KVM
saying it had the ability to support guests using XIVE natively, but
it wouldn't actually work since we don't have all the code that is in
the following patches.
Thus, in order to avoid breaking bisection, you should either add the
capability now but have it always return false until the rest of the
code is in place, or else defer the addition of the capability until
the end of the patch series.
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 8c69af10f91d..a38a643a24dd 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> case KVM_CAP_PPC_GET_CPU_CHAR:
> r = 1;
> break;
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_CAP_PPC_IRQ_XIVE:
> + /* only for PowerNV */
> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
Shouldn't this be r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE)
(or alternatively r = xics_on_xive(), though that would be confusing
to the reader)?
As it stands this would report true on POWER8, unless I'm missing
something.
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-02-25 4:35 ` Paul Mackerras
0 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-25 4:35 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> The user interface exposes a new capability to let QEMU connect the
> vCPU to the XIVE KVM device if required. The capability is only
> advertised on a PowerNV Hypervisor as support for nested guests
> (pseries KVM Hypervisor) is not yet available.
If a bisection happened to land on this commit, we would have KVM
saying it had the ability to support guests using XIVE natively, but
it wouldn't actually work since we don't have all the code that is in
the following patches.
Thus, in order to avoid breaking bisection, you should either add the
capability now but have it always return false until the rest of the
code is in place, or else defer the addition of the capability until
the end of the patch series.
> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
> index 8c69af10f91d..a38a643a24dd 100644
> --- a/arch/powerpc/kvm/powerpc.c
> +++ b/arch/powerpc/kvm/powerpc.c
> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> case KVM_CAP_PPC_GET_CPU_CHAR:
> r = 1;
> break;
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_CAP_PPC_IRQ_XIVE:
> + /* only for PowerNV */
> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
Shouldn't this be r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE)
(or alternatively r = xics_on_xive(), though that would be confusing
to the reader)?
As it stands this would report true on POWER8, unless I'm missing
something.
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-02-25 4:35 ` Paul Mackerras
@ 2019-03-13 8:34 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:34 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On 2/25/19 5:35 AM, Paul Mackerras wrote:
> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
>> The user interface exposes a new capability to let QEMU connect the
>> vCPU to the XIVE KVM device if required. The capability is only
>> advertised on a PowerNV Hypervisor as support for nested guests
>> (pseries KVM Hypervisor) is not yet available.
>
> If a bisection happened to land on this commit, we would have KVM
> saying it had the ability to support guests using XIVE natively, but
> it wouldn't actually work since we don't have all the code that is in
> the following patches.
OK. I didn't think migration was a must-have for bisection. I will move
the enablement at end.
> Thus, in order to avoid breaking bisection, you should either add the
> capability now but have it always return false until the rest of the
> code is in place, or else defer the addition of the capability until
> the end of the patch series.
I will introduce the capability early in the patchset and return false
as you are proposing. It seems to be the best approach.
>> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
>> index 8c69af10f91d..a38a643a24dd 100644
>> --- a/arch/powerpc/kvm/powerpc.c
>> +++ b/arch/powerpc/kvm/powerpc.c
>> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>> case KVM_CAP_PPC_GET_CPU_CHAR:
>> r = 1;
>> break;
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_CAP_PPC_IRQ_XIVE:
>> + /* only for PowerNV */
>> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
>
> Shouldn't this be r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE)
yes. we need the '__xive_enabled' toggle to be set also :/
It can set to off with the "xive=off" on the command line and on old P9
skiboot. That could be simplified one day.
> (or alternatively r = xics_on_xive(), though that would be confusing
> to the reader)?
This is correct. I didn't want to use the xics_on_xive() which is not
the capability we are activating.
I will keep the open-coded version.
> As it stands this would report true on POWER8, unless I'm missing
> something.
Ah yes. I forgot this combination also.
This should not be too complex to fix.
Thanks,
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-03-13 8:34 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:34 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On 2/25/19 5:35 AM, Paul Mackerras wrote:
> On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
>> The user interface exposes a new capability to let QEMU connect the
>> vCPU to the XIVE KVM device if required. The capability is only
>> advertised on a PowerNV Hypervisor as support for nested guests
>> (pseries KVM Hypervisor) is not yet available.
>
> If a bisection happened to land on this commit, we would have KVM
> saying it had the ability to support guests using XIVE natively, but
> it wouldn't actually work since we don't have all the code that is in
> the following patches.
OK. I didn't think migration was a must-have for bisection. I will move
the enablement at end.
> Thus, in order to avoid breaking bisection, you should either add the
> capability now but have it always return false until the rest of the
> code is in place, or else defer the addition of the capability until
> the end of the patch series.
I will introduce the capability early in the patchset and return false
as you are proposing. It seems to be the best approach.
>> diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
>> index 8c69af10f91d..a38a643a24dd 100644
>> --- a/arch/powerpc/kvm/powerpc.c
>> +++ b/arch/powerpc/kvm/powerpc.c
>> @@ -570,6 +570,12 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>> case KVM_CAP_PPC_GET_CPU_CHAR:
>> r = 1;
>> break;
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_CAP_PPC_IRQ_XIVE:
>> + /* only for PowerNV */
>> + r = !!cpu_has_feature(CPU_FTR_HVMODE);
>
> Shouldn't this be r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE)
yes. we need the '__xive_enabled' toggle to be set also :/
It can set to off with the "xive=off" on the command line and on old P9
skiboot. That could be simplified one day.
> (or alternatively r = xics_on_xive(), though that would be confusing
> to the reader)?
This is correct. I didn't want to use the xics_on_xive() which is not
the capability we are activating.
I will keep the open-coded version.
> As it stands this would report true on POWER8, unless I'm missing
> something.
Ah yes. I forgot this combination also.
This should not be too complex to fix.
Thanks,
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-03-13 8:34 ` Cédric Le Goater
@ 2019-03-14 2:29 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:29 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]
On Wed, Mar 13, 2019 at 09:34:53AM +0100, Cédric Le Goater wrote:
> On 2/25/19 5:35 AM, Paul Mackerras wrote:
> > On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> >> The user interface exposes a new capability to let QEMU connect the
> >> vCPU to the XIVE KVM device if required. The capability is only
> >> advertised on a PowerNV Hypervisor as support for nested guests
> >> (pseries KVM Hypervisor) is not yet available.
> >
> > If a bisection happened to land on this commit, we would have KVM
> > saying it had the ability to support guests using XIVE natively, but
> > it wouldn't actually work since we don't have all the code that is in
> > the following patches.
>
> OK. I didn't think migration was a must-have for bisection. I will move
> the enablement at end.
Any temporary feature regression potentially breaks bisection, because
you don't know what we'll want to bisect for. Obviously we're never
going to get that perfectly right, but that doesn't mean we shouldn't
try when we do see the problem in advance.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-03-14 2:29 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:29 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]
On Wed, Mar 13, 2019 at 09:34:53AM +0100, Cédric Le Goater wrote:
> On 2/25/19 5:35 AM, Paul Mackerras wrote:
> > On Fri, Feb 22, 2019 at 12:28:27PM +0100, Cédric Le Goater wrote:
> >> The user interface exposes a new capability to let QEMU connect the
> >> vCPU to the XIVE KVM device if required. The capability is only
> >> advertised on a PowerNV Hypervisor as support for nested guests
> >> (pseries KVM Hypervisor) is not yet available.
> >
> > If a bisection happened to land on this commit, we would have KVM
> > saying it had the ability to support guests using XIVE natively, but
> > it wouldn't actually work since we don't have all the code that is in
> > the following patches.
>
> OK. I didn't think migration was a must-have for bisection. I will move
> the enablement at end.
Any temporary feature regression potentially breaks bisection, because
you don't know what we'll want to bisect for. Obviously we're never
going to get that perfectly right, but that doesn't mean we shouldn't
try when we do see the problem in advance.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The associated HW interrupt source is simply allocated at the OPAL/HW
level and then MASKED. KVM only needs to know about its type: LSI or
MSI.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 5 +
arch/powerpc/kvm/book3s_xive.h | 10 ++
arch/powerpc/kvm/book3s_xive.c | 8 +-
arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 +++
5 files changed, 148 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index b002c0c67787..a9ad99f2a11b 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
+
+/* Layout of 64-bit XIVE source attribute values */
+#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
+#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index bcb1bbcf0359..f22f2d46d0f0 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -12,6 +12,13 @@
#ifdef CONFIG_KVM_XICS
#include "book3s_xics.h"
+/*
+ * The XIVE IRQ number space is aligned with the XICS IRQ number
+ * space, CPU IPIs being allocated in the first 4K.
+ */
+#define KVMPPC_XIVE_FIRST_IRQ 0
+#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
+
/*
* State for one guest irq source.
*
@@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
*/
void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq);
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index d1cc18a5b1c4..6f950ecb3592 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
return 0;
}
-static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
- int irq)
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq)
{
struct kvm *kvm = xive->kvm;
struct kvmppc_xive_src_block *sb;
@@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
sb = kvmppc_xive_find_source(xive, irq, &idx);
if (!sb) {
pr_devel("No source, creating source block...\n");
- sb = xive_create_src_block(xive, irq);
+ sb = kvmppc_xive_create_src_block(xive, irq);
if (!sb) {
pr_devel("Failed to create block...\n");
return -ENOMEM;
@@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
xive_cleanup_irq_data(xd);
}
-static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
{
int i;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 1f3da47a4a6a..a9b2d2d9af99 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -31,6 +31,29 @@
#include "book3s_xive.h"
+/*
+ * TODO: introduce a common template file with the XIVE native layer
+ * and the XICS-on-XIVE glue for the utility functions
+ */
+#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
+#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
+#define __x_readq __raw_readq
+#define __x_writeq __raw_writeq
+
+static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = __x_readq(__x_eoi_page(xd) + offset);
+#ifdef __LITTLE_ENDIAN__
+ val >>= 64-8;
+#endif
+ return (u8)val;
+}
+
static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
@@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
+ u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u64 val;
+ u16 idx;
+
+ pr_devel("%s irq=0x%lx\n", __func__, irq);
+
+ if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
+ return -E2BIG;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb) {
+ pr_debug("No source, creating source block...\n");
+ sb = kvmppc_xive_create_src_block(xive, irq);
+ if (!sb) {
+ pr_err("Failed to create block...\n");
+ return -ENOMEM;
+ }
+ }
+ state = &sb->irq_state[idx];
+
+ if (get_user(val, ubufp)) {
+ pr_err("fault getting user info !\n");
+ return -EFAULT;
+ }
+
+ /*
+ * If the source doesn't already have an IPI, allocate
+ * one and get the corresponding data
+ */
+ if (!state->ipi_number) {
+ state->ipi_number = xive_native_alloc_irq();
+ if (state->ipi_number == 0) {
+ pr_err("Failed to allocate IRQ !\n");
+ return -ENXIO;
+ }
+ xive_native_populate_irq_data(state->ipi_number,
+ &state->ipi_data);
+ pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
+ state->ipi_number, irq);
+ }
+
+ arch_spin_lock(&sb->lock);
+
+ /* Restore LSI state */
+ if (val & KVM_XIVE_LEVEL_SENSITIVE) {
+ state->lsi = true;
+ if (val & KVM_XIVE_LEVEL_ASSERTED)
+ state->asserted = true;
+ pr_devel(" LSI ! Asserted=%d\n", state->asserted);
+ }
+
+ /* Mask IRQ to start with */
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+
+ /* Increment the number of valid sources and mark this one valid */
+ if (!state->valid)
+ xive->src_count++;
+ state->valid = true;
+
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ return kvmppc_xive_native_set_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
+ attr->attr < KVMPPC_XIVE_NR_IRQS)
+ return 0;
+ break;
}
return -ENXIO;
}
@@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ int i;
debugfs_remove(xive->dentry);
@@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
if (kvm)
kvm->arch.xive = NULL;
+ /* Mask and free interrupts */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
if (xive->vp_base != XIVE_INVALID_VP)
xive_native_free_vp_block(xive->vp_base);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index fdbd2ff92a88..cd8bfc37b72e 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+
+ 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
+ Initializes a new source in the XIVE device and mask it.
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 2 | 1 | 0
+ values: | unused | level | type
+ - type: 0:MSI 1:LSI
+ - level: assertion level in case of an LSI.
+ Errors:
+ -E2BIG: Interrupt source number is out of range
+ -ENOMEM: Could not create a new source block
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: Could not allocate underlying HW interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The associated HW interrupt source is simply allocated at the OPAL/HW
level and then MASKED. KVM only needs to know about its type: LSI or
MSI.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 5 +
arch/powerpc/kvm/book3s_xive.h | 10 ++
arch/powerpc/kvm/book3s_xive.c | 8 +-
arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 +++
5 files changed, 148 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index b002c0c67787..a9ad99f2a11b 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
+
+/* Layout of 64-bit XIVE source attribute values */
+#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
+#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index bcb1bbcf0359..f22f2d46d0f0 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -12,6 +12,13 @@
#ifdef CONFIG_KVM_XICS
#include "book3s_xics.h"
+/*
+ * The XIVE IRQ number space is aligned with the XICS IRQ number
+ * space, CPU IPIs being allocated in the first 4K.
+ */
+#define KVMPPC_XIVE_FIRST_IRQ 0
+#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
+
/*
* State for one guest irq source.
*
@@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
*/
void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq);
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index d1cc18a5b1c4..6f950ecb3592 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
return 0;
}
-static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
- int irq)
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq)
{
struct kvm *kvm = xive->kvm;
struct kvmppc_xive_src_block *sb;
@@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
sb = kvmppc_xive_find_source(xive, irq, &idx);
if (!sb) {
pr_devel("No source, creating source block...\n");
- sb = xive_create_src_block(xive, irq);
+ sb = kvmppc_xive_create_src_block(xive, irq);
if (!sb) {
pr_devel("Failed to create block...\n");
return -ENOMEM;
@@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
xive_cleanup_irq_data(xd);
}
-static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
{
int i;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 1f3da47a4a6a..a9b2d2d9af99 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -31,6 +31,29 @@
#include "book3s_xive.h"
+/*
+ * TODO: introduce a common template file with the XIVE native layer
+ * and the XICS-on-XIVE glue for the utility functions
+ */
+#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
+#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
+#define __x_readq __raw_readq
+#define __x_writeq __raw_writeq
+
+static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = __x_readq(__x_eoi_page(xd) + offset);
+#ifdef __LITTLE_ENDIAN__
+ val >>= 64-8;
+#endif
+ return (u8)val;
+}
+
static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
@@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
+ u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u64 val;
+ u16 idx;
+
+ pr_devel("%s irq=0x%lx\n", __func__, irq);
+
+ if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
+ return -E2BIG;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb) {
+ pr_debug("No source, creating source block...\n");
+ sb = kvmppc_xive_create_src_block(xive, irq);
+ if (!sb) {
+ pr_err("Failed to create block...\n");
+ return -ENOMEM;
+ }
+ }
+ state = &sb->irq_state[idx];
+
+ if (get_user(val, ubufp)) {
+ pr_err("fault getting user info !\n");
+ return -EFAULT;
+ }
+
+ /*
+ * If the source doesn't already have an IPI, allocate
+ * one and get the corresponding data
+ */
+ if (!state->ipi_number) {
+ state->ipi_number = xive_native_alloc_irq();
+ if (state->ipi_number = 0) {
+ pr_err("Failed to allocate IRQ !\n");
+ return -ENXIO;
+ }
+ xive_native_populate_irq_data(state->ipi_number,
+ &state->ipi_data);
+ pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
+ state->ipi_number, irq);
+ }
+
+ arch_spin_lock(&sb->lock);
+
+ /* Restore LSI state */
+ if (val & KVM_XIVE_LEVEL_SENSITIVE) {
+ state->lsi = true;
+ if (val & KVM_XIVE_LEVEL_ASSERTED)
+ state->asserted = true;
+ pr_devel(" LSI ! Asserted=%d\n", state->asserted);
+ }
+
+ /* Mask IRQ to start with */
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+
+ /* Increment the number of valid sources and mark this one valid */
+ if (!state->valid)
+ xive->src_count++;
+ state->valid = true;
+
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ return kvmppc_xive_native_set_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
+ attr->attr < KVMPPC_XIVE_NR_IRQS)
+ return 0;
+ break;
}
return -ENXIO;
}
@@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ int i;
debugfs_remove(xive->dentry);
@@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
if (kvm)
kvm->arch.xive = NULL;
+ /* Mask and free interrupts */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
if (xive->vp_base != XIVE_INVALID_VP)
xive_native_free_vp_block(xive->vp_base);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index fdbd2ff92a88..cd8bfc37b72e 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+
+ 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
+ Initializes a new source in the XIVE device and mask it.
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 2 | 1 | 0
+ values: | unused | level | type
+ - type: 0:MSI 1:LSI
+ - level: assertion level in case of an LSI.
+ Errors:
+ -E2BIG: Interrupt source number is out of range
+ -ENOMEM: Could not create a new source block
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: Could not allocate underlying HW interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 2:10 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:10 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 10424 bytes --]
On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> The associated HW interrupt source is simply allocated at the OPAL/HW
> level and then MASKED. KVM only needs to know about its type: LSI or
> MSI.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 5 +
> arch/powerpc/kvm/book3s_xive.h | 10 ++
> arch/powerpc/kvm/book3s_xive.c | 8 +-
> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 15 +++
> 5 files changed, 148 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index b002c0c67787..a9ad99f2a11b 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
>
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> +
> +/* Layout of 64-bit XIVE source attribute values */
> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index bcb1bbcf0359..f22f2d46d0f0 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -12,6 +12,13 @@
> #ifdef CONFIG_KVM_XICS
> #include "book3s_xics.h"
>
> +/*
> + * The XIVE IRQ number space is aligned with the XICS IRQ number
> + * space, CPU IPIs being allocated in the first 4K.
We do align these in qemu, but I don't see that the kernel part
cares: as far as it's concerned only one of XICS or XIVE is active at
a time, and the irq numbers are chosen by userspace.
> + */
> +#define KVMPPC_XIVE_FIRST_IRQ 0
> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
> +
> /*
> * State for one guest irq source.
> *
> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> */
> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq);
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index d1cc18a5b1c4..6f950ecb3592 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
I wonder if we should rename this book3s_xics_on_xive.c or something
at some point, I keep getting confused because I forget that this is
only dealing with host xive, not guest xive.
> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> return 0;
> }
>
> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> - int irq)
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq)
> {
> struct kvm *kvm = xive->kvm;
> struct kvmppc_xive_src_block *sb;
It's odd that this function, now used from the xive-on-xive path as
well as the xics-on-xive path references KVMPPC_XICS_ICS_SHIFT a few
lines down from this change.
> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> sb = kvmppc_xive_find_source(xive, irq, &idx);
> if (!sb) {
> pr_devel("No source, creating source block...\n");
> - sb = xive_create_src_block(xive, irq);
> + sb = kvmppc_xive_create_src_block(xive, irq);
> if (!sb) {
> pr_devel("Failed to create block...\n");
> return -ENOMEM;
> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> xive_cleanup_irq_data(xd);
> }
>
> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> {
> int i;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 1f3da47a4a6a..a9b2d2d9af99 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -31,6 +31,29 @@
>
> #include "book3s_xive.h"
>
> +/*
> + * TODO: introduce a common template file with the XIVE native layer
> + * and the XICS-on-XIVE glue for the utility functions
> + */
> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> +#define __x_readq __raw_readq
> +#define __x_writeq __raw_writeq
> +
> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = __x_readq(__x_eoi_page(xd) + offset);
> +#ifdef __LITTLE_ENDIAN__
> + val >>= 64-8;
> +#endif
> + return (u8)val;
> +}
> +
> static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> @@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> + u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u64 __user *ubufp = (u64 __user *) addr;
> + u64 val;
> + u16 idx;
> +
> + pr_devel("%s irq=0x%lx\n", __func__, irq);
> +
> + if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
> + return -E2BIG;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb) {
> + pr_debug("No source, creating source block...\n");
> + sb = kvmppc_xive_create_src_block(xive, irq);
> + if (!sb) {
> + pr_err("Failed to create block...\n");
> + return -ENOMEM;
> + }
> + }
> + state = &sb->irq_state[idx];
> +
> + if (get_user(val, ubufp)) {
> + pr_err("fault getting user info !\n");
> + return -EFAULT;
> + }
You should validate the value loaded here to check it doesn't have any
bits set we don't know about.
> +
> + /*
> + * If the source doesn't already have an IPI, allocate
> + * one and get the corresponding data
> + */
> + if (!state->ipi_number) {
> + state->ipi_number = xive_native_alloc_irq();
> + if (state->ipi_number == 0) {
> + pr_err("Failed to allocate IRQ !\n");
> + return -ENXIO;
> + }
> + xive_native_populate_irq_data(state->ipi_number,
> + &state->ipi_data);
> + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> + state->ipi_number, irq);
> + }
> +
> + arch_spin_lock(&sb->lock);
Why the direct call to arch_spin_lock() rather than just spin_lock()?
> +
> + /* Restore LSI state */
> + if (val & KVM_XIVE_LEVEL_SENSITIVE) {
> + state->lsi = true;
> + if (val & KVM_XIVE_LEVEL_ASSERTED)
> + state->asserted = true;
> + pr_devel(" LSI ! Asserted=%d\n", state->asserted);
> + }
> +
> + /* Mask IRQ to start with */
> + state->act_server = 0;
> + state->act_priority = MASKED;
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> +
> + /* Increment the number of valid sources and mark this one valid */
> + if (!state->valid)
> + xive->src_count++;
> + state->valid = true;
> +
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> break;
> + case KVM_DEV_XIVE_GRP_SOURCE:
> + return kvmppc_xive_native_set_source(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> break;
> + case KVM_DEV_XIVE_GRP_SOURCE:
> + if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> + attr->attr < KVMPPC_XIVE_NR_IRQS)
> + return 0;
> + break;
> }
> return -ENXIO;
> }
> @@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> {
> struct kvmppc_xive *xive = dev->private;
> struct kvm *kvm = xive->kvm;
> + int i;
>
> debugfs_remove(xive->dentry);
>
> @@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> if (kvm)
> kvm->arch.xive = NULL;
>
> + /* Mask and free interrupts */
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_free_sources(xive->src_blocks[i]);
> + kfree(xive->src_blocks[i]);
> + xive->src_blocks[i] = NULL;
> + }
> +
> if (xive->vp_base != XIVE_INVALID_VP)
> xive_native_free_vp_block(xive->vp_base);
>
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index fdbd2ff92a88..cd8bfc37b72e 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>
> 1. KVM_DEV_XIVE_GRP_CTRL
> Provides global controls on the device
> +
> + 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> + Initializes a new source in the XIVE device and mask it.
> + Attributes:
> + Interrupt source number (64-bit)
> + The kvm_device_attr.addr points to a __u64 value:
> + bits: | 63 .... 2 | 1 | 0
> + values: | unused | level | type
> + - type: 0:MSI 1:LSI
> + - level: assertion level in case of an LSI.
> + Errors:
> + -E2BIG: Interrupt source number is out of range
> + -ENOMEM: Could not create a new source block
> + -EFAULT: Invalid user pointer for attr->addr.
> + -ENXIO: Could not allocate underlying HW interrupt
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-02-25 2:10 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:10 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 10424 bytes --]
On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> The associated HW interrupt source is simply allocated at the OPAL/HW
> level and then MASKED. KVM only needs to know about its type: LSI or
> MSI.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 5 +
> arch/powerpc/kvm/book3s_xive.h | 10 ++
> arch/powerpc/kvm/book3s_xive.c | 8 +-
> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 15 +++
> 5 files changed, 148 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index b002c0c67787..a9ad99f2a11b 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
>
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> +
> +/* Layout of 64-bit XIVE source attribute values */
> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index bcb1bbcf0359..f22f2d46d0f0 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -12,6 +12,13 @@
> #ifdef CONFIG_KVM_XICS
> #include "book3s_xics.h"
>
> +/*
> + * The XIVE IRQ number space is aligned with the XICS IRQ number
> + * space, CPU IPIs being allocated in the first 4K.
We do align these in qemu, but I don't see that the kernel part
cares: as far as it's concerned only one of XICS or XIVE is active at
a time, and the irq numbers are chosen by userspace.
> + */
> +#define KVMPPC_XIVE_FIRST_IRQ 0
> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
> +
> /*
> * State for one guest irq source.
> *
> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> */
> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq);
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index d1cc18a5b1c4..6f950ecb3592 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
I wonder if we should rename this book3s_xics_on_xive.c or something
at some point, I keep getting confused because I forget that this is
only dealing with host xive, not guest xive.
> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> return 0;
> }
>
> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> - int irq)
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq)
> {
> struct kvm *kvm = xive->kvm;
> struct kvmppc_xive_src_block *sb;
It's odd that this function, now used from the xive-on-xive path as
well as the xics-on-xive path references KVMPPC_XICS_ICS_SHIFT a few
lines down from this change.
> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> sb = kvmppc_xive_find_source(xive, irq, &idx);
> if (!sb) {
> pr_devel("No source, creating source block...\n");
> - sb = xive_create_src_block(xive, irq);
> + sb = kvmppc_xive_create_src_block(xive, irq);
> if (!sb) {
> pr_devel("Failed to create block...\n");
> return -ENOMEM;
> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> xive_cleanup_irq_data(xd);
> }
>
> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> {
> int i;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 1f3da47a4a6a..a9b2d2d9af99 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -31,6 +31,29 @@
>
> #include "book3s_xive.h"
>
> +/*
> + * TODO: introduce a common template file with the XIVE native layer
> + * and the XICS-on-XIVE glue for the utility functions
> + */
> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> +#define __x_readq __raw_readq
> +#define __x_writeq __raw_writeq
> +
> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = __x_readq(__x_eoi_page(xd) + offset);
> +#ifdef __LITTLE_ENDIAN__
> + val >>= 64-8;
> +#endif
> + return (u8)val;
> +}
> +
> static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> @@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> + u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u64 __user *ubufp = (u64 __user *) addr;
> + u64 val;
> + u16 idx;
> +
> + pr_devel("%s irq=0x%lx\n", __func__, irq);
> +
> + if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
> + return -E2BIG;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> + if (!sb) {
> + pr_debug("No source, creating source block...\n");
> + sb = kvmppc_xive_create_src_block(xive, irq);
> + if (!sb) {
> + pr_err("Failed to create block...\n");
> + return -ENOMEM;
> + }
> + }
> + state = &sb->irq_state[idx];
> +
> + if (get_user(val, ubufp)) {
> + pr_err("fault getting user info !\n");
> + return -EFAULT;
> + }
You should validate the value loaded here to check it doesn't have any
bits set we don't know about.
> +
> + /*
> + * If the source doesn't already have an IPI, allocate
> + * one and get the corresponding data
> + */
> + if (!state->ipi_number) {
> + state->ipi_number = xive_native_alloc_irq();
> + if (state->ipi_number == 0) {
> + pr_err("Failed to allocate IRQ !\n");
> + return -ENXIO;
> + }
> + xive_native_populate_irq_data(state->ipi_number,
> + &state->ipi_data);
> + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> + state->ipi_number, irq);
> + }
> +
> + arch_spin_lock(&sb->lock);
Why the direct call to arch_spin_lock() rather than just spin_lock()?
> +
> + /* Restore LSI state */
> + if (val & KVM_XIVE_LEVEL_SENSITIVE) {
> + state->lsi = true;
> + if (val & KVM_XIVE_LEVEL_ASSERTED)
> + state->asserted = true;
> + pr_devel(" LSI ! Asserted=%d\n", state->asserted);
> + }
> +
> + /* Mask IRQ to start with */
> + state->act_server = 0;
> + state->act_priority = MASKED;
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> +
> + /* Increment the number of valid sources and mark this one valid */
> + if (!state->valid)
> + xive->src_count++;
> + state->valid = true;
> +
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> break;
> + case KVM_DEV_XIVE_GRP_SOURCE:
> + return kvmppc_xive_native_set_source(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> break;
> + case KVM_DEV_XIVE_GRP_SOURCE:
> + if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> + attr->attr < KVMPPC_XIVE_NR_IRQS)
> + return 0;
> + break;
> }
> return -ENXIO;
> }
> @@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> {
> struct kvmppc_xive *xive = dev->private;
> struct kvm *kvm = xive->kvm;
> + int i;
>
> debugfs_remove(xive->dentry);
>
> @@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> if (kvm)
> kvm->arch.xive = NULL;
>
> + /* Mask and free interrupts */
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_free_sources(xive->src_blocks[i]);
> + kfree(xive->src_blocks[i]);
> + xive->src_blocks[i] = NULL;
> + }
> +
> if (xive->vp_base != XIVE_INVALID_VP)
> xive_native_free_vp_block(xive->vp_base);
>
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index fdbd2ff92a88..cd8bfc37b72e 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>
> 1. KVM_DEV_XIVE_GRP_CTRL
> Provides global controls on the device
> +
> + 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> + Initializes a new source in the XIVE device and mask it.
> + Attributes:
> + Interrupt source number (64-bit)
> + The kvm_device_attr.addr points to a __u64 value:
> + bits: | 63 .... 2 | 1 | 0
> + values: | unused | level | type
> + - type: 0:MSI 1:LSI
> + - level: assertion level in case of an LSI.
> + Errors:
> + -E2BIG: Interrupt source number is out of range
> + -ENOMEM: Could not create a new source block
> + -EFAULT: Invalid user pointer for attr->addr.
> + -ENXIO: Could not allocate underlying HW interrupt
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-02-25 2:10 ` David Gibson
@ 2019-02-26 4:25 ` Paul Mackerras
-1 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-26 4:25 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Cédric Le Goater, linuxppc-dev
On Mon, Feb 25, 2019 at 01:10:12PM +1100, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> > + /*
> > + * If the source doesn't already have an IPI, allocate
> > + * one and get the corresponding data
> > + */
> > + if (!state->ipi_number) {
> > + state->ipi_number = xive_native_alloc_irq();
> > + if (state->ipi_number == 0) {
> > + pr_err("Failed to allocate IRQ !\n");
> > + return -ENXIO;
> > + }
> > + xive_native_populate_irq_data(state->ipi_number,
> > + &state->ipi_data);
> > + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> > + state->ipi_number, irq);
> > + }
> > +
> > + arch_spin_lock(&sb->lock);
>
> Why the direct call to arch_spin_lock() rather than just spin_lock()?
He's sharing data structures with the xics-on-xive code, and that code
has a real-mode variant, and in real mode we don't want to risk
invoking lockdep code. Hence sb->lock is an arch_spinlock_t, and he
has to use arch_spin_lock() on it.
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-02-26 4:25 ` Paul Mackerras
0 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-26 4:25 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Cédric Le Goater, linuxppc-dev
On Mon, Feb 25, 2019 at 01:10:12PM +1100, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> > + /*
> > + * If the source doesn't already have an IPI, allocate
> > + * one and get the corresponding data
> > + */
> > + if (!state->ipi_number) {
> > + state->ipi_number = xive_native_alloc_irq();
> > + if (state->ipi_number = 0) {
> > + pr_err("Failed to allocate IRQ !\n");
> > + return -ENXIO;
> > + }
> > + xive_native_populate_irq_data(state->ipi_number,
> > + &state->ipi_data);
> > + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> > + state->ipi_number, irq);
> > + }
> > +
> > + arch_spin_lock(&sb->lock);
>
> Why the direct call to arch_spin_lock() rather than just spin_lock()?
He's sharing data structures with the xics-on-xive code, and that code
has a real-mode variant, and in real mode we don't want to risk
invoking lockdep code. Hence sb->lock is an arch_spinlock_t, and he
has to use arch_spin_lock() on it.
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-02-26 4:25 ` Paul Mackerras
@ 2019-02-26 23:20 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-26 23:20 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, Cédric Le Goater, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1363 bytes --]
On Tue, Feb 26, 2019 at 03:25:15PM +1100, Paul Mackerras wrote:
> On Mon, Feb 25, 2019 at 01:10:12PM +1100, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> > > + /*
> > > + * If the source doesn't already have an IPI, allocate
> > > + * one and get the corresponding data
> > > + */
> > > + if (!state->ipi_number) {
> > > + state->ipi_number = xive_native_alloc_irq();
> > > + if (state->ipi_number == 0) {
> > > + pr_err("Failed to allocate IRQ !\n");
> > > + return -ENXIO;
> > > + }
> > > + xive_native_populate_irq_data(state->ipi_number,
> > > + &state->ipi_data);
> > > + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> > > + state->ipi_number, irq);
> > > + }
> > > +
> > > + arch_spin_lock(&sb->lock);
> >
> > Why the direct call to arch_spin_lock() rather than just spin_lock()?
>
> He's sharing data structures with the xics-on-xive code, and that code
> has a real-mode variant, and in real mode we don't want to risk
> invoking lockdep code. Hence sb->lock is an arch_spinlock_t, and he
> has to use arch_spin_lock() on it.
Ah, right.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-02-26 23:20 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-26 23:20 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, Cédric Le Goater, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1363 bytes --]
On Tue, Feb 26, 2019 at 03:25:15PM +1100, Paul Mackerras wrote:
> On Mon, Feb 25, 2019 at 01:10:12PM +1100, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> > > + /*
> > > + * If the source doesn't already have an IPI, allocate
> > > + * one and get the corresponding data
> > > + */
> > > + if (!state->ipi_number) {
> > > + state->ipi_number = xive_native_alloc_irq();
> > > + if (state->ipi_number == 0) {
> > > + pr_err("Failed to allocate IRQ !\n");
> > > + return -ENXIO;
> > > + }
> > > + xive_native_populate_irq_data(state->ipi_number,
> > > + &state->ipi_data);
> > > + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> > > + state->ipi_number, irq);
> > > + }
> > > +
> > > + arch_spin_lock(&sb->lock);
> >
> > Why the direct call to arch_spin_lock() rather than just spin_lock()?
>
> He's sharing data structures with the xics-on-xive code, and that code
> has a real-mode variant, and in real mode we don't want to risk
> invoking lockdep code. Hence sb->lock is an arch_spinlock_t, and he
> has to use arch_spin_lock() on it.
Ah, right.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-02-25 2:10 ` David Gibson
@ 2019-03-12 15:19 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 15:19 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 3:10 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
>> The associated HW interrupt source is simply allocated at the OPAL/HW
>> level and then MASKED. KVM only needs to know about its type: LSI or
>> MSI.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/uapi/asm/kvm.h | 5 +
>> arch/powerpc/kvm/book3s_xive.h | 10 ++
>> arch/powerpc/kvm/book3s_xive.c | 8 +-
>> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 15 +++
>> 5 files changed, 148 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index b002c0c67787..a9ad99f2a11b 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
>>
>> /* POWER9 XIVE Native Interrupt Controller */
>> #define KVM_DEV_XIVE_GRP_CTRL 1
>> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
>> +
>> +/* Layout of 64-bit XIVE source attribute values */
>> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
>> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>>
>> #endif /* __LINUX_KVM_POWERPC_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
>> index bcb1bbcf0359..f22f2d46d0f0 100644
>> --- a/arch/powerpc/kvm/book3s_xive.h
>> +++ b/arch/powerpc/kvm/book3s_xive.h
>> @@ -12,6 +12,13 @@
>> #ifdef CONFIG_KVM_XICS
>> #include "book3s_xics.h"
>>
>> +/*
>> + * The XIVE IRQ number space is aligned with the XICS IRQ number
>> + * space, CPU IPIs being allocated in the first 4K.
>
> We do align these in qemu, but I don't see that the kernel part
> cares: as far as it's concerned only one of XICS or XIVE is active at
> a time, and the irq numbers are chosen by userspace.
There is some relation with userspace nevertheless. The KVM device does
not remap the numbers to some other range today and the limits are fixed
values. Checks are being done in the has_attr() and the set_attr().
>> + */
>> +#define KVMPPC_XIVE_FIRST_IRQ 0
>> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
>> +
>> /*
>> * State for one guest irq source.
>> *
>> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
>> */
>> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
>> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
>> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>> + struct kvmppc_xive *xive, int irq);
>> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
>>
>> #endif /* CONFIG_KVM_XICS */
>> #endif /* _KVM_PPC_BOOK3S_XICS_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index d1cc18a5b1c4..6f950ecb3592 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>
> I wonder if we should rename this book3s_xics_on_xive.c or something
> at some point, I keep getting confused because I forget that this is
> only dealing with host xive, not guest xive.
I am fine with renaming. Any objections ? book3s_xics_p9.c ?
>> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
>> return 0;
>> }
>>
>> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
>> - int irq)
>> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>> + struct kvmppc_xive *xive, int irq)
>> {
>> struct kvm *kvm = xive->kvm;
>> struct kvmppc_xive_src_block *sb;
>
> It's odd that this function, now used from the xive-on-xive path as
> well as the xics-on-xive path references KVMPPC_XICS_ICS_SHIFT a few
> lines down from this change.
Yes. This is because of the definition of the struct kvmppc_xive_src_block.
We could introduce new defines for XIVE or a common set of defines for
XICS and XIVE.
>> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
>> sb = kvmppc_xive_find_source(xive, irq, &idx);
>> if (!sb) {
>> pr_devel("No source, creating source block...\n");
>> - sb = xive_create_src_block(xive, irq);
>> + sb = kvmppc_xive_create_src_block(xive, irq);
>> if (!sb) {
>> pr_devel("Failed to create block...\n");
>> return -ENOMEM;
>> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
>> xive_cleanup_irq_data(xd);
>> }
>>
>> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
>> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
>> {
>> int i;
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index 1f3da47a4a6a..a9b2d2d9af99 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -31,6 +31,29 @@
>>
>> #include "book3s_xive.h"
>>
>> +/*
>> + * TODO: introduce a common template file with the XIVE native layer
>> + * and the XICS-on-XIVE glue for the utility functions
>> + */
>> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
>> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
>> +#define __x_readq __raw_readq
>> +#define __x_writeq __raw_writeq
>> +
>> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
>> +{
>> + u64 val;
>> +
>> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
>> + offset |= offset << 4;
>> +
>> + val = __x_readq(__x_eoi_page(xd) + offset);
>> +#ifdef __LITTLE_ENDIAN__
>> + val >>= 64-8;
>> +#endif
>> + return (u8)val;
>> +}
>> +
>> static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> @@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> return rc;
>> }
>>
>> +static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
>> + u64 addr)
>> +{
>> + struct kvmppc_xive_src_block *sb;
>> + struct kvmppc_xive_irq_state *state;
>> + u64 __user *ubufp = (u64 __user *) addr;
>> + u64 val;
>> + u16 idx;
>> +
>> + pr_devel("%s irq=0x%lx\n", __func__, irq);
>> +
>> + if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
>> + return -E2BIG;
>> +
>> + sb = kvmppc_xive_find_source(xive, irq, &idx);
>> + if (!sb) {
>> + pr_debug("No source, creating source block...\n");
>> + sb = kvmppc_xive_create_src_block(xive, irq);
>> + if (!sb) {
>> + pr_err("Failed to create block...\n");
>> + return -ENOMEM;
>> + }
>> + }
>> + state = &sb->irq_state[idx];
>> +
>> + if (get_user(val, ubufp)) {
>> + pr_err("fault getting user info !\n");
>> + return -EFAULT;
>> + }
>
> You should validate the value loaded here to check it doesn't have any
> bits set we don't know about.
ok
>
>> +
>> + /*
>> + * If the source doesn't already have an IPI, allocate
>> + * one and get the corresponding data
>> + */
>> + if (!state->ipi_number) {
>> + state->ipi_number = xive_native_alloc_irq();
>> + if (state->ipi_number == 0) {
>> + pr_err("Failed to allocate IRQ !\n");
>> + return -ENXIO;
>> + }
>> + xive_native_populate_irq_data(state->ipi_number,
>> + &state->ipi_data);
>> + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
>> + state->ipi_number, irq);
>> + }
>> +
>> + arch_spin_lock(&sb->lock);
>
> Why the direct call to arch_spin_lock() rather than just spin_lock()?
Paul answered this question but may be I should make the effort to
decouple both devices on this aspect.
Thanks,
C.
>> +
>> + /* Restore LSI state */
>> + if (val & KVM_XIVE_LEVEL_SENSITIVE) {
>> + state->lsi = true;
>> + if (val & KVM_XIVE_LEVEL_ASSERTED)
>> + state->asserted = true;
>> + pr_devel(" LSI ! Asserted=%d\n", state->asserted);
>> + }
>> +
>> + /* Mask IRQ to start with */
>> + state->act_server = 0;
>> + state->act_priority = MASKED;
>> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
>> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
>> +
>> + /* Increment the number of valid sources and mark this one valid */
>> + if (!state->valid)
>> + xive->src_count++;
>> + state->valid = true;
>> +
>> + arch_spin_unlock(&sb->lock);
>> +
>> + return 0;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> + struct kvmppc_xive *xive = dev->private;
>> +
>> switch (attr->group) {
>> case KVM_DEV_XIVE_GRP_CTRL:
>> break;
>> + case KVM_DEV_XIVE_GRP_SOURCE:
>> + return kvmppc_xive_native_set_source(xive, attr->attr,
>> + attr->addr);
>> }
>> return -ENXIO;
>> }
>> @@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> switch (attr->group) {
>> case KVM_DEV_XIVE_GRP_CTRL:
>> break;
>> + case KVM_DEV_XIVE_GRP_SOURCE:
>> + if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
>> + attr->attr < KVMPPC_XIVE_NR_IRQS)
>> + return 0;
>> + break;
>> }
>> return -ENXIO;
>> }
>> @@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
>> {
>> struct kvmppc_xive *xive = dev->private;
>> struct kvm *kvm = xive->kvm;
>> + int i;
>>
>> debugfs_remove(xive->dentry);
>>
>> @@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
>> if (kvm)
>> kvm->arch.xive = NULL;
>>
>> + /* Mask and free interrupts */
>> + for (i = 0; i <= xive->max_sbid; i++) {
>> + if (xive->src_blocks[i])
>> + kvmppc_xive_free_sources(xive->src_blocks[i]);
>> + kfree(xive->src_blocks[i]);
>> + xive->src_blocks[i] = NULL;
>> + }
>> +
>> if (xive->vp_base != XIVE_INVALID_VP)
>> xive_native_free_vp_block(xive->vp_base);
>>
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index fdbd2ff92a88..cd8bfc37b72e 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>>
>> 1. KVM_DEV_XIVE_GRP_CTRL
>> Provides global controls on the device
>> +
>> + 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
>> + Initializes a new source in the XIVE device and mask it.
>> + Attributes:
>> + Interrupt source number (64-bit)
>> + The kvm_device_attr.addr points to a __u64 value:
>> + bits: | 63 .... 2 | 1 | 0
>> + values: | unused | level | type
>> + - type: 0:MSI 1:LSI
>> + - level: assertion level in case of an LSI.
>> + Errors:
>> + -E2BIG: Interrupt source number is out of range
>> + -ENOMEM: Could not create a new source block
>> + -EFAULT: Invalid user pointer for attr->addr.
>> + -ENXIO: Could not allocate underlying HW interrupt
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-03-12 15:19 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 15:19 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 3:10 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
>> The associated HW interrupt source is simply allocated at the OPAL/HW
>> level and then MASKED. KVM only needs to know about its type: LSI or
>> MSI.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/uapi/asm/kvm.h | 5 +
>> arch/powerpc/kvm/book3s_xive.h | 10 ++
>> arch/powerpc/kvm/book3s_xive.c | 8 +-
>> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 15 +++
>> 5 files changed, 148 insertions(+), 4 deletions(-)
>>
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index b002c0c67787..a9ad99f2a11b 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
>>
>> /* POWER9 XIVE Native Interrupt Controller */
>> #define KVM_DEV_XIVE_GRP_CTRL 1
>> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
>> +
>> +/* Layout of 64-bit XIVE source attribute values */
>> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
>> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>>
>> #endif /* __LINUX_KVM_POWERPC_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
>> index bcb1bbcf0359..f22f2d46d0f0 100644
>> --- a/arch/powerpc/kvm/book3s_xive.h
>> +++ b/arch/powerpc/kvm/book3s_xive.h
>> @@ -12,6 +12,13 @@
>> #ifdef CONFIG_KVM_XICS
>> #include "book3s_xics.h"
>>
>> +/*
>> + * The XIVE IRQ number space is aligned with the XICS IRQ number
>> + * space, CPU IPIs being allocated in the first 4K.
>
> We do align these in qemu, but I don't see that the kernel part
> cares: as far as it's concerned only one of XICS or XIVE is active at
> a time, and the irq numbers are chosen by userspace.
There is some relation with userspace nevertheless. The KVM device does
not remap the numbers to some other range today and the limits are fixed
values. Checks are being done in the has_attr() and the set_attr().
>> + */
>> +#define KVMPPC_XIVE_FIRST_IRQ 0
>> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
>> +
>> /*
>> * State for one guest irq source.
>> *
>> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
>> */
>> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
>> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
>> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>> + struct kvmppc_xive *xive, int irq);
>> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
>>
>> #endif /* CONFIG_KVM_XICS */
>> #endif /* _KVM_PPC_BOOK3S_XICS_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index d1cc18a5b1c4..6f950ecb3592 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>
> I wonder if we should rename this book3s_xics_on_xive.c or something
> at some point, I keep getting confused because I forget that this is
> only dealing with host xive, not guest xive.
I am fine with renaming. Any objections ? book3s_xics_p9.c ?
>> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
>> return 0;
>> }
>>
>> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
>> - int irq)
>> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>> + struct kvmppc_xive *xive, int irq)
>> {
>> struct kvm *kvm = xive->kvm;
>> struct kvmppc_xive_src_block *sb;
>
> It's odd that this function, now used from the xive-on-xive path as
> well as the xics-on-xive path references KVMPPC_XICS_ICS_SHIFT a few
> lines down from this change.
Yes. This is because of the definition of the struct kvmppc_xive_src_block.
We could introduce new defines for XIVE or a common set of defines for
XICS and XIVE.
>> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
>> sb = kvmppc_xive_find_source(xive, irq, &idx);
>> if (!sb) {
>> pr_devel("No source, creating source block...\n");
>> - sb = xive_create_src_block(xive, irq);
>> + sb = kvmppc_xive_create_src_block(xive, irq);
>> if (!sb) {
>> pr_devel("Failed to create block...\n");
>> return -ENOMEM;
>> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
>> xive_cleanup_irq_data(xd);
>> }
>>
>> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
>> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
>> {
>> int i;
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index 1f3da47a4a6a..a9b2d2d9af99 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -31,6 +31,29 @@
>>
>> #include "book3s_xive.h"
>>
>> +/*
>> + * TODO: introduce a common template file with the XIVE native layer
>> + * and the XICS-on-XIVE glue for the utility functions
>> + */
>> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
>> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
>> +#define __x_readq __raw_readq
>> +#define __x_writeq __raw_writeq
>> +
>> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
>> +{
>> + u64 val;
>> +
>> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
>> + offset |= offset << 4;
>> +
>> + val = __x_readq(__x_eoi_page(xd) + offset);
>> +#ifdef __LITTLE_ENDIAN__
>> + val >>= 64-8;
>> +#endif
>> + return (u8)val;
>> +}
>> +
>> static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> @@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> return rc;
>> }
>>
>> +static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
>> + u64 addr)
>> +{
>> + struct kvmppc_xive_src_block *sb;
>> + struct kvmppc_xive_irq_state *state;
>> + u64 __user *ubufp = (u64 __user *) addr;
>> + u64 val;
>> + u16 idx;
>> +
>> + pr_devel("%s irq=0x%lx\n", __func__, irq);
>> +
>> + if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
>> + return -E2BIG;
>> +
>> + sb = kvmppc_xive_find_source(xive, irq, &idx);
>> + if (!sb) {
>> + pr_debug("No source, creating source block...\n");
>> + sb = kvmppc_xive_create_src_block(xive, irq);
>> + if (!sb) {
>> + pr_err("Failed to create block...\n");
>> + return -ENOMEM;
>> + }
>> + }
>> + state = &sb->irq_state[idx];
>> +
>> + if (get_user(val, ubufp)) {
>> + pr_err("fault getting user info !\n");
>> + return -EFAULT;
>> + }
>
> You should validate the value loaded here to check it doesn't have any
> bits set we don't know about.
ok
>
>> +
>> + /*
>> + * If the source doesn't already have an IPI, allocate
>> + * one and get the corresponding data
>> + */
>> + if (!state->ipi_number) {
>> + state->ipi_number = xive_native_alloc_irq();
>> + if (state->ipi_number = 0) {
>> + pr_err("Failed to allocate IRQ !\n");
>> + return -ENXIO;
>> + }
>> + xive_native_populate_irq_data(state->ipi_number,
>> + &state->ipi_data);
>> + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
>> + state->ipi_number, irq);
>> + }
>> +
>> + arch_spin_lock(&sb->lock);
>
> Why the direct call to arch_spin_lock() rather than just spin_lock()?
Paul answered this question but may be I should make the effort to
decouple both devices on this aspect.
Thanks,
C.
>> +
>> + /* Restore LSI state */
>> + if (val & KVM_XIVE_LEVEL_SENSITIVE) {
>> + state->lsi = true;
>> + if (val & KVM_XIVE_LEVEL_ASSERTED)
>> + state->asserted = true;
>> + pr_devel(" LSI ! Asserted=%d\n", state->asserted);
>> + }
>> +
>> + /* Mask IRQ to start with */
>> + state->act_server = 0;
>> + state->act_priority = MASKED;
>> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
>> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
>> +
>> + /* Increment the number of valid sources and mark this one valid */
>> + if (!state->valid)
>> + xive->src_count++;
>> + state->valid = true;
>> +
>> + arch_spin_unlock(&sb->lock);
>> +
>> + return 0;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> + struct kvmppc_xive *xive = dev->private;
>> +
>> switch (attr->group) {
>> case KVM_DEV_XIVE_GRP_CTRL:
>> break;
>> + case KVM_DEV_XIVE_GRP_SOURCE:
>> + return kvmppc_xive_native_set_source(xive, attr->attr,
>> + attr->addr);
>> }
>> return -ENXIO;
>> }
>> @@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> switch (attr->group) {
>> case KVM_DEV_XIVE_GRP_CTRL:
>> break;
>> + case KVM_DEV_XIVE_GRP_SOURCE:
>> + if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
>> + attr->attr < KVMPPC_XIVE_NR_IRQS)
>> + return 0;
>> + break;
>> }
>> return -ENXIO;
>> }
>> @@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
>> {
>> struct kvmppc_xive *xive = dev->private;
>> struct kvm *kvm = xive->kvm;
>> + int i;
>>
>> debugfs_remove(xive->dentry);
>>
>> @@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
>> if (kvm)
>> kvm->arch.xive = NULL;
>>
>> + /* Mask and free interrupts */
>> + for (i = 0; i <= xive->max_sbid; i++) {
>> + if (xive->src_blocks[i])
>> + kvmppc_xive_free_sources(xive->src_blocks[i]);
>> + kfree(xive->src_blocks[i]);
>> + xive->src_blocks[i] = NULL;
>> + }
>> +
>> if (xive->vp_base != XIVE_INVALID_VP)
>> xive_native_free_vp_block(xive->vp_base);
>>
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index fdbd2ff92a88..cd8bfc37b72e 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>>
>> 1. KVM_DEV_XIVE_GRP_CTRL
>> Provides global controls on the device
>> +
>> + 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
>> + Initializes a new source in the XIVE device and mask it.
>> + Attributes:
>> + Interrupt source number (64-bit)
>> + The kvm_device_attr.addr points to a __u64 value:
>> + bits: | 63 .... 2 | 1 | 0
>> + values: | unused | level | type
>> + - type: 0:MSI 1:LSI
>> + - level: assertion level in case of an LSI.
>> + Errors:
>> + -E2BIG: Interrupt source number is out of range
>> + -ENOMEM: Could not create a new source block
>> + -EFAULT: Invalid user pointer for attr->addr.
>> + -ENXIO: Could not allocate underlying HW interrupt
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-03-12 15:19 ` Cédric Le Goater
@ 2019-03-14 2:15 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:15 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 12358 bytes --]
On Tue, Mar 12, 2019 at 04:19:35PM +0100, Cédric Le Goater wrote:
> On 2/25/19 3:10 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> >> The associated HW interrupt source is simply allocated at the OPAL/HW
> >> level and then MASKED. KVM only needs to know about its type: LSI or
> >> MSI.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/uapi/asm/kvm.h | 5 +
> >> arch/powerpc/kvm/book3s_xive.h | 10 ++
> >> arch/powerpc/kvm/book3s_xive.c | 8 +-
> >> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
> >> Documentation/virtual/kvm/devices/xive.txt | 15 +++
> >> 5 files changed, 148 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> >> index b002c0c67787..a9ad99f2a11b 100644
> >> --- a/arch/powerpc/include/uapi/asm/kvm.h
> >> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> >> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
> >>
> >> /* POWER9 XIVE Native Interrupt Controller */
> >> #define KVM_DEV_XIVE_GRP_CTRL 1
> >> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> >> +
> >> +/* Layout of 64-bit XIVE source attribute values */
> >> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> >> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
> >>
> >> #endif /* __LINUX_KVM_POWERPC_H */
> >> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> >> index bcb1bbcf0359..f22f2d46d0f0 100644
> >> --- a/arch/powerpc/kvm/book3s_xive.h
> >> +++ b/arch/powerpc/kvm/book3s_xive.h
> >> @@ -12,6 +12,13 @@
> >> #ifdef CONFIG_KVM_XICS
> >> #include "book3s_xics.h"
> >>
> >> +/*
> >> + * The XIVE IRQ number space is aligned with the XICS IRQ number
> >> + * space, CPU IPIs being allocated in the first 4K.
> >
> > We do align these in qemu, but I don't see that the kernel part
> > cares: as far as it's concerned only one of XICS or XIVE is active at
> > a time, and the irq numbers are chosen by userspace.
>
> There is some relation with userspace nevertheless. The KVM device does
> not remap the numbers to some other range today and the limits are fixed
> values. Checks are being done in the has_attr() and the set_attr().
Hrm. I still think the comment needs to describe what the constraint
is from the point of view of the kernel, which this isn't. Maybe, "we
allow irqs in the range X..Y (allowing userspace to do ...)"
>
> >> + */
> >> +#define KVMPPC_XIVE_FIRST_IRQ 0
> >> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
> >> +
> >> /*
> >> * State for one guest irq source.
> >> *
> >> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> >> */
> >> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> >> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> >> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> >> + struct kvmppc_xive *xive, int irq);
> >> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> >>
> >> #endif /* CONFIG_KVM_XICS */
> >> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> >> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> >> index d1cc18a5b1c4..6f950ecb3592 100644
> >> --- a/arch/powerpc/kvm/book3s_xive.c
> >> +++ b/arch/powerpc/kvm/book3s_xive.c
> >
> > I wonder if we should rename this book3s_xics_on_xive.c or something
> > at some point, I keep getting confused because I forget that this is
> > only dealing with host xive, not guest xive.
>
> I am fine with renaming. Any objections ? book3s_xics_p9.c ?
Hm, maybe?
> >> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> >> return 0;
> >> }
> >>
> >> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> >> - int irq)
> >> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> >> + struct kvmppc_xive *xive, int irq)
> >> {
> >> struct kvm *kvm = xive->kvm;
> >> struct kvmppc_xive_src_block *sb;
> >
> > It's odd that this function, now used from the xive-on-xive path as
> > well as the xics-on-xive path references KVMPPC_XICS_ICS_SHIFT a few
> > lines down from this change.
>
> Yes. This is because of the definition of the struct kvmppc_xive_src_block.
>
> We could introduce new defines for XIVE or a common set of defines for
> XICS and XIVE.
I think making common definitions would be best, if possible.
>
> >> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> >> sb = kvmppc_xive_find_source(xive, irq, &idx);
> >> if (!sb) {
> >> pr_devel("No source, creating source block...\n");
> >> - sb = xive_create_src_block(xive, irq);
> >> + sb = kvmppc_xive_create_src_block(xive, irq);
> >> if (!sb) {
> >> pr_devel("Failed to create block...\n");
> >> return -ENOMEM;
> >> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> >> xive_cleanup_irq_data(xd);
> >> }
> >>
> >> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> >> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> >> {
> >> int i;
> >>
> >> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> >> index 1f3da47a4a6a..a9b2d2d9af99 100644
> >> --- a/arch/powerpc/kvm/book3s_xive_native.c
> >> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> >> @@ -31,6 +31,29 @@
> >>
> >> #include "book3s_xive.h"
> >>
> >> +/*
> >> + * TODO: introduce a common template file with the XIVE native layer
> >> + * and the XICS-on-XIVE glue for the utility functions
> >> + */
> >> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> >> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> >> +#define __x_readq __raw_readq
> >> +#define __x_writeq __raw_writeq
> >> +
> >> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
> >> +{
> >> + u64 val;
> >> +
> >> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> >> + offset |= offset << 4;
> >> +
> >> + val = __x_readq(__x_eoi_page(xd) + offset);
> >> +#ifdef __LITTLE_ENDIAN__
> >> + val >>= 64-8;
> >> +#endif
> >> + return (u8)val;
> >> +}
> >> +
> >> static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
> >> {
> >> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> @@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> return rc;
> >> }
> >>
> >> +static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> >> + u64 addr)
> >> +{
> >> + struct kvmppc_xive_src_block *sb;
> >> + struct kvmppc_xive_irq_state *state;
> >> + u64 __user *ubufp = (u64 __user *) addr;
> >> + u64 val;
> >> + u16 idx;
> >> +
> >> + pr_devel("%s irq=0x%lx\n", __func__, irq);
> >> +
> >> + if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
> >> + return -E2BIG;
> >> +
> >> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> >> + if (!sb) {
> >> + pr_debug("No source, creating source block...\n");
> >> + sb = kvmppc_xive_create_src_block(xive, irq);
> >> + if (!sb) {
> >> + pr_err("Failed to create block...\n");
> >> + return -ENOMEM;
> >> + }
> >> + }
> >> + state = &sb->irq_state[idx];
> >> +
> >> + if (get_user(val, ubufp)) {
> >> + pr_err("fault getting user info !\n");
> >> + return -EFAULT;
> >> + }
> >
> > You should validate the value loaded here to check it doesn't have any
> > bits set we don't know about.
>
> ok
>
> >
> >> +
> >> + /*
> >> + * If the source doesn't already have an IPI, allocate
> >> + * one and get the corresponding data
> >> + */
> >> + if (!state->ipi_number) {
> >> + state->ipi_number = xive_native_alloc_irq();
> >> + if (state->ipi_number == 0) {
> >> + pr_err("Failed to allocate IRQ !\n");
> >> + return -ENXIO;
> >> + }
> >> + xive_native_populate_irq_data(state->ipi_number,
> >> + &state->ipi_data);
> >> + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> >> + state->ipi_number, irq);
> >> + }
> >> +
> >> + arch_spin_lock(&sb->lock);
> >
> > Why the direct call to arch_spin_lock() rather than just spin_lock()?
>
> Paul answered this question but may be I should make the effort to
> decouple both devices on this aspect.
>
> Thanks,
>
> C.
>
> >> +
> >> + /* Restore LSI state */
> >> + if (val & KVM_XIVE_LEVEL_SENSITIVE) {
> >> + state->lsi = true;
> >> + if (val & KVM_XIVE_LEVEL_ASSERTED)
> >> + state->asserted = true;
> >> + pr_devel(" LSI ! Asserted=%d\n", state->asserted);
> >> + }
> >> +
> >> + /* Mask IRQ to start with */
> >> + state->act_server = 0;
> >> + state->act_priority = MASKED;
> >> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> >> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> >> +
> >> + /* Increment the number of valid sources and mark this one valid */
> >> + if (!state->valid)
> >> + xive->src_count++;
> >> + state->valid = true;
> >> +
> >> + arch_spin_unlock(&sb->lock);
> >> +
> >> + return 0;
> >> +}
> >> +
> >> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> >> struct kvm_device_attr *attr)
> >> {
> >> + struct kvmppc_xive *xive = dev->private;
> >> +
> >> switch (attr->group) {
> >> case KVM_DEV_XIVE_GRP_CTRL:
> >> break;
> >> + case KVM_DEV_XIVE_GRP_SOURCE:
> >> + return kvmppc_xive_native_set_source(xive, attr->attr,
> >> + attr->addr);
> >> }
> >> return -ENXIO;
> >> }
> >> @@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> >> switch (attr->group) {
> >> case KVM_DEV_XIVE_GRP_CTRL:
> >> break;
> >> + case KVM_DEV_XIVE_GRP_SOURCE:
> >> + if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> >> + attr->attr < KVMPPC_XIVE_NR_IRQS)
> >> + return 0;
> >> + break;
> >> }
> >> return -ENXIO;
> >> }
> >> @@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> >> {
> >> struct kvmppc_xive *xive = dev->private;
> >> struct kvm *kvm = xive->kvm;
> >> + int i;
> >>
> >> debugfs_remove(xive->dentry);
> >>
> >> @@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> >> if (kvm)
> >> kvm->arch.xive = NULL;
> >>
> >> + /* Mask and free interrupts */
> >> + for (i = 0; i <= xive->max_sbid; i++) {
> >> + if (xive->src_blocks[i])
> >> + kvmppc_xive_free_sources(xive->src_blocks[i]);
> >> + kfree(xive->src_blocks[i]);
> >> + xive->src_blocks[i] = NULL;
> >> + }
> >> +
> >> if (xive->vp_base != XIVE_INVALID_VP)
> >> xive_native_free_vp_block(xive->vp_base);
> >>
> >> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> >> index fdbd2ff92a88..cd8bfc37b72e 100644
> >> --- a/Documentation/virtual/kvm/devices/xive.txt
> >> +++ b/Documentation/virtual/kvm/devices/xive.txt
> >> @@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> >>
> >> 1. KVM_DEV_XIVE_GRP_CTRL
> >> Provides global controls on the device
> >> +
> >> + 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> >> + Initializes a new source in the XIVE device and mask it.
> >> + Attributes:
> >> + Interrupt source number (64-bit)
> >> + The kvm_device_attr.addr points to a __u64 value:
> >> + bits: | 63 .... 2 | 1 | 0
> >> + values: | unused | level | type
> >> + - type: 0:MSI 1:LSI
> >> + - level: assertion level in case of an LSI.
> >> + Errors:
> >> + -E2BIG: Interrupt source number is out of range
> >> + -ENOMEM: Could not create a new source block
> >> + -EFAULT: Invalid user pointer for attr->addr.
> >> + -ENXIO: Could not allocate underlying HW interrupt
> >
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-03-14 2:15 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:15 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 12358 bytes --]
On Tue, Mar 12, 2019 at 04:19:35PM +0100, Cédric Le Goater wrote:
> On 2/25/19 3:10 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> >> The associated HW interrupt source is simply allocated at the OPAL/HW
> >> level and then MASKED. KVM only needs to know about its type: LSI or
> >> MSI.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/uapi/asm/kvm.h | 5 +
> >> arch/powerpc/kvm/book3s_xive.h | 10 ++
> >> arch/powerpc/kvm/book3s_xive.c | 8 +-
> >> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
> >> Documentation/virtual/kvm/devices/xive.txt | 15 +++
> >> 5 files changed, 148 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> >> index b002c0c67787..a9ad99f2a11b 100644
> >> --- a/arch/powerpc/include/uapi/asm/kvm.h
> >> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> >> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
> >>
> >> /* POWER9 XIVE Native Interrupt Controller */
> >> #define KVM_DEV_XIVE_GRP_CTRL 1
> >> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> >> +
> >> +/* Layout of 64-bit XIVE source attribute values */
> >> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> >> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
> >>
> >> #endif /* __LINUX_KVM_POWERPC_H */
> >> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> >> index bcb1bbcf0359..f22f2d46d0f0 100644
> >> --- a/arch/powerpc/kvm/book3s_xive.h
> >> +++ b/arch/powerpc/kvm/book3s_xive.h
> >> @@ -12,6 +12,13 @@
> >> #ifdef CONFIG_KVM_XICS
> >> #include "book3s_xics.h"
> >>
> >> +/*
> >> + * The XIVE IRQ number space is aligned with the XICS IRQ number
> >> + * space, CPU IPIs being allocated in the first 4K.
> >
> > We do align these in qemu, but I don't see that the kernel part
> > cares: as far as it's concerned only one of XICS or XIVE is active at
> > a time, and the irq numbers are chosen by userspace.
>
> There is some relation with userspace nevertheless. The KVM device does
> not remap the numbers to some other range today and the limits are fixed
> values. Checks are being done in the has_attr() and the set_attr().
Hrm. I still think the comment needs to describe what the constraint
is from the point of view of the kernel, which this isn't. Maybe, "we
allow irqs in the range X..Y (allowing userspace to do ...)"
>
> >> + */
> >> +#define KVMPPC_XIVE_FIRST_IRQ 0
> >> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
> >> +
> >> /*
> >> * State for one guest irq source.
> >> *
> >> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> >> */
> >> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> >> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> >> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> >> + struct kvmppc_xive *xive, int irq);
> >> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> >>
> >> #endif /* CONFIG_KVM_XICS */
> >> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> >> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> >> index d1cc18a5b1c4..6f950ecb3592 100644
> >> --- a/arch/powerpc/kvm/book3s_xive.c
> >> +++ b/arch/powerpc/kvm/book3s_xive.c
> >
> > I wonder if we should rename this book3s_xics_on_xive.c or something
> > at some point, I keep getting confused because I forget that this is
> > only dealing with host xive, not guest xive.
>
> I am fine with renaming. Any objections ? book3s_xics_p9.c ?
Hm, maybe?
> >> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> >> return 0;
> >> }
> >>
> >> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> >> - int irq)
> >> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> >> + struct kvmppc_xive *xive, int irq)
> >> {
> >> struct kvm *kvm = xive->kvm;
> >> struct kvmppc_xive_src_block *sb;
> >
> > It's odd that this function, now used from the xive-on-xive path as
> > well as the xics-on-xive path references KVMPPC_XICS_ICS_SHIFT a few
> > lines down from this change.
>
> Yes. This is because of the definition of the struct kvmppc_xive_src_block.
>
> We could introduce new defines for XIVE or a common set of defines for
> XICS and XIVE.
I think making common definitions would be best, if possible.
>
> >> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> >> sb = kvmppc_xive_find_source(xive, irq, &idx);
> >> if (!sb) {
> >> pr_devel("No source, creating source block...\n");
> >> - sb = xive_create_src_block(xive, irq);
> >> + sb = kvmppc_xive_create_src_block(xive, irq);
> >> if (!sb) {
> >> pr_devel("Failed to create block...\n");
> >> return -ENOMEM;
> >> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> >> xive_cleanup_irq_data(xd);
> >> }
> >>
> >> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> >> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> >> {
> >> int i;
> >>
> >> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> >> index 1f3da47a4a6a..a9b2d2d9af99 100644
> >> --- a/arch/powerpc/kvm/book3s_xive_native.c
> >> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> >> @@ -31,6 +31,29 @@
> >>
> >> #include "book3s_xive.h"
> >>
> >> +/*
> >> + * TODO: introduce a common template file with the XIVE native layer
> >> + * and the XICS-on-XIVE glue for the utility functions
> >> + */
> >> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> >> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> >> +#define __x_readq __raw_readq
> >> +#define __x_writeq __raw_writeq
> >> +
> >> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
> >> +{
> >> + u64 val;
> >> +
> >> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> >> + offset |= offset << 4;
> >> +
> >> + val = __x_readq(__x_eoi_page(xd) + offset);
> >> +#ifdef __LITTLE_ENDIAN__
> >> + val >>= 64-8;
> >> +#endif
> >> + return (u8)val;
> >> +}
> >> +
> >> static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
> >> {
> >> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> @@ -153,12 +176,89 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> return rc;
> >> }
> >>
> >> +static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> >> + u64 addr)
> >> +{
> >> + struct kvmppc_xive_src_block *sb;
> >> + struct kvmppc_xive_irq_state *state;
> >> + u64 __user *ubufp = (u64 __user *) addr;
> >> + u64 val;
> >> + u16 idx;
> >> +
> >> + pr_devel("%s irq=0x%lx\n", __func__, irq);
> >> +
> >> + if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
> >> + return -E2BIG;
> >> +
> >> + sb = kvmppc_xive_find_source(xive, irq, &idx);
> >> + if (!sb) {
> >> + pr_debug("No source, creating source block...\n");
> >> + sb = kvmppc_xive_create_src_block(xive, irq);
> >> + if (!sb) {
> >> + pr_err("Failed to create block...\n");
> >> + return -ENOMEM;
> >> + }
> >> + }
> >> + state = &sb->irq_state[idx];
> >> +
> >> + if (get_user(val, ubufp)) {
> >> + pr_err("fault getting user info !\n");
> >> + return -EFAULT;
> >> + }
> >
> > You should validate the value loaded here to check it doesn't have any
> > bits set we don't know about.
>
> ok
>
> >
> >> +
> >> + /*
> >> + * If the source doesn't already have an IPI, allocate
> >> + * one and get the corresponding data
> >> + */
> >> + if (!state->ipi_number) {
> >> + state->ipi_number = xive_native_alloc_irq();
> >> + if (state->ipi_number == 0) {
> >> + pr_err("Failed to allocate IRQ !\n");
> >> + return -ENXIO;
> >> + }
> >> + xive_native_populate_irq_data(state->ipi_number,
> >> + &state->ipi_data);
> >> + pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
> >> + state->ipi_number, irq);
> >> + }
> >> +
> >> + arch_spin_lock(&sb->lock);
> >
> > Why the direct call to arch_spin_lock() rather than just spin_lock()?
>
> Paul answered this question but may be I should make the effort to
> decouple both devices on this aspect.
>
> Thanks,
>
> C.
>
> >> +
> >> + /* Restore LSI state */
> >> + if (val & KVM_XIVE_LEVEL_SENSITIVE) {
> >> + state->lsi = true;
> >> + if (val & KVM_XIVE_LEVEL_ASSERTED)
> >> + state->asserted = true;
> >> + pr_devel(" LSI ! Asserted=%d\n", state->asserted);
> >> + }
> >> +
> >> + /* Mask IRQ to start with */
> >> + state->act_server = 0;
> >> + state->act_priority = MASKED;
> >> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> >> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> >> +
> >> + /* Increment the number of valid sources and mark this one valid */
> >> + if (!state->valid)
> >> + xive->src_count++;
> >> + state->valid = true;
> >> +
> >> + arch_spin_unlock(&sb->lock);
> >> +
> >> + return 0;
> >> +}
> >> +
> >> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> >> struct kvm_device_attr *attr)
> >> {
> >> + struct kvmppc_xive *xive = dev->private;
> >> +
> >> switch (attr->group) {
> >> case KVM_DEV_XIVE_GRP_CTRL:
> >> break;
> >> + case KVM_DEV_XIVE_GRP_SOURCE:
> >> + return kvmppc_xive_native_set_source(xive, attr->attr,
> >> + attr->addr);
> >> }
> >> return -ENXIO;
> >> }
> >> @@ -175,6 +275,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> >> switch (attr->group) {
> >> case KVM_DEV_XIVE_GRP_CTRL:
> >> break;
> >> + case KVM_DEV_XIVE_GRP_SOURCE:
> >> + if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> >> + attr->attr < KVMPPC_XIVE_NR_IRQS)
> >> + return 0;
> >> + break;
> >> }
> >> return -ENXIO;
> >> }
> >> @@ -183,6 +288,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> >> {
> >> struct kvmppc_xive *xive = dev->private;
> >> struct kvm *kvm = xive->kvm;
> >> + int i;
> >>
> >> debugfs_remove(xive->dentry);
> >>
> >> @@ -191,6 +297,14 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> >> if (kvm)
> >> kvm->arch.xive = NULL;
> >>
> >> + /* Mask and free interrupts */
> >> + for (i = 0; i <= xive->max_sbid; i++) {
> >> + if (xive->src_blocks[i])
> >> + kvmppc_xive_free_sources(xive->src_blocks[i]);
> >> + kfree(xive->src_blocks[i]);
> >> + xive->src_blocks[i] = NULL;
> >> + }
> >> +
> >> if (xive->vp_base != XIVE_INVALID_VP)
> >> xive_native_free_vp_block(xive->vp_base);
> >>
> >> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> >> index fdbd2ff92a88..cd8bfc37b72e 100644
> >> --- a/Documentation/virtual/kvm/devices/xive.txt
> >> +++ b/Documentation/virtual/kvm/devices/xive.txt
> >> @@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> >>
> >> 1. KVM_DEV_XIVE_GRP_CTRL
> >> Provides global controls on the device
> >> +
> >> + 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> >> + Initializes a new source in the XIVE device and mask it.
> >> + Attributes:
> >> + Interrupt source number (64-bit)
> >> + The kvm_device_attr.addr points to a __u64 value:
> >> + bits: | 63 .... 2 | 1 | 0
> >> + values: | unused | level | type
> >> + - type: 0:MSI 1:LSI
> >> + - level: assertion level in case of an LSI.
> >> + Errors:
> >> + -E2BIG: Interrupt source number is out of range
> >> + -ENOMEM: Could not create a new source block
> >> + -EFAULT: Invalid user pointer for attr->addr.
> >> + -ENXIO: Could not allocate underlying HW interrupt
> >
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 5:30 ` Paul Mackerras
-1 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-25 5:30 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> The associated HW interrupt source is simply allocated at the OPAL/HW
> level and then MASKED. KVM only needs to know about its type: LSI or
> MSI.
I think it would be helpful to explain to the reader here that with
XIVE, all interrupts have a hardware source, even IPIs and virtual
device interrupts, for which we allocate a software-triggerable
interrupt source in the XIVE hardware.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 5 +
> arch/powerpc/kvm/book3s_xive.h | 10 ++
> arch/powerpc/kvm/book3s_xive.c | 8 +-
> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 15 +++
> 5 files changed, 148 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index b002c0c67787..a9ad99f2a11b 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
>
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> +
> +/* Layout of 64-bit XIVE source attribute values */
> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index bcb1bbcf0359..f22f2d46d0f0 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -12,6 +12,13 @@
> #ifdef CONFIG_KVM_XICS
> #include "book3s_xics.h"
>
> +/*
> + * The XIVE IRQ number space is aligned with the XICS IRQ number
> + * space, CPU IPIs being allocated in the first 4K.
> + */
> +#define KVMPPC_XIVE_FIRST_IRQ 0
> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
> +
> /*
> * State for one guest irq source.
> *
> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> */
> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq);
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
So we're using the same source block data structure (effectively a
2-level tree) that the XICS-on-XIVE code uses. That would be worth
mentioning as a design choice in the patch description.
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index d1cc18a5b1c4..6f950ecb3592 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> return 0;
> }
>
> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> - int irq)
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq)
> {
> struct kvm *kvm = xive->kvm;
> struct kvmppc_xive_src_block *sb;
> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> sb = kvmppc_xive_find_source(xive, irq, &idx);
> if (!sb) {
> pr_devel("No source, creating source block...\n");
> - sb = xive_create_src_block(xive, irq);
> + sb = kvmppc_xive_create_src_block(xive, irq);
> if (!sb) {
> pr_devel("Failed to create block...\n");
> return -ENOMEM;
> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> xive_cleanup_irq_data(xd);
> }
>
> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> {
> int i;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 1f3da47a4a6a..a9b2d2d9af99 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -31,6 +31,29 @@
>
> #include "book3s_xive.h"
>
> +/*
> + * TODO: introduce a common template file with the XIVE native layer
> + * and the XICS-on-XIVE glue for the utility functions
> + */
> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> +#define __x_readq __raw_readq
> +#define __x_writeq __raw_writeq
> +
> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = __x_readq(__x_eoi_page(xd) + offset);
Ben originally defined the __x_* macros so that he could use the same
source code twice, once for real mode and once for virtual mode.
Since you're not doing that, is there really any reason for this
indirection? Why not just __raw_readq, __raw_writeq etc. directly?
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-02-25 5:30 ` Paul Mackerras
0 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-25 5:30 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On Fri, Feb 22, 2019 at 12:28:28PM +0100, Cédric Le Goater wrote:
> The associated HW interrupt source is simply allocated at the OPAL/HW
> level and then MASKED. KVM only needs to know about its type: LSI or
> MSI.
I think it would be helpful to explain to the reader here that with
XIVE, all interrupts have a hardware source, even IPIs and virtual
device interrupts, for which we allocate a software-triggerable
interrupt source in the XIVE hardware.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 5 +
> arch/powerpc/kvm/book3s_xive.h | 10 ++
> arch/powerpc/kvm/book3s_xive.c | 8 +-
> arch/powerpc/kvm/book3s_xive_native.c | 114 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 15 +++
> 5 files changed, 148 insertions(+), 4 deletions(-)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index b002c0c67787..a9ad99f2a11b 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -677,5 +677,10 @@ struct kvm_ppc_cpu_char {
>
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> +#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> +
> +/* Layout of 64-bit XIVE source attribute values */
> +#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> +#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index bcb1bbcf0359..f22f2d46d0f0 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -12,6 +12,13 @@
> #ifdef CONFIG_KVM_XICS
> #include "book3s_xics.h"
>
> +/*
> + * The XIVE IRQ number space is aligned with the XICS IRQ number
> + * space, CPU IPIs being allocated in the first 4K.
> + */
> +#define KVMPPC_XIVE_FIRST_IRQ 0
> +#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
> +
> /*
> * State for one guest irq source.
> *
> @@ -253,6 +260,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
> */
> void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
> int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq);
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
So we're using the same source block data structure (effectively a
2-level tree) that the XICS-on-XIVE code uses. That would be worth
mentioning as a design choice in the patch description.
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index d1cc18a5b1c4..6f950ecb3592 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1485,8 +1485,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
> return 0;
> }
>
> -static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
> - int irq)
> +struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> + struct kvmppc_xive *xive, int irq)
> {
> struct kvm *kvm = xive->kvm;
> struct kvmppc_xive_src_block *sb;
> @@ -1565,7 +1565,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
> sb = kvmppc_xive_find_source(xive, irq, &idx);
> if (!sb) {
> pr_devel("No source, creating source block...\n");
> - sb = xive_create_src_block(xive, irq);
> + sb = kvmppc_xive_create_src_block(xive, irq);
> if (!sb) {
> pr_devel("Failed to create block...\n");
> return -ENOMEM;
> @@ -1789,7 +1789,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
> xive_cleanup_irq_data(xd);
> }
>
> -static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> +void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
> {
> int i;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 1f3da47a4a6a..a9b2d2d9af99 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -31,6 +31,29 @@
>
> #include "book3s_xive.h"
>
> +/*
> + * TODO: introduce a common template file with the XIVE native layer
> + * and the XICS-on-XIVE glue for the utility functions
> + */
> +#define __x_eoi_page(xd) ((void __iomem *)((xd)->eoi_mmio))
> +#define __x_trig_page(xd) ((void __iomem *)((xd)->trig_mmio))
> +#define __x_readq __raw_readq
> +#define __x_writeq __raw_writeq
> +
> +static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
> +{
> + u64 val;
> +
> + if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
> + offset |= offset << 4;
> +
> + val = __x_readq(__x_eoi_page(xd) + offset);
Ben originally defined the __x_* macros so that he could use the same
source code twice, once for real mode and once for virtual mode.
Since you're not doing that, is there really any reason for this
indirection? Why not just __raw_readq, __raw_writeq etc. directly?
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
QEMU and also to restore the configuration of the source in the KVM
device.
The XIVE internal IRQ structure is extended with the value of the
Effective Interrupt Source Number. The EISN is the interrupt number
pushed in the event queue that the guest OS will use to dispatch
events internally. Caching the EISN value in KVM ease the test when
checking if a reconfiguration is indeed needed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 11 +++
arch/powerpc/kvm/book3s_xive.h | 4 +
arch/powerpc/kvm/book3s_xive.c | 5 +-
arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 20 +++++
5 files changed, 135 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index a9ad99f2a11b..91899c7f9abd 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -678,9 +678,20 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
+#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
+/* Layout of 64-bit XIVE source configuration attribute values */
+#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
+#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
+#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
+#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
+#define KVM_XIVE_SOURCE_MASK_SHIFT 32
+#define KVM_XIVE_SOURCE_MASK_MASK 0x100000000ULL
+#define KVM_XIVE_SOURCE_EISN_SHIFT 33
+#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index f22f2d46d0f0..ab3ac152980d 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
bool saved_p;
bool saved_q;
u8 saved_scan_prio;
+
+ /* Xive native */
+ u32 eisn; /* Guest Effective IRQ number */
};
/* Select the "right" interrupt (IPI vs. passthrough) */
@@ -263,6 +266,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 6f950ecb3592..086da91d7c6e 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
}
-static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
{
struct kvm_vcpu *vcpu;
int i, rc;
@@ -535,7 +535,7 @@ static int xive_target_interrupt(struct kvm *kvm,
* priority. The count for that new target will have
* already been incremented.
*/
- rc = xive_select_target(kvm, &server, prio);
+ rc = kvmppc_xive_select_target(kvm, &server, prio);
/*
* We failed to find a target ? Not much we can do
@@ -1509,6 +1509,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
+ sb->irq_state[i].eisn = 0;
sb->irq_state[i].guest_priority = MASKED;
sb->irq_state[i].saved_priority = MASKED;
sb->irq_state[i].act_priority = MASKED;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index a9b2d2d9af99..cb5a5c6e05af 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -248,6 +248,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
return 0;
}
+static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state,
+ u32 server,
+ u8 priority,
+ u32 eisn)
+{
+ struct kvm *kvm = xive->kvm;
+ u32 hw_num;
+ int rc = 0;
+
+ /*
+ * TODO: Do we need to safely mask and unmask a source ? can
+ * we just let the guest handle the possible races ?
+ */
+ arch_spin_lock(&sb->lock);
+
+ if (state->act_server == server && state->act_priority == priority &&
+ state->eisn == eisn)
+ goto unlock;
+
+ pr_devel("new_act_prio=%d new_act_server=%d act_server=%d act_prio=%d\n",
+ priority, server, state->act_server, state->act_priority);
+
+ kvmppc_xive_select_irq(state, &hw_num, NULL);
+
+ if (priority != MASKED) {
+ rc = kvmppc_xive_select_target(kvm, &server, priority);
+ if (rc)
+ goto unlock;
+
+ state->act_priority = priority;
+ state->act_server = server;
+ state->eisn = eisn;
+
+ rc = xive_native_configure_irq(hw_num, xive->vp_base + server,
+ priority, eisn);
+ } else {
+ state->act_priority = MASKED;
+ state->act_server = 0;
+ state->eisn = 0;
+
+ rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
+ }
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
+static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u16 src;
+ u64 kvm_cfg;
+ u32 server;
+ u8 priority;
+ u32 eisn;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ if (get_user(kvm_cfg, ubufp))
+ return -EFAULT;
+
+ pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
+
+ priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
+ KVM_XIVE_SOURCE_PRIORITY_SHIFT;
+ server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
+ KVM_XIVE_SOURCE_SERVER_SHIFT;
+ eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
+ KVM_XIVE_SOURCE_EISN_SHIFT;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+
+ return kvmppc_xive_native_update_source_config(xive, sb, state, server,
+ priority, eisn);
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -259,6 +352,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ return kvmppc_xive_native_set_source_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -276,6 +372,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
break;
case KVM_DEV_XIVE_GRP_SOURCE:
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cd8bfc37b72e..4f513a1880c7 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -32,3 +32,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOMEM: Could not create a new source block
-EFAULT: Invalid user pointer for attr->addr.
-ENXIO: Could not allocate underlying HW interrupt
+
+ 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
+ Configures source targeting
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
+ values: | eisn | mask | server | priority
+ - priority: 0-7 interrupt priority level
+ - server: CPU number chosen to handle the interrupt
+ - mask: mask flag (unused)
+ - eisn: Effective Interrupt Source Number
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number, invalid priority or
+ invalid CPU number.
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: CPU event queues not configured or configuration of the
+ underlying HW interrupt failed
+ -EBUSY: No CPU available to serve interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
QEMU and also to restore the configuration of the source in the KVM
device.
The XIVE internal IRQ structure is extended with the value of the
Effective Interrupt Source Number. The EISN is the interrupt number
pushed in the event queue that the guest OS will use to dispatch
events internally. Caching the EISN value in KVM ease the test when
checking if a reconfiguration is indeed needed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 11 +++
arch/powerpc/kvm/book3s_xive.h | 4 +
arch/powerpc/kvm/book3s_xive.c | 5 +-
arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 20 +++++
5 files changed, 135 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index a9ad99f2a11b..91899c7f9abd 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -678,9 +678,20 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
+#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
+/* Layout of 64-bit XIVE source configuration attribute values */
+#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
+#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
+#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
+#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
+#define KVM_XIVE_SOURCE_MASK_SHIFT 32
+#define KVM_XIVE_SOURCE_MASK_MASK 0x100000000ULL
+#define KVM_XIVE_SOURCE_EISN_SHIFT 33
+#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index f22f2d46d0f0..ab3ac152980d 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
bool saved_p;
bool saved_q;
u8 saved_scan_prio;
+
+ /* Xive native */
+ u32 eisn; /* Guest Effective IRQ number */
};
/* Select the "right" interrupt (IPI vs. passthrough) */
@@ -263,6 +266,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 6f950ecb3592..086da91d7c6e 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
}
-static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
{
struct kvm_vcpu *vcpu;
int i, rc;
@@ -535,7 +535,7 @@ static int xive_target_interrupt(struct kvm *kvm,
* priority. The count for that new target will have
* already been incremented.
*/
- rc = xive_select_target(kvm, &server, prio);
+ rc = kvmppc_xive_select_target(kvm, &server, prio);
/*
* We failed to find a target ? Not much we can do
@@ -1509,6 +1509,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
+ sb->irq_state[i].eisn = 0;
sb->irq_state[i].guest_priority = MASKED;
sb->irq_state[i].saved_priority = MASKED;
sb->irq_state[i].act_priority = MASKED;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index a9b2d2d9af99..cb5a5c6e05af 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -248,6 +248,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
return 0;
}
+static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state,
+ u32 server,
+ u8 priority,
+ u32 eisn)
+{
+ struct kvm *kvm = xive->kvm;
+ u32 hw_num;
+ int rc = 0;
+
+ /*
+ * TODO: Do we need to safely mask and unmask a source ? can
+ * we just let the guest handle the possible races ?
+ */
+ arch_spin_lock(&sb->lock);
+
+ if (state->act_server = server && state->act_priority = priority &&
+ state->eisn = eisn)
+ goto unlock;
+
+ pr_devel("new_act_prio=%d new_act_server=%d act_server=%d act_prio=%d\n",
+ priority, server, state->act_server, state->act_priority);
+
+ kvmppc_xive_select_irq(state, &hw_num, NULL);
+
+ if (priority != MASKED) {
+ rc = kvmppc_xive_select_target(kvm, &server, priority);
+ if (rc)
+ goto unlock;
+
+ state->act_priority = priority;
+ state->act_server = server;
+ state->eisn = eisn;
+
+ rc = xive_native_configure_irq(hw_num, xive->vp_base + server,
+ priority, eisn);
+ } else {
+ state->act_priority = MASKED;
+ state->act_server = 0;
+ state->eisn = 0;
+
+ rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
+ }
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
+static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u16 src;
+ u64 kvm_cfg;
+ u32 server;
+ u8 priority;
+ u32 eisn;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ if (get_user(kvm_cfg, ubufp))
+ return -EFAULT;
+
+ pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
+
+ priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
+ KVM_XIVE_SOURCE_PRIORITY_SHIFT;
+ server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
+ KVM_XIVE_SOURCE_SERVER_SHIFT;
+ eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
+ KVM_XIVE_SOURCE_EISN_SHIFT;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+
+ return kvmppc_xive_native_update_source_config(xive, sb, state, server,
+ priority, eisn);
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -259,6 +352,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ return kvmppc_xive_native_set_source_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -276,6 +372,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
break;
case KVM_DEV_XIVE_GRP_SOURCE:
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cd8bfc37b72e..4f513a1880c7 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -32,3 +32,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOMEM: Could not create a new source block
-EFAULT: Invalid user pointer for attr->addr.
-ENXIO: Could not allocate underlying HW interrupt
+
+ 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
+ Configures source targeting
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
+ values: | eisn | mask | server | priority
+ - priority: 0-7 interrupt priority level
+ - server: CPU number chosen to handle the interrupt
+ - mask: mask flag (unused)
+ - eisn: Effective Interrupt Source Number
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number, invalid priority or
+ invalid CPU number.
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: CPU event queues not configured or configuration of the
+ underlying HW interrupt failed
+ -EBUSY: No CPU available to serve interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 2:21 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:21 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 9807 bytes --]
On Fri, Feb 22, 2019 at 12:28:29PM +0100, Cédric Le Goater wrote:
> This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
> QEMU and also to restore the configuration of the source in the KVM
> device.
>
> The XIVE internal IRQ structure is extended with the value of the
> Effective Interrupt Source Number. The EISN is the interrupt number
> pushed in the event queue that the guest OS will use to dispatch
> events internally. Caching the EISN value in KVM ease the test when
> checking if a reconfiguration is indeed needed.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 11 +++
> arch/powerpc/kvm/book3s_xive.h | 4 +
> arch/powerpc/kvm/book3s_xive.c | 5 +-
> arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 20 +++++
> 5 files changed, 135 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index a9ad99f2a11b..91899c7f9abd 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -678,9 +678,20 @@ struct kvm_ppc_cpu_char {
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> +#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> #define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>
> +/* Layout of 64-bit XIVE source configuration attribute values */
> +#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
> +#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
> +#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
> +#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
> +#define KVM_XIVE_SOURCE_MASK_SHIFT 32
> +#define KVM_XIVE_SOURCE_MASK_MASK 0x100000000ULL
> +#define KVM_XIVE_SOURCE_EISN_SHIFT 33
> +#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index f22f2d46d0f0..ab3ac152980d 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
> bool saved_p;
> bool saved_q;
> u8 saved_scan_prio;
> +
> + /* Xive native */
> + u32 eisn; /* Guest Effective IRQ number */
> };
>
> /* Select the "right" interrupt (IPI vs. passthrough) */
> @@ -263,6 +266,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> +int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 6f950ecb3592..086da91d7c6e 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
> return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
> }
>
> -static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
> +int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
> {
> struct kvm_vcpu *vcpu;
> int i, rc;
> @@ -535,7 +535,7 @@ static int xive_target_interrupt(struct kvm *kvm,
> * priority. The count for that new target will have
> * already been incremented.
> */
> - rc = xive_select_target(kvm, &server, prio);
> + rc = kvmppc_xive_select_target(kvm, &server, prio);
>
> /*
> * We failed to find a target ? Not much we can do
> @@ -1509,6 +1509,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>
> for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
> sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
> + sb->irq_state[i].eisn = 0;
> sb->irq_state[i].guest_priority = MASKED;
> sb->irq_state[i].saved_priority = MASKED;
> sb->irq_state[i].act_priority = MASKED;
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index a9b2d2d9af99..cb5a5c6e05af 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -248,6 +248,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> return 0;
> }
>
> +static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
> + struct kvmppc_xive_src_block *sb,
> + struct kvmppc_xive_irq_state *state,
> + u32 server,
> + u8 priority,
> + u32 eisn)
> +{
> + struct kvm *kvm = xive->kvm;
> + u32 hw_num;
> + int rc = 0;
> +
> + /*
> + * TODO: Do we need to safely mask and unmask a source ? can
> + * we just let the guest handle the possible races ?
> + */
> + arch_spin_lock(&sb->lock);
> +
> + if (state->act_server == server && state->act_priority == priority &&
> + state->eisn == eisn)
> + goto unlock;
> +
> + pr_devel("new_act_prio=%d new_act_server=%d act_server=%d act_prio=%d\n",
> + priority, server, state->act_server, state->act_priority);
> +
> + kvmppc_xive_select_irq(state, &hw_num, NULL);
> +
> + if (priority != MASKED) {
> + rc = kvmppc_xive_select_target(kvm, &server, priority);
> + if (rc)
> + goto unlock;
> +
> + state->act_priority = priority;
> + state->act_server = server;
> + state->eisn = eisn;
> +
> + rc = xive_native_configure_irq(hw_num, xive->vp_base + server,
> + priority, eisn);
> + } else {
> + state->act_priority = MASKED;
> + state->act_server = 0;
> + state->eisn = 0;
> +
> + rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
> + }
> +
> +unlock:
> + arch_spin_unlock(&sb->lock);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> + long irq, u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u64 __user *ubufp = (u64 __user *) addr;
> + u16 src;
> + u64 kvm_cfg;
> + u32 server;
> + u8 priority;
> + u32 eisn;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb)
> + return -ENOENT;
> +
> + state = &sb->irq_state[src];
> +
> + if (!state->valid)
> + return -EINVAL;
> +
> + if (get_user(kvm_cfg, ubufp))
> + return -EFAULT;
> +
> + pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
> +
> + priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
> + KVM_XIVE_SOURCE_PRIORITY_SHIFT;
> + server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
> + KVM_XIVE_SOURCE_SERVER_SHIFT;
> + eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
> + KVM_XIVE_SOURCE_EISN_SHIFT;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> +
> + return kvmppc_xive_native_update_source_config(xive, sb, state, server,
> + priority, eisn);
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -259,6 +352,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE:
> return kvmppc_xive_native_set_source(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> + return kvmppc_xive_native_set_source_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -276,6 +372,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_CTRL:
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> + case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index cd8bfc37b72e..4f513a1880c7 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -32,3 +32,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENOMEM: Could not create a new source block
> -EFAULT: Invalid user pointer for attr->addr.
> -ENXIO: Could not allocate underlying HW interrupt
> +
> + 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
> + Configures source targeting
> + Attributes:
> + Interrupt source number (64-bit)
> + The kvm_device_attr.addr points to a __u64 value:
> + bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
> + values: | eisn | mask | server | priority
> + - priority: 0-7 interrupt priority level
> + - server: CPU number chosen to handle the interrupt
> + - mask: mask flag (unused)
> + - eisn: Effective Interrupt Source Number
> + Errors:
> + -ENOENT: Unknown source number
> + -EINVAL: Not initialized source number, invalid priority or
> + invalid CPU number.
> + -EFAULT: Invalid user pointer for attr->addr.
> + -ENXIO: CPU event queues not configured or configuration of the
> + underlying HW interrupt failed
> + -EBUSY: No CPU available to serve interrupt
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
@ 2019-02-25 2:21 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:21 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 9807 bytes --]
On Fri, Feb 22, 2019 at 12:28:29PM +0100, Cédric Le Goater wrote:
> This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
> QEMU and also to restore the configuration of the source in the KVM
> device.
>
> The XIVE internal IRQ structure is extended with the value of the
> Effective Interrupt Source Number. The EISN is the interrupt number
> pushed in the event queue that the guest OS will use to dispatch
> events internally. Caching the EISN value in KVM ease the test when
> checking if a reconfiguration is indeed needed.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 11 +++
> arch/powerpc/kvm/book3s_xive.h | 4 +
> arch/powerpc/kvm/book3s_xive.c | 5 +-
> arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 20 +++++
> 5 files changed, 135 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index a9ad99f2a11b..91899c7f9abd 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -678,9 +678,20 @@ struct kvm_ppc_cpu_char {
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> +#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> #define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
>
> +/* Layout of 64-bit XIVE source configuration attribute values */
> +#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
> +#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
> +#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
> +#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
> +#define KVM_XIVE_SOURCE_MASK_SHIFT 32
> +#define KVM_XIVE_SOURCE_MASK_MASK 0x100000000ULL
> +#define KVM_XIVE_SOURCE_EISN_SHIFT 33
> +#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index f22f2d46d0f0..ab3ac152980d 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
> bool saved_p;
> bool saved_q;
> u8 saved_scan_prio;
> +
> + /* Xive native */
> + u32 eisn; /* Guest Effective IRQ number */
> };
>
> /* Select the "right" interrupt (IPI vs. passthrough) */
> @@ -263,6 +266,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
> struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> +int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 6f950ecb3592..086da91d7c6e 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
> return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
> }
>
> -static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
> +int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
> {
> struct kvm_vcpu *vcpu;
> int i, rc;
> @@ -535,7 +535,7 @@ static int xive_target_interrupt(struct kvm *kvm,
> * priority. The count for that new target will have
> * already been incremented.
> */
> - rc = xive_select_target(kvm, &server, prio);
> + rc = kvmppc_xive_select_target(kvm, &server, prio);
>
> /*
> * We failed to find a target ? Not much we can do
> @@ -1509,6 +1509,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>
> for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
> sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
> + sb->irq_state[i].eisn = 0;
> sb->irq_state[i].guest_priority = MASKED;
> sb->irq_state[i].saved_priority = MASKED;
> sb->irq_state[i].act_priority = MASKED;
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index a9b2d2d9af99..cb5a5c6e05af 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -248,6 +248,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> return 0;
> }
>
> +static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
> + struct kvmppc_xive_src_block *sb,
> + struct kvmppc_xive_irq_state *state,
> + u32 server,
> + u8 priority,
> + u32 eisn)
> +{
> + struct kvm *kvm = xive->kvm;
> + u32 hw_num;
> + int rc = 0;
> +
> + /*
> + * TODO: Do we need to safely mask and unmask a source ? can
> + * we just let the guest handle the possible races ?
> + */
> + arch_spin_lock(&sb->lock);
> +
> + if (state->act_server == server && state->act_priority == priority &&
> + state->eisn == eisn)
> + goto unlock;
> +
> + pr_devel("new_act_prio=%d new_act_server=%d act_server=%d act_prio=%d\n",
> + priority, server, state->act_server, state->act_priority);
> +
> + kvmppc_xive_select_irq(state, &hw_num, NULL);
> +
> + if (priority != MASKED) {
> + rc = kvmppc_xive_select_target(kvm, &server, priority);
> + if (rc)
> + goto unlock;
> +
> + state->act_priority = priority;
> + state->act_server = server;
> + state->eisn = eisn;
> +
> + rc = xive_native_configure_irq(hw_num, xive->vp_base + server,
> + priority, eisn);
> + } else {
> + state->act_priority = MASKED;
> + state->act_server = 0;
> + state->eisn = 0;
> +
> + rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
> + }
> +
> +unlock:
> + arch_spin_unlock(&sb->lock);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> + long irq, u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + u64 __user *ubufp = (u64 __user *) addr;
> + u16 src;
> + u64 kvm_cfg;
> + u32 server;
> + u8 priority;
> + u32 eisn;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb)
> + return -ENOENT;
> +
> + state = &sb->irq_state[src];
> +
> + if (!state->valid)
> + return -EINVAL;
> +
> + if (get_user(kvm_cfg, ubufp))
> + return -EFAULT;
> +
> + pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
> +
> + priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
> + KVM_XIVE_SOURCE_PRIORITY_SHIFT;
> + server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
> + KVM_XIVE_SOURCE_SERVER_SHIFT;
> + eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
> + KVM_XIVE_SOURCE_EISN_SHIFT;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> +
> + return kvmppc_xive_native_update_source_config(xive, sb, state, server,
> + priority, eisn);
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -259,6 +352,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE:
> return kvmppc_xive_native_set_source(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> + return kvmppc_xive_native_set_source_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -276,6 +372,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_CTRL:
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> + case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index cd8bfc37b72e..4f513a1880c7 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -32,3 +32,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENOMEM: Could not create a new source block
> -EFAULT: Invalid user pointer for attr->addr.
> -ENXIO: Could not allocate underlying HW interrupt
> +
> + 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
> + Configures source targeting
> + Attributes:
> + Interrupt source number (64-bit)
> + The kvm_device_attr.addr points to a __u64 value:
> + bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
> + values: | eisn | mask | server | priority
> + - priority: 0-7 interrupt priority level
> + - server: CPU number chosen to handle the interrupt
> + - mask: mask flag (unused)
> + - eisn: Effective Interrupt Source Number
> + Errors:
> + -ENOENT: Unknown source number
> + -EINVAL: Not initialized source number, invalid priority or
> + invalid CPU number.
> + -EFAULT: Invalid user pointer for attr->addr.
> + -ENXIO: CPU event queues not configured or configuration of the
> + underlying HW interrupt failed
> + -EBUSY: No CPU available to serve interrupt
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
These controls will be used by the H_INT_SET_QUEUE_CONFIG and
H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
restore the configuration of the XIVE EQs in the KVM device and to
capture the internal runtime state of the EQs. Both 'get' and 'set'
rely on an OPAL call to access from the XIVE interrupt controller the
EQ toggle bit and EQ index which are updated by the HW when event
notifications are enqueued in the EQ.
The value of the guest physical address of the event queue is saved in
the XIVE internal xive_q structure for later use. That is when
migration needs to mark the EQ pages dirty to capture a consistent
memory state of the VM.
To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
but restoring the EQ state will.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/xive.h | 2 +
arch/powerpc/include/uapi/asm/kvm.h | 21 +++
arch/powerpc/kvm/book3s_xive.h | 2 +
arch/powerpc/kvm/book3s_xive.c | 15 +-
arch/powerpc/kvm/book3s_xive_native.c | 207 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 +++
6 files changed, 270 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index b579a943407b..46891f321606 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -73,6 +73,8 @@ struct xive_q {
u32 esc_irq;
atomic_t count;
atomic_t pending_count;
+ u64 guest_qpage;
+ u32 guest_qsize;
};
/* Global enable flags for the XIVE support */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 91899c7f9abd..177e43f3edaf 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
+#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
@@ -694,4 +695,24 @@ struct kvm_ppc_cpu_char {
#define KVM_XIVE_SOURCE_EISN_SHIFT 33
#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+/* Layout of 64-bit eq attribute */
+#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
+#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
+#define KVM_XIVE_EQ_SERVER_SHIFT 3
+#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
+
+/* Layout of 64-bit eq attribute values */
+struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qsize;
+ __u64 qpage;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+};
+
+#define KVM_XIVE_EQ_FLAG_ENABLED 0x00000001
+#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
+#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index ab3ac152980d..6660d138c6b7 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -267,6 +267,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 086da91d7c6e..7431e31bc541 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
return IRQ_HANDLED;
}
-static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct xive_q *q = &xc->queues[prio];
@@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
return -EIO;
}
- if (xc->xive->single_escalation)
+ if (single_escalation)
name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
vcpu->kvm->arch.lpid, xc->server_num);
else
@@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
* interrupt, thus leaving it effectively masked after
* it fires once.
*/
- if (xc->xive->single_escalation) {
+ if (single_escalation) {
struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
@@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
continue;
rc = xive_provision_queue(vcpu, prio);
if (rc == 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, prio);
+ kvmppc_xive_attach_escalation(vcpu, prio,
+ xive->single_escalation);
if (rc)
return rc;
}
@@ -1219,7 +1221,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
if (xive->qmap & (1 << i)) {
r = xive_provision_queue(vcpu, i);
if (r == 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, i);
+ kvmppc_xive_attach_escalation(
+ vcpu, i, xive->single_escalation);
if (r)
goto bail;
} else {
@@ -1234,7 +1237,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
/* If not done above, attach priority 0 escalation */
- r = xive_attach_escalation(vcpu, 0);
+ r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
if (r)
goto bail;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index cb5a5c6e05af..34a35bcf550c 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -341,6 +341,201 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, eisn);
}
+static int xive_native_validate_queue_size(u32 qsize)
+{
+ switch (qsize) {
+ case 12:
+ case 16:
+ case 21:
+ case 24:
+ case 0:
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ int rc;
+ __be32 *qaddr = 0;
+ struct page *page;
+ struct xive_q *q;
+
+ /*
+ * Demangle priority/server tuple from the EQ index
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("Trying to restore invalid queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ rc = xive_native_validate_queue_size(kvm_eq.qsize);
+ if (rc) {
+ pr_err("invalid queue size %d\n", kvm_eq.qsize);
+ return rc;
+ }
+
+ /* reset queue and disable queueing */
+ if (!kvm_eq.qsize) {
+ q->guest_qpage = 0;
+ q->guest_qsize = 0;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ NULL, 0, true);
+ if (rc) {
+ pr_err("Failed to reset queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ return rc;
+ }
+
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+
+ return 0;
+ }
+
+
+ page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
+ if (is_error_page(page)) {
+ pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
+ return -ENOMEM;
+ }
+ qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
+
+ /* Backup queue page guest address for migration */
+ q->guest_qpage = kvm_eq.qpage;
+ q->guest_qsize = kvm_eq.qsize;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ (__be32 *) qaddr, kvm_eq.qsize, true);
+ if (rc) {
+ pr_err("Failed to configure queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ put_page(page);
+ return rc;
+ }
+
+ rc = xive_native_set_queue_state(xc->vp_id, priority, kvm_eq.qtoggle,
+ kvm_eq.qindex);
+ if (rc)
+ goto error;
+
+ rc = kvmppc_xive_attach_escalation(vcpu, priority,
+ xive->single_escalation);
+error:
+ if (rc)
+ kvmppc_xive_native_cleanup_queue(vcpu, priority);
+ return rc;
+}
+
+static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ struct xive_q *q;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ u64 qpage;
+ u64 qsize;
+ u64 qeoi_page;
+ u32 escalate_irq;
+ u64 qflags;
+ int rc;
+
+ /*
+ * Demangle priority/server tuple from the EQ index
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ memset(&kvm_eq, 0, sizeof(kvm_eq));
+
+ if (!q->qpage)
+ return 0;
+
+ rc = xive_native_get_queue_info(xc->vp_id, priority, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc)
+ return rc;
+
+ kvm_eq.flags = 0;
+ if (qflags & OPAL_XIVE_EQ_ENABLED)
+ kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ENABLED;
+ if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
+ kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY;
+ if (qflags & OPAL_XIVE_EQ_ESCALATE)
+ kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ESCALATE;
+
+ kvm_eq.qsize = q->guest_qsize;
+ kvm_eq.qpage = q->guest_qpage;
+
+ rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
+ &kvm_eq.qindex);
+ if (rc)
+ return rc;
+
+ pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -355,6 +550,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
return kvmppc_xive_native_set_source_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_set_queue_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -362,6 +560,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_get_queue_config(xive, attr->attr,
+ attr->addr);
+ }
return -ENXIO;
}
@@ -377,6 +582,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
break;
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return 0;
}
return -ENXIO;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 4f513a1880c7..c0b5d9bd43fb 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -52,3 +52,32 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENXIO: CPU event queues not configured or configuration of the
underlying HW interrupt failed
-EBUSY: No CPU available to serve interrupt
+
+ 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
+ Configures an event queue of a CPU
+ Attributes:
+ EQ descriptor identifier (64-bit)
+ The EQ descriptor identifier is a tuple (server, priority) :
+ bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
+ values: | unused | server | priority
+ The kvm_device_attr.addr points to :
+ struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qsize;
+ __u64 qpage;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+ };
+ - flags: queue flags
+ - qsize: queue size (power of 2)
+ - qpage: real address of queue
+ - qtoggle: current queue toggle bit
+ - qindex: current queue index
+ - pad: reserved for future use
+ Errors:
+ -ENOENT: Invalid CPU number
+ -EINVAL: Invalid priority or invalid queue size
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENOMEM: Invalid queue address
+ -EIO: Configuration of the underlying HW failed
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
These controls will be used by the H_INT_SET_QUEUE_CONFIG and
H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
restore the configuration of the XIVE EQs in the KVM device and to
capture the internal runtime state of the EQs. Both 'get' and 'set'
rely on an OPAL call to access from the XIVE interrupt controller the
EQ toggle bit and EQ index which are updated by the HW when event
notifications are enqueued in the EQ.
The value of the guest physical address of the event queue is saved in
the XIVE internal xive_q structure for later use. That is when
migration needs to mark the EQ pages dirty to capture a consistent
memory state of the VM.
To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
but restoring the EQ state will.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/xive.h | 2 +
arch/powerpc/include/uapi/asm/kvm.h | 21 +++
arch/powerpc/kvm/book3s_xive.h | 2 +
arch/powerpc/kvm/book3s_xive.c | 15 +-
arch/powerpc/kvm/book3s_xive_native.c | 207 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 +++
6 files changed, 270 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index b579a943407b..46891f321606 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -73,6 +73,8 @@ struct xive_q {
u32 esc_irq;
atomic_t count;
atomic_t pending_count;
+ u64 guest_qpage;
+ u32 guest_qsize;
};
/* Global enable flags for the XIVE support */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 91899c7f9abd..177e43f3edaf 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
+#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
@@ -694,4 +695,24 @@ struct kvm_ppc_cpu_char {
#define KVM_XIVE_SOURCE_EISN_SHIFT 33
#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+/* Layout of 64-bit eq attribute */
+#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
+#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
+#define KVM_XIVE_EQ_SERVER_SHIFT 3
+#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
+
+/* Layout of 64-bit eq attribute values */
+struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qsize;
+ __u64 qpage;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+};
+
+#define KVM_XIVE_EQ_FLAG_ENABLED 0x00000001
+#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
+#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index ab3ac152980d..6660d138c6b7 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -267,6 +267,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 086da91d7c6e..7431e31bc541 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
return IRQ_HANDLED;
}
-static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct xive_q *q = &xc->queues[prio];
@@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
return -EIO;
}
- if (xc->xive->single_escalation)
+ if (single_escalation)
name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
vcpu->kvm->arch.lpid, xc->server_num);
else
@@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
* interrupt, thus leaving it effectively masked after
* it fires once.
*/
- if (xc->xive->single_escalation) {
+ if (single_escalation) {
struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
@@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
continue;
rc = xive_provision_queue(vcpu, prio);
if (rc = 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, prio);
+ kvmppc_xive_attach_escalation(vcpu, prio,
+ xive->single_escalation);
if (rc)
return rc;
}
@@ -1219,7 +1221,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
if (xive->qmap & (1 << i)) {
r = xive_provision_queue(vcpu, i);
if (r = 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, i);
+ kvmppc_xive_attach_escalation(
+ vcpu, i, xive->single_escalation);
if (r)
goto bail;
} else {
@@ -1234,7 +1237,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
/* If not done above, attach priority 0 escalation */
- r = xive_attach_escalation(vcpu, 0);
+ r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
if (r)
goto bail;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index cb5a5c6e05af..34a35bcf550c 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -341,6 +341,201 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, eisn);
}
+static int xive_native_validate_queue_size(u32 qsize)
+{
+ switch (qsize) {
+ case 12:
+ case 16:
+ case 21:
+ case 24:
+ case 0:
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
+static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ int rc;
+ __be32 *qaddr = 0;
+ struct page *page;
+ struct xive_q *q;
+
+ /*
+ * Demangle priority/server tuple from the EQ index
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("Trying to restore invalid queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ rc = xive_native_validate_queue_size(kvm_eq.qsize);
+ if (rc) {
+ pr_err("invalid queue size %d\n", kvm_eq.qsize);
+ return rc;
+ }
+
+ /* reset queue and disable queueing */
+ if (!kvm_eq.qsize) {
+ q->guest_qpage = 0;
+ q->guest_qsize = 0;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ NULL, 0, true);
+ if (rc) {
+ pr_err("Failed to reset queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ return rc;
+ }
+
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+
+ return 0;
+ }
+
+
+ page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
+ if (is_error_page(page)) {
+ pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
+ return -ENOMEM;
+ }
+ qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
+
+ /* Backup queue page guest address for migration */
+ q->guest_qpage = kvm_eq.qpage;
+ q->guest_qsize = kvm_eq.qsize;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ (__be32 *) qaddr, kvm_eq.qsize, true);
+ if (rc) {
+ pr_err("Failed to configure queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ put_page(page);
+ return rc;
+ }
+
+ rc = xive_native_set_queue_state(xc->vp_id, priority, kvm_eq.qtoggle,
+ kvm_eq.qindex);
+ if (rc)
+ goto error;
+
+ rc = kvmppc_xive_attach_escalation(vcpu, priority,
+ xive->single_escalation);
+error:
+ if (rc)
+ kvmppc_xive_native_cleanup_queue(vcpu, priority);
+ return rc;
+}
+
+static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ struct xive_q *q;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ u64 qpage;
+ u64 qsize;
+ u64 qeoi_page;
+ u32 escalate_irq;
+ u64 qflags;
+ int rc;
+
+ /*
+ * Demangle priority/server tuple from the EQ index
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ memset(&kvm_eq, 0, sizeof(kvm_eq));
+
+ if (!q->qpage)
+ return 0;
+
+ rc = xive_native_get_queue_info(xc->vp_id, priority, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc)
+ return rc;
+
+ kvm_eq.flags = 0;
+ if (qflags & OPAL_XIVE_EQ_ENABLED)
+ kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ENABLED;
+ if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
+ kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY;
+ if (qflags & OPAL_XIVE_EQ_ESCALATE)
+ kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ESCALATE;
+
+ kvm_eq.qsize = q->guest_qsize;
+ kvm_eq.qpage = q->guest_qpage;
+
+ rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
+ &kvm_eq.qindex);
+ if (rc)
+ return rc;
+
+ pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -355,6 +550,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
return kvmppc_xive_native_set_source_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_set_queue_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -362,6 +560,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_get_queue_config(xive, attr->attr,
+ attr->addr);
+ }
return -ENXIO;
}
@@ -377,6 +582,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
break;
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return 0;
}
return -ENXIO;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 4f513a1880c7..c0b5d9bd43fb 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -52,3 +52,32 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENXIO: CPU event queues not configured or configuration of the
underlying HW interrupt failed
-EBUSY: No CPU available to serve interrupt
+
+ 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
+ Configures an event queue of a CPU
+ Attributes:
+ EQ descriptor identifier (64-bit)
+ The EQ descriptor identifier is a tuple (server, priority) :
+ bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
+ values: | unused | server | priority
+ The kvm_device_attr.addr points to :
+ struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qsize;
+ __u64 qpage;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+ };
+ - flags: queue flags
+ - qsize: queue size (power of 2)
+ - qpage: real address of queue
+ - qtoggle: current queue toggle bit
+ - qindex: current queue index
+ - pad: reserved for future use
+ Errors:
+ -ENOENT: Invalid CPU number
+ -EINVAL: Invalid priority or invalid queue size
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENOMEM: Invalid queue address
+ -EIO: Configuration of the underlying HW failed
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 2:39 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:39 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 15529 bytes --]
On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> restore the configuration of the XIVE EQs in the KVM device and to
> capture the internal runtime state of the EQs. Both 'get' and 'set'
> rely on an OPAL call to access from the XIVE interrupt controller the
> EQ toggle bit and EQ index which are updated by the HW when event
> notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/xive.h | 2 +
> arch/powerpc/include/uapi/asm/kvm.h | 21 +++
> arch/powerpc/kvm/book3s_xive.h | 2 +
> arch/powerpc/kvm/book3s_xive.c | 15 +-
> arch/powerpc/kvm/book3s_xive_native.c | 207 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 29 +++
> 6 files changed, 270 insertions(+), 6 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index b579a943407b..46891f321606 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -73,6 +73,8 @@ struct xive_q {
> u32 esc_irq;
> atomic_t count;
> atomic_t pending_count;
> + u64 guest_qpage;
> + u32 guest_qsize;
> };
>
> /* Global enable flags for the XIVE support */
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 91899c7f9abd..177e43f3edaf 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> @@ -694,4 +695,24 @@ struct kvm_ppc_cpu_char {
> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>
> +/* Layout of 64-bit eq attribute */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of 64-bit eq attribute values */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qsize;
> + __u64 qpage;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
> +
> +#define KVM_XIVE_EQ_FLAG_ENABLED 0x00000001
> +#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
> +#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index ab3ac152980d..6660d138c6b7 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -267,6 +267,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 086da91d7c6e..7431e31bc541 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
> return IRQ_HANDLED;
> }
>
> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct xive_q *q = &xc->queues[prio];
> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> return -EIO;
> }
>
> - if (xc->xive->single_escalation)
> + if (single_escalation)
> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
> vcpu->kvm->arch.lpid, xc->server_num);
> else
> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> * interrupt, thus leaving it effectively masked after
> * it fires once.
> */
> - if (xc->xive->single_escalation) {
> + if (single_escalation) {
> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>
> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
> continue;
> rc = xive_provision_queue(vcpu, prio);
> if (rc == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, prio);
> + kvmppc_xive_attach_escalation(vcpu, prio,
> + xive->single_escalation);
> if (rc)
> return rc;
> }
> @@ -1219,7 +1221,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> if (xive->qmap & (1 << i)) {
> r = xive_provision_queue(vcpu, i);
> if (r == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, i);
> + kvmppc_xive_attach_escalation(
> + vcpu, i, xive->single_escalation);
> if (r)
> goto bail;
> } else {
> @@ -1234,7 +1237,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
>
> /* If not done above, attach priority 0 escalation */
> - r = xive_attach_escalation(vcpu, 0);
> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
> if (r)
> goto bail;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index cb5a5c6e05af..34a35bcf550c 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -341,6 +341,201 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, eisn);
> }
>
> +static int xive_native_validate_queue_size(u32 qsize)
> +{
> + switch (qsize) {
> + case 12:
> + case 16:
> + case 21:
> + case 24:
> + case 0:
> + return 0;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + int rc;
> + __be32 *qaddr = 0;
> + struct page *page;
> + struct xive_q *q;
> +
> + /*
> + * Demangle priority/server tuple from the EQ index
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
You need to validate the 'flags' field (AFAICT we don't actually have
any flags yet, so it's only valid it if is 0.
> +
> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + rc = xive_native_validate_queue_size(kvm_eq.qsize);
> + if (rc) {
> + pr_err("invalid queue size %d\n", kvm_eq.qsize);
> + return rc;
> + }
> +
> + /* reset queue and disable queueing */
> + if (!kvm_eq.qsize) {
> + q->guest_qpage = 0;
> + q->guest_qsize = 0;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + NULL, 0, true);
> + if (rc) {
> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + return rc;
> + }
> +
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +
> + return 0;
> + }
> +
> +
> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> + if (is_error_page(page)) {
> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> + return -ENOMEM;
Nit: I don't think ENOMEM is the right error here. ENOMEM indicates
that the kernel couldn't allocate enough memory to complete whatever
you asked. Here the problem is the user supplied a bad guest address,
which is a rather different error. EFAULT is closer, but still not
quite right, since it could be a valid user address but not a valid
guest address. There are probably existing KVM calls that could hit
this problem, I wonder what they use.
> + }
> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
> +
> + /* Backup queue page guest address for migration */
> + q->guest_qpage = kvm_eq.qpage;
> + q->guest_qsize = kvm_eq.qsize;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + (__be32 *) qaddr, kvm_eq.qsize, true);
> + if (rc) {
> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + put_page(page);
> + return rc;
> + }
> +
> + rc = xive_native_set_queue_state(xc->vp_id, priority, kvm_eq.qtoggle,
> + kvm_eq.qindex);
> + if (rc)
> + goto error;
> +
> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
> + xive->single_escalation);
> +error:
> + if (rc)
> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + struct xive_q *q;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + u64 qpage;
> + u64 qsize;
> + u64 qeoi_page;
> + u32 escalate_irq;
> + u64 qflags;
> + int rc;
> +
> + /*
> + * Demangle priority/server tuple from the EQ index
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + memset(&kvm_eq, 0, sizeof(kvm_eq));
> +
> + if (!q->qpage)
> + return 0;
> +
> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qpage, &qsize,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc)
> + return rc;
> +
> + kvm_eq.flags = 0;
> + if (qflags & OPAL_XIVE_EQ_ENABLED)
> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ENABLED;
> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY;
> + if (qflags & OPAL_XIVE_EQ_ESCALATE)
> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ESCALATE;
> +
> + kvm_eq.qsize = q->guest_qsize;
> + kvm_eq.qpage = q->guest_qpage;
> +
> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
> + &kvm_eq.qindex);
> + if (rc)
> + return rc;
> +
> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -355,6 +550,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> return kvmppc_xive_native_set_source_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -362,6 +560,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
> + attr->addr);
> + }
> return -ENXIO;
> }
>
> @@ -377,6 +582,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> break;
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return 0;
> }
> return -ENXIO;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 4f513a1880c7..c0b5d9bd43fb 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -52,3 +52,32 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENXIO: CPU event queues not configured or configuration of the
> underlying HW interrupt failed
> -EBUSY: No CPU available to serve interrupt
> +
> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
> + Configures an event queue of a CPU
> + Attributes:
> + EQ descriptor identifier (64-bit)
> + The EQ descriptor identifier is a tuple (server, priority) :
> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
> + values: | unused | server | priority
> + The kvm_device_attr.addr points to :
> + struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qsize;
> + __u64 qpage;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> + };
> + - flags: queue flags
> + - qsize: queue size (power of 2)
> + - qpage: real address of queue
> + - qtoggle: current queue toggle bit
> + - qindex: current queue index
> + - pad: reserved for future use
> + Errors:
> + -ENOENT: Invalid CPU number
> + -EINVAL: Invalid priority or invalid queue size
> + -EFAULT: Invalid user pointer for attr->addr.
> + -ENOMEM: Invalid queue address
> + -EIO: Configuration of the underlying HW failed
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-02-25 2:39 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:39 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 15529 bytes --]
On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> restore the configuration of the XIVE EQs in the KVM device and to
> capture the internal runtime state of the EQs. Both 'get' and 'set'
> rely on an OPAL call to access from the XIVE interrupt controller the
> EQ toggle bit and EQ index which are updated by the HW when event
> notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/xive.h | 2 +
> arch/powerpc/include/uapi/asm/kvm.h | 21 +++
> arch/powerpc/kvm/book3s_xive.h | 2 +
> arch/powerpc/kvm/book3s_xive.c | 15 +-
> arch/powerpc/kvm/book3s_xive_native.c | 207 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 29 +++
> 6 files changed, 270 insertions(+), 6 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index b579a943407b..46891f321606 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -73,6 +73,8 @@ struct xive_q {
> u32 esc_irq;
> atomic_t count;
> atomic_t pending_count;
> + u64 guest_qpage;
> + u32 guest_qsize;
> };
>
> /* Global enable flags for the XIVE support */
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 91899c7f9abd..177e43f3edaf 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> @@ -694,4 +695,24 @@ struct kvm_ppc_cpu_char {
> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>
> +/* Layout of 64-bit eq attribute */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of 64-bit eq attribute values */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qsize;
> + __u64 qpage;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
> +
> +#define KVM_XIVE_EQ_FLAG_ENABLED 0x00000001
> +#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
> +#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index ab3ac152980d..6660d138c6b7 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -267,6 +267,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 086da91d7c6e..7431e31bc541 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
> return IRQ_HANDLED;
> }
>
> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct xive_q *q = &xc->queues[prio];
> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> return -EIO;
> }
>
> - if (xc->xive->single_escalation)
> + if (single_escalation)
> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
> vcpu->kvm->arch.lpid, xc->server_num);
> else
> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> * interrupt, thus leaving it effectively masked after
> * it fires once.
> */
> - if (xc->xive->single_escalation) {
> + if (single_escalation) {
> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>
> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
> continue;
> rc = xive_provision_queue(vcpu, prio);
> if (rc == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, prio);
> + kvmppc_xive_attach_escalation(vcpu, prio,
> + xive->single_escalation);
> if (rc)
> return rc;
> }
> @@ -1219,7 +1221,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> if (xive->qmap & (1 << i)) {
> r = xive_provision_queue(vcpu, i);
> if (r == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, i);
> + kvmppc_xive_attach_escalation(
> + vcpu, i, xive->single_escalation);
> if (r)
> goto bail;
> } else {
> @@ -1234,7 +1237,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
>
> /* If not done above, attach priority 0 escalation */
> - r = xive_attach_escalation(vcpu, 0);
> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
> if (r)
> goto bail;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index cb5a5c6e05af..34a35bcf550c 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -341,6 +341,201 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, eisn);
> }
>
> +static int xive_native_validate_queue_size(u32 qsize)
> +{
> + switch (qsize) {
> + case 12:
> + case 16:
> + case 21:
> + case 24:
> + case 0:
> + return 0;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + int rc;
> + __be32 *qaddr = 0;
> + struct page *page;
> + struct xive_q *q;
> +
> + /*
> + * Demangle priority/server tuple from the EQ index
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
You need to validate the 'flags' field (AFAICT we don't actually have
any flags yet, so it's only valid it if is 0.
> +
> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + rc = xive_native_validate_queue_size(kvm_eq.qsize);
> + if (rc) {
> + pr_err("invalid queue size %d\n", kvm_eq.qsize);
> + return rc;
> + }
> +
> + /* reset queue and disable queueing */
> + if (!kvm_eq.qsize) {
> + q->guest_qpage = 0;
> + q->guest_qsize = 0;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + NULL, 0, true);
> + if (rc) {
> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + return rc;
> + }
> +
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +
> + return 0;
> + }
> +
> +
> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> + if (is_error_page(page)) {
> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> + return -ENOMEM;
Nit: I don't think ENOMEM is the right error here. ENOMEM indicates
that the kernel couldn't allocate enough memory to complete whatever
you asked. Here the problem is the user supplied a bad guest address,
which is a rather different error. EFAULT is closer, but still not
quite right, since it could be a valid user address but not a valid
guest address. There are probably existing KVM calls that could hit
this problem, I wonder what they use.
> + }
> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
> +
> + /* Backup queue page guest address for migration */
> + q->guest_qpage = kvm_eq.qpage;
> + q->guest_qsize = kvm_eq.qsize;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + (__be32 *) qaddr, kvm_eq.qsize, true);
> + if (rc) {
> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + put_page(page);
> + return rc;
> + }
> +
> + rc = xive_native_set_queue_state(xc->vp_id, priority, kvm_eq.qtoggle,
> + kvm_eq.qindex);
> + if (rc)
> + goto error;
> +
> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
> + xive->single_escalation);
> +error:
> + if (rc)
> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + struct xive_q *q;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + u64 qpage;
> + u64 qsize;
> + u64 qeoi_page;
> + u32 escalate_irq;
> + u64 qflags;
> + int rc;
> +
> + /*
> + * Demangle priority/server tuple from the EQ index
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + memset(&kvm_eq, 0, sizeof(kvm_eq));
> +
> + if (!q->qpage)
> + return 0;
> +
> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qpage, &qsize,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc)
> + return rc;
> +
> + kvm_eq.flags = 0;
> + if (qflags & OPAL_XIVE_EQ_ENABLED)
> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ENABLED;
> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY;
> + if (qflags & OPAL_XIVE_EQ_ESCALATE)
> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ESCALATE;
> +
> + kvm_eq.qsize = q->guest_qsize;
> + kvm_eq.qpage = q->guest_qpage;
> +
> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
> + &kvm_eq.qindex);
> + if (rc)
> + return rc;
> +
> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -355,6 +550,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> return kvmppc_xive_native_set_source_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -362,6 +560,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
> + attr->addr);
> + }
> return -ENXIO;
> }
>
> @@ -377,6 +582,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> break;
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return 0;
> }
> return -ENXIO;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 4f513a1880c7..c0b5d9bd43fb 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -52,3 +52,32 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENXIO: CPU event queues not configured or configuration of the
> underlying HW interrupt failed
> -EBUSY: No CPU available to serve interrupt
> +
> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
> + Configures an event queue of a CPU
> + Attributes:
> + EQ descriptor identifier (64-bit)
> + The EQ descriptor identifier is a tuple (server, priority) :
> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
> + values: | unused | server | priority
> + The kvm_device_attr.addr points to :
> + struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qsize;
> + __u64 qpage;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> + };
> + - flags: queue flags
> + - qsize: queue size (power of 2)
> + - qpage: real address of queue
> + - qtoggle: current queue toggle bit
> + - qindex: current queue index
> + - pad: reserved for future use
> + Errors:
> + -ENOENT: Invalid CPU number
> + -EINVAL: Invalid priority or invalid queue size
> + -EFAULT: Invalid user pointer for attr->addr.
> + -ENOMEM: Invalid queue address
> + -EIO: Configuration of the underlying HW failed
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-02-25 2:39 ` David Gibson
@ 2019-03-12 17:00 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 17:00 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 3:39 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>> restore the configuration of the XIVE EQs in the KVM device and to
>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>> rely on an OPAL call to access from the XIVE interrupt controller the
>> EQ toggle bit and EQ index which are updated by the HW when event
>> notifications are enqueued in the EQ.
>>
>> The value of the guest physical address of the event queue is saved in
>> the XIVE internal xive_q structure for later use. That is when
>> migration needs to mark the EQ pages dirty to capture a consistent
>> memory state of the VM.
>>
>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>> but restoring the EQ state will.
I think we need to add some kind of flags to differentiate the hcall
H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
not need OPAL support call and this could help in the code transition.
But without OPAL support, we won't have migration. Would that be of
any use ?
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/xive.h | 2 +
>> arch/powerpc/include/uapi/asm/kvm.h | 21 +++
>> arch/powerpc/kvm/book3s_xive.h | 2 +
>> arch/powerpc/kvm/book3s_xive.c | 15 +-
>> arch/powerpc/kvm/book3s_xive_native.c | 207 +++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 29 +++
>> 6 files changed, 270 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
>> index b579a943407b..46891f321606 100644
>> --- a/arch/powerpc/include/asm/xive.h
>> +++ b/arch/powerpc/include/asm/xive.h
>> @@ -73,6 +73,8 @@ struct xive_q {
>> u32 esc_irq;
>> atomic_t count;
>> atomic_t pending_count;
>> + u64 guest_qpage;
>> + u32 guest_qsize;
>> };
>>
>> /* Global enable flags for the XIVE support */
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index 91899c7f9abd..177e43f3edaf 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
>> #define KVM_DEV_XIVE_GRP_CTRL 1
>> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
>> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
>> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
>>
>> /* Layout of 64-bit XIVE source attribute values */
>> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
>> @@ -694,4 +695,24 @@ struct kvm_ppc_cpu_char {
>> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
>> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>>
>> +/* Layout of 64-bit eq attribute */
>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
>> +
>> +/* Layout of 64-bit eq attribute values */
>> +struct kvm_ppc_xive_eq {
>> + __u32 flags;
>> + __u32 qsize;
>> + __u64 qpage;
>> + __u32 qtoggle;
>> + __u32 qindex;
>> + __u8 pad[40];
>> +};
>> +
>> +#define KVM_XIVE_EQ_FLAG_ENABLED 0x00000001
>> +#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
>> +#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
>> +
>> #endif /* __LINUX_KVM_POWERPC_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
>> index ab3ac152980d..6660d138c6b7 100644
>> --- a/arch/powerpc/kvm/book3s_xive.h
>> +++ b/arch/powerpc/kvm/book3s_xive.h
>> @@ -267,6 +267,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>> struct kvmppc_xive *xive, int irq);
>> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
>> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
>> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
>> + bool single_escalation);
>>
>> #endif /* CONFIG_KVM_XICS */
>> #endif /* _KVM_PPC_BOOK3S_XICS_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index 086da91d7c6e..7431e31bc541 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
>> return IRQ_HANDLED;
>> }
>>
>> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
>> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
>> + bool single_escalation)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> struct xive_q *q = &xc->queues[prio];
>> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
>> return -EIO;
>> }
>>
>> - if (xc->xive->single_escalation)
>> + if (single_escalation)
>> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
>> vcpu->kvm->arch.lpid, xc->server_num);
>> else
>> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
>> * interrupt, thus leaving it effectively masked after
>> * it fires once.
>> */
>> - if (xc->xive->single_escalation) {
>> + if (single_escalation) {
>> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
>> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>>
>> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
>> continue;
>> rc = xive_provision_queue(vcpu, prio);
>> if (rc == 0 && !xive->single_escalation)
>> - xive_attach_escalation(vcpu, prio);
>> + kvmppc_xive_attach_escalation(vcpu, prio,
>> + xive->single_escalation);
>> if (rc)
>> return rc;
>> }
>> @@ -1219,7 +1221,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> if (xive->qmap & (1 << i)) {
>> r = xive_provision_queue(vcpu, i);
>> if (r == 0 && !xive->single_escalation)
>> - xive_attach_escalation(vcpu, i);
>> + kvmppc_xive_attach_escalation(
>> + vcpu, i, xive->single_escalation);
>> if (r)
>> goto bail;
>> } else {
>> @@ -1234,7 +1237,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> }
>>
>> /* If not done above, attach priority 0 escalation */
>> - r = xive_attach_escalation(vcpu, 0);
>> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
>> if (r)
>> goto bail;
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index cb5a5c6e05af..34a35bcf550c 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -341,6 +341,201 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
>> priority, eisn);
>> }
>>
>> +static int xive_native_validate_queue_size(u32 qsize)
>> +{
>> + switch (qsize) {
>> + case 12:
>> + case 16:
>> + case 21:
>> + case 24:
>> + case 0:
>> + return 0;
>> + default:
>> + return -EINVAL;
>> + }
>> +}
>> +
>> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
>> + long eq_idx, u64 addr)
>> +{
>> + struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + struct kvmppc_xive_vcpu *xc;
>> + void __user *ubufp = (u64 __user *) addr;
>> + u32 server;
>> + u8 priority;
>> + struct kvm_ppc_xive_eq kvm_eq;
>> + int rc;
>> + __be32 *qaddr = 0;
>> + struct page *page;
>> + struct xive_q *q;
>> +
>> + /*
>> + * Demangle priority/server tuple from the EQ index
>> + */
>> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
>> + KVM_XIVE_EQ_PRIORITY_SHIFT;
>> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
>> + KVM_XIVE_EQ_SERVER_SHIFT;
>> +
>> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
>> + return -EFAULT;
>> +
>> + vcpu = kvmppc_xive_find_server(kvm, server);
>> + if (!vcpu) {
>> + pr_err("Can't find server %d\n", server);
>> + return -ENOENT;
>> + }
>> + xc = vcpu->arch.xive_vcpu;
>> +
>> + if (priority != xive_prio_from_guest(priority)) {
>> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
>> + priority, server);
>> + return -EINVAL;
>> + }
>> + q = &xc->queues[priority];
>
> You need to validate the 'flags' field (AFAICT we don't actually have
> any flags yet, so it's only valid it if is 0.
Well, we should set the ESCALATE flags and ALWAYS_NOTIFY also but this
is already set by xive_native_configure_queue(). I will take a closer look
and document.
>> +
>> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
>> + __func__, server, priority, kvm_eq.flags,
>> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
>> +
>> + rc = xive_native_validate_queue_size(kvm_eq.qsize);
>> + if (rc) {
>> + pr_err("invalid queue size %d\n", kvm_eq.qsize);
>> + return rc;
>> + }
>> +
>> + /* reset queue and disable queueing */
>> + if (!kvm_eq.qsize) {
>> + q->guest_qpage = 0;
>> + q->guest_qsize = 0;
>> +
>> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
>> + NULL, 0, true);
>> + if (rc) {
>> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
>> + priority, xc->server_num, rc);
>> + return rc;
>> + }
>> +
>> + if (q->qpage) {
>> + put_page(virt_to_page(q->qpage));
>> + q->qpage = NULL;
>> + }
>> +
>> + return 0;
>> + }
>> +
>> +
>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
>> + if (is_error_page(page)) {
>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
>> + return -ENOMEM;
>
> Nit: I don't think ENOMEM is the right error here. ENOMEM indicates
> that the kernel couldn't allocate enough memory to complete whatever
> you asked. Here the problem is the user supplied a bad guest address,
> which is a rather different error. EFAULT is closer, but still not
> quite right, since it could be a valid user address but not a valid
> guest address. There are probably existing KVM calls that could hit
> this problem, I wonder what they use.
I have seen a -EINVAL (s390) and a -EFAULT (x86)
Thanks,
C.
>> + }
>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
>> +
>> + /* Backup queue page guest address for migration */
>> + q->guest_qpage = kvm_eq.qpage;
>> + q->guest_qsize = kvm_eq.qsize;
>> +
>> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
>> + (__be32 *) qaddr, kvm_eq.qsize, true);
>> + if (rc) {
>> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
>> + priority, xc->server_num, rc);
>> + put_page(page);
>> + return rc;
>> + }
>> +
>> + rc = xive_native_set_queue_state(xc->vp_id, priority, kvm_eq.qtoggle,
>> + kvm_eq.qindex);
>> + if (rc)
>> + goto error;
>> +
>> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
>> + xive->single_escalation);
>> +error:
>> + if (rc)
>> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
>> + return rc;
>> +}
>> +
>> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
>> + long eq_idx, u64 addr)
>> +{
>> + struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + struct kvmppc_xive_vcpu *xc;
>> + struct xive_q *q;
>> + void __user *ubufp = (u64 __user *) addr;
>> + u32 server;
>> + u8 priority;
>> + struct kvm_ppc_xive_eq kvm_eq;
>> + u64 qpage;
>> + u64 qsize;
>> + u64 qeoi_page;
>> + u32 escalate_irq;
>> + u64 qflags;
>> + int rc;
>> +
>> + /*
>> + * Demangle priority/server tuple from the EQ index
>> + */
>> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
>> + KVM_XIVE_EQ_PRIORITY_SHIFT;
>> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
>> + KVM_XIVE_EQ_SERVER_SHIFT;
>> +
>> + vcpu = kvmppc_xive_find_server(kvm, server);
>> + if (!vcpu) {
>> + pr_err("Can't find server %d\n", server);
>> + return -ENOENT;
>> + }
>> + xc = vcpu->arch.xive_vcpu;
>> +
>> + if (priority != xive_prio_from_guest(priority)) {
>> + pr_err("invalid priority for queue %d for VCPU %d\n",
>> + priority, server);
>> + return -EINVAL;
>> + }
>> + q = &xc->queues[priority];
>> +
>> + memset(&kvm_eq, 0, sizeof(kvm_eq));
>> +
>> + if (!q->qpage)
>> + return 0;
>> +
>> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qpage, &qsize,
>> + &qeoi_page, &escalate_irq, &qflags);
>> + if (rc)
>> + return rc;
>> +
>> + kvm_eq.flags = 0;
>> + if (qflags & OPAL_XIVE_EQ_ENABLED)
>> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ENABLED;
>> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
>> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY;
>> + if (qflags & OPAL_XIVE_EQ_ESCALATE)
>> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ESCALATE;
>> +
>> + kvm_eq.qsize = q->guest_qsize;
>> + kvm_eq.qpage = q->guest_qpage;
>> +
>> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
>> + &kvm_eq.qindex);
>> + if (rc)
>> + return rc;
>> +
>> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
>> + __func__, server, priority, kvm_eq.flags,
>> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
>> +
>> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
>> + return -EFAULT;
>> +
>> + return 0;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> @@ -355,6 +550,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
>> return kvmppc_xive_native_set_source_config(xive, attr->attr,
>> attr->addr);
>> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
>> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
>> + attr->addr);
>> }
>> return -ENXIO;
>> }
>> @@ -362,6 +560,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> + struct kvmppc_xive *xive = dev->private;
>> +
>> + switch (attr->group) {
>> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
>> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
>> + attr->addr);
>> + }
>> return -ENXIO;
>> }
>>
>> @@ -377,6 +582,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> attr->attr < KVMPPC_XIVE_NR_IRQS)
>> return 0;
>> break;
>> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
>> + return 0;
>> }
>> return -ENXIO;
>> }
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index 4f513a1880c7..c0b5d9bd43fb 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -52,3 +52,32 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> -ENXIO: CPU event queues not configured or configuration of the
>> underlying HW interrupt failed
>> -EBUSY: No CPU available to serve interrupt
>> +
>> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
>> + Configures an event queue of a CPU
>> + Attributes:
>> + EQ descriptor identifier (64-bit)
>> + The EQ descriptor identifier is a tuple (server, priority) :
>> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
>> + values: | unused | server | priority
>> + The kvm_device_attr.addr points to :
>> + struct kvm_ppc_xive_eq {
>> + __u32 flags;
>> + __u32 qsize;
>> + __u64 qpage;
>> + __u32 qtoggle;
>> + __u32 qindex;
>> + __u8 pad[40];
>> + };
>> + - flags: queue flags
>> + - qsize: queue size (power of 2)
>> + - qpage: real address of queue
>> + - qtoggle: current queue toggle bit
>> + - qindex: current queue index
>> + - pad: reserved for future use
>> + Errors:
>> + -ENOENT: Invalid CPU number
>> + -EINVAL: Invalid priority or invalid queue size
>> + -EFAULT: Invalid user pointer for attr->addr.
>> + -ENOMEM: Invalid queue address
>> + -EIO: Configuration of the underlying HW failed
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-12 17:00 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-12 17:00 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 3:39 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>> restore the configuration of the XIVE EQs in the KVM device and to
>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>> rely on an OPAL call to access from the XIVE interrupt controller the
>> EQ toggle bit and EQ index which are updated by the HW when event
>> notifications are enqueued in the EQ.
>>
>> The value of the guest physical address of the event queue is saved in
>> the XIVE internal xive_q structure for later use. That is when
>> migration needs to mark the EQ pages dirty to capture a consistent
>> memory state of the VM.
>>
>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>> but restoring the EQ state will.
I think we need to add some kind of flags to differentiate the hcall
H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
not need OPAL support call and this could help in the code transition.
But without OPAL support, we won't have migration. Would that be of
any use ?
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/xive.h | 2 +
>> arch/powerpc/include/uapi/asm/kvm.h | 21 +++
>> arch/powerpc/kvm/book3s_xive.h | 2 +
>> arch/powerpc/kvm/book3s_xive.c | 15 +-
>> arch/powerpc/kvm/book3s_xive_native.c | 207 +++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 29 +++
>> 6 files changed, 270 insertions(+), 6 deletions(-)
>>
>> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
>> index b579a943407b..46891f321606 100644
>> --- a/arch/powerpc/include/asm/xive.h
>> +++ b/arch/powerpc/include/asm/xive.h
>> @@ -73,6 +73,8 @@ struct xive_q {
>> u32 esc_irq;
>> atomic_t count;
>> atomic_t pending_count;
>> + u64 guest_qpage;
>> + u32 guest_qsize;
>> };
>>
>> /* Global enable flags for the XIVE support */
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index 91899c7f9abd..177e43f3edaf 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
>> #define KVM_DEV_XIVE_GRP_CTRL 1
>> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
>> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
>> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
>>
>> /* Layout of 64-bit XIVE source attribute values */
>> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
>> @@ -694,4 +695,24 @@ struct kvm_ppc_cpu_char {
>> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
>> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>>
>> +/* Layout of 64-bit eq attribute */
>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
>> +
>> +/* Layout of 64-bit eq attribute values */
>> +struct kvm_ppc_xive_eq {
>> + __u32 flags;
>> + __u32 qsize;
>> + __u64 qpage;
>> + __u32 qtoggle;
>> + __u32 qindex;
>> + __u8 pad[40];
>> +};
>> +
>> +#define KVM_XIVE_EQ_FLAG_ENABLED 0x00000001
>> +#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
>> +#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
>> +
>> #endif /* __LINUX_KVM_POWERPC_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
>> index ab3ac152980d..6660d138c6b7 100644
>> --- a/arch/powerpc/kvm/book3s_xive.h
>> +++ b/arch/powerpc/kvm/book3s_xive.h
>> @@ -267,6 +267,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
>> struct kvmppc_xive *xive, int irq);
>> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
>> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
>> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
>> + bool single_escalation);
>>
>> #endif /* CONFIG_KVM_XICS */
>> #endif /* _KVM_PPC_BOOK3S_XICS_H */
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index 086da91d7c6e..7431e31bc541 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
>> return IRQ_HANDLED;
>> }
>>
>> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
>> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
>> + bool single_escalation)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> struct xive_q *q = &xc->queues[prio];
>> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
>> return -EIO;
>> }
>>
>> - if (xc->xive->single_escalation)
>> + if (single_escalation)
>> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
>> vcpu->kvm->arch.lpid, xc->server_num);
>> else
>> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
>> * interrupt, thus leaving it effectively masked after
>> * it fires once.
>> */
>> - if (xc->xive->single_escalation) {
>> + if (single_escalation) {
>> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
>> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>>
>> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
>> continue;
>> rc = xive_provision_queue(vcpu, prio);
>> if (rc = 0 && !xive->single_escalation)
>> - xive_attach_escalation(vcpu, prio);
>> + kvmppc_xive_attach_escalation(vcpu, prio,
>> + xive->single_escalation);
>> if (rc)
>> return rc;
>> }
>> @@ -1219,7 +1221,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> if (xive->qmap & (1 << i)) {
>> r = xive_provision_queue(vcpu, i);
>> if (r = 0 && !xive->single_escalation)
>> - xive_attach_escalation(vcpu, i);
>> + kvmppc_xive_attach_escalation(
>> + vcpu, i, xive->single_escalation);
>> if (r)
>> goto bail;
>> } else {
>> @@ -1234,7 +1237,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> }
>>
>> /* If not done above, attach priority 0 escalation */
>> - r = xive_attach_escalation(vcpu, 0);
>> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
>> if (r)
>> goto bail;
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index cb5a5c6e05af..34a35bcf550c 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -341,6 +341,201 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
>> priority, eisn);
>> }
>>
>> +static int xive_native_validate_queue_size(u32 qsize)
>> +{
>> + switch (qsize) {
>> + case 12:
>> + case 16:
>> + case 21:
>> + case 24:
>> + case 0:
>> + return 0;
>> + default:
>> + return -EINVAL;
>> + }
>> +}
>> +
>> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
>> + long eq_idx, u64 addr)
>> +{
>> + struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + struct kvmppc_xive_vcpu *xc;
>> + void __user *ubufp = (u64 __user *) addr;
>> + u32 server;
>> + u8 priority;
>> + struct kvm_ppc_xive_eq kvm_eq;
>> + int rc;
>> + __be32 *qaddr = 0;
>> + struct page *page;
>> + struct xive_q *q;
>> +
>> + /*
>> + * Demangle priority/server tuple from the EQ index
>> + */
>> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
>> + KVM_XIVE_EQ_PRIORITY_SHIFT;
>> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
>> + KVM_XIVE_EQ_SERVER_SHIFT;
>> +
>> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
>> + return -EFAULT;
>> +
>> + vcpu = kvmppc_xive_find_server(kvm, server);
>> + if (!vcpu) {
>> + pr_err("Can't find server %d\n", server);
>> + return -ENOENT;
>> + }
>> + xc = vcpu->arch.xive_vcpu;
>> +
>> + if (priority != xive_prio_from_guest(priority)) {
>> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
>> + priority, server);
>> + return -EINVAL;
>> + }
>> + q = &xc->queues[priority];
>
> You need to validate the 'flags' field (AFAICT we don't actually have
> any flags yet, so it's only valid it if is 0.
Well, we should set the ESCALATE flags and ALWAYS_NOTIFY also but this
is already set by xive_native_configure_queue(). I will take a closer look
and document.
>> +
>> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
>> + __func__, server, priority, kvm_eq.flags,
>> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
>> +
>> + rc = xive_native_validate_queue_size(kvm_eq.qsize);
>> + if (rc) {
>> + pr_err("invalid queue size %d\n", kvm_eq.qsize);
>> + return rc;
>> + }
>> +
>> + /* reset queue and disable queueing */
>> + if (!kvm_eq.qsize) {
>> + q->guest_qpage = 0;
>> + q->guest_qsize = 0;
>> +
>> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
>> + NULL, 0, true);
>> + if (rc) {
>> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
>> + priority, xc->server_num, rc);
>> + return rc;
>> + }
>> +
>> + if (q->qpage) {
>> + put_page(virt_to_page(q->qpage));
>> + q->qpage = NULL;
>> + }
>> +
>> + return 0;
>> + }
>> +
>> +
>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
>> + if (is_error_page(page)) {
>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
>> + return -ENOMEM;
>
> Nit: I don't think ENOMEM is the right error here. ENOMEM indicates
> that the kernel couldn't allocate enough memory to complete whatever
> you asked. Here the problem is the user supplied a bad guest address,
> which is a rather different error. EFAULT is closer, but still not
> quite right, since it could be a valid user address but not a valid
> guest address. There are probably existing KVM calls that could hit
> this problem, I wonder what they use.
I have seen a -EINVAL (s390) and a -EFAULT (x86)
Thanks,
C.
>> + }
>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
>> +
>> + /* Backup queue page guest address for migration */
>> + q->guest_qpage = kvm_eq.qpage;
>> + q->guest_qsize = kvm_eq.qsize;
>> +
>> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
>> + (__be32 *) qaddr, kvm_eq.qsize, true);
>> + if (rc) {
>> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
>> + priority, xc->server_num, rc);
>> + put_page(page);
>> + return rc;
>> + }
>> +
>> + rc = xive_native_set_queue_state(xc->vp_id, priority, kvm_eq.qtoggle,
>> + kvm_eq.qindex);
>> + if (rc)
>> + goto error;
>> +
>> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
>> + xive->single_escalation);
>> +error:
>> + if (rc)
>> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
>> + return rc;
>> +}
>> +
>> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
>> + long eq_idx, u64 addr)
>> +{
>> + struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + struct kvmppc_xive_vcpu *xc;
>> + struct xive_q *q;
>> + void __user *ubufp = (u64 __user *) addr;
>> + u32 server;
>> + u8 priority;
>> + struct kvm_ppc_xive_eq kvm_eq;
>> + u64 qpage;
>> + u64 qsize;
>> + u64 qeoi_page;
>> + u32 escalate_irq;
>> + u64 qflags;
>> + int rc;
>> +
>> + /*
>> + * Demangle priority/server tuple from the EQ index
>> + */
>> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
>> + KVM_XIVE_EQ_PRIORITY_SHIFT;
>> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
>> + KVM_XIVE_EQ_SERVER_SHIFT;
>> +
>> + vcpu = kvmppc_xive_find_server(kvm, server);
>> + if (!vcpu) {
>> + pr_err("Can't find server %d\n", server);
>> + return -ENOENT;
>> + }
>> + xc = vcpu->arch.xive_vcpu;
>> +
>> + if (priority != xive_prio_from_guest(priority)) {
>> + pr_err("invalid priority for queue %d for VCPU %d\n",
>> + priority, server);
>> + return -EINVAL;
>> + }
>> + q = &xc->queues[priority];
>> +
>> + memset(&kvm_eq, 0, sizeof(kvm_eq));
>> +
>> + if (!q->qpage)
>> + return 0;
>> +
>> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qpage, &qsize,
>> + &qeoi_page, &escalate_irq, &qflags);
>> + if (rc)
>> + return rc;
>> +
>> + kvm_eq.flags = 0;
>> + if (qflags & OPAL_XIVE_EQ_ENABLED)
>> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ENABLED;
>> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
>> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY;
>> + if (qflags & OPAL_XIVE_EQ_ESCALATE)
>> + kvm_eq.flags |= KVM_XIVE_EQ_FLAG_ESCALATE;
>> +
>> + kvm_eq.qsize = q->guest_qsize;
>> + kvm_eq.qpage = q->guest_qpage;
>> +
>> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
>> + &kvm_eq.qindex);
>> + if (rc)
>> + return rc;
>> +
>> + pr_devel("%s VCPU %d priority %d fl:%x sz:%d addr:%llx g:%d idx:%d\n",
>> + __func__, server, priority, kvm_eq.flags,
>> + kvm_eq.qsize, kvm_eq.qpage, kvm_eq.qtoggle, kvm_eq.qindex);
>> +
>> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
>> + return -EFAULT;
>> +
>> + return 0;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> @@ -355,6 +550,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
>> return kvmppc_xive_native_set_source_config(xive, attr->attr,
>> attr->addr);
>> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
>> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
>> + attr->addr);
>> }
>> return -ENXIO;
>> }
>> @@ -362,6 +560,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> + struct kvmppc_xive *xive = dev->private;
>> +
>> + switch (attr->group) {
>> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
>> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
>> + attr->addr);
>> + }
>> return -ENXIO;
>> }
>>
>> @@ -377,6 +582,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> attr->attr < KVMPPC_XIVE_NR_IRQS)
>> return 0;
>> break;
>> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
>> + return 0;
>> }
>> return -ENXIO;
>> }
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index 4f513a1880c7..c0b5d9bd43fb 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -52,3 +52,32 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> -ENXIO: CPU event queues not configured or configuration of the
>> underlying HW interrupt failed
>> -EBUSY: No CPU available to serve interrupt
>> +
>> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
>> + Configures an event queue of a CPU
>> + Attributes:
>> + EQ descriptor identifier (64-bit)
>> + The EQ descriptor identifier is a tuple (server, priority) :
>> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
>> + values: | unused | server | priority
>> + The kvm_device_attr.addr points to :
>> + struct kvm_ppc_xive_eq {
>> + __u32 flags;
>> + __u32 qsize;
>> + __u64 qpage;
>> + __u32 qtoggle;
>> + __u32 qindex;
>> + __u8 pad[40];
>> + };
>> + - flags: queue flags
>> + - qsize: queue size (power of 2)
>> + - qpage: real address of queue
>> + - qtoggle: current queue toggle bit
>> + - qindex: current queue index
>> + - pad: reserved for future use
>> + Errors:
>> + -ENOENT: Invalid CPU number
>> + -EINVAL: Invalid priority or invalid queue size
>> + -EFAULT: Invalid user pointer for attr->addr.
>> + -ENOMEM: Invalid queue address
>> + -EIO: Configuration of the underlying HW failed
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-03-12 17:00 ` Cédric Le Goater
@ 2019-03-13 4:03 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-13 4:03 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1680 bytes --]
On Tue, Mar 12, 2019 at 06:00:38PM +0100, Cédric Le Goater wrote:
> On 2/25/19 3:39 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >> restore the configuration of the XIVE EQs in the KVM device and to
> >> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >> rely on an OPAL call to access from the XIVE interrupt controller the
> >> EQ toggle bit and EQ index which are updated by the HW when event
> >> notifications are enqueued in the EQ.
> >>
> >> The value of the guest physical address of the event queue is saved in
> >> the XIVE internal xive_q structure for later use. That is when
> >> migration needs to mark the EQ pages dirty to capture a consistent
> >> memory state of the VM.
> >>
> >> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >> but restoring the EQ state will.
>
> I think we need to add some kind of flags to differentiate the hcall
> H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
> not need OPAL support call and this could help in the code
> transition.
Hrm. What's the actual difference in the semantics between the two
cases. The guest shouldn't have awareness of whether or not OPAL is
involved.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-13 4:03 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-13 4:03 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 1680 bytes --]
On Tue, Mar 12, 2019 at 06:00:38PM +0100, Cédric Le Goater wrote:
> On 2/25/19 3:39 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >> restore the configuration of the XIVE EQs in the KVM device and to
> >> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >> rely on an OPAL call to access from the XIVE interrupt controller the
> >> EQ toggle bit and EQ index which are updated by the HW when event
> >> notifications are enqueued in the EQ.
> >>
> >> The value of the guest physical address of the event queue is saved in
> >> the XIVE internal xive_q structure for later use. That is when
> >> migration needs to mark the EQ pages dirty to capture a consistent
> >> memory state of the VM.
> >>
> >> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >> but restoring the EQ state will.
>
> I think we need to add some kind of flags to differentiate the hcall
> H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
> not need OPAL support call and this could help in the code
> transition.
Hrm. What's the actual difference in the semantics between the two
cases. The guest shouldn't have awareness of whether or not OPAL is
involved.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-03-13 4:03 ` David Gibson
@ 2019-03-13 8:46 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:46 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 3/13/19 5:03 AM, David Gibson wrote:
> On Tue, Mar 12, 2019 at 06:00:38PM +0100, Cédric Le Goater wrote:
>> On 2/25/19 3:39 AM, David Gibson wrote:
>>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>>>> restore the configuration of the XIVE EQs in the KVM device and to
>>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>>>> rely on an OPAL call to access from the XIVE interrupt controller the
>>>> EQ toggle bit and EQ index which are updated by the HW when event
>>>> notifications are enqueued in the EQ.
>>>>
>>>> The value of the guest physical address of the event queue is saved in
>>>> the XIVE internal xive_q structure for later use. That is when
>>>> migration needs to mark the EQ pages dirty to capture a consistent
>>>> memory state of the VM.
>>>>
>>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>>>> but restoring the EQ state will.
>>
>> I think we need to add some kind of flags to differentiate the hcall
>> H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
>> not need OPAL support call and this could help in the code
>> transition.
>
> Hrm. What's the actual difference in the semantics between the two
> cases.
None.
But we don't need to set the EQ state in the case of the HCALL and it's
(very) practical to run guests with XIVE enabled without the OPAL support.
The latter is the main reason clearly.
Thinking of it, I could test the EQ toggle bit and index passed to KVM
and skip the OPAL call which restores the EQ state if they are zero.
This is because I know that the OPAL call configuring the EQ resets them.
That will do. No need for a flag.
> The guest shouldn't have awareness of whether or not OPAL is involved.
yes.
Thanks,
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-13 8:46 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:46 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 3/13/19 5:03 AM, David Gibson wrote:
> On Tue, Mar 12, 2019 at 06:00:38PM +0100, Cédric Le Goater wrote:
>> On 2/25/19 3:39 AM, David Gibson wrote:
>>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>>>> restore the configuration of the XIVE EQs in the KVM device and to
>>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>>>> rely on an OPAL call to access from the XIVE interrupt controller the
>>>> EQ toggle bit and EQ index which are updated by the HW when event
>>>> notifications are enqueued in the EQ.
>>>>
>>>> The value of the guest physical address of the event queue is saved in
>>>> the XIVE internal xive_q structure for later use. That is when
>>>> migration needs to mark the EQ pages dirty to capture a consistent
>>>> memory state of the VM.
>>>>
>>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>>>> but restoring the EQ state will.
>>
>> I think we need to add some kind of flags to differentiate the hcall
>> H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
>> not need OPAL support call and this could help in the code
>> transition.
>
> Hrm. What's the actual difference in the semantics between the two
> cases.
None.
But we don't need to set the EQ state in the case of the HCALL and it's
(very) practical to run guests with XIVE enabled without the OPAL support.
The latter is the main reason clearly.
Thinking of it, I could test the EQ toggle bit and index passed to KVM
and skip the OPAL call which restores the EQ state if they are zero.
This is because I know that the OPAL call configuring the EQ resets them.
That will do. No need for a flag.
> The guest shouldn't have awareness of whether or not OPAL is involved.
yes.
Thanks,
C.
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-03-13 8:46 ` Cédric Le Goater
@ 2019-03-14 3:29 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 3:29 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]
On Wed, Mar 13, 2019 at 09:46:08AM +0100, Cédric Le Goater wrote:
> On 3/13/19 5:03 AM, David Gibson wrote:
> > On Tue, Mar 12, 2019 at 06:00:38PM +0100, Cédric Le Goater wrote:
> >> On 2/25/19 3:39 AM, David Gibson wrote:
> >>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >>>> restore the configuration of the XIVE EQs in the KVM device and to
> >>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >>>> rely on an OPAL call to access from the XIVE interrupt controller the
> >>>> EQ toggle bit and EQ index which are updated by the HW when event
> >>>> notifications are enqueued in the EQ.
> >>>>
> >>>> The value of the guest physical address of the event queue is saved in
> >>>> the XIVE internal xive_q structure for later use. That is when
> >>>> migration needs to mark the EQ pages dirty to capture a consistent
> >>>> memory state of the VM.
> >>>>
> >>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >>>> but restoring the EQ state will.
> >>
> >> I think we need to add some kind of flags to differentiate the hcall
> >> H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
> >> not need OPAL support call and this could help in the code
> >> transition.
> >
> > Hrm. What's the actual difference in the semantics between the two
> > cases.
>
> None.
>
> But we don't need to set the EQ state in the case of the HCALL and it's
> (very) practical to run guests with XIVE enabled without the OPAL support.
> The latter is the main reason clearly.
>
> Thinking of it, I could test the EQ toggle bit and index passed to KVM
> and skip the OPAL call which restores the EQ state if they are zero.
> This is because I know that the OPAL call configuring the EQ resets them.
>
> That will do. No need for a flag.
That's a much better idea.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-14 3:29 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 3:29 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2302 bytes --]
On Wed, Mar 13, 2019 at 09:46:08AM +0100, Cédric Le Goater wrote:
> On 3/13/19 5:03 AM, David Gibson wrote:
> > On Tue, Mar 12, 2019 at 06:00:38PM +0100, Cédric Le Goater wrote:
> >> On 2/25/19 3:39 AM, David Gibson wrote:
> >>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >>>> restore the configuration of the XIVE EQs in the KVM device and to
> >>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >>>> rely on an OPAL call to access from the XIVE interrupt controller the
> >>>> EQ toggle bit and EQ index which are updated by the HW when event
> >>>> notifications are enqueued in the EQ.
> >>>>
> >>>> The value of the guest physical address of the event queue is saved in
> >>>> the XIVE internal xive_q structure for later use. That is when
> >>>> migration needs to mark the EQ pages dirty to capture a consistent
> >>>> memory state of the VM.
> >>>>
> >>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >>>> but restoring the EQ state will.
> >>
> >> I think we need to add some kind of flags to differentiate the hcall
> >> H_INT_SET_QUEUE_CONFIG from the restore of the EQ. The hcall does
> >> not need OPAL support call and this could help in the code
> >> transition.
> >
> > Hrm. What's the actual difference in the semantics between the two
> > cases.
>
> None.
>
> But we don't need to set the EQ state in the case of the HCALL and it's
> (very) practical to run guests with XIVE enabled without the OPAL support.
> The latter is the main reason clearly.
>
> Thinking of it, I could test the EQ toggle bit and index passed to KVM
> and skip the OPAL call which restores the EQ state if they are zero.
> This is because I know that the OPAL call configuring the EQ resets them.
>
> That will do. No need for a flag.
That's a much better idea.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-26 5:24 ` Paul Mackerras
-1 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-26 5:24 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> restore the configuration of the XIVE EQs in the KVM device and to
> capture the internal runtime state of the EQs. Both 'get' and 'set'
> rely on an OPAL call to access from the XIVE interrupt controller the
> EQ toggle bit and EQ index which are updated by the HW when event
> notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
[snip]
> +/* Layout of 64-bit eq attribute */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of 64-bit eq attribute values */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qsize;
> + __u64 qpage;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
This is confusing. What's the difference between an "eq attribute"
and an "eq attribute value"? Is the first actually a queue index or
a queue identifier?
Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
wrong. Maybe you meant "64-byte"?
[snip]
> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> + if (is_error_page(page)) {
> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> + return -ENOMEM;
> + }
> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
Isn't this assuming that we can map the whole queue with a single
gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
What happens if kvm_eq.qsize > PAGE_SHIFT?
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-02-26 5:24 ` Paul Mackerras
0 siblings, 0 replies; 142+ messages in thread
From: Paul Mackerras @ 2019-02-26 5:24 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> restore the configuration of the XIVE EQs in the KVM device and to
> capture the internal runtime state of the EQs. Both 'get' and 'set'
> rely on an OPAL call to access from the XIVE interrupt controller the
> EQ toggle bit and EQ index which are updated by the HW when event
> notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
[snip]
> +/* Layout of 64-bit eq attribute */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of 64-bit eq attribute values */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qsize;
> + __u64 qpage;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
This is confusing. What's the difference between an "eq attribute"
and an "eq attribute value"? Is the first actually a queue index or
a queue identifier?
Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
wrong. Maybe you meant "64-byte"?
[snip]
> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> + if (is_error_page(page)) {
> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> + return -ENOMEM;
> + }
> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
Isn't this assuming that we can map the whole queue with a single
gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
What happens if kvm_eq.qsize > PAGE_SHIFT?
Paul.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-02-26 5:24 ` Paul Mackerras
@ 2019-03-13 9:40 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 9:40 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On 2/26/19 6:24 AM, Paul Mackerras wrote:
> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>> restore the configuration of the XIVE EQs in the KVM device and to
>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>> rely on an OPAL call to access from the XIVE interrupt controller the
>> EQ toggle bit and EQ index which are updated by the HW when event
>> notifications are enqueued in the EQ.
>>
>> The value of the guest physical address of the event queue is saved in
>> the XIVE internal xive_q structure for later use. That is when
>> migration needs to mark the EQ pages dirty to capture a consistent
>> memory state of the VM.
>>
>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>> but restoring the EQ state will.
>
> [snip]
>
>> +/* Layout of 64-bit eq attribute */
>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
>> +
>> +/* Layout of 64-bit eq attribute values */
>> +struct kvm_ppc_xive_eq {
>> + __u32 flags;
>> + __u32 qsize;
>> + __u64 qpage;
>> + __u32 qtoggle;
>> + __u32 qindex;
>> + __u8 pad[40];
>> +};
>
> This is confusing. What's the difference between an "eq attribute"
> and an "eq attribute value"? Is the first actually a queue index or
> a queue identifier?
The "attribute" qualifier comes from the {get,set,has}_addr methods
of the KVM device. But it is not a well chosen name for the group
KVM_DEV_XIVE_GRP_EQ_CONFIG.
I should be using "eq identifier" and "eq values" or "eq state".
> Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
> wrong. Maybe you meant "64-byte"?
That was a bad copy paste. I have padded the structure to twice the size
of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
I thought that one extra u64 was not enough room for future.
>
> [snip]
>
>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
>> + if (is_error_page(page)) {
>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
>> + return -ENOMEM;
>> + }
>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
>
> Isn't this assuming that we can map the whole queue with a single
> gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
> What happens if kvm_eq.qsize > PAGE_SHIFT?
Ah yes. Theoretically, it should not happen because we only advertise
64K in the DT for the moment. I should at least add a check. So I will
change the helper xive_native_validate_queue_size() to return -EINVAL
for other page sizes.
Do you think it would be complex to support XIVE EQs using a page larger
than the default one on the guest ?
Thanks,
C.
>
> Paul.
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-13 9:40 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 9:40 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, linuxppc-dev, David Gibson
On 2/26/19 6:24 AM, Paul Mackerras wrote:
> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>> restore the configuration of the XIVE EQs in the KVM device and to
>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>> rely on an OPAL call to access from the XIVE interrupt controller the
>> EQ toggle bit and EQ index which are updated by the HW when event
>> notifications are enqueued in the EQ.
>>
>> The value of the guest physical address of the event queue is saved in
>> the XIVE internal xive_q structure for later use. That is when
>> migration needs to mark the EQ pages dirty to capture a consistent
>> memory state of the VM.
>>
>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>> but restoring the EQ state will.
>
> [snip]
>
>> +/* Layout of 64-bit eq attribute */
>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
>> +
>> +/* Layout of 64-bit eq attribute values */
>> +struct kvm_ppc_xive_eq {
>> + __u32 flags;
>> + __u32 qsize;
>> + __u64 qpage;
>> + __u32 qtoggle;
>> + __u32 qindex;
>> + __u8 pad[40];
>> +};
>
> This is confusing. What's the difference between an "eq attribute"
> and an "eq attribute value"? Is the first actually a queue index or
> a queue identifier?
The "attribute" qualifier comes from the {get,set,has}_addr methods
of the KVM device. But it is not a well chosen name for the group
KVM_DEV_XIVE_GRP_EQ_CONFIG.
I should be using "eq identifier" and "eq values" or "eq state".
> Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
> wrong. Maybe you meant "64-byte"?
That was a bad copy paste. I have padded the structure to twice the size
of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
I thought that one extra u64 was not enough room for future.
>
> [snip]
>
>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
>> + if (is_error_page(page)) {
>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
>> + return -ENOMEM;
>> + }
>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
>
> Isn't this assuming that we can map the whole queue with a single
> gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
> What happens if kvm_eq.qsize > PAGE_SHIFT?
Ah yes. Theoretically, it should not happen because we only advertise
64K in the DT for the moment. I should at least add a check. So I will
change the helper xive_native_validate_queue_size() to return -EINVAL
for other page sizes.
Do you think it would be complex to support XIVE EQs using a page larger
than the default one on the guest ?
Thanks,
C.
>
> Paul.
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-03-13 9:40 ` Cédric Le Goater
@ 2019-03-14 2:32 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:32 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3679 bytes --]
On Wed, Mar 13, 2019 at 10:40:19AM +0100, Cédric Le Goater wrote:
> On 2/26/19 6:24 AM, Paul Mackerras wrote:
> > On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >> restore the configuration of the XIVE EQs in the KVM device and to
> >> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >> rely on an OPAL call to access from the XIVE interrupt controller the
> >> EQ toggle bit and EQ index which are updated by the HW when event
> >> notifications are enqueued in the EQ.
> >>
> >> The value of the guest physical address of the event queue is saved in
> >> the XIVE internal xive_q structure for later use. That is when
> >> migration needs to mark the EQ pages dirty to capture a consistent
> >> memory state of the VM.
> >>
> >> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >> but restoring the EQ state will.
> >
> > [snip]
> >
> >> +/* Layout of 64-bit eq attribute */
> >> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> >> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> >> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> >> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> >> +
> >> +/* Layout of 64-bit eq attribute values */
> >> +struct kvm_ppc_xive_eq {
> >> + __u32 flags;
> >> + __u32 qsize;
> >> + __u64 qpage;
> >> + __u32 qtoggle;
> >> + __u32 qindex;
> >> + __u8 pad[40];
> >> +};
> >
> > This is confusing. What's the difference between an "eq attribute"
> > and an "eq attribute value"? Is the first actually a queue index or
> > a queue identifier?
>
> The "attribute" qualifier comes from the {get,set,has}_addr methods
> of the KVM device. But it is not a well chosen name for the group
> KVM_DEV_XIVE_GRP_EQ_CONFIG.
>
> I should be using "eq identifier" and "eq values" or "eq state".
Yeah, that seems clearer.
> > Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
> > wrong. Maybe you meant "64-byte"?
>
> That was a bad copy paste. I have padded the structure to twice the size
> of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
> I thought that one extra u64 was not enough room for future.
>
> >
> > [snip]
> >
> >> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> >> + if (is_error_page(page)) {
> >> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> >> + return -ENOMEM;
> >> + }
> >> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
> >
> > Isn't this assuming that we can map the whole queue with a single
> > gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
> > What happens if kvm_eq.qsize > PAGE_SHIFT?
>
> Ah yes. Theoretically, it should not happen because we only advertise
> 64K in the DT for the moment. I should at least add a check. So I will
> change the helper xive_native_validate_queue_size() to return -EINVAL
> for other page sizes.
Ok.
> Do you think it would be complex to support XIVE EQs using a page larger
> than the default one on the guest ?
Hm. The queue has to be physically contiguous from the host point of
view, in order for the XIVE hardware to write to it, doesn't it? If
so then supporting queues bigger than the guest page size would be
very difficult.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-14 2:32 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:32 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3679 bytes --]
On Wed, Mar 13, 2019 at 10:40:19AM +0100, Cédric Le Goater wrote:
> On 2/26/19 6:24 AM, Paul Mackerras wrote:
> > On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >> restore the configuration of the XIVE EQs in the KVM device and to
> >> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >> rely on an OPAL call to access from the XIVE interrupt controller the
> >> EQ toggle bit and EQ index which are updated by the HW when event
> >> notifications are enqueued in the EQ.
> >>
> >> The value of the guest physical address of the event queue is saved in
> >> the XIVE internal xive_q structure for later use. That is when
> >> migration needs to mark the EQ pages dirty to capture a consistent
> >> memory state of the VM.
> >>
> >> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >> but restoring the EQ state will.
> >
> > [snip]
> >
> >> +/* Layout of 64-bit eq attribute */
> >> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> >> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> >> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> >> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> >> +
> >> +/* Layout of 64-bit eq attribute values */
> >> +struct kvm_ppc_xive_eq {
> >> + __u32 flags;
> >> + __u32 qsize;
> >> + __u64 qpage;
> >> + __u32 qtoggle;
> >> + __u32 qindex;
> >> + __u8 pad[40];
> >> +};
> >
> > This is confusing. What's the difference between an "eq attribute"
> > and an "eq attribute value"? Is the first actually a queue index or
> > a queue identifier?
>
> The "attribute" qualifier comes from the {get,set,has}_addr methods
> of the KVM device. But it is not a well chosen name for the group
> KVM_DEV_XIVE_GRP_EQ_CONFIG.
>
> I should be using "eq identifier" and "eq values" or "eq state".
Yeah, that seems clearer.
> > Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
> > wrong. Maybe you meant "64-byte"?
>
> That was a bad copy paste. I have padded the structure to twice the size
> of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
> I thought that one extra u64 was not enough room for future.
>
> >
> > [snip]
> >
> >> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> >> + if (is_error_page(page)) {
> >> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> >> + return -ENOMEM;
> >> + }
> >> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
> >
> > Isn't this assuming that we can map the whole queue with a single
> > gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
> > What happens if kvm_eq.qsize > PAGE_SHIFT?
>
> Ah yes. Theoretically, it should not happen because we only advertise
> 64K in the DT for the moment. I should at least add a check. So I will
> change the helper xive_native_validate_queue_size() to return -EINVAL
> for other page sizes.
Ok.
> Do you think it would be complex to support XIVE EQs using a page larger
> than the default one on the guest ?
Hm. The queue has to be physically contiguous from the host point of
view, in order for the XIVE hardware to write to it, doesn't it? If
so then supporting queues bigger than the guest page size would be
very difficult.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-03-14 2:32 ` David Gibson
@ 2019-03-14 7:11 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-14 7:11 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, linuxppc-dev
On 3/14/19 3:32 AM, David Gibson wrote:
> On Wed, Mar 13, 2019 at 10:40:19AM +0100, Cédric Le Goater wrote:
>> On 2/26/19 6:24 AM, Paul Mackerras wrote:
>>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>>>> restore the configuration of the XIVE EQs in the KVM device and to
>>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>>>> rely on an OPAL call to access from the XIVE interrupt controller the
>>>> EQ toggle bit and EQ index which are updated by the HW when event
>>>> notifications are enqueued in the EQ.
>>>>
>>>> The value of the guest physical address of the event queue is saved in
>>>> the XIVE internal xive_q structure for later use. That is when
>>>> migration needs to mark the EQ pages dirty to capture a consistent
>>>> memory state of the VM.
>>>>
>>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>>>> but restoring the EQ state will.
>>>
>>> [snip]
>>>
>>>> +/* Layout of 64-bit eq attribute */
>>>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
>>>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
>>>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
>>>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
>>>> +
>>>> +/* Layout of 64-bit eq attribute values */
>>>> +struct kvm_ppc_xive_eq {
>>>> + __u32 flags;
>>>> + __u32 qsize;
>>>> + __u64 qpage;
>>>> + __u32 qtoggle;
>>>> + __u32 qindex;
>>>> + __u8 pad[40];
>>>> +};
>>>
>>> This is confusing. What's the difference between an "eq attribute"
>>> and an "eq attribute value"? Is the first actually a queue index or
>>> a queue identifier?
>>
>> The "attribute" qualifier comes from the {get,set,has}_addr methods
>> of the KVM device. But it is not a well chosen name for the group
>> KVM_DEV_XIVE_GRP_EQ_CONFIG.
>>
>> I should be using "eq identifier" and "eq values" or "eq state".
>
> Yeah, that seems clearer.
>
>>> Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
>>> wrong. Maybe you meant "64-byte"?
>>
>> That was a bad copy paste. I have padded the structure to twice the size
>> of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
>> I thought that one extra u64 was not enough room for future.
>>
>>>
>>> [snip]
>>>
>>>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
>>>> + if (is_error_page(page)) {
>>>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
>>>> + return -ENOMEM;
>>>> + }
>>>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
>>>
>>> Isn't this assuming that we can map the whole queue with a single
>>> gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
>>> What happens if kvm_eq.qsize > PAGE_SHIFT?
>>
>> Ah yes. Theoretically, it should not happen because we only advertise
>> 64K in the DT for the moment. I should at least add a check. So I will
>> change the helper xive_native_validate_queue_size() to return -EINVAL
>> for other page sizes.
>
> Ok.
>
>> Do you think it would be complex to support XIVE EQs using a page larger
>> than the default one on the guest ?
>
> Hm. The queue has to be physically contiguous from the host point of
> view, in order for the XIVE hardware to write to it, doesn't it? If
> so then supporting queues bigger than the guest page size would be
> very difficult.
The queue is only *one* page.
C.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-14 7:11 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-14 7:11 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, linuxppc-dev
On 3/14/19 3:32 AM, David Gibson wrote:
> On Wed, Mar 13, 2019 at 10:40:19AM +0100, Cédric Le Goater wrote:
>> On 2/26/19 6:24 AM, Paul Mackerras wrote:
>>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
>>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
>>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
>>>> restore the configuration of the XIVE EQs in the KVM device and to
>>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
>>>> rely on an OPAL call to access from the XIVE interrupt controller the
>>>> EQ toggle bit and EQ index which are updated by the HW when event
>>>> notifications are enqueued in the EQ.
>>>>
>>>> The value of the guest physical address of the event queue is saved in
>>>> the XIVE internal xive_q structure for later use. That is when
>>>> migration needs to mark the EQ pages dirty to capture a consistent
>>>> memory state of the VM.
>>>>
>>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
>>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
>>>> but restoring the EQ state will.
>>>
>>> [snip]
>>>
>>>> +/* Layout of 64-bit eq attribute */
>>>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
>>>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
>>>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
>>>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
>>>> +
>>>> +/* Layout of 64-bit eq attribute values */
>>>> +struct kvm_ppc_xive_eq {
>>>> + __u32 flags;
>>>> + __u32 qsize;
>>>> + __u64 qpage;
>>>> + __u32 qtoggle;
>>>> + __u32 qindex;
>>>> + __u8 pad[40];
>>>> +};
>>>
>>> This is confusing. What's the difference between an "eq attribute"
>>> and an "eq attribute value"? Is the first actually a queue index or
>>> a queue identifier?
>>
>> The "attribute" qualifier comes from the {get,set,has}_addr methods
>> of the KVM device. But it is not a well chosen name for the group
>> KVM_DEV_XIVE_GRP_EQ_CONFIG.
>>
>> I should be using "eq identifier" and "eq values" or "eq state".
>
> Yeah, that seems clearer.
>
>>> Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
>>> wrong. Maybe you meant "64-byte"?
>>
>> That was a bad copy paste. I have padded the structure to twice the size
>> of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
>> I thought that one extra u64 was not enough room for future.
>>
>>>
>>> [snip]
>>>
>>>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
>>>> + if (is_error_page(page)) {
>>>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
>>>> + return -ENOMEM;
>>>> + }
>>>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
>>>
>>> Isn't this assuming that we can map the whole queue with a single
>>> gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
>>> What happens if kvm_eq.qsize > PAGE_SHIFT?
>>
>> Ah yes. Theoretically, it should not happen because we only advertise
>> 64K in the DT for the moment. I should at least add a check. So I will
>> change the helper xive_native_validate_queue_size() to return -EINVAL
>> for other page sizes.
>
> Ok.
>
>> Do you think it would be complex to support XIVE EQs using a page larger
>> than the default one on the guest ?
>
> Hm. The queue has to be physically contiguous from the host point of
> view, in order for the XIVE hardware to write to it, doesn't it? If
> so then supporting queues bigger than the guest page size would be
> very difficult.
The queue is only *one* page.
C.
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-03-14 7:11 ` Cédric Le Goater
@ 2019-03-15 0:29 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-15 0:29 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4218 bytes --]
On Thu, Mar 14, 2019 at 08:11:17AM +0100, Cédric Le Goater wrote:
> On 3/14/19 3:32 AM, David Gibson wrote:
> > On Wed, Mar 13, 2019 at 10:40:19AM +0100, Cédric Le Goater wrote:
> >> On 2/26/19 6:24 AM, Paul Mackerras wrote:
> >>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >>>> restore the configuration of the XIVE EQs in the KVM device and to
> >>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >>>> rely on an OPAL call to access from the XIVE interrupt controller the
> >>>> EQ toggle bit and EQ index which are updated by the HW when event
> >>>> notifications are enqueued in the EQ.
> >>>>
> >>>> The value of the guest physical address of the event queue is saved in
> >>>> the XIVE internal xive_q structure for later use. That is when
> >>>> migration needs to mark the EQ pages dirty to capture a consistent
> >>>> memory state of the VM.
> >>>>
> >>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >>>> but restoring the EQ state will.
> >>>
> >>> [snip]
> >>>
> >>>> +/* Layout of 64-bit eq attribute */
> >>>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> >>>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> >>>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> >>>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> >>>> +
> >>>> +/* Layout of 64-bit eq attribute values */
> >>>> +struct kvm_ppc_xive_eq {
> >>>> + __u32 flags;
> >>>> + __u32 qsize;
> >>>> + __u64 qpage;
> >>>> + __u32 qtoggle;
> >>>> + __u32 qindex;
> >>>> + __u8 pad[40];
> >>>> +};
> >>>
> >>> This is confusing. What's the difference between an "eq attribute"
> >>> and an "eq attribute value"? Is the first actually a queue index or
> >>> a queue identifier?
> >>
> >> The "attribute" qualifier comes from the {get,set,has}_addr methods
> >> of the KVM device. But it is not a well chosen name for the group
> >> KVM_DEV_XIVE_GRP_EQ_CONFIG.
> >>
> >> I should be using "eq identifier" and "eq values" or "eq state".
> >
> > Yeah, that seems clearer.
> >
> >>> Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
> >>> wrong. Maybe you meant "64-byte"?
> >>
> >> That was a bad copy paste. I have padded the structure to twice the size
> >> of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
> >> I thought that one extra u64 was not enough room for future.
> >>
> >>>
> >>> [snip]
> >>>
> >>>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> >>>> + if (is_error_page(page)) {
> >>>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> >>>> + return -ENOMEM;
> >>>> + }
> >>>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
> >>>
> >>> Isn't this assuming that we can map the whole queue with a single
> >>> gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
> >>> What happens if kvm_eq.qsize > PAGE_SHIFT?
> >>
> >> Ah yes. Theoretically, it should not happen because we only advertise
> >> 64K in the DT for the moment. I should at least add a check. So I will
> >> change the helper xive_native_validate_queue_size() to return -EINVAL
> >> for other page sizes.
> >
> > Ok.
> >
> >> Do you think it would be complex to support XIVE EQs using a page larger
> >> than the default one on the guest ?
> >
> > Hm. The queue has to be physically contiguous from the host point of
> > view, in order for the XIVE hardware to write to it, doesn't it? If
> > so then supporting queues bigger than the guest page size would be
> > very difficult.
>
> The queue is only *one* page.
Right, but it's one *host* page, right, which is by nature host
physically contiguous. If the guest page size is different a single
guest page might not be host physically contiguous.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-03-15 0:29 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-15 0:29 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4218 bytes --]
On Thu, Mar 14, 2019 at 08:11:17AM +0100, Cédric Le Goater wrote:
> On 3/14/19 3:32 AM, David Gibson wrote:
> > On Wed, Mar 13, 2019 at 10:40:19AM +0100, Cédric Le Goater wrote:
> >> On 2/26/19 6:24 AM, Paul Mackerras wrote:
> >>> On Fri, Feb 22, 2019 at 12:28:30PM +0100, Cédric Le Goater wrote:
> >>>> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> >>>> H_INT_GET_QUEUE_CONFIG hcalls from QEMU. They will also be used to
> >>>> restore the configuration of the XIVE EQs in the KVM device and to
> >>>> capture the internal runtime state of the EQs. Both 'get' and 'set'
> >>>> rely on an OPAL call to access from the XIVE interrupt controller the
> >>>> EQ toggle bit and EQ index which are updated by the HW when event
> >>>> notifications are enqueued in the EQ.
> >>>>
> >>>> The value of the guest physical address of the event queue is saved in
> >>>> the XIVE internal xive_q structure for later use. That is when
> >>>> migration needs to mark the EQ pages dirty to capture a consistent
> >>>> memory state of the VM.
> >>>>
> >>>> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> >>>> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> >>>> but restoring the EQ state will.
> >>>
> >>> [snip]
> >>>
> >>>> +/* Layout of 64-bit eq attribute */
> >>>> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> >>>> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> >>>> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> >>>> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> >>>> +
> >>>> +/* Layout of 64-bit eq attribute values */
> >>>> +struct kvm_ppc_xive_eq {
> >>>> + __u32 flags;
> >>>> + __u32 qsize;
> >>>> + __u64 qpage;
> >>>> + __u32 qtoggle;
> >>>> + __u32 qindex;
> >>>> + __u8 pad[40];
> >>>> +};
> >>>
> >>> This is confusing. What's the difference between an "eq attribute"
> >>> and an "eq attribute value"? Is the first actually a queue index or
> >>> a queue identifier?
> >>
> >> The "attribute" qualifier comes from the {get,set,has}_addr methods
> >> of the KVM device. But it is not a well chosen name for the group
> >> KVM_DEV_XIVE_GRP_EQ_CONFIG.
> >>
> >> I should be using "eq identifier" and "eq values" or "eq state".
> >
> > Yeah, that seems clearer.
> >
> >>> Also, the kvm_ppc_xive_eq is not 64 bits, so the comment above it is
> >>> wrong. Maybe you meant "64-byte"?
> >>
> >> That was a bad copy paste. I have padded the structure to twice the size
> >> of the XIVE END (the XIVE EQ descriptor in HW) which size is 32 bytes.
> >> I thought that one extra u64 was not enough room for future.
> >>
> >>>
> >>> [snip]
> >>>
> >>>> + page = gfn_to_page(kvm, gpa_to_gfn(kvm_eq.qpage));
> >>>> + if (is_error_page(page)) {
> >>>> + pr_warn("Couldn't get guest page for %llx!\n", kvm_eq.qpage);
> >>>> + return -ENOMEM;
> >>>> + }
> >>>> + qaddr = page_to_virt(page) + (kvm_eq.qpage & ~PAGE_MASK);
> >>>
> >>> Isn't this assuming that we can map the whole queue with a single
> >>> gfn_to_page? That would only be true if kvm_eq.qsize <= PAGE_SHIFT.
> >>> What happens if kvm_eq.qsize > PAGE_SHIFT?
> >>
> >> Ah yes. Theoretically, it should not happen because we only advertise
> >> 64K in the DT for the moment. I should at least add a check. So I will
> >> change the helper xive_native_validate_queue_size() to return -EINVAL
> >> for other page sizes.
> >
> > Ok.
> >
> >> Do you think it would be complex to support XIVE EQs using a page larger
> >> than the default one on the guest ?
> >
> > Hm. The queue has to be physically contiguous from the host point of
> > view, in order for the XIVE hardware to write to it, doesn't it? If
> > so then supporting queues bigger than the guest page size would be
> > very difficult.
>
> The queue is only *one* page.
Right, but it's one *host* page, right, which is by nature host
physically contiguous. If the guest page size is different a single
guest page might not be host physically contiguous.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control is to be used by the H_INT_RESET hcall from QEMU. Its
purpose is to clear all configuration of the sources and EQs. This is
necessary in case of a kexec (for a kdump kernel for instance) to make
sure that no remaining configuration is left from the previous boot
setup so that the new kernel can start safely from a clean state.
The queue 7 is ignored when the KVM device is configured to run in
single escalation mode. Prio 7 is used by escalations.
The XIVE VP is kept enabled as the vCPU is still active and connected
to the XIVE device.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 5 ++
3 files changed, 88 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 177e43f3edaf..7ae8cb22af7d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,6 +677,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_RESET 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 34a35bcf550c..bb3e121c918a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -536,6 +536,80 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
return 0;
}
+static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
+{
+ int i;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
+
+ if (!state->valid)
+ continue;
+
+ if (state->act_priority == MASKED)
+ continue;
+
+ arch_spin_lock(&sb->lock);
+ state->eisn = 0;
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+ if (state->pt_number) {
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->pt_number,
+ 0, MASKED, 0);
+ }
+ arch_spin_unlock(&sb->lock);
+ }
+}
+
+static int kvmppc_xive_reset(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ continue;
+
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+
+ /* Single escalation, no queue 7 */
+ if (prio == 7 && xive->single_escalation)
+ break;
+
+ if (xc->esc_virq[prio]) {
+ free_irq(xc->esc_virq[prio], vcpu);
+ irq_dispose_mapping(xc->esc_virq[prio]);
+ kfree(xc->esc_virq_names[prio]);
+ xc->esc_virq[prio] = 0;
+ }
+
+ kvmppc_xive_native_cleanup_queue(vcpu, prio);
+ }
+ }
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_reset_sources(xive->src_blocks[i]);
+ }
+
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -543,6 +617,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return kvmppc_xive_reset(xive);
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
@@ -575,6 +653,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
{
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return 0;
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index c0b5d9bd43fb..f1d007f485a9 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+ Attributes:
+ 1.1 KVM_DEV_XIVE_RESET (write only)
+ Resets the interrupt controller configuration for sources and event
+ queues. To be used by kexec and kdump.
+ Errors: none
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control is to be used by the H_INT_RESET hcall from QEMU. Its
purpose is to clear all configuration of the sources and EQs. This is
necessary in case of a kexec (for a kdump kernel for instance) to make
sure that no remaining configuration is left from the previous boot
setup so that the new kernel can start safely from a clean state.
The queue 7 is ignored when the KVM device is configured to run in
single escalation mode. Prio 7 is used by escalations.
The XIVE VP is kept enabled as the vCPU is still active and connected
to the XIVE device.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 5 ++
3 files changed, 88 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 177e43f3edaf..7ae8cb22af7d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,6 +677,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_RESET 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 34a35bcf550c..bb3e121c918a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -536,6 +536,80 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
return 0;
}
+static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
+{
+ int i;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
+
+ if (!state->valid)
+ continue;
+
+ if (state->act_priority = MASKED)
+ continue;
+
+ arch_spin_lock(&sb->lock);
+ state->eisn = 0;
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+ if (state->pt_number) {
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->pt_number,
+ 0, MASKED, 0);
+ }
+ arch_spin_unlock(&sb->lock);
+ }
+}
+
+static int kvmppc_xive_reset(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ continue;
+
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+
+ /* Single escalation, no queue 7 */
+ if (prio = 7 && xive->single_escalation)
+ break;
+
+ if (xc->esc_virq[prio]) {
+ free_irq(xc->esc_virq[prio], vcpu);
+ irq_dispose_mapping(xc->esc_virq[prio]);
+ kfree(xc->esc_virq_names[prio]);
+ xc->esc_virq[prio] = 0;
+ }
+
+ kvmppc_xive_native_cleanup_queue(vcpu, prio);
+ }
+ }
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_reset_sources(xive->src_blocks[i]);
+ }
+
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -543,6 +617,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return kvmppc_xive_reset(xive);
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
@@ -575,6 +653,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
{
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return 0;
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index c0b5d9bd43fb..f1d007f485a9 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+ Attributes:
+ 1.1 KVM_DEV_XIVE_RESET (write only)
+ Resets the interrupt controller configuration for sources and event
+ queues. To be used by kexec and kdump.
+ Errors: none
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 2:43 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:43 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 5657 bytes --]
On Fri, Feb 22, 2019 at 12:28:31PM +0100, Cédric Le Goater wrote:
> This control is to be used by the H_INT_RESET hcall from QEMU. Its
> purpose is to clear all configuration of the sources and EQs. This is
> necessary in case of a kexec (for a kdump kernel for instance) to make
> sure that no remaining configuration is left from the previous boot
> setup so that the new kernel can start safely from a clean state.
>
> The queue 7 is ignored when the KVM device is configured to run in
> single escalation mode. Prio 7 is used by escalations.
>
> The XIVE VP is kept enabled as the vCPU is still active and connected
> to the XIVE device.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 5 ++
> 3 files changed, 88 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 177e43f3edaf..7ae8cb22af7d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -677,6 +677,7 @@ struct kvm_ppc_cpu_char {
>
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> +#define KVM_DEV_XIVE_RESET 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 34a35bcf550c..bb3e121c918a 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -536,6 +536,80 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> return 0;
> }
>
> +static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
> +{
> + int i;
> +
> + for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
> +
> + if (!state->valid)
> + continue;
> +
> + if (state->act_priority == MASKED)
You're checking some things in state before you take the lock. Could
this race?
> + continue;
> +
> + arch_spin_lock(&sb->lock);
> + state->eisn = 0;
> + state->act_server = 0;
> + state->act_priority = MASKED;
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> + if (state->pt_number) {
> + xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->pt_number,
> + 0, MASKED, 0);
> + }
> + arch_spin_unlock(&sb->lock);
> + }
> +}
> +
> +static int kvmppc_xive_reset(struct kvmppc_xive *xive)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + unsigned int i;
> +
> + pr_devel("%s\n", __func__);
> +
> + mutex_lock(&kvm->lock);
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + unsigned int prio;
> +
> + if (!xc)
> + continue;
> +
> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
> +
> + for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
> +
> + /* Single escalation, no queue 7 */
> + if (prio == 7 && xive->single_escalation)
> + break;
> +
> + if (xc->esc_virq[prio]) {
> + free_irq(xc->esc_virq[prio], vcpu);
> + irq_dispose_mapping(xc->esc_virq[prio]);
> + kfree(xc->esc_virq_names[prio]);
> + xc->esc_virq[prio] = 0;
> + }
> +
> + kvmppc_xive_native_cleanup_queue(vcpu, prio);
> + }
> + }
> +
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_reset_sources(xive->src_blocks[i]);
> + }
> +
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -543,6 +617,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> + switch (attr->attr) {
> + case KVM_DEV_XIVE_RESET:
> + return kvmppc_xive_reset(xive);
> + }
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> return kvmppc_xive_native_set_source(xive, attr->attr,
> @@ -575,6 +653,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> {
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> + switch (attr->attr) {
> + case KVM_DEV_XIVE_RESET:
> + return 0;
> + }
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index c0b5d9bd43fb..f1d007f485a9 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>
> 1. KVM_DEV_XIVE_GRP_CTRL
> Provides global controls on the device
> + Attributes:
> + 1.1 KVM_DEV_XIVE_RESET (write only)
> + Resets the interrupt controller configuration for sources and event
> + queues. To be used by kexec and kdump.
> + Errors: none
>
> 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> Initializes a new source in the XIVE device and mask it.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
@ 2019-02-25 2:43 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:43 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 5657 bytes --]
On Fri, Feb 22, 2019 at 12:28:31PM +0100, Cédric Le Goater wrote:
> This control is to be used by the H_INT_RESET hcall from QEMU. Its
> purpose is to clear all configuration of the sources and EQs. This is
> necessary in case of a kexec (for a kdump kernel for instance) to make
> sure that no remaining configuration is left from the previous boot
> setup so that the new kernel can start safely from a clean state.
>
> The queue 7 is ignored when the KVM device is configured to run in
> single escalation mode. Prio 7 is used by escalations.
>
> The XIVE VP is kept enabled as the vCPU is still active and connected
> to the XIVE device.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 5 ++
> 3 files changed, 88 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 177e43f3edaf..7ae8cb22af7d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -677,6 +677,7 @@ struct kvm_ppc_cpu_char {
>
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> +#define KVM_DEV_XIVE_RESET 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 34a35bcf550c..bb3e121c918a 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -536,6 +536,80 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> return 0;
> }
>
> +static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
> +{
> + int i;
> +
> + for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
> +
> + if (!state->valid)
> + continue;
> +
> + if (state->act_priority == MASKED)
You're checking some things in state before you take the lock. Could
this race?
> + continue;
> +
> + arch_spin_lock(&sb->lock);
> + state->eisn = 0;
> + state->act_server = 0;
> + state->act_priority = MASKED;
> + xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
> + if (state->pt_number) {
> + xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
> + xive_native_configure_irq(state->pt_number,
> + 0, MASKED, 0);
> + }
> + arch_spin_unlock(&sb->lock);
> + }
> +}
> +
> +static int kvmppc_xive_reset(struct kvmppc_xive *xive)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + unsigned int i;
> +
> + pr_devel("%s\n", __func__);
> +
> + mutex_lock(&kvm->lock);
> +
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + unsigned int prio;
> +
> + if (!xc)
> + continue;
> +
> + kvmppc_xive_disable_vcpu_interrupts(vcpu);
> +
> + for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
> +
> + /* Single escalation, no queue 7 */
> + if (prio == 7 && xive->single_escalation)
> + break;
> +
> + if (xc->esc_virq[prio]) {
> + free_irq(xc->esc_virq[prio], vcpu);
> + irq_dispose_mapping(xc->esc_virq[prio]);
> + kfree(xc->esc_virq_names[prio]);
> + xc->esc_virq[prio] = 0;
> + }
> +
> + kvmppc_xive_native_cleanup_queue(vcpu, prio);
> + }
> + }
> +
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_reset_sources(xive->src_blocks[i]);
> + }
> +
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -543,6 +617,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> + switch (attr->attr) {
> + case KVM_DEV_XIVE_RESET:
> + return kvmppc_xive_reset(xive);
> + }
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> return kvmppc_xive_native_set_source(xive, attr->attr,
> @@ -575,6 +653,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> {
> switch (attr->group) {
> case KVM_DEV_XIVE_GRP_CTRL:
> + switch (attr->attr) {
> + case KVM_DEV_XIVE_RESET:
> + return 0;
> + }
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index c0b5d9bd43fb..f1d007f485a9 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>
> 1. KVM_DEV_XIVE_GRP_CTRL
> Provides global controls on the device
> + Attributes:
> + 1.1 KVM_DEV_XIVE_RESET (write only)
> + Resets the interrupt controller configuration for sources and event
> + queues. To be used by kexec and kdump.
> + Errors: none
>
> 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> Initializes a new source in the XIVE device and mask it.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control will be used by the H_INT_SYNC hcall from QEMU.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 34 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 9 ++++++
3 files changed, 44 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 7ae8cb22af7d..289c504b7c1d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
+#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source attributes */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index bb3e121c918a..dd2a9d411fe7 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -341,6 +341,36 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, eisn);
}
+static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+
+ pr_devel("%s irq=0x%lx", __func__, irq);
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ arch_spin_lock(&sb->lock);
+
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
static int xive_native_validate_queue_size(u32 qsize)
{
switch (qsize) {
@@ -631,6 +661,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_EQ_CONFIG:
return kvmppc_xive_native_set_queue_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
+ return kvmppc_xive_native_sync_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -660,6 +693,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index f1d007f485a9..267634eae9e0 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -86,3 +86,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EFAULT: Invalid user pointer for attr->addr.
-ENOMEM: Invalid queue address
-EIO: Configuration of the underlying HW failed
+
+ 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
+ Synchronize the source to flush event notification
+ Attributes:
+ Interrupt source number (64-bit)
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number, invalid priority or
+ invalid CPU number.
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control will be used by the H_INT_SYNC hcall from QEMU.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 34 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 9 ++++++
3 files changed, 44 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 7ae8cb22af7d..289c504b7c1d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
+#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source attributes */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index bb3e121c918a..dd2a9d411fe7 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -341,6 +341,36 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, eisn);
}
+static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+
+ pr_devel("%s irq=0x%lx", __func__, irq);
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ arch_spin_lock(&sb->lock);
+
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+
+ arch_spin_unlock(&sb->lock);
+
+ return 0;
+}
+
static int xive_native_validate_queue_size(u32 qsize)
{
switch (qsize) {
@@ -631,6 +661,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_EQ_CONFIG:
return kvmppc_xive_native_set_queue_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
+ return kvmppc_xive_native_sync_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -660,6 +693,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index f1d007f485a9..267634eae9e0 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -86,3 +86,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EFAULT: Invalid user pointer for attr->addr.
-ENOMEM: Invalid queue address
-EIO: Configuration of the underlying HW failed
+
+ 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
+ Synchronize the source to flush event notification
+ Attributes:
+ Interrupt source number (64-bit)
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number, invalid priority or
+ invalid CPU number.
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 2:45 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:45 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4012 bytes --]
On Fri, Feb 22, 2019 at 12:28:32PM +0100, Cédric Le Goater wrote:
> This control will be used by the H_INT_SYNC hcall from QEMU.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 34 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 9 ++++++
> 3 files changed, 44 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 7ae8cb22af7d..289c504b7c1d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> +#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source attributes */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index bb3e121c918a..dd2a9d411fe7 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -341,6 +341,36 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, eisn);
> }
>
> +static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
> + long irq, u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + struct xive_irq_data *xd;
> + u32 hw_num;
> + u16 src;
> +
> + pr_devel("%s irq=0x%lx", __func__, irq);
> +
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb)
> + return -ENOENT;
> +
> + state = &sb->irq_state[src];
> +
> + if (!state->valid)
Is testing this before you take the lock safe?
> + return -EINVAL;
> +
> + arch_spin_lock(&sb->lock);
> +
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> + xive_native_sync_source(hw_num);
> +
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> static int xive_native_validate_queue_size(u32 qsize)
> {
> switch (qsize) {
> @@ -631,6 +661,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
> + return kvmppc_xive_native_sync_source(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -660,6 +693,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> + case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
> if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index f1d007f485a9..267634eae9e0 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -86,3 +86,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -EFAULT: Invalid user pointer for attr->addr.
> -ENOMEM: Invalid queue address
> -EIO: Configuration of the underlying HW failed
> +
> + 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
> + Synchronize the source to flush event notification
> + Attributes:
> + Interrupt source number (64-bit)
> + Errors:
> + -ENOENT: Unknown source number
> + -EINVAL: Not initialized source number, invalid priority or
> + invalid CPU number.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
@ 2019-02-25 2:45 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:45 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4012 bytes --]
On Fri, Feb 22, 2019 at 12:28:32PM +0100, Cédric Le Goater wrote:
> This control will be used by the H_INT_SYNC hcall from QEMU.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 34 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 9 ++++++
> 3 files changed, 44 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 7ae8cb22af7d..289c504b7c1d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> +#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source attributes */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index bb3e121c918a..dd2a9d411fe7 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -341,6 +341,36 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, eisn);
> }
>
> +static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
> + long irq, u64 addr)
> +{
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + struct xive_irq_data *xd;
> + u32 hw_num;
> + u16 src;
> +
> + pr_devel("%s irq=0x%lx", __func__, irq);
> +
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb)
> + return -ENOENT;
> +
> + state = &sb->irq_state[src];
> +
> + if (!state->valid)
Is testing this before you take the lock safe?
> + return -EINVAL;
> +
> + arch_spin_lock(&sb->lock);
> +
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> + xive_native_sync_source(hw_num);
> +
> + arch_spin_unlock(&sb->lock);
> +
> + return 0;
> +}
> +
> static int xive_native_validate_queue_size(u32 qsize)
> {
> switch (qsize) {
> @@ -631,6 +661,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
> + return kvmppc_xive_native_sync_source(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -660,6 +693,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> + case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
> if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index f1d007f485a9..267634eae9e0 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -86,3 +86,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -EFAULT: Invalid user pointer for attr->addr.
> -ENOMEM: Invalid queue address
> -EIO: Configuration of the underlying HW failed
> +
> + 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
> + Synchronize the source to flush event notification
> + Attributes:
> + Interrupt source number (64-bit)
> + Errors:
> + -ENOENT: Unknown source number
> + -EINVAL: Not initialized source number, invalid priority or
> + invalid CPU number.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
When migration of a VM is initiated, a first copy of the RAM is
transferred to the destination before the VM is stopped, but there is
no guarantee that the EQ pages in which the event notification are
queued have not been modified.
To make sure migration will capture a consistent memory state, the
XIVE device should perform a XIVE quiesce sequence to stop the flow of
event notifications and stabilize the EQs. This is the purpose of the
KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
to force their transfer.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
3 files changed, 97 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 289c504b7c1d..cd78ad1020fe 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_RESET 1
+#define KVM_DEV_XIVE_EQ_SYNC 2
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index dd2a9d411fe7..3debc876d5a0 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
return 0;
}
+static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
+{
+ int j;
+
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
+ struct xive_irq_data *xd;
+ u32 hw_num;
+
+ if (!state->valid)
+ continue;
+ if (state->act_priority == MASKED)
+ continue;
+
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ xive_native_sync_queue(hw_num);
+ arch_spin_unlock(&sb->lock);
+ }
+}
+
+static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ return -ENOENT;
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+ struct xive_q *q = &xc->queues[prio];
+
+ if (!q->qpage)
+ continue;
+
+ /* Mark EQ page dirty for migration */
+ mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qpage));
+ }
+ return 0;
+}
+
+static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_native_sync_sources(xive->src_blocks[i]);
+ }
+
+ mutex_lock(&kvm->lock);
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ kvmppc_xive_native_vcpu_eq_sync(vcpu);
+ }
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -650,6 +714,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
return kvmppc_xive_reset(xive);
+ case KVM_DEV_XIVE_EQ_SYNC:
+ return kvmppc_xive_native_eq_sync(xive);
}
break;
case KVM_DEV_XIVE_GRP_SOURCE:
@@ -688,6 +754,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
+ case KVM_DEV_XIVE_EQ_SYNC:
return 0;
}
break;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 267634eae9e0..a26be635cff9 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
queues. To be used by kexec and kdump.
Errors: none
+ 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
+ Sync all the sources and queues and mark the EQ pages dirty. This
+ to make sure that a consistent memory state is captured when
+ migrating the VM.
+ Errors: none
+
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
Attributes:
@@ -95,3 +101,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOENT: Unknown source number
-EINVAL: Not initialized source number, invalid priority or
invalid CPU number.
+
+* Migration:
+
+ Saving the state of a VM using the XIVE native exploitation mode
+ should follow a specific sequence. When the VM is stopped :
+
+ 1. Mask all sources (PQ=01) to stop the flow of events.
+
+ 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
+ flush any in-flight event notification and to stabilize the EQs. At
+ this stage, the EQ pages are marked dirty to make sure they are
+ transferred in the migration sequence.
+
+ 3. Capture the state of the source targeting, the EQs configuration
+ and the state of thread interrupt context registers.
+
+ Restore is similar :
+
+ 1. Restore the EQ configuration. As targeting depends on it.
+ 2. Restore targeting
+ 3. Restore the thread interrupt contexts
+ 4. Restore the source states
+ 5. Let the vCPU run
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
When migration of a VM is initiated, a first copy of the RAM is
transferred to the destination before the VM is stopped, but there is
no guarantee that the EQ pages in which the event notification are
queued have not been modified.
To make sure migration will capture a consistent memory state, the
XIVE device should perform a XIVE quiesce sequence to stop the flow of
event notifications and stabilize the EQs. This is the purpose of the
KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
to force their transfer.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
3 files changed, 97 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 289c504b7c1d..cd78ad1020fe 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_RESET 1
+#define KVM_DEV_XIVE_EQ_SYNC 2
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index dd2a9d411fe7..3debc876d5a0 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
return 0;
}
+static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
+{
+ int j;
+
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
+ struct xive_irq_data *xd;
+ u32 hw_num;
+
+ if (!state->valid)
+ continue;
+ if (state->act_priority = MASKED)
+ continue;
+
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ xive_native_sync_queue(hw_num);
+ arch_spin_unlock(&sb->lock);
+ }
+}
+
+static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ return -ENOENT;
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+ struct xive_q *q = &xc->queues[prio];
+
+ if (!q->qpage)
+ continue;
+
+ /* Mark EQ page dirty for migration */
+ mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qpage));
+ }
+ return 0;
+}
+
+static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_native_sync_sources(xive->src_blocks[i]);
+ }
+
+ mutex_lock(&kvm->lock);
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ kvmppc_xive_native_vcpu_eq_sync(vcpu);
+ }
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -650,6 +714,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
return kvmppc_xive_reset(xive);
+ case KVM_DEV_XIVE_EQ_SYNC:
+ return kvmppc_xive_native_eq_sync(xive);
}
break;
case KVM_DEV_XIVE_GRP_SOURCE:
@@ -688,6 +754,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
+ case KVM_DEV_XIVE_EQ_SYNC:
return 0;
}
break;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 267634eae9e0..a26be635cff9 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
queues. To be used by kexec and kdump.
Errors: none
+ 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
+ Sync all the sources and queues and mark the EQ pages dirty. This
+ to make sure that a consistent memory state is captured when
+ migrating the VM.
+ Errors: none
+
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
Attributes:
@@ -95,3 +101,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOENT: Unknown source number
-EINVAL: Not initialized source number, invalid priority or
invalid CPU number.
+
+* Migration:
+
+ Saving the state of a VM using the XIVE native exploitation mode
+ should follow a specific sequence. When the VM is stopped :
+
+ 1. Mask all sources (PQ\x01) to stop the flow of events.
+
+ 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
+ flush any in-flight event notification and to stabilize the EQs. At
+ this stage, the EQ pages are marked dirty to make sure they are
+ transferred in the migration sequence.
+
+ 3. Capture the state of the source targeting, the EQs configuration
+ and the state of thread interrupt context registers.
+
+ Restore is similar :
+
+ 1. Restore the EQ configuration. As targeting depends on it.
+ 2. Restore targeting
+ 3. Restore the thread interrupt contexts
+ 4. Restore the source states
+ 5. Let the vCPU run
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 2:53 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:53 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 6528 bytes --]
On Fri, Feb 22, 2019 at 12:28:33PM +0100, Cédric Le Goater wrote:
> When migration of a VM is initiated, a first copy of the RAM is
> transferred to the destination before the VM is stopped, but there is
> no guarantee that the EQ pages in which the event notification are
> queued have not been modified.
>
> To make sure migration will capture a consistent memory state, the
> XIVE device should perform a XIVE quiesce sequence to stop the flow of
> event notifications and stabilize the EQs. This is the purpose of the
> KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
> to force their transfer.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
> 3 files changed, 97 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 289c504b7c1d..cd78ad1020fe 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_RESET 1
> +#define KVM_DEV_XIVE_EQ_SYNC 2
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index dd2a9d411fe7..3debc876d5a0 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
> return 0;
> }
>
> +static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
> +{
> + int j;
> +
> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
> + struct xive_irq_data *xd;
> + u32 hw_num;
> +
> + if (!state->valid)
> + continue;
> + if (state->act_priority == MASKED)
Is this correct? If you masked an irq, then immediately did a sync,
couldn't there still be some of the irqs in flight? I thought the
reason we needed a sync was that masking and other such operations
_didn't_ implicitly synchronize.
> + continue;
> +
> + arch_spin_lock(&sb->lock);
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> + xive_native_sync_source(hw_num);
> + xive_native_sync_queue(hw_num);
> + arch_spin_unlock(&sb->lock);
> + }
> +}
> +
> +static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + unsigned int prio;
> +
> + if (!xc)
> + return -ENOENT;
> +
> + for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
> + struct xive_q *q = &xc->queues[prio];
> +
> + if (!q->qpage)
> + continue;
> +
> + /* Mark EQ page dirty for migration */
> + mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qpage));
> + }
> + return 0;
> +}
> +
> +static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + unsigned int i;
> +
> + pr_devel("%s\n", __func__);
> +
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_native_sync_sources(xive->src_blocks[i]);
> + }
> +
> + mutex_lock(&kvm->lock);
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + kvmppc_xive_native_vcpu_eq_sync(vcpu);
> + }
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -650,6 +714,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> switch (attr->attr) {
> case KVM_DEV_XIVE_RESET:
> return kvmppc_xive_reset(xive);
> + case KVM_DEV_XIVE_EQ_SYNC:
> + return kvmppc_xive_native_eq_sync(xive);
> }
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> @@ -688,6 +754,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_CTRL:
> switch (attr->attr) {
> case KVM_DEV_XIVE_RESET:
> + case KVM_DEV_XIVE_EQ_SYNC:
> return 0;
> }
> break;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 267634eae9e0..a26be635cff9 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> queues. To be used by kexec and kdump.
> Errors: none
>
> + 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
> + Sync all the sources and queues and mark the EQ pages dirty. This
> + to make sure that a consistent memory state is captured when
> + migrating the VM.
> + Errors: none
> +
> 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> Initializes a new source in the XIVE device and mask it.
> Attributes:
> @@ -95,3 +101,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENOENT: Unknown source number
> -EINVAL: Not initialized source number, invalid priority or
> invalid CPU number.
> +
> +* Migration:
> +
> + Saving the state of a VM using the XIVE native exploitation mode
> + should follow a specific sequence. When the VM is stopped :
> +
> + 1. Mask all sources (PQ=01) to stop the flow of events.
> +
> + 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
> + flush any in-flight event notification and to stabilize the EQs. At
> + this stage, the EQ pages are marked dirty to make sure they are
> + transferred in the migration sequence.
> +
> + 3. Capture the state of the source targeting, the EQs configuration
> + and the state of thread interrupt context registers.
> +
> + Restore is similar :
> +
> + 1. Restore the EQ configuration. As targeting depends on it.
> + 2. Restore targeting
> + 3. Restore the thread interrupt contexts
> + 4. Restore the source states
> + 5. Let the vCPU run
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
@ 2019-02-25 2:53 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 2:53 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 6528 bytes --]
On Fri, Feb 22, 2019 at 12:28:33PM +0100, Cédric Le Goater wrote:
> When migration of a VM is initiated, a first copy of the RAM is
> transferred to the destination before the VM is stopped, but there is
> no guarantee that the EQ pages in which the event notification are
> queued have not been modified.
>
> To make sure migration will capture a consistent memory state, the
> XIVE device should perform a XIVE quiesce sequence to stop the flow of
> event notifications and stabilize the EQs. This is the purpose of the
> KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
> to force their transfer.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
> 3 files changed, 97 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 289c504b7c1d..cd78ad1020fe 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
> /* POWER9 XIVE Native Interrupt Controller */
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_RESET 1
> +#define KVM_DEV_XIVE_EQ_SYNC 2
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index dd2a9d411fe7..3debc876d5a0 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
> return 0;
> }
>
> +static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
> +{
> + int j;
> +
> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
> + struct xive_irq_data *xd;
> + u32 hw_num;
> +
> + if (!state->valid)
> + continue;
> + if (state->act_priority == MASKED)
Is this correct? If you masked an irq, then immediately did a sync,
couldn't there still be some of the irqs in flight? I thought the
reason we needed a sync was that masking and other such operations
_didn't_ implicitly synchronize.
> + continue;
> +
> + arch_spin_lock(&sb->lock);
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> + xive_native_sync_source(hw_num);
> + xive_native_sync_queue(hw_num);
> + arch_spin_unlock(&sb->lock);
> + }
> +}
> +
> +static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + unsigned int prio;
> +
> + if (!xc)
> + return -ENOENT;
> +
> + for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
> + struct xive_q *q = &xc->queues[prio];
> +
> + if (!q->qpage)
> + continue;
> +
> + /* Mark EQ page dirty for migration */
> + mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qpage));
> + }
> + return 0;
> +}
> +
> +static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + unsigned int i;
> +
> + pr_devel("%s\n", __func__);
> +
> + for (i = 0; i <= xive->max_sbid; i++) {
> + if (xive->src_blocks[i])
> + kvmppc_xive_native_sync_sources(xive->src_blocks[i]);
> + }
> +
> + mutex_lock(&kvm->lock);
> + kvm_for_each_vcpu(i, vcpu, kvm) {
> + kvmppc_xive_native_vcpu_eq_sync(vcpu);
> + }
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -650,6 +714,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> switch (attr->attr) {
> case KVM_DEV_XIVE_RESET:
> return kvmppc_xive_reset(xive);
> + case KVM_DEV_XIVE_EQ_SYNC:
> + return kvmppc_xive_native_eq_sync(xive);
> }
> break;
> case KVM_DEV_XIVE_GRP_SOURCE:
> @@ -688,6 +754,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_CTRL:
> switch (attr->attr) {
> case KVM_DEV_XIVE_RESET:
> + case KVM_DEV_XIVE_EQ_SYNC:
> return 0;
> }
> break;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 267634eae9e0..a26be635cff9 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> queues. To be used by kexec and kdump.
> Errors: none
>
> + 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
> + Sync all the sources and queues and mark the EQ pages dirty. This
> + to make sure that a consistent memory state is captured when
> + migrating the VM.
> + Errors: none
> +
> 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
> Initializes a new source in the XIVE device and mask it.
> Attributes:
> @@ -95,3 +101,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENOENT: Unknown source number
> -EINVAL: Not initialized source number, invalid priority or
> invalid CPU number.
> +
> +* Migration:
> +
> + Saving the state of a VM using the XIVE native exploitation mode
> + should follow a specific sequence. When the VM is stopped :
> +
> + 1. Mask all sources (PQ=01) to stop the flow of events.
> +
> + 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
> + flush any in-flight event notification and to stabilize the EQs. At
> + this stage, the EQ pages are marked dirty to make sure they are
> + transferred in the migration sequence.
> +
> + 3. Capture the state of the source targeting, the EQs configuration
> + and the state of thread interrupt context registers.
> +
> + Restore is similar :
> +
> + 1. Restore the EQ configuration. As targeting depends on it.
> + 2. Restore targeting
> + 3. Restore the thread interrupt contexts
> + 4. Restore the source states
> + 5. Let the vCPU run
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
2019-02-25 2:53 ` David Gibson
@ 2019-03-13 11:48 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 11:48 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 3:53 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:33PM +0100, Cédric Le Goater wrote:
>> When migration of a VM is initiated, a first copy of the RAM is
>> transferred to the destination before the VM is stopped, but there is
>> no guarantee that the EQ pages in which the event notification are
>> queued have not been modified.
>>
>> To make sure migration will capture a consistent memory state, the
>> XIVE device should perform a XIVE quiesce sequence to stop the flow of
>> event notifications and stabilize the EQs. This is the purpose of the
>> KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
>> to force their transfer.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/uapi/asm/kvm.h | 1 +
>> arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
>> 3 files changed, 97 insertions(+)
>>
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index 289c504b7c1d..cd78ad1020fe 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
>> /* POWER9 XIVE Native Interrupt Controller */
>> #define KVM_DEV_XIVE_GRP_CTRL 1
>> #define KVM_DEV_XIVE_RESET 1
>> +#define KVM_DEV_XIVE_EQ_SYNC 2
>> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
>> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
>> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index dd2a9d411fe7..3debc876d5a0 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
>> return 0;
>> }
>>
>> +static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
>> +{
>> + int j;
>> +
>> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
>> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
>> + struct xive_irq_data *xd;
>> + u32 hw_num;
>> +
>> + if (!state->valid)
>> + continue;
>> + if (state->act_priority == MASKED)
>
> Is this correct? If you masked an irq, then immediately did a sync,
> couldn't there still be some of the irqs in flight? I thought the
> reason we needed a sync was that masking and other such operations
> _didn't_ implicitly synchronize.
The struct kvmppc_xive_irq_state reflects the state of the EAS
configuration and not the state of the source. The source is masked
setting the PQ bits to '-Q', which is what is being done before calling
the KVM_DEV_XIVE_EQ_SYNC control.
If a source EAS is configured, OPAL syncs the XIVE IC of the source and
the XIVE IC of the previous target if any.
So I think we are fine.
C.
>> + continue;
>> +
>> + arch_spin_lock(&sb->lock);
>> + kvmppc_xive_select_irq(state, &hw_num, &xd);
>> + xive_native_sync_source(hw_num);
>> + xive_native_sync_queue(hw_num);
>> + arch_spin_unlock(&sb->lock);
>> + }
>> +}
>> +
>> +static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + unsigned int prio;
>> +
>> + if (!xc)
>> + return -ENOENT;
>> +
>> + for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
>> + struct xive_q *q = &xc->queues[prio];
>> +
>> + if (!q->qpage)
>> + continue;
>> +
>> + /* Mark EQ page dirty for migration */
>> + mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qpage));
>> + }
>> + return 0;
>> +}
>> +
>> +static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
>> +{
>> + struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + unsigned int i;
>> +
>> + pr_devel("%s\n", __func__);
>> +
>> + for (i = 0; i <= xive->max_sbid; i++) {
>> + if (xive->src_blocks[i])
>> + kvmppc_xive_native_sync_sources(xive->src_blocks[i]);
>> + }
>> +
>> + mutex_lock(&kvm->lock);
>> + kvm_for_each_vcpu(i, vcpu, kvm) {
>> + kvmppc_xive_native_vcpu_eq_sync(vcpu);
>> + }
>> + mutex_unlock(&kvm->lock);
>> +
>> + return 0;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> @@ -650,6 +714,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> switch (attr->attr) {
>> case KVM_DEV_XIVE_RESET:
>> return kvmppc_xive_reset(xive);
>> + case KVM_DEV_XIVE_EQ_SYNC:
>> + return kvmppc_xive_native_eq_sync(xive);
>> }
>> break;
>> case KVM_DEV_XIVE_GRP_SOURCE:
>> @@ -688,6 +754,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> case KVM_DEV_XIVE_GRP_CTRL:
>> switch (attr->attr) {
>> case KVM_DEV_XIVE_RESET:
>> + case KVM_DEV_XIVE_EQ_SYNC:
>> return 0;
>> }
>> break;
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index 267634eae9e0..a26be635cff9 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> queues. To be used by kexec and kdump.
>> Errors: none
>>
>> + 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
>> + Sync all the sources and queues and mark the EQ pages dirty. This
>> + to make sure that a consistent memory state is captured when
>> + migrating the VM.
>> + Errors: none
>> +
>> 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
>> Initializes a new source in the XIVE device and mask it.
>> Attributes:
>> @@ -95,3 +101,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> -ENOENT: Unknown source number
>> -EINVAL: Not initialized source number, invalid priority or
>> invalid CPU number.
>> +
>> +* Migration:
>> +
>> + Saving the state of a VM using the XIVE native exploitation mode
>> + should follow a specific sequence. When the VM is stopped :
>> +
>> + 1. Mask all sources (PQ=01) to stop the flow of events.
>> +
>> + 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
>> + flush any in-flight event notification and to stabilize the EQs. At
>> + this stage, the EQ pages are marked dirty to make sure they are
>> + transferred in the migration sequence.
>> +
>> + 3. Capture the state of the source targeting, the EQs configuration
>> + and the state of thread interrupt context registers.
>> +
>> + Restore is similar :
>> +
>> + 1. Restore the EQ configuration. As targeting depends on it.
>> + 2. Restore targeting
>> + 3. Restore the thread interrupt contexts
>> + 4. Restore the source states
>> + 5. Let the vCPU run
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
@ 2019-03-13 11:48 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 11:48 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 3:53 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:33PM +0100, Cédric Le Goater wrote:
>> When migration of a VM is initiated, a first copy of the RAM is
>> transferred to the destination before the VM is stopped, but there is
>> no guarantee that the EQ pages in which the event notification are
>> queued have not been modified.
>>
>> To make sure migration will capture a consistent memory state, the
>> XIVE device should perform a XIVE quiesce sequence to stop the flow of
>> event notifications and stabilize the EQs. This is the purpose of the
>> KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
>> to force their transfer.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/uapi/asm/kvm.h | 1 +
>> arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
>> 3 files changed, 97 insertions(+)
>>
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index 289c504b7c1d..cd78ad1020fe 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
>> /* POWER9 XIVE Native Interrupt Controller */
>> #define KVM_DEV_XIVE_GRP_CTRL 1
>> #define KVM_DEV_XIVE_RESET 1
>> +#define KVM_DEV_XIVE_EQ_SYNC 2
>> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
>> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
>> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index dd2a9d411fe7..3debc876d5a0 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
>> return 0;
>> }
>>
>> +static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
>> +{
>> + int j;
>> +
>> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
>> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
>> + struct xive_irq_data *xd;
>> + u32 hw_num;
>> +
>> + if (!state->valid)
>> + continue;
>> + if (state->act_priority = MASKED)
>
> Is this correct? If you masked an irq, then immediately did a sync,
> couldn't there still be some of the irqs in flight? I thought the
> reason we needed a sync was that masking and other such operations
> _didn't_ implicitly synchronize.
The struct kvmppc_xive_irq_state reflects the state of the EAS
configuration and not the state of the source. The source is masked
setting the PQ bits to '-Q', which is what is being done before calling
the KVM_DEV_XIVE_EQ_SYNC control.
If a source EAS is configured, OPAL syncs the XIVE IC of the source and
the XIVE IC of the previous target if any.
So I think we are fine.
C.
>> + continue;
>> +
>> + arch_spin_lock(&sb->lock);
>> + kvmppc_xive_select_irq(state, &hw_num, &xd);
>> + xive_native_sync_source(hw_num);
>> + xive_native_sync_queue(hw_num);
>> + arch_spin_unlock(&sb->lock);
>> + }
>> +}
>> +
>> +static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + unsigned int prio;
>> +
>> + if (!xc)
>> + return -ENOENT;
>> +
>> + for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
>> + struct xive_q *q = &xc->queues[prio];
>> +
>> + if (!q->qpage)
>> + continue;
>> +
>> + /* Mark EQ page dirty for migration */
>> + mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qpage));
>> + }
>> + return 0;
>> +}
>> +
>> +static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
>> +{
>> + struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> + unsigned int i;
>> +
>> + pr_devel("%s\n", __func__);
>> +
>> + for (i = 0; i <= xive->max_sbid; i++) {
>> + if (xive->src_blocks[i])
>> + kvmppc_xive_native_sync_sources(xive->src_blocks[i]);
>> + }
>> +
>> + mutex_lock(&kvm->lock);
>> + kvm_for_each_vcpu(i, vcpu, kvm) {
>> + kvmppc_xive_native_vcpu_eq_sync(vcpu);
>> + }
>> + mutex_unlock(&kvm->lock);
>> +
>> + return 0;
>> +}
>> +
>> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> struct kvm_device_attr *attr)
>> {
>> @@ -650,6 +714,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
>> switch (attr->attr) {
>> case KVM_DEV_XIVE_RESET:
>> return kvmppc_xive_reset(xive);
>> + case KVM_DEV_XIVE_EQ_SYNC:
>> + return kvmppc_xive_native_eq_sync(xive);
>> }
>> break;
>> case KVM_DEV_XIVE_GRP_SOURCE:
>> @@ -688,6 +754,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
>> case KVM_DEV_XIVE_GRP_CTRL:
>> switch (attr->attr) {
>> case KVM_DEV_XIVE_RESET:
>> + case KVM_DEV_XIVE_EQ_SYNC:
>> return 0;
>> }
>> break;
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index 267634eae9e0..a26be635cff9 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> queues. To be used by kexec and kdump.
>> Errors: none
>>
>> + 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
>> + Sync all the sources and queues and mark the EQ pages dirty. This
>> + to make sure that a consistent memory state is captured when
>> + migrating the VM.
>> + Errors: none
>> +
>> 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
>> Initializes a new source in the XIVE device and mask it.
>> Attributes:
>> @@ -95,3 +101,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> -ENOENT: Unknown source number
>> -EINVAL: Not initialized source number, invalid priority or
>> invalid CPU number.
>> +
>> +* Migration:
>> +
>> + Saving the state of a VM using the XIVE native exploitation mode
>> + should follow a specific sequence. When the VM is stopped :
>> +
>> + 1. Mask all sources (PQ\x01) to stop the flow of events.
>> +
>> + 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
>> + flush any in-flight event notification and to stabilize the EQs. At
>> + this stage, the EQ pages are marked dirty to make sure they are
>> + transferred in the migration sequence.
>> +
>> + 3. Capture the state of the source targeting, the EQs configuration
>> + and the state of thread interrupt context registers.
>> +
>> + Restore is similar :
>> +
>> + 1. Restore the EQ configuration. As targeting depends on it.
>> + 2. Restore targeting
>> + 3. Restore the thread interrupt contexts
>> + 4. Restore the source states
>> + 5. Let the vCPU run
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
2019-03-13 11:48 ` Cédric Le Goater
@ 2019-03-14 2:33 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:33 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]
On Wed, Mar 13, 2019 at 12:48:57PM +0100, Cédric Le Goater wrote:
> On 2/25/19 3:53 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:33PM +0100, Cédric Le Goater wrote:
> >> When migration of a VM is initiated, a first copy of the RAM is
> >> transferred to the destination before the VM is stopped, but there is
> >> no guarantee that the EQ pages in which the event notification are
> >> queued have not been modified.
> >>
> >> To make sure migration will capture a consistent memory state, the
> >> XIVE device should perform a XIVE quiesce sequence to stop the flow of
> >> event notifications and stabilize the EQs. This is the purpose of the
> >> KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
> >> to force their transfer.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> >> arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
> >> Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
> >> 3 files changed, 97 insertions(+)
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> >> index 289c504b7c1d..cd78ad1020fe 100644
> >> --- a/arch/powerpc/include/uapi/asm/kvm.h
> >> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> >> @@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
> >> /* POWER9 XIVE Native Interrupt Controller */
> >> #define KVM_DEV_XIVE_GRP_CTRL 1
> >> #define KVM_DEV_XIVE_RESET 1
> >> +#define KVM_DEV_XIVE_EQ_SYNC 2
> >> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> >> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> >> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> >> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> >> index dd2a9d411fe7..3debc876d5a0 100644
> >> --- a/arch/powerpc/kvm/book3s_xive_native.c
> >> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> >> @@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
> >> return 0;
> >> }
> >>
> >> +static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
> >> +{
> >> + int j;
> >> +
> >> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
> >> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
> >> + struct xive_irq_data *xd;
> >> + u32 hw_num;
> >> +
> >> + if (!state->valid)
> >> + continue;
> >> + if (state->act_priority == MASKED)
> >
> > Is this correct? If you masked an irq, then immediately did a sync,
> > couldn't there still be some of the irqs in flight? I thought the
> > reason we needed a sync was that masking and other such operations
> > _didn't_ implicitly synchronize.
>
> The struct kvmppc_xive_irq_state reflects the state of the EAS
> configuration and not the state of the source. The source is masked
> setting the PQ bits to '-Q', which is what is being done before calling
> the KVM_DEV_XIVE_EQ_SYNC control.
>
> If a source EAS is configured, OPAL syncs the XIVE IC of the source and
> the XIVE IC of the previous target if any.
>
> So I think we are fine.
Ok.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
@ 2019-03-14 2:33 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:33 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3411 bytes --]
On Wed, Mar 13, 2019 at 12:48:57PM +0100, Cédric Le Goater wrote:
> On 2/25/19 3:53 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:33PM +0100, Cédric Le Goater wrote:
> >> When migration of a VM is initiated, a first copy of the RAM is
> >> transferred to the destination before the VM is stopped, but there is
> >> no guarantee that the EQ pages in which the event notification are
> >> queued have not been modified.
> >>
> >> To make sure migration will capture a consistent memory state, the
> >> XIVE device should perform a XIVE quiesce sequence to stop the flow of
> >> event notifications and stabilize the EQs. This is the purpose of the
> >> KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
> >> to force their transfer.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> >> arch/powerpc/kvm/book3s_xive_native.c | 67 ++++++++++++++++++++++
> >> Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++++
> >> 3 files changed, 97 insertions(+)
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> >> index 289c504b7c1d..cd78ad1020fe 100644
> >> --- a/arch/powerpc/include/uapi/asm/kvm.h
> >> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> >> @@ -678,6 +678,7 @@ struct kvm_ppc_cpu_char {
> >> /* POWER9 XIVE Native Interrupt Controller */
> >> #define KVM_DEV_XIVE_GRP_CTRL 1
> >> #define KVM_DEV_XIVE_RESET 1
> >> +#define KVM_DEV_XIVE_EQ_SYNC 2
> >> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source attributes */
> >> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source attributes */
> >> #define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit eq attributes */
> >> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> >> index dd2a9d411fe7..3debc876d5a0 100644
> >> --- a/arch/powerpc/kvm/book3s_xive_native.c
> >> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> >> @@ -640,6 +640,70 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
> >> return 0;
> >> }
> >>
> >> +static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
> >> +{
> >> + int j;
> >> +
> >> + for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
> >> + struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
> >> + struct xive_irq_data *xd;
> >> + u32 hw_num;
> >> +
> >> + if (!state->valid)
> >> + continue;
> >> + if (state->act_priority == MASKED)
> >
> > Is this correct? If you masked an irq, then immediately did a sync,
> > couldn't there still be some of the irqs in flight? I thought the
> > reason we needed a sync was that masking and other such operations
> > _didn't_ implicitly synchronize.
>
> The struct kvmppc_xive_irq_state reflects the state of the EAS
> configuration and not the state of the source. The source is masked
> setting the PQ bits to '-Q', which is what is being done before calling
> the KVM_DEV_XIVE_EQ_SYNC control.
>
> If a source EAS is configured, OPAL syncs the XIVE IC of the source and
> the XIVE IC of the previous target if any.
>
> So I think we are fine.
Ok.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
At a VCPU level, the state of the thread interrupt management
registers needs to be collected. These registers are cached under the
'xive_saved_state.w01' field of the VCPU when the VPCU context is
pulled from the HW thread. An OPAL call retrieves the backup of the
IPB register in the underlying XIVE NVT structure and merges it in the
KVM state.
The structures of the interface between QEMU and KVM provisions some
extra room (two u64) for further extensions if more state needs to be
transferred back to QEMU.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_ppc.h | 11 +++
arch/powerpc/include/uapi/asm/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 24 +++++++
arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 +++++
5 files changed, 138 insertions(+)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 1e61877fe147..664c65051612 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -272,6 +272,7 @@ union kvmppc_one_reg {
u64 addr;
u64 length;
} vpaval;
+ u64 xive_timaval[4];
};
struct kvmppc_ops {
@@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
+extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
+extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
@@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
+static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return 0; }
+static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return -ENOENT; }
#endif /* CONFIG_KVM_XIVE */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index cd78ad1020fe..42d4ef93ec2d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
#define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
#define KVM_REG_PPC_ICP_PPRI_MASK 0xff
+#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
+
/* Device control API: PPC-specific devices */
#define KVM_DEV_MPIC_GRP_MISC 1
#define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 96d43f091255..f85a9211f30c 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
*val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_get_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
*val = get_reg_val(id, vcpu->arch.fscr);
break;
@@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_set_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
vcpu->arch.fscr = set_reg_val(id, *val);
break;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 3debc876d5a0..132bff52d70a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
return ret;
}
+/*
+ * Interrupt Pending Buffer (IPB) offset
+ */
+#define TM_IPB_SHIFT 40
+#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
+
+int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u64 opal_state;
+ int rc;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc)
+ return -ENOENT;
+
+ /* Thread context registers. We only care about IPB and CPPR */
+ val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
+
+ /*
+ * Return the OS CAM line to print out the VP identifier in
+ * the QEMU monitor. This is not restored.
+ */
+ val->xive_timaval[1] = vcpu->arch.xive_cam_word;
+
+ /* Get the VP state from OPAL */
+ rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
+ if (rc)
+ return rc;
+
+ /*
+ * Capture the backup of IPB register in the NVT structure and
+ * merge it in our KVM VP state.
+ */
+ val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
+
+ pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
+ __func__,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word, opal_state);
+
+ return 0;
+}
+
+int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+
+ pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
+ val->xive_timaval[0], val->xive_timaval[1]);
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc || !xive)
+ return -ENOENT;
+
+ /* We can't update the state of a "pushed" VCPU */
+ if (WARN_ON(vcpu->arch.xive_pushed))
+ return -EIO;
+
+ /*
+ * Restore the thread context registers. IPB and CPPR should
+ * be the only ones that matter.
+ */
+ vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
+
+ /*
+ * There is no need to restore the XIVE internal state (IPB
+ * stored in the NVT) as the IPB register was merged in KVM VP
+ * state when captured.
+ */
+ return 0;
+}
+
static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index a26be635cff9..1b8957c50c53 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EINVAL: Not initialized source number, invalid priority or
invalid CPU number.
+* VCPU state
+
+ The XIVE IC maintains VP interrupt state in an internal structure
+ called the NVT. When a VP is not dispatched on a HW processor
+ thread, this structure can be updated by HW if the VP is the target
+ of an event notification.
+
+ It is important for migration to capture the cached IPB from the NVT
+ as it synthesizes the priorities of the pending interrupts. We
+ capture a bit more to report debug information.
+
+ KVM_REG_PPC_VP_STATE (4 * 64bits)
+ bits: | 63 .... 32 | 31 .... 0 |
+ values: | TIMA word0 | TIMA word1 |
+ bits: | 127 .......... 64 |
+ values: | VP CAM Line |
+ bits: | 255 .......... 128 |
+ values: | unused |
+
* Migration:
Saving the state of a VM using the XIVE native exploitation mode
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
At a VCPU level, the state of the thread interrupt management
registers needs to be collected. These registers are cached under the
'xive_saved_state.w01' field of the VCPU when the VPCU context is
pulled from the HW thread. An OPAL call retrieves the backup of the
IPB register in the underlying XIVE NVT structure and merges it in the
KVM state.
The structures of the interface between QEMU and KVM provisions some
extra room (two u64) for further extensions if more state needs to be
transferred back to QEMU.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_ppc.h | 11 +++
arch/powerpc/include/uapi/asm/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 24 +++++++
arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 +++++
5 files changed, 138 insertions(+)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 1e61877fe147..664c65051612 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -272,6 +272,7 @@ union kvmppc_one_reg {
u64 addr;
u64 length;
} vpaval;
+ u64 xive_timaval[4];
};
struct kvmppc_ops {
@@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
+extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
+extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
@@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
+static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return 0; }
+static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return -ENOENT; }
#endif /* CONFIG_KVM_XIVE */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index cd78ad1020fe..42d4ef93ec2d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
#define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
#define KVM_REG_PPC_ICP_PPRI_MASK 0xff
+#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
+
/* Device control API: PPC-specific devices */
#define KVM_DEV_MPIC_GRP_MISC 1
#define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 96d43f091255..f85a9211f30c 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
*val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_get_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
*val = get_reg_val(id, vcpu->arch.fscr);
break;
@@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_set_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
vcpu->arch.fscr = set_reg_val(id, *val);
break;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 3debc876d5a0..132bff52d70a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
return ret;
}
+/*
+ * Interrupt Pending Buffer (IPB) offset
+ */
+#define TM_IPB_SHIFT 40
+#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
+
+int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u64 opal_state;
+ int rc;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc)
+ return -ENOENT;
+
+ /* Thread context registers. We only care about IPB and CPPR */
+ val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
+
+ /*
+ * Return the OS CAM line to print out the VP identifier in
+ * the QEMU monitor. This is not restored.
+ */
+ val->xive_timaval[1] = vcpu->arch.xive_cam_word;
+
+ /* Get the VP state from OPAL */
+ rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
+ if (rc)
+ return rc;
+
+ /*
+ * Capture the backup of IPB register in the NVT structure and
+ * merge it in our KVM VP state.
+ */
+ val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
+
+ pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
+ __func__,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word, opal_state);
+
+ return 0;
+}
+
+int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+
+ pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
+ val->xive_timaval[0], val->xive_timaval[1]);
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc || !xive)
+ return -ENOENT;
+
+ /* We can't update the state of a "pushed" VCPU */
+ if (WARN_ON(vcpu->arch.xive_pushed))
+ return -EIO;
+
+ /*
+ * Restore the thread context registers. IPB and CPPR should
+ * be the only ones that matter.
+ */
+ vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
+
+ /*
+ * There is no need to restore the XIVE internal state (IPB
+ * stored in the NVT) as the IPB register was merged in KVM VP
+ * state when captured.
+ */
+ return 0;
+}
+
static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index a26be635cff9..1b8957c50c53 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EINVAL: Not initialized source number, invalid priority or
invalid CPU number.
+* VCPU state
+
+ The XIVE IC maintains VP interrupt state in an internal structure
+ called the NVT. When a VP is not dispatched on a HW processor
+ thread, this structure can be updated by HW if the VP is the target
+ of an event notification.
+
+ It is important for migration to capture the cached IPB from the NVT
+ as it synthesizes the priorities of the pending interrupts. We
+ capture a bit more to report debug information.
+
+ KVM_REG_PPC_VP_STATE (4 * 64bits)
+ bits: | 63 .... 32 | 31 .... 0 |
+ values: | TIMA word0 | TIMA word1 |
+ bits: | 127 .......... 64 |
+ values: | VP CAM Line |
+ bits: | 255 .......... 128 |
+ values: | unused |
+
* Migration:
Saving the state of a VM using the XIVE native exploitation mode
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 3:31 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:31 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 9623 bytes --]
On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
> At a VCPU level, the state of the thread interrupt management
> registers needs to be collected. These registers are cached under the
> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
> pulled from the HW thread. An OPAL call retrieves the backup of the
> IPB register in the underlying XIVE NVT structure and merges it in the
> KVM state.
>
> The structures of the interface between QEMU and KVM provisions some
> extra room (two u64) for further extensions if more state needs to be
> transferred back to QEMU.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
> arch/powerpc/include/uapi/asm/kvm.h | 2 +
> arch/powerpc/kvm/book3s.c | 24 +++++++
> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
> 5 files changed, 138 insertions(+)
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index 1e61877fe147..664c65051612 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
> u64 addr;
> u64 length;
> } vpaval;
> + u64 xive_timaval[4];
This is doubling the size of the userspace visible one_reg union. Is
that safe?
> };
>
> struct kvmppc_ops {
> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
> extern void kvmppc_xive_native_init_module(void);
> extern void kvmppc_xive_native_exit_module(void);
> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val);
> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val);
>
> #else
> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
> static inline void kvmppc_xive_native_init_module(void) { }
> static inline void kvmppc_xive_native_exit_module(void) { }
> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val)
> +{ return 0; }
> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val)
> +{ return -ENOENT; }
>
> #endif /* CONFIG_KVM_XIVE */
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index cd78ad1020fe..42d4ef93ec2d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
>
> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
> +
> /* Device control API: PPC-specific devices */
> #define KVM_DEV_MPIC_GRP_MISC 1
> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> index 96d43f091255..f85a9211f30c 100644
> --- a/arch/powerpc/kvm/book3s.c
> +++ b/arch/powerpc/kvm/book3s.c
> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
> break;
> #endif /* CONFIG_KVM_XICS */
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_REG_PPC_VP_STATE:
> + if (!vcpu->arch.xive_vcpu) {
> + r = -ENXIO;
> + break;
> + }
> + if (xive_enabled())
> + r = kvmppc_xive_native_get_vp(vcpu, val);
> + else
> + r = -ENXIO;
> + break;
> +#endif /* CONFIG_KVM_XIVE */
> case KVM_REG_PPC_FSCR:
> *val = get_reg_val(id, vcpu->arch.fscr);
> break;
> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
> break;
> #endif /* CONFIG_KVM_XICS */
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_REG_PPC_VP_STATE:
> + if (!vcpu->arch.xive_vcpu) {
> + r = -ENXIO;
> + break;
> + }
> + if (xive_enabled())
> + r = kvmppc_xive_native_set_vp(vcpu, val);
> + else
> + r = -ENXIO;
> + break;
> +#endif /* CONFIG_KVM_XIVE */
> case KVM_REG_PPC_FSCR:
> vcpu->arch.fscr = set_reg_val(id, *val);
> break;
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 3debc876d5a0..132bff52d70a 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> return ret;
> }
>
> +/*
> + * Interrupt Pending Buffer (IPB) offset
> + */
> +#define TM_IPB_SHIFT 40
> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
> +
> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + u64 opal_state;
> + int rc;
> +
> + if (!kvmppc_xive_enabled(vcpu))
> + return -EPERM;
> +
> + if (!xc)
> + return -ENOENT;
> +
> + /* Thread context registers. We only care about IPB and CPPR */
> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
> +
> + /*
> + * Return the OS CAM line to print out the VP identifier in
> + * the QEMU monitor. This is not restored.
> + */
> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
I'm pretty dubious about this mixing of vital state information with
what's basically debug information. Doubly so since it requires
changing the ABI to increase the one_reg union's size.
Might be better to have this control only return the 0th and 2nd u64s
from the TIMA, with the CAM debug information returned via some other
mechanism.
> +
> + /* Get the VP state from OPAL */
> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
> + if (rc)
> + return rc;
> +
> + /*
> + * Capture the backup of IPB register in the NVT structure and
> + * merge it in our KVM VP state.
> + */
> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
> +
> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
> + __func__,
> + vcpu->arch.xive_saved_state.nsr,
> + vcpu->arch.xive_saved_state.cppr,
> + vcpu->arch.xive_saved_state.ipb,
> + vcpu->arch.xive_saved_state.pipr,
> + vcpu->arch.xive_saved_state.w01,
> + (u32) vcpu->arch.xive_cam_word, opal_state);
Hrm.. except you don't seem to be using the last half of the timaval
field anyway.
> +
> + return 0;
> +}
> +
> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
> +
> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
> + val->xive_timaval[0], val->xive_timaval[1]);
> +
> + if (!kvmppc_xive_enabled(vcpu))
> + return -EPERM;
> +
> + if (!xc || !xive)
> + return -ENOENT;
> +
> + /* We can't update the state of a "pushed" VCPU */
> + if (WARN_ON(vcpu->arch.xive_pushed))
What prevents userspace from tripping this WARN_ON()?
> + return -EIO;
EBUSY might be more appropriate here.
> +
> + /*
> + * Restore the thread context registers. IPB and CPPR should
> + * be the only ones that matter.
> + */
> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
> +
> + /*
> + * There is no need to restore the XIVE internal state (IPB
> + * stored in the NVT) as the IPB register was merged in KVM VP
> + * state when captured.
> + */
> + return 0;
> +}
> +
> static int xive_native_debug_show(struct seq_file *m, void *private)
> {
> struct kvmppc_xive *xive = m->private;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index a26be635cff9..1b8957c50c53 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -EINVAL: Not initialized source number, invalid priority or
> invalid CPU number.
>
> +* VCPU state
> +
> + The XIVE IC maintains VP interrupt state in an internal structure
> + called the NVT. When a VP is not dispatched on a HW processor
> + thread, this structure can be updated by HW if the VP is the target
> + of an event notification.
> +
> + It is important for migration to capture the cached IPB from the NVT
> + as it synthesizes the priorities of the pending interrupts. We
> + capture a bit more to report debug information.
> +
> + KVM_REG_PPC_VP_STATE (4 * 64bits)
> + bits: | 63 .... 32 | 31 .... 0 |
> + values: | TIMA word0 | TIMA word1 |
> + bits: | 127 .......... 64 |
> + values: | VP CAM Line |
> + bits: | 255 .......... 128 |
> + values: | unused |
> +
> * Migration:
>
> Saving the state of a VM using the XIVE native exploitation mode
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-02-25 3:31 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:31 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 9623 bytes --]
On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
> At a VCPU level, the state of the thread interrupt management
> registers needs to be collected. These registers are cached under the
> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
> pulled from the HW thread. An OPAL call retrieves the backup of the
> IPB register in the underlying XIVE NVT structure and merges it in the
> KVM state.
>
> The structures of the interface between QEMU and KVM provisions some
> extra room (two u64) for further extensions if more state needs to be
> transferred back to QEMU.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
> arch/powerpc/include/uapi/asm/kvm.h | 2 +
> arch/powerpc/kvm/book3s.c | 24 +++++++
> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
> 5 files changed, 138 insertions(+)
>
> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> index 1e61877fe147..664c65051612 100644
> --- a/arch/powerpc/include/asm/kvm_ppc.h
> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
> u64 addr;
> u64 length;
> } vpaval;
> + u64 xive_timaval[4];
This is doubling the size of the userspace visible one_reg union. Is
that safe?
> };
>
> struct kvmppc_ops {
> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
> extern void kvmppc_xive_native_init_module(void);
> extern void kvmppc_xive_native_exit_module(void);
> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val);
> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val);
>
> #else
> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
> static inline void kvmppc_xive_native_init_module(void) { }
> static inline void kvmppc_xive_native_exit_module(void) { }
> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val)
> +{ return 0; }
> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> + union kvmppc_one_reg *val)
> +{ return -ENOENT; }
>
> #endif /* CONFIG_KVM_XIVE */
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index cd78ad1020fe..42d4ef93ec2d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
>
> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
> +
> /* Device control API: PPC-specific devices */
> #define KVM_DEV_MPIC_GRP_MISC 1
> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> index 96d43f091255..f85a9211f30c 100644
> --- a/arch/powerpc/kvm/book3s.c
> +++ b/arch/powerpc/kvm/book3s.c
> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
> break;
> #endif /* CONFIG_KVM_XICS */
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_REG_PPC_VP_STATE:
> + if (!vcpu->arch.xive_vcpu) {
> + r = -ENXIO;
> + break;
> + }
> + if (xive_enabled())
> + r = kvmppc_xive_native_get_vp(vcpu, val);
> + else
> + r = -ENXIO;
> + break;
> +#endif /* CONFIG_KVM_XIVE */
> case KVM_REG_PPC_FSCR:
> *val = get_reg_val(id, vcpu->arch.fscr);
> break;
> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
> break;
> #endif /* CONFIG_KVM_XICS */
> +#ifdef CONFIG_KVM_XIVE
> + case KVM_REG_PPC_VP_STATE:
> + if (!vcpu->arch.xive_vcpu) {
> + r = -ENXIO;
> + break;
> + }
> + if (xive_enabled())
> + r = kvmppc_xive_native_set_vp(vcpu, val);
> + else
> + r = -ENXIO;
> + break;
> +#endif /* CONFIG_KVM_XIVE */
> case KVM_REG_PPC_FSCR:
> vcpu->arch.fscr = set_reg_val(id, *val);
> break;
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 3debc876d5a0..132bff52d70a 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> return ret;
> }
>
> +/*
> + * Interrupt Pending Buffer (IPB) offset
> + */
> +#define TM_IPB_SHIFT 40
> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
> +
> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + u64 opal_state;
> + int rc;
> +
> + if (!kvmppc_xive_enabled(vcpu))
> + return -EPERM;
> +
> + if (!xc)
> + return -ENOENT;
> +
> + /* Thread context registers. We only care about IPB and CPPR */
> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
> +
> + /*
> + * Return the OS CAM line to print out the VP identifier in
> + * the QEMU monitor. This is not restored.
> + */
> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
I'm pretty dubious about this mixing of vital state information with
what's basically debug information. Doubly so since it requires
changing the ABI to increase the one_reg union's size.
Might be better to have this control only return the 0th and 2nd u64s
from the TIMA, with the CAM debug information returned via some other
mechanism.
> +
> + /* Get the VP state from OPAL */
> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
> + if (rc)
> + return rc;
> +
> + /*
> + * Capture the backup of IPB register in the NVT structure and
> + * merge it in our KVM VP state.
> + */
> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
> +
> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
> + __func__,
> + vcpu->arch.xive_saved_state.nsr,
> + vcpu->arch.xive_saved_state.cppr,
> + vcpu->arch.xive_saved_state.ipb,
> + vcpu->arch.xive_saved_state.pipr,
> + vcpu->arch.xive_saved_state.w01,
> + (u32) vcpu->arch.xive_cam_word, opal_state);
Hrm.. except you don't seem to be using the last half of the timaval
field anyway.
> +
> + return 0;
> +}
> +
> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> +{
> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
> +
> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
> + val->xive_timaval[0], val->xive_timaval[1]);
> +
> + if (!kvmppc_xive_enabled(vcpu))
> + return -EPERM;
> +
> + if (!xc || !xive)
> + return -ENOENT;
> +
> + /* We can't update the state of a "pushed" VCPU */
> + if (WARN_ON(vcpu->arch.xive_pushed))
What prevents userspace from tripping this WARN_ON()?
> + return -EIO;
EBUSY might be more appropriate here.
> +
> + /*
> + * Restore the thread context registers. IPB and CPPR should
> + * be the only ones that matter.
> + */
> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
> +
> + /*
> + * There is no need to restore the XIVE internal state (IPB
> + * stored in the NVT) as the IPB register was merged in KVM VP
> + * state when captured.
> + */
> + return 0;
> +}
> +
> static int xive_native_debug_show(struct seq_file *m, void *private)
> {
> struct kvmppc_xive *xive = m->private;
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index a26be635cff9..1b8957c50c53 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -EINVAL: Not initialized source number, invalid priority or
> invalid CPU number.
>
> +* VCPU state
> +
> + The XIVE IC maintains VP interrupt state in an internal structure
> + called the NVT. When a VP is not dispatched on a HW processor
> + thread, this structure can be updated by HW if the VP is the target
> + of an event notification.
> +
> + It is important for migration to capture the cached IPB from the NVT
> + as it synthesizes the priorities of the pending interrupts. We
> + capture a bit more to report debug information.
> +
> + KVM_REG_PPC_VP_STATE (4 * 64bits)
> + bits: | 63 .... 32 | 31 .... 0 |
> + values: | TIMA word0 | TIMA word1 |
> + bits: | 127 .......... 64 |
> + values: | VP CAM Line |
> + bits: | 255 .......... 128 |
> + values: | unused |
> +
> * Migration:
>
> Saving the state of a VM using the XIVE native exploitation mode
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
2019-02-25 3:31 ` David Gibson
@ 2019-03-13 13:19 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 13:19 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 4:31 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
>> At a VCPU level, the state of the thread interrupt management
>> registers needs to be collected. These registers are cached under the
>> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
>> pulled from the HW thread. An OPAL call retrieves the backup of the
>> IPB register in the underlying XIVE NVT structure and merges it in the
>> KVM state.
>>
>> The structures of the interface between QEMU and KVM provisions some
>> extra room (two u64) for further extensions if more state needs to be
>> transferred back to QEMU.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
>> arch/powerpc/include/uapi/asm/kvm.h | 2 +
>> arch/powerpc/kvm/book3s.c | 24 +++++++
>> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
>> 5 files changed, 138 insertions(+)
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index 1e61877fe147..664c65051612 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
>> u64 addr;
>> u64 length;
>> } vpaval;
>> + u64 xive_timaval[4];
>
> This is doubling the size of the userspace visible one_reg union. Is
> that safe?
'safe' as in compatibility on an older KVM which would still use the old
kvmppc_one_reg definition ?
It should be fine as KVM_REG_PPC_VP_STATE would not be handled. Am I wrong ?
>> };
>>
>> struct kvmppc_ops {
>> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
>> extern void kvmppc_xive_native_init_module(void);
>> extern void kvmppc_xive_native_exit_module(void);
>> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val);
>> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val);
>>
>> #else
>> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
>> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
>> static inline void kvmppc_xive_native_init_module(void) { }
>> static inline void kvmppc_xive_native_exit_module(void) { }
>> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val)
>> +{ return 0; }
>> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val)
>> +{ return -ENOENT; }
>>
>> #endif /* CONFIG_KVM_XIVE */
>>
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index cd78ad1020fe..42d4ef93ec2d 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
>> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
>> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
>>
>> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
>> +
>> /* Device control API: PPC-specific devices */
>> #define KVM_DEV_MPIC_GRP_MISC 1
>> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
>> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
>> index 96d43f091255..f85a9211f30c 100644
>> --- a/arch/powerpc/kvm/book3s.c
>> +++ b/arch/powerpc/kvm/book3s.c
>> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
>> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
>> break;
>> #endif /* CONFIG_KVM_XICS */
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_REG_PPC_VP_STATE:
>> + if (!vcpu->arch.xive_vcpu) {
>> + r = -ENXIO;
>> + break;
>> + }
>> + if (xive_enabled())
>> + r = kvmppc_xive_native_get_vp(vcpu, val);
>> + else
>> + r = -ENXIO;
>> + break;
>> +#endif /* CONFIG_KVM_XIVE */
>> case KVM_REG_PPC_FSCR:
>> *val = get_reg_val(id, vcpu->arch.fscr);
>> break;
>> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
>> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
>> break;
>> #endif /* CONFIG_KVM_XICS */
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_REG_PPC_VP_STATE:
>> + if (!vcpu->arch.xive_vcpu) {
>> + r = -ENXIO;
>> + break;
>> + }
>> + if (xive_enabled())
>> + r = kvmppc_xive_native_set_vp(vcpu, val);
>> + else
>> + r = -ENXIO;
>> + break;
>> +#endif /* CONFIG_KVM_XIVE */
>> case KVM_REG_PPC_FSCR:
>> vcpu->arch.fscr = set_reg_val(id, *val);
>> break;
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index 3debc876d5a0..132bff52d70a 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
>> return ret;
>> }
>>
>> +/*
>> + * Interrupt Pending Buffer (IPB) offset
>> + */
>> +#define TM_IPB_SHIFT 40
>> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
>> +
>> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + u64 opal_state;
>> + int rc;
>> +
>> + if (!kvmppc_xive_enabled(vcpu))
>> + return -EPERM;
>> +
>> + if (!xc)
>> + return -ENOENT;
>> +
>> + /* Thread context registers. We only care about IPB and CPPR */
>> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
>> +
>> + /*
>> + * Return the OS CAM line to print out the VP identifier in
>> + * the QEMU monitor. This is not restored.
>> + */
>> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
>
> I'm pretty dubious about this mixing of vital state information with
> what's basically debug information.
I think QEMU deserves to know about the OS CAM line value. I was even
thinking about adding the POOL CAM line value for future use (nested)
> Doubly so since it requires changing the ABI to increase
> the one_reg union's size.
OK. That's one argument.
> Might be better to have this control only return the 0th and 2nd u64s
> from the TIMA, with the CAM debug information returned via some other
> mechanism.
Like an extra reg : KVM_REG_PPC_VP_CAM ?
>> +
>> + /* Get the VP state from OPAL */
>> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
>> + if (rc)
>> + return rc;
>> +
>> + /*
>> + * Capture the backup of IPB register in the NVT structure and
>> + * merge it in our KVM VP state.
>> + */
>> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
>> +
>> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
>> + __func__,
>> + vcpu->arch.xive_saved_state.nsr,
>> + vcpu->arch.xive_saved_state.cppr,
>> + vcpu->arch.xive_saved_state.ipb,
>> + vcpu->arch.xive_saved_state.pipr,
>> + vcpu->arch.xive_saved_state.w01,
>> + (u32) vcpu->arch.xive_cam_word, opal_state);
>
> Hrm.. except you don't seem to be using the last half of the timaval
> field anyway.
Yes. The two u64 are extras. We can do without.
Would that be ok if I stored the w01 regs in the first u64, the CAM line(s)
in the second and remove the extra two u64 ?
>> +
>> + return 0;
>> +}
>> +
>> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
>> +
>> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
>> + val->xive_timaval[0], val->xive_timaval[1]);
>> +
>> + if (!kvmppc_xive_enabled(vcpu))
>> + return -EPERM;
>> +
>> + if (!xc || !xive)
>> + return -ENOENT;
>> +
>> + /* We can't update the state of a "pushed" VCPU */
>> + if (WARN_ON(vcpu->arch.xive_pushed))
>
> What prevents userspace from tripping this WARN_ON()?
if the vCPU is executing a vCPU ioctl, it means that it exited the guest
and that its interrupt context has been pulled out of XIVE.
>> + return -EIO;
>
> EBUSY might be more appropriate here.
OK.
Thanks,
C.
>
>> +
>> + /*
>> + * Restore the thread context registers. IPB and CPPR should
>> + * be the only ones that matter.
>> + */
>> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
>> +
>> + /*
>> + * There is no need to restore the XIVE internal state (IPB
>> + * stored in the NVT) as the IPB register was merged in KVM VP
>> + * state when captured.
>> + */
>> + return 0;
>> +}
>> +
>> static int xive_native_debug_show(struct seq_file *m, void *private)
>> {
>> struct kvmppc_xive *xive = m->private;
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index a26be635cff9..1b8957c50c53 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> -EINVAL: Not initialized source number, invalid priority or
>> invalid CPU number.
>>
>> +* VCPU state
>> +
>> + The XIVE IC maintains VP interrupt state in an internal structure
>> + called the NVT. When a VP is not dispatched on a HW processor
>> + thread, this structure can be updated by HW if the VP is the target
>> + of an event notification.
>> +
>> + It is important for migration to capture the cached IPB from the NVT
>> + as it synthesizes the priorities of the pending interrupts. We
>> + capture a bit more to report debug information.
>> +
>> + KVM_REG_PPC_VP_STATE (4 * 64bits)
>> + bits: | 63 .... 32 | 31 .... 0 |
>> + values: | TIMA word0 | TIMA word1 |
>> + bits: | 127 .......... 64 |
>> + values: | VP CAM Line |
>> + bits: | 255 .......... 128 |
>> + values: | unused |
>> +
>> * Migration:
>>
>> Saving the state of a VM using the XIVE native exploitation mode
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-03-13 13:19 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 13:19 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 4:31 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
>> At a VCPU level, the state of the thread interrupt management
>> registers needs to be collected. These registers are cached under the
>> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
>> pulled from the HW thread. An OPAL call retrieves the backup of the
>> IPB register in the underlying XIVE NVT structure and merges it in the
>> KVM state.
>>
>> The structures of the interface between QEMU and KVM provisions some
>> extra room (two u64) for further extensions if more state needs to be
>> transferred back to QEMU.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
>> arch/powerpc/include/uapi/asm/kvm.h | 2 +
>> arch/powerpc/kvm/book3s.c | 24 +++++++
>> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
>> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
>> 5 files changed, 138 insertions(+)
>>
>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>> index 1e61877fe147..664c65051612 100644
>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
>> u64 addr;
>> u64 length;
>> } vpaval;
>> + u64 xive_timaval[4];
>
> This is doubling the size of the userspace visible one_reg union. Is
> that safe?
'safe' as in compatibility on an older KVM which would still use the old
kvmppc_one_reg definition ?
It should be fine as KVM_REG_PPC_VP_STATE would not be handled. Am I wrong ?
>> };
>>
>> struct kvmppc_ops {
>> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
>> extern void kvmppc_xive_native_init_module(void);
>> extern void kvmppc_xive_native_exit_module(void);
>> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val);
>> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val);
>>
>> #else
>> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
>> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
>> static inline void kvmppc_xive_native_init_module(void) { }
>> static inline void kvmppc_xive_native_exit_module(void) { }
>> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val)
>> +{ return 0; }
>> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>> + union kvmppc_one_reg *val)
>> +{ return -ENOENT; }
>>
>> #endif /* CONFIG_KVM_XIVE */
>>
>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>> index cd78ad1020fe..42d4ef93ec2d 100644
>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
>> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
>> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
>>
>> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
>> +
>> /* Device control API: PPC-specific devices */
>> #define KVM_DEV_MPIC_GRP_MISC 1
>> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
>> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
>> index 96d43f091255..f85a9211f30c 100644
>> --- a/arch/powerpc/kvm/book3s.c
>> +++ b/arch/powerpc/kvm/book3s.c
>> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
>> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
>> break;
>> #endif /* CONFIG_KVM_XICS */
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_REG_PPC_VP_STATE:
>> + if (!vcpu->arch.xive_vcpu) {
>> + r = -ENXIO;
>> + break;
>> + }
>> + if (xive_enabled())
>> + r = kvmppc_xive_native_get_vp(vcpu, val);
>> + else
>> + r = -ENXIO;
>> + break;
>> +#endif /* CONFIG_KVM_XIVE */
>> case KVM_REG_PPC_FSCR:
>> *val = get_reg_val(id, vcpu->arch.fscr);
>> break;
>> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
>> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
>> break;
>> #endif /* CONFIG_KVM_XICS */
>> +#ifdef CONFIG_KVM_XIVE
>> + case KVM_REG_PPC_VP_STATE:
>> + if (!vcpu->arch.xive_vcpu) {
>> + r = -ENXIO;
>> + break;
>> + }
>> + if (xive_enabled())
>> + r = kvmppc_xive_native_set_vp(vcpu, val);
>> + else
>> + r = -ENXIO;
>> + break;
>> +#endif /* CONFIG_KVM_XIVE */
>> case KVM_REG_PPC_FSCR:
>> vcpu->arch.fscr = set_reg_val(id, *val);
>> break;
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index 3debc876d5a0..132bff52d70a 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
>> return ret;
>> }
>>
>> +/*
>> + * Interrupt Pending Buffer (IPB) offset
>> + */
>> +#define TM_IPB_SHIFT 40
>> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
>> +
>> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + u64 opal_state;
>> + int rc;
>> +
>> + if (!kvmppc_xive_enabled(vcpu))
>> + return -EPERM;
>> +
>> + if (!xc)
>> + return -ENOENT;
>> +
>> + /* Thread context registers. We only care about IPB and CPPR */
>> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
>> +
>> + /*
>> + * Return the OS CAM line to print out the VP identifier in
>> + * the QEMU monitor. This is not restored.
>> + */
>> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
>
> I'm pretty dubious about this mixing of vital state information with
> what's basically debug information.
I think QEMU deserves to know about the OS CAM line value. I was even
thinking about adding the POOL CAM line value for future use (nested)
> Doubly so since it requires changing the ABI to increase
> the one_reg union's size.
OK. That's one argument.
> Might be better to have this control only return the 0th and 2nd u64s
> from the TIMA, with the CAM debug information returned via some other
> mechanism.
Like an extra reg : KVM_REG_PPC_VP_CAM ?
>> +
>> + /* Get the VP state from OPAL */
>> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
>> + if (rc)
>> + return rc;
>> +
>> + /*
>> + * Capture the backup of IPB register in the NVT structure and
>> + * merge it in our KVM VP state.
>> + */
>> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
>> +
>> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
>> + __func__,
>> + vcpu->arch.xive_saved_state.nsr,
>> + vcpu->arch.xive_saved_state.cppr,
>> + vcpu->arch.xive_saved_state.ipb,
>> + vcpu->arch.xive_saved_state.pipr,
>> + vcpu->arch.xive_saved_state.w01,
>> + (u32) vcpu->arch.xive_cam_word, opal_state);
>
> Hrm.. except you don't seem to be using the last half of the timaval
> field anyway.
Yes. The two u64 are extras. We can do without.
Would that be ok if I stored the w01 regs in the first u64, the CAM line(s)
in the second and remove the extra two u64 ?
>> +
>> + return 0;
>> +}
>> +
>> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>> +{
>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
>> +
>> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
>> + val->xive_timaval[0], val->xive_timaval[1]);
>> +
>> + if (!kvmppc_xive_enabled(vcpu))
>> + return -EPERM;
>> +
>> + if (!xc || !xive)
>> + return -ENOENT;
>> +
>> + /* We can't update the state of a "pushed" VCPU */
>> + if (WARN_ON(vcpu->arch.xive_pushed))
>
> What prevents userspace from tripping this WARN_ON()?
if the vCPU is executing a vCPU ioctl, it means that it exited the guest
and that its interrupt context has been pulled out of XIVE.
>> + return -EIO;
>
> EBUSY might be more appropriate here.
OK.
Thanks,
C.
>
>> +
>> + /*
>> + * Restore the thread context registers. IPB and CPPR should
>> + * be the only ones that matter.
>> + */
>> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
>> +
>> + /*
>> + * There is no need to restore the XIVE internal state (IPB
>> + * stored in the NVT) as the IPB register was merged in KVM VP
>> + * state when captured.
>> + */
>> + return 0;
>> +}
>> +
>> static int xive_native_debug_show(struct seq_file *m, void *private)
>> {
>> struct kvmppc_xive *xive = m->private;
>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>> index a26be635cff9..1b8957c50c53 100644
>> --- a/Documentation/virtual/kvm/devices/xive.txt
>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>> -EINVAL: Not initialized source number, invalid priority or
>> invalid CPU number.
>>
>> +* VCPU state
>> +
>> + The XIVE IC maintains VP interrupt state in an internal structure
>> + called the NVT. When a VP is not dispatched on a HW processor
>> + thread, this structure can be updated by HW if the VP is the target
>> + of an event notification.
>> +
>> + It is important for migration to capture the cached IPB from the NVT
>> + as it synthesizes the priorities of the pending interrupts. We
>> + capture a bit more to report debug information.
>> +
>> + KVM_REG_PPC_VP_STATE (4 * 64bits)
>> + bits: | 63 .... 32 | 31 .... 0 |
>> + values: | TIMA word0 | TIMA word1 |
>> + bits: | 127 .......... 64 |
>> + values: | VP CAM Line |
>> + bits: | 255 .......... 128 |
>> + values: | unused |
>> +
>> * Migration:
>>
>> Saving the state of a VM using the XIVE native exploitation mode
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
2019-03-13 13:19 ` Cédric Le Goater
@ 2019-03-14 3:09 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 3:09 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 11922 bytes --]
On Wed, Mar 13, 2019 at 02:19:13PM +0100, Cédric Le Goater wrote:
> On 2/25/19 4:31 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
> >> At a VCPU level, the state of the thread interrupt management
> >> registers needs to be collected. These registers are cached under the
> >> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
> >> pulled from the HW thread. An OPAL call retrieves the backup of the
> >> IPB register in the underlying XIVE NVT structure and merges it in the
> >> KVM state.
> >>
> >> The structures of the interface between QEMU and KVM provisions some
> >> extra room (two u64) for further extensions if more state needs to be
> >> transferred back to QEMU.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
> >> arch/powerpc/include/uapi/asm/kvm.h | 2 +
> >> arch/powerpc/kvm/book3s.c | 24 +++++++
> >> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
> >> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
> >> 5 files changed, 138 insertions(+)
> >>
> >> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> >> index 1e61877fe147..664c65051612 100644
> >> --- a/arch/powerpc/include/asm/kvm_ppc.h
> >> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> >> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
> >> u64 addr;
> >> u64 length;
> >> } vpaval;
> >> + u64 xive_timaval[4];
> >
> > This is doubling the size of the userspace visible one_reg union. Is
> > that safe?
>
> 'safe' as in compatibility on an older KVM which would still use the old
> kvmppc_one_reg definition ?
I was more thinking of old qemu with a new kernel.
> It should be fine as KVM_REG_PPC_VP_STATE would not be handled. Am I
> wrong ?
Looks like it should be ok, because we only partially copy the
structure to/from userspace due to the one_reg_size() logic. If the
whole union was always copied, it would be hilariously unsafe.
>
> >> };
> >>
> >> struct kvmppc_ops {
> >> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
> >> extern void kvmppc_xive_native_init_module(void);
> >> extern void kvmppc_xive_native_exit_module(void);
> >> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val);
> >> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val);
> >>
> >> #else
> >> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> >> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
> >> static inline void kvmppc_xive_native_init_module(void) { }
> >> static inline void kvmppc_xive_native_exit_module(void) { }
> >> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val)
> >> +{ return 0; }
> >> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val)
> >> +{ return -ENOENT; }
> >>
> >> #endif /* CONFIG_KVM_XIVE */
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> >> index cd78ad1020fe..42d4ef93ec2d 100644
> >> --- a/arch/powerpc/include/uapi/asm/kvm.h
> >> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> >> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
> >> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
> >> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
> >>
> >> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
> >> +
> >> /* Device control API: PPC-specific devices */
> >> #define KVM_DEV_MPIC_GRP_MISC 1
> >> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
> >> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> >> index 96d43f091255..f85a9211f30c 100644
> >> --- a/arch/powerpc/kvm/book3s.c
> >> +++ b/arch/powerpc/kvm/book3s.c
> >> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
> >> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
> >> break;
> >> #endif /* CONFIG_KVM_XICS */
> >> +#ifdef CONFIG_KVM_XIVE
> >> + case KVM_REG_PPC_VP_STATE:
> >> + if (!vcpu->arch.xive_vcpu) {
> >> + r = -ENXIO;
> >> + break;
> >> + }
> >> + if (xive_enabled())
> >> + r = kvmppc_xive_native_get_vp(vcpu, val);
> >> + else
> >> + r = -ENXIO;
> >> + break;
> >> +#endif /* CONFIG_KVM_XIVE */
> >> case KVM_REG_PPC_FSCR:
> >> *val = get_reg_val(id, vcpu->arch.fscr);
> >> break;
> >> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
> >> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
> >> break;
> >> #endif /* CONFIG_KVM_XICS */
> >> +#ifdef CONFIG_KVM_XIVE
> >> + case KVM_REG_PPC_VP_STATE:
> >> + if (!vcpu->arch.xive_vcpu) {
> >> + r = -ENXIO;
> >> + break;
> >> + }
> >> + if (xive_enabled())
> >> + r = kvmppc_xive_native_set_vp(vcpu, val);
> >> + else
> >> + r = -ENXIO;
> >> + break;
> >> +#endif /* CONFIG_KVM_XIVE */
> >> case KVM_REG_PPC_FSCR:
> >> vcpu->arch.fscr = set_reg_val(id, *val);
> >> break;
> >> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> >> index 3debc876d5a0..132bff52d70a 100644
> >> --- a/arch/powerpc/kvm/book3s_xive_native.c
> >> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> >> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> >> return ret;
> >> }
> >>
> >> +/*
> >> + * Interrupt Pending Buffer (IPB) offset
> >> + */
> >> +#define TM_IPB_SHIFT 40
> >> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
> >> +
> >> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> >> +{
> >> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> + u64 opal_state;
> >> + int rc;
> >> +
> >> + if (!kvmppc_xive_enabled(vcpu))
> >> + return -EPERM;
> >> +
> >> + if (!xc)
> >> + return -ENOENT;
> >> +
> >> + /* Thread context registers. We only care about IPB and CPPR */
> >> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
> >> +
> >> + /*
> >> + * Return the OS CAM line to print out the VP identifier in
> >> + * the QEMU monitor. This is not restored.
> >> + */
> >> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
> >
> > I'm pretty dubious about this mixing of vital state information with
> > what's basically debug information.
>
> I think QEMU deserves to know about the OS CAM line value. I was even
> thinking about adding the POOL CAM line value for future use (nested)
>
> > Doubly so since it requires changing the ABI to increase
> > the one_reg union's size.
>
> OK. That's one argument.
>
> > Might be better to have this control only return the 0th and 2nd u64s
> > from the TIMA, with the CAM debug information returned via some other
> > mechanism.
>
> Like an extra reg : KVM_REG_PPC_VP_CAM ?
That would be the obvious choice, yes.
> >> +
> >> + /* Get the VP state from OPAL */
> >> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
> >> + if (rc)
> >> + return rc;
> >> +
> >> + /*
> >> + * Capture the backup of IPB register in the NVT structure and
> >> + * merge it in our KVM VP state.
> >> + */
> >> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
> >> +
> >> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
> >> + __func__,
> >> + vcpu->arch.xive_saved_state.nsr,
> >> + vcpu->arch.xive_saved_state.cppr,
> >> + vcpu->arch.xive_saved_state.ipb,
> >> + vcpu->arch.xive_saved_state.pipr,
> >> + vcpu->arch.xive_saved_state.w01,
> >> + (u32) vcpu->arch.xive_cam_word, opal_state);
> >
> > Hrm.. except you don't seem to be using the last half of the timaval
> > field anyway.
>
> Yes. The two u64 are extras. We can do without.
>
> Would that be ok if I stored the w01 regs in the first u64, the CAM line(s)
> in the second and remove the extra two u64 ?
I'd still prefer them in separate regs. They kind of belong to
different categories of information, and I can't think of any
particular reason you'd have to update or fetch them as a unit.
>
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> >> +{
> >> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
> >> +
> >> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
> >> + val->xive_timaval[0], val->xive_timaval[1]);
> >> +
> >> + if (!kvmppc_xive_enabled(vcpu))
> >> + return -EPERM;
> >> +
> >> + if (!xc || !xive)
> >> + return -ENOENT;
> >> +
> >> + /* We can't update the state of a "pushed" VCPU */
> >> + if (WARN_ON(vcpu->arch.xive_pushed))
> >
> > What prevents userspace from tripping this WARN_ON()?
>
> if the vCPU is executing a vCPU ioctl, it means that it exited the guest
> and that its interrupt context has been pulled out of XIVE.
But couldn't one user thread call the vcpu ioctl() while another is
inside the guest?
> >> + return -EIO;
> >
> > EBUSY might be more appropriate here.
>
> OK.
>
> Thanks,
>
> C.
>
> >
> >> +
> >> + /*
> >> + * Restore the thread context registers. IPB and CPPR should
> >> + * be the only ones that matter.
> >> + */
> >> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
> >> +
> >> + /*
> >> + * There is no need to restore the XIVE internal state (IPB
> >> + * stored in the NVT) as the IPB register was merged in KVM VP
> >> + * state when captured.
> >> + */
> >> + return 0;
> >> +}
> >> +
> >> static int xive_native_debug_show(struct seq_file *m, void *private)
> >> {
> >> struct kvmppc_xive *xive = m->private;
> >> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> >> index a26be635cff9..1b8957c50c53 100644
> >> --- a/Documentation/virtual/kvm/devices/xive.txt
> >> +++ b/Documentation/virtual/kvm/devices/xive.txt
> >> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> >> -EINVAL: Not initialized source number, invalid priority or
> >> invalid CPU number.
> >>
> >> +* VCPU state
> >> +
> >> + The XIVE IC maintains VP interrupt state in an internal structure
> >> + called the NVT. When a VP is not dispatched on a HW processor
> >> + thread, this structure can be updated by HW if the VP is the target
> >> + of an event notification.
> >> +
> >> + It is important for migration to capture the cached IPB from the NVT
> >> + as it synthesizes the priorities of the pending interrupts. We
> >> + capture a bit more to report debug information.
> >> +
> >> + KVM_REG_PPC_VP_STATE (4 * 64bits)
> >> + bits: | 63 .... 32 | 31 .... 0 |
> >> + values: | TIMA word0 | TIMA word1 |
> >> + bits: | 127 .......... 64 |
> >> + values: | VP CAM Line |
> >> + bits: | 255 .......... 128 |
> >> + values: | unused |
> >> +
> >> * Migration:
> >>
> >> Saving the state of a VM using the XIVE native exploitation mode
> >
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-03-14 3:09 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 3:09 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 11922 bytes --]
On Wed, Mar 13, 2019 at 02:19:13PM +0100, Cédric Le Goater wrote:
> On 2/25/19 4:31 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
> >> At a VCPU level, the state of the thread interrupt management
> >> registers needs to be collected. These registers are cached under the
> >> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
> >> pulled from the HW thread. An OPAL call retrieves the backup of the
> >> IPB register in the underlying XIVE NVT structure and merges it in the
> >> KVM state.
> >>
> >> The structures of the interface between QEMU and KVM provisions some
> >> extra room (two u64) for further extensions if more state needs to be
> >> transferred back to QEMU.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
> >> arch/powerpc/include/uapi/asm/kvm.h | 2 +
> >> arch/powerpc/kvm/book3s.c | 24 +++++++
> >> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
> >> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
> >> 5 files changed, 138 insertions(+)
> >>
> >> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
> >> index 1e61877fe147..664c65051612 100644
> >> --- a/arch/powerpc/include/asm/kvm_ppc.h
> >> +++ b/arch/powerpc/include/asm/kvm_ppc.h
> >> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
> >> u64 addr;
> >> u64 length;
> >> } vpaval;
> >> + u64 xive_timaval[4];
> >
> > This is doubling the size of the userspace visible one_reg union. Is
> > that safe?
>
> 'safe' as in compatibility on an older KVM which would still use the old
> kvmppc_one_reg definition ?
I was more thinking of old qemu with a new kernel.
> It should be fine as KVM_REG_PPC_VP_STATE would not be handled. Am I
> wrong ?
Looks like it should be ok, because we only partially copy the
structure to/from userspace due to the one_reg_size() logic. If the
whole union was always copied, it would be hilariously unsafe.
>
> >> };
> >>
> >> struct kvmppc_ops {
> >> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
> >> extern void kvmppc_xive_native_init_module(void);
> >> extern void kvmppc_xive_native_exit_module(void);
> >> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val);
> >> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val);
> >>
> >> #else
> >> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
> >> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> >> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
> >> static inline void kvmppc_xive_native_init_module(void) { }
> >> static inline void kvmppc_xive_native_exit_module(void) { }
> >> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val)
> >> +{ return 0; }
> >> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
> >> + union kvmppc_one_reg *val)
> >> +{ return -ENOENT; }
> >>
> >> #endif /* CONFIG_KVM_XIVE */
> >>
> >> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> >> index cd78ad1020fe..42d4ef93ec2d 100644
> >> --- a/arch/powerpc/include/uapi/asm/kvm.h
> >> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> >> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
> >> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
> >> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
> >>
> >> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
> >> +
> >> /* Device control API: PPC-specific devices */
> >> #define KVM_DEV_MPIC_GRP_MISC 1
> >> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
> >> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
> >> index 96d43f091255..f85a9211f30c 100644
> >> --- a/arch/powerpc/kvm/book3s.c
> >> +++ b/arch/powerpc/kvm/book3s.c
> >> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
> >> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
> >> break;
> >> #endif /* CONFIG_KVM_XICS */
> >> +#ifdef CONFIG_KVM_XIVE
> >> + case KVM_REG_PPC_VP_STATE:
> >> + if (!vcpu->arch.xive_vcpu) {
> >> + r = -ENXIO;
> >> + break;
> >> + }
> >> + if (xive_enabled())
> >> + r = kvmppc_xive_native_get_vp(vcpu, val);
> >> + else
> >> + r = -ENXIO;
> >> + break;
> >> +#endif /* CONFIG_KVM_XIVE */
> >> case KVM_REG_PPC_FSCR:
> >> *val = get_reg_val(id, vcpu->arch.fscr);
> >> break;
> >> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
> >> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
> >> break;
> >> #endif /* CONFIG_KVM_XICS */
> >> +#ifdef CONFIG_KVM_XIVE
> >> + case KVM_REG_PPC_VP_STATE:
> >> + if (!vcpu->arch.xive_vcpu) {
> >> + r = -ENXIO;
> >> + break;
> >> + }
> >> + if (xive_enabled())
> >> + r = kvmppc_xive_native_set_vp(vcpu, val);
> >> + else
> >> + r = -ENXIO;
> >> + break;
> >> +#endif /* CONFIG_KVM_XIVE */
> >> case KVM_REG_PPC_FSCR:
> >> vcpu->arch.fscr = set_reg_val(id, *val);
> >> break;
> >> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> >> index 3debc876d5a0..132bff52d70a 100644
> >> --- a/arch/powerpc/kvm/book3s_xive_native.c
> >> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> >> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> >> return ret;
> >> }
> >>
> >> +/*
> >> + * Interrupt Pending Buffer (IPB) offset
> >> + */
> >> +#define TM_IPB_SHIFT 40
> >> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
> >> +
> >> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> >> +{
> >> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> + u64 opal_state;
> >> + int rc;
> >> +
> >> + if (!kvmppc_xive_enabled(vcpu))
> >> + return -EPERM;
> >> +
> >> + if (!xc)
> >> + return -ENOENT;
> >> +
> >> + /* Thread context registers. We only care about IPB and CPPR */
> >> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
> >> +
> >> + /*
> >> + * Return the OS CAM line to print out the VP identifier in
> >> + * the QEMU monitor. This is not restored.
> >> + */
> >> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
> >
> > I'm pretty dubious about this mixing of vital state information with
> > what's basically debug information.
>
> I think QEMU deserves to know about the OS CAM line value. I was even
> thinking about adding the POOL CAM line value for future use (nested)
>
> > Doubly so since it requires changing the ABI to increase
> > the one_reg union's size.
>
> OK. That's one argument.
>
> > Might be better to have this control only return the 0th and 2nd u64s
> > from the TIMA, with the CAM debug information returned via some other
> > mechanism.
>
> Like an extra reg : KVM_REG_PPC_VP_CAM ?
That would be the obvious choice, yes.
> >> +
> >> + /* Get the VP state from OPAL */
> >> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
> >> + if (rc)
> >> + return rc;
> >> +
> >> + /*
> >> + * Capture the backup of IPB register in the NVT structure and
> >> + * merge it in our KVM VP state.
> >> + */
> >> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
> >> +
> >> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
> >> + __func__,
> >> + vcpu->arch.xive_saved_state.nsr,
> >> + vcpu->arch.xive_saved_state.cppr,
> >> + vcpu->arch.xive_saved_state.ipb,
> >> + vcpu->arch.xive_saved_state.pipr,
> >> + vcpu->arch.xive_saved_state.w01,
> >> + (u32) vcpu->arch.xive_cam_word, opal_state);
> >
> > Hrm.. except you don't seem to be using the last half of the timaval
> > field anyway.
>
> Yes. The two u64 are extras. We can do without.
>
> Would that be ok if I stored the w01 regs in the first u64, the CAM line(s)
> in the second and remove the extra two u64 ?
I'd still prefer them in separate regs. They kind of belong to
different categories of information, and I can't think of any
particular reason you'd have to update or fetch them as a unit.
>
> >> +
> >> + return 0;
> >> +}
> >> +
> >> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
> >> +{
> >> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
> >> +
> >> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
> >> + val->xive_timaval[0], val->xive_timaval[1]);
> >> +
> >> + if (!kvmppc_xive_enabled(vcpu))
> >> + return -EPERM;
> >> +
> >> + if (!xc || !xive)
> >> + return -ENOENT;
> >> +
> >> + /* We can't update the state of a "pushed" VCPU */
> >> + if (WARN_ON(vcpu->arch.xive_pushed))
> >
> > What prevents userspace from tripping this WARN_ON()?
>
> if the vCPU is executing a vCPU ioctl, it means that it exited the guest
> and that its interrupt context has been pulled out of XIVE.
But couldn't one user thread call the vcpu ioctl() while another is
inside the guest?
> >> + return -EIO;
> >
> > EBUSY might be more appropriate here.
>
> OK.
>
> Thanks,
>
> C.
>
> >
> >> +
> >> + /*
> >> + * Restore the thread context registers. IPB and CPPR should
> >> + * be the only ones that matter.
> >> + */
> >> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
> >> +
> >> + /*
> >> + * There is no need to restore the XIVE internal state (IPB
> >> + * stored in the NVT) as the IPB register was merged in KVM VP
> >> + * state when captured.
> >> + */
> >> + return 0;
> >> +}
> >> +
> >> static int xive_native_debug_show(struct seq_file *m, void *private)
> >> {
> >> struct kvmppc_xive *xive = m->private;
> >> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> >> index a26be635cff9..1b8957c50c53 100644
> >> --- a/Documentation/virtual/kvm/devices/xive.txt
> >> +++ b/Documentation/virtual/kvm/devices/xive.txt
> >> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> >> -EINVAL: Not initialized source number, invalid priority or
> >> invalid CPU number.
> >>
> >> +* VCPU state
> >> +
> >> + The XIVE IC maintains VP interrupt state in an internal structure
> >> + called the NVT. When a VP is not dispatched on a HW processor
> >> + thread, this structure can be updated by HW if the VP is the target
> >> + of an event notification.
> >> +
> >> + It is important for migration to capture the cached IPB from the NVT
> >> + as it synthesizes the priorities of the pending interrupts. We
> >> + capture a bit more to report debug information.
> >> +
> >> + KVM_REG_PPC_VP_STATE (4 * 64bits)
> >> + bits: | 63 .... 32 | 31 .... 0 |
> >> + values: | TIMA word0 | TIMA word1 |
> >> + bits: | 127 .......... 64 |
> >> + values: | VP CAM Line |
> >> + bits: | 255 .......... 128 |
> >> + values: | unused |
> >> +
> >> * Migration:
> >>
> >> Saving the state of a VM using the XIVE native exploitation mode
> >
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
2019-03-14 3:09 ` David Gibson
@ 2019-03-14 7:08 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-14 7:08 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 3/14/19 4:09 AM, David Gibson wrote:
> On Wed, Mar 13, 2019 at 02:19:13PM +0100, Cédric Le Goater wrote:
>> On 2/25/19 4:31 AM, David Gibson wrote:
>>> On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
>>>> At a VCPU level, the state of the thread interrupt management
>>>> registers needs to be collected. These registers are cached under the
>>>> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
>>>> pulled from the HW thread. An OPAL call retrieves the backup of the
>>>> IPB register in the underlying XIVE NVT structure and merges it in the
>>>> KVM state.
>>>>
>>>> The structures of the interface between QEMU and KVM provisions some
>>>> extra room (two u64) for further extensions if more state needs to be
>>>> transferred back to QEMU.
>>>>
>>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>>> ---
>>>> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
>>>> arch/powerpc/include/uapi/asm/kvm.h | 2 +
>>>> arch/powerpc/kvm/book3s.c | 24 +++++++
>>>> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
>>>> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
>>>> 5 files changed, 138 insertions(+)
>>>>
>>>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>>>> index 1e61877fe147..664c65051612 100644
>>>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>>>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>>>> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
>>>> u64 addr;
>>>> u64 length;
>>>> } vpaval;
>>>> + u64 xive_timaval[4];
>>>
>>> This is doubling the size of the userspace visible one_reg union. Is
>>> that safe?
>>
>> 'safe' as in compatibility on an older KVM which would still use the old
>> kvmppc_one_reg definition ?
>
> I was more thinking of old qemu with a new kernel.
>
>> It should be fine as KVM_REG_PPC_VP_STATE would not be handled. Am I
>> wrong ?
>
> Looks like it should be ok, because we only partially copy the
> structure to/from userspace due to the one_reg_size() logic. If the
> whole union was always copied, it would be hilariously unsafe.
>
>>
>>>> };
>>>>
>>>> struct kvmppc_ops {
>>>> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>>>> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
>>>> extern void kvmppc_xive_native_init_module(void);
>>>> extern void kvmppc_xive_native_exit_module(void);
>>>> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val);
>>>> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val);
>>>>
>>>> #else
>>>> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
>>>> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>>>> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
>>>> static inline void kvmppc_xive_native_init_module(void) { }
>>>> static inline void kvmppc_xive_native_exit_module(void) { }
>>>> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val)
>>>> +{ return 0; }
>>>> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val)
>>>> +{ return -ENOENT; }
>>>>
>>>> #endif /* CONFIG_KVM_XIVE */
>>>>
>>>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>>>> index cd78ad1020fe..42d4ef93ec2d 100644
>>>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>>>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>>>> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
>>>> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
>>>> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
>>>>
>>>> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
>>>> +
>>>> /* Device control API: PPC-specific devices */
>>>> #define KVM_DEV_MPIC_GRP_MISC 1
>>>> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
>>>> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
>>>> index 96d43f091255..f85a9211f30c 100644
>>>> --- a/arch/powerpc/kvm/book3s.c
>>>> +++ b/arch/powerpc/kvm/book3s.c
>>>> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
>>>> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
>>>> break;
>>>> #endif /* CONFIG_KVM_XICS */
>>>> +#ifdef CONFIG_KVM_XIVE
>>>> + case KVM_REG_PPC_VP_STATE:
>>>> + if (!vcpu->arch.xive_vcpu) {
>>>> + r = -ENXIO;
>>>> + break;
>>>> + }
>>>> + if (xive_enabled())
>>>> + r = kvmppc_xive_native_get_vp(vcpu, val);
>>>> + else
>>>> + r = -ENXIO;
>>>> + break;
>>>> +#endif /* CONFIG_KVM_XIVE */
>>>> case KVM_REG_PPC_FSCR:
>>>> *val = get_reg_val(id, vcpu->arch.fscr);
>>>> break;
>>>> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
>>>> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
>>>> break;
>>>> #endif /* CONFIG_KVM_XICS */
>>>> +#ifdef CONFIG_KVM_XIVE
>>>> + case KVM_REG_PPC_VP_STATE:
>>>> + if (!vcpu->arch.xive_vcpu) {
>>>> + r = -ENXIO;
>>>> + break;
>>>> + }
>>>> + if (xive_enabled())
>>>> + r = kvmppc_xive_native_set_vp(vcpu, val);
>>>> + else
>>>> + r = -ENXIO;
>>>> + break;
>>>> +#endif /* CONFIG_KVM_XIVE */
>>>> case KVM_REG_PPC_FSCR:
>>>> vcpu->arch.fscr = set_reg_val(id, *val);
>>>> break;
>>>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>>>> index 3debc876d5a0..132bff52d70a 100644
>>>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>>>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>>>> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
>>>> return ret;
>>>> }
>>>>
>>>> +/*
>>>> + * Interrupt Pending Buffer (IPB) offset
>>>> + */
>>>> +#define TM_IPB_SHIFT 40
>>>> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
>>>> +
>>>> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>>>> +{
>>>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>>> + u64 opal_state;
>>>> + int rc;
>>>> +
>>>> + if (!kvmppc_xive_enabled(vcpu))
>>>> + return -EPERM;
>>>> +
>>>> + if (!xc)
>>>> + return -ENOENT;
>>>> +
>>>> + /* Thread context registers. We only care about IPB and CPPR */
>>>> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
>>>> +
>>>> + /*
>>>> + * Return the OS CAM line to print out the VP identifier in
>>>> + * the QEMU monitor. This is not restored.
>>>> + */
>>>> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
>>>
>>> I'm pretty dubious about this mixing of vital state information with
>>> what's basically debug information.
>>
>> I think QEMU deserves to know about the OS CAM line value. I was even
>> thinking about adding the POOL CAM line value for future use (nested)
>>
>>> Doubly so since it requires changing the ABI to increase
>>> the one_reg union's size.
>>
>> OK. That's one argument.
>>
>>> Might be better to have this control only return the 0th and 2nd u64s
>>> from the TIMA, with the CAM debug information returned via some other
>>> mechanism.
>>
>> Like an extra reg : KVM_REG_PPC_VP_CAM ?
>
> That would be the obvious choice, yes.
OK. Let's keep that in mind but I think it is overkill. I would rather
have one reg per ring instead.
>>>> +
>>>> + /* Get the VP state from OPAL */
>>>> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
>>>> + if (rc)
>>>> + return rc;
>>>> +
>>>> + /*
>>>> + * Capture the backup of IPB register in the NVT structure and
>>>> + * merge it in our KVM VP state.
>>>> + */
>>>> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
>>>> +
>>>> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
>>>> + __func__,
>>>> + vcpu->arch.xive_saved_state.nsr,
>>>> + vcpu->arch.xive_saved_state.cppr,
>>>> + vcpu->arch.xive_saved_state.ipb,
>>>> + vcpu->arch.xive_saved_state.pipr,
>>>> + vcpu->arch.xive_saved_state.w01,
>>>> + (u32) vcpu->arch.xive_cam_word, opal_state);
>>>
>>> Hrm.. except you don't seem to be using the last half of the timaval
>>> field anyway.
>>
>> Yes. The two u64 are extras. We can do without.
>>
>> Would that be ok if I stored the w01 regs in the first u64, the CAM line(s)
>> in the second and remove the extra two u64 ?
>
> I'd still prefer them in separate regs. They kind of belong to
> different categories of information, and I can't think of any
> particular reason you'd have to update or fetch them as a unit.
Because they belong to the same thread interrupt context and the same
ring (OS) even if only the hypervisor can set the OS CAM line. The OS
can only set the CPPR. QEMU operates at the hypervisor level so it is
not violating any privilege level.
>>
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>>>> +{
>>>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>>> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
>>>> +
>>>> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
>>>> + val->xive_timaval[0], val->xive_timaval[1]);
>>>> +
>>>> + if (!kvmppc_xive_enabled(vcpu))
>>>> + return -EPERM;
>>>> +
>>>> + if (!xc || !xive)
>>>> + return -ENOENT;
>>>> +
>>>> + /* We can't update the state of a "pushed" VCPU */
>>>> + if (WARN_ON(vcpu->arch.xive_pushed))
>>>
>>> What prevents userspace from tripping this WARN_ON()?
>>
>> if the vCPU is executing a vCPU ioctl, it means that it exited the guest
>> and that its interrupt context has been pulled out of XIVE.
>
> But couldn't one user thread call the vcpu ioctl() while another is
> inside the guest?
Not while setting the VP state. The guest is not resumed.
Thanks,
C.
>
>>>> + return -EIO;
>>>
>>> EBUSY might be more appropriate here.
>>
>> OK.
>>
>> Thanks,
>>
>> C.
>>
>>>
>>>> +
>>>> + /*
>>>> + * Restore the thread context registers. IPB and CPPR should
>>>> + * be the only ones that matter.
>>>> + */
>>>> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
>>>> +
>>>> + /*
>>>> + * There is no need to restore the XIVE internal state (IPB
>>>> + * stored in the NVT) as the IPB register was merged in KVM VP
>>>> + * state when captured.
>>>> + */
>>>> + return 0;
>>>> +}
>>>> +
>>>> static int xive_native_debug_show(struct seq_file *m, void *private)
>>>> {
>>>> struct kvmppc_xive *xive = m->private;
>>>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>>>> index a26be635cff9..1b8957c50c53 100644
>>>> --- a/Documentation/virtual/kvm/devices/xive.txt
>>>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>>>> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>>>> -EINVAL: Not initialized source number, invalid priority or
>>>> invalid CPU number.
>>>>
>>>> +* VCPU state
>>>> +
>>>> + The XIVE IC maintains VP interrupt state in an internal structure
>>>> + called the NVT. When a VP is not dispatched on a HW processor
>>>> + thread, this structure can be updated by HW if the VP is the target
>>>> + of an event notification.
>>>> +
>>>> + It is important for migration to capture the cached IPB from the NVT
>>>> + as it synthesizes the priorities of the pending interrupts. We
>>>> + capture a bit more to report debug information.
>>>> +
>>>> + KVM_REG_PPC_VP_STATE (4 * 64bits)
>>>> + bits: | 63 .... 32 | 31 .... 0 |
>>>> + values: | TIMA word0 | TIMA word1 |
>>>> + bits: | 127 .......... 64 |
>>>> + values: | VP CAM Line |
>>>> + bits: | 255 .......... 128 |
>>>> + values: | unused |
>>>> +
>>>> * Migration:
>>>>
>>>> Saving the state of a VM using the XIVE native exploitation mode
>>>
>>
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-03-14 7:08 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-14 7:08 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 3/14/19 4:09 AM, David Gibson wrote:
> On Wed, Mar 13, 2019 at 02:19:13PM +0100, Cédric Le Goater wrote:
>> On 2/25/19 4:31 AM, David Gibson wrote:
>>> On Fri, Feb 22, 2019 at 12:28:34PM +0100, Cédric Le Goater wrote:
>>>> At a VCPU level, the state of the thread interrupt management
>>>> registers needs to be collected. These registers are cached under the
>>>> 'xive_saved_state.w01' field of the VCPU when the VPCU context is
>>>> pulled from the HW thread. An OPAL call retrieves the backup of the
>>>> IPB register in the underlying XIVE NVT structure and merges it in the
>>>> KVM state.
>>>>
>>>> The structures of the interface between QEMU and KVM provisions some
>>>> extra room (two u64) for further extensions if more state needs to be
>>>> transferred back to QEMU.
>>>>
>>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>>> ---
>>>> arch/powerpc/include/asm/kvm_ppc.h | 11 +++
>>>> arch/powerpc/include/uapi/asm/kvm.h | 2 +
>>>> arch/powerpc/kvm/book3s.c | 24 +++++++
>>>> arch/powerpc/kvm/book3s_xive_native.c | 82 ++++++++++++++++++++++
>>>> Documentation/virtual/kvm/devices/xive.txt | 19 +++++
>>>> 5 files changed, 138 insertions(+)
>>>>
>>>> diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
>>>> index 1e61877fe147..664c65051612 100644
>>>> --- a/arch/powerpc/include/asm/kvm_ppc.h
>>>> +++ b/arch/powerpc/include/asm/kvm_ppc.h
>>>> @@ -272,6 +272,7 @@ union kvmppc_one_reg {
>>>> u64 addr;
>>>> u64 length;
>>>> } vpaval;
>>>> + u64 xive_timaval[4];
>>>
>>> This is doubling the size of the userspace visible one_reg union. Is
>>> that safe?
>>
>> 'safe' as in compatibility on an older KVM which would still use the old
>> kvmppc_one_reg definition ?
>
> I was more thinking of old qemu with a new kernel.
>
>> It should be fine as KVM_REG_PPC_VP_STATE would not be handled. Am I
>> wrong ?
>
> Looks like it should be ok, because we only partially copy the
> structure to/from userspace due to the one_reg_size() logic. If the
> whole union was always copied, it would be hilariously unsafe.
>
>>
>>>> };
>>>>
>>>> struct kvmppc_ops {
>>>> @@ -604,6 +605,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>>>> extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
>>>> extern void kvmppc_xive_native_init_module(void);
>>>> extern void kvmppc_xive_native_exit_module(void);
>>>> +extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val);
>>>> +extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val);
>>>>
>>>> #else
>>>> static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
>>>> @@ -636,6 +641,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
>>>> static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
>>>> static inline void kvmppc_xive_native_init_module(void) { }
>>>> static inline void kvmppc_xive_native_exit_module(void) { }
>>>> +static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val)
>>>> +{ return 0; }
>>>> +static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
>>>> + union kvmppc_one_reg *val)
>>>> +{ return -ENOENT; }
>>>>
>>>> #endif /* CONFIG_KVM_XIVE */
>>>>
>>>> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
>>>> index cd78ad1020fe..42d4ef93ec2d 100644
>>>> --- a/arch/powerpc/include/uapi/asm/kvm.h
>>>> +++ b/arch/powerpc/include/uapi/asm/kvm.h
>>>> @@ -480,6 +480,8 @@ struct kvm_ppc_cpu_char {
>>>> #define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
>>>> #define KVM_REG_PPC_ICP_PPRI_MASK 0xff
>>>>
>>>> +#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U256 | 0x8d)
>>>> +
>>>> /* Device control API: PPC-specific devices */
>>>> #define KVM_DEV_MPIC_GRP_MISC 1
>>>> #define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
>>>> diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
>>>> index 96d43f091255..f85a9211f30c 100644
>>>> --- a/arch/powerpc/kvm/book3s.c
>>>> +++ b/arch/powerpc/kvm/book3s.c
>>>> @@ -641,6 +641,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
>>>> *val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
>>>> break;
>>>> #endif /* CONFIG_KVM_XICS */
>>>> +#ifdef CONFIG_KVM_XIVE
>>>> + case KVM_REG_PPC_VP_STATE:
>>>> + if (!vcpu->arch.xive_vcpu) {
>>>> + r = -ENXIO;
>>>> + break;
>>>> + }
>>>> + if (xive_enabled())
>>>> + r = kvmppc_xive_native_get_vp(vcpu, val);
>>>> + else
>>>> + r = -ENXIO;
>>>> + break;
>>>> +#endif /* CONFIG_KVM_XIVE */
>>>> case KVM_REG_PPC_FSCR:
>>>> *val = get_reg_val(id, vcpu->arch.fscr);
>>>> break;
>>>> @@ -714,6 +726,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
>>>> r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
>>>> break;
>>>> #endif /* CONFIG_KVM_XICS */
>>>> +#ifdef CONFIG_KVM_XIVE
>>>> + case KVM_REG_PPC_VP_STATE:
>>>> + if (!vcpu->arch.xive_vcpu) {
>>>> + r = -ENXIO;
>>>> + break;
>>>> + }
>>>> + if (xive_enabled())
>>>> + r = kvmppc_xive_native_set_vp(vcpu, val);
>>>> + else
>>>> + r = -ENXIO;
>>>> + break;
>>>> +#endif /* CONFIG_KVM_XIVE */
>>>> case KVM_REG_PPC_FSCR:
>>>> vcpu->arch.fscr = set_reg_val(id, *val);
>>>> break;
>>>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>>>> index 3debc876d5a0..132bff52d70a 100644
>>>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>>>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>>>> @@ -845,6 +845,88 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
>>>> return ret;
>>>> }
>>>>
>>>> +/*
>>>> + * Interrupt Pending Buffer (IPB) offset
>>>> + */
>>>> +#define TM_IPB_SHIFT 40
>>>> +#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
>>>> +
>>>> +int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>>>> +{
>>>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>>> + u64 opal_state;
>>>> + int rc;
>>>> +
>>>> + if (!kvmppc_xive_enabled(vcpu))
>>>> + return -EPERM;
>>>> +
>>>> + if (!xc)
>>>> + return -ENOENT;
>>>> +
>>>> + /* Thread context registers. We only care about IPB and CPPR */
>>>> + val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
>>>> +
>>>> + /*
>>>> + * Return the OS CAM line to print out the VP identifier in
>>>> + * the QEMU monitor. This is not restored.
>>>> + */
>>>> + val->xive_timaval[1] = vcpu->arch.xive_cam_word;
>>>
>>> I'm pretty dubious about this mixing of vital state information with
>>> what's basically debug information.
>>
>> I think QEMU deserves to know about the OS CAM line value. I was even
>> thinking about adding the POOL CAM line value for future use (nested)
>>
>>> Doubly so since it requires changing the ABI to increase
>>> the one_reg union's size.
>>
>> OK. That's one argument.
>>
>>> Might be better to have this control only return the 0th and 2nd u64s
>>> from the TIMA, with the CAM debug information returned via some other
>>> mechanism.
>>
>> Like an extra reg : KVM_REG_PPC_VP_CAM ?
>
> That would be the obvious choice, yes.
OK. Let's keep that in mind but I think it is overkill. I would rather
have one reg per ring instead.
>>>> +
>>>> + /* Get the VP state from OPAL */
>>>> + rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
>>>> + if (rc)
>>>> + return rc;
>>>> +
>>>> + /*
>>>> + * Capture the backup of IPB register in the NVT structure and
>>>> + * merge it in our KVM VP state.
>>>> + */
>>>> + val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
>>>> +
>>>> + pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
>>>> + __func__,
>>>> + vcpu->arch.xive_saved_state.nsr,
>>>> + vcpu->arch.xive_saved_state.cppr,
>>>> + vcpu->arch.xive_saved_state.ipb,
>>>> + vcpu->arch.xive_saved_state.pipr,
>>>> + vcpu->arch.xive_saved_state.w01,
>>>> + (u32) vcpu->arch.xive_cam_word, opal_state);
>>>
>>> Hrm.. except you don't seem to be using the last half of the timaval
>>> field anyway.
>>
>> Yes. The two u64 are extras. We can do without.
>>
>> Would that be ok if I stored the w01 regs in the first u64, the CAM line(s)
>> in the second and remove the extra two u64 ?
>
> I'd still prefer them in separate regs. They kind of belong to
> different categories of information, and I can't think of any
> particular reason you'd have to update or fetch them as a unit.
Because they belong to the same thread interrupt context and the same
ring (OS) even if only the hypervisor can set the OS CAM line. The OS
can only set the CPPR. QEMU operates at the hypervisor level so it is
not violating any privilege level.
>>
>>>> +
>>>> + return 0;
>>>> +}
>>>> +
>>>> +int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
>>>> +{
>>>> + struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>>> + struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
>>>> +
>>>> + pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
>>>> + val->xive_timaval[0], val->xive_timaval[1]);
>>>> +
>>>> + if (!kvmppc_xive_enabled(vcpu))
>>>> + return -EPERM;
>>>> +
>>>> + if (!xc || !xive)
>>>> + return -ENOENT;
>>>> +
>>>> + /* We can't update the state of a "pushed" VCPU */
>>>> + if (WARN_ON(vcpu->arch.xive_pushed))
>>>
>>> What prevents userspace from tripping this WARN_ON()?
>>
>> if the vCPU is executing a vCPU ioctl, it means that it exited the guest
>> and that its interrupt context has been pulled out of XIVE.
>
> But couldn't one user thread call the vcpu ioctl() while another is
> inside the guest?
Not while setting the VP state. The guest is not resumed.
Thanks,
C.
>
>>>> + return -EIO;
>>>
>>> EBUSY might be more appropriate here.
>>
>> OK.
>>
>> Thanks,
>>
>> C.
>>
>>>
>>>> +
>>>> + /*
>>>> + * Restore the thread context registers. IPB and CPPR should
>>>> + * be the only ones that matter.
>>>> + */
>>>> + vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
>>>> +
>>>> + /*
>>>> + * There is no need to restore the XIVE internal state (IPB
>>>> + * stored in the NVT) as the IPB register was merged in KVM VP
>>>> + * state when captured.
>>>> + */
>>>> + return 0;
>>>> +}
>>>> +
>>>> static int xive_native_debug_show(struct seq_file *m, void *private)
>>>> {
>>>> struct kvmppc_xive *xive = m->private;
>>>> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
>>>> index a26be635cff9..1b8957c50c53 100644
>>>> --- a/Documentation/virtual/kvm/devices/xive.txt
>>>> +++ b/Documentation/virtual/kvm/devices/xive.txt
>>>> @@ -102,6 +102,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
>>>> -EINVAL: Not initialized source number, invalid priority or
>>>> invalid CPU number.
>>>>
>>>> +* VCPU state
>>>> +
>>>> + The XIVE IC maintains VP interrupt state in an internal structure
>>>> + called the NVT. When a VP is not dispatched on a HW processor
>>>> + thread, this structure can be updated by HW if the VP is the target
>>>> + of an event notification.
>>>> +
>>>> + It is important for migration to capture the cached IPB from the NVT
>>>> + as it synthesizes the priorities of the pending interrupts. We
>>>> + capture a bit more to report debug information.
>>>> +
>>>> + KVM_REG_PPC_VP_STATE (4 * 64bits)
>>>> + bits: | 63 .... 32 | 31 .... 0 |
>>>> + values: | TIMA word0 | TIMA word1 |
>>>> + bits: | 127 .......... 64 |
>>>> + values: | VP CAM Line |
>>>> + bits: | 255 .......... 128 |
>>>> + values: | unused |
>>>> +
>>>> * Migration:
>>>>
>>>> Saving the state of a VM using the XIVE native exploitation mode
>>>
>>
>
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Some KVM devices will want to handle special mappings related to the
underlying HW. For instance, the XIVE interrupt controller of the
POWER9 processor has MMIO pages for thread interrupt management and
for interrupt source control that need to be exposed to the guest when
the OS has the required support.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
include/linux/kvm_host.h | 1 +
virt/kvm/kvm_main.c | 11 +++++++++++
2 files changed, 12 insertions(+)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index c38cc5eb7e73..cbf81487b69f 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1223,6 +1223,7 @@ struct kvm_device_ops {
int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
+ int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 585845203db8..84717d8cb5e4 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2878,6 +2878,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
}
#endif
+static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct kvm_device *dev = filp->private_data;
+
+ if (dev->ops->mmap)
+ return dev->ops->mmap(dev, vma);
+
+ return -ENODEV;
+}
+
static int kvm_device_ioctl_attr(struct kvm_device *dev,
int (*accessor)(struct kvm_device *dev,
struct kvm_device_attr *attr),
@@ -2927,6 +2937,7 @@ static const struct file_operations kvm_device_fops = {
.unlocked_ioctl = kvm_device_ioctl,
.release = kvm_device_release,
KVM_COMPAT(kvm_device_ioctl),
+ .mmap = kvm_device_mmap,
};
struct kvm_device *kvm_device_from_filp(struct file *filp)
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Some KVM devices will want to handle special mappings related to the
underlying HW. For instance, the XIVE interrupt controller of the
POWER9 processor has MMIO pages for thread interrupt management and
for interrupt source control that need to be exposed to the guest when
the OS has the required support.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
include/linux/kvm_host.h | 1 +
virt/kvm/kvm_main.c | 11 +++++++++++
2 files changed, 12 insertions(+)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index c38cc5eb7e73..cbf81487b69f 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1223,6 +1223,7 @@ struct kvm_device_ops {
int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
+ int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 585845203db8..84717d8cb5e4 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2878,6 +2878,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
}
#endif
+static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct kvm_device *dev = filp->private_data;
+
+ if (dev->ops->mmap)
+ return dev->ops->mmap(dev, vma);
+
+ return -ENODEV;
+}
+
static int kvm_device_ioctl_attr(struct kvm_device *dev,
int (*accessor)(struct kvm_device *dev,
struct kvm_device_attr *attr),
@@ -2927,6 +2937,7 @@ static const struct file_operations kvm_device_fops = {
.unlocked_ioctl = kvm_device_ioctl,
.release = kvm_device_release,
KVM_COMPAT(kvm_device_ioctl),
+ .mmap = kvm_device_mmap,
};
struct kvm_device *kvm_device_from_filp(struct file *filp)
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 3:33 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:33 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2451 bytes --]
On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
> Some KVM devices will want to handle special mappings related to the
> underlying HW. For instance, the XIVE interrupt controller of the
> POWER9 processor has MMIO pages for thread interrupt management and
> for interrupt source control that need to be exposed to the guest when
> the OS has the required support.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Ah, when I suggested mmap() on the base device fd, I hadn't realized
there wasn't a facility for that yet.
Have you discussed this with Paolo? We'll need some core KVM buy in
to merge this.
> ---
> include/linux/kvm_host.h | 1 +
> virt/kvm/kvm_main.c | 11 +++++++++++
> 2 files changed, 12 insertions(+)
>
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index c38cc5eb7e73..cbf81487b69f 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1223,6 +1223,7 @@ struct kvm_device_ops {
> int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
> long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
> unsigned long arg);
> + int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
> };
>
> void kvm_device_get(struct kvm_device *dev);
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 585845203db8..84717d8cb5e4 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2878,6 +2878,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
> }
> #endif
>
> +static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
> +{
> + struct kvm_device *dev = filp->private_data;
> +
> + if (dev->ops->mmap)
> + return dev->ops->mmap(dev, vma);
> +
> + return -ENODEV;
> +}
> +
> static int kvm_device_ioctl_attr(struct kvm_device *dev,
> int (*accessor)(struct kvm_device *dev,
> struct kvm_device_attr *attr),
> @@ -2927,6 +2937,7 @@ static const struct file_operations kvm_device_fops = {
> .unlocked_ioctl = kvm_device_ioctl,
> .release = kvm_device_release,
> KVM_COMPAT(kvm_device_ioctl),
> + .mmap = kvm_device_mmap,
> };
>
> struct kvm_device *kvm_device_from_filp(struct file *filp)
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-02-25 3:33 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:33 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2451 bytes --]
On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
> Some KVM devices will want to handle special mappings related to the
> underlying HW. For instance, the XIVE interrupt controller of the
> POWER9 processor has MMIO pages for thread interrupt management and
> for interrupt source control that need to be exposed to the guest when
> the OS has the required support.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Ah, when I suggested mmap() on the base device fd, I hadn't realized
there wasn't a facility for that yet.
Have you discussed this with Paolo? We'll need some core KVM buy in
to merge this.
> ---
> include/linux/kvm_host.h | 1 +
> virt/kvm/kvm_main.c | 11 +++++++++++
> 2 files changed, 12 insertions(+)
>
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index c38cc5eb7e73..cbf81487b69f 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1223,6 +1223,7 @@ struct kvm_device_ops {
> int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
> long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
> unsigned long arg);
> + int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
> };
>
> void kvm_device_get(struct kvm_device *dev);
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 585845203db8..84717d8cb5e4 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2878,6 +2878,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
> }
> #endif
>
> +static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
> +{
> + struct kvm_device *dev = filp->private_data;
> +
> + if (dev->ops->mmap)
> + return dev->ops->mmap(dev, vma);
> +
> + return -ENODEV;
> +}
> +
> static int kvm_device_ioctl_attr(struct kvm_device *dev,
> int (*accessor)(struct kvm_device *dev,
> struct kvm_device_attr *attr),
> @@ -2927,6 +2937,7 @@ static const struct file_operations kvm_device_fops = {
> .unlocked_ioctl = kvm_device_ioctl,
> .release = kvm_device_release,
> KVM_COMPAT(kvm_device_ioctl),
> + .mmap = kvm_device_mmap,
> };
>
> struct kvm_device *kvm_device_from_filp(struct file *filp)
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
2019-02-25 3:33 ` David Gibson
@ 2019-02-25 10:57 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-25 10:57 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, Paolo Bonzini, linuxppc-dev
Hello Paolo,
On 2/25/19 4:33 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
>> Some KVM devices will want to handle special mappings related to the
>> underlying HW. For instance, the XIVE interrupt controller of the
>> POWER9 processor has MMIO pages for thread interrupt management and
>> for interrupt source control that need to be exposed to the guest when
>> the OS has the required support.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>
> Ah, when I suggested mmap() on the base device fd, I hadn't realized
> there wasn't a facility for that yet.
>
> Have you discussed this with Paolo?
Not yet.
> We'll need some core KVM buy in to merge this.
Here is an extension of the KVM device to allow special mappings.
Something we would need for the support of the POWER9 XIVE interrupt
controller.
There are two MMIOs we need to expose to the guest :
1. HW MMIO controlling of the interrupt presenter registers (TIMA)
2. HW MMIO of the interrupt sources for interrupt management (ESB)
The TIMA could have been exposed with a page offset in the vCPU mapping
but as it only makes sense when the XIVE interrupt mode is active, we
chose to use directly the KVM device fd for that. Is that ok ?
An alternate solution is to use a device ioctl to allocate an anon fd
and do the mapping, but that seems like extra fuss for the same result.
Thanks,
C.
>> ---
>> include/linux/kvm_host.h | 1 +
>> virt/kvm/kvm_main.c | 11 +++++++++++
>> 2 files changed, 12 insertions(+)
>>
>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>> index c38cc5eb7e73..cbf81487b69f 100644
>> --- a/include/linux/kvm_host.h
>> +++ b/include/linux/kvm_host.h
>> @@ -1223,6 +1223,7 @@ struct kvm_device_ops {
>> int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
>> long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
>> unsigned long arg);
>> + int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
>> };
>>
>> void kvm_device_get(struct kvm_device *dev);
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 585845203db8..84717d8cb5e4 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -2878,6 +2878,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
>> }
>> #endif
>>
>> +static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
>> +{
>> + struct kvm_device *dev = filp->private_data;
>> +
>> + if (dev->ops->mmap)
>> + return dev->ops->mmap(dev, vma);
>> +
>> + return -ENODEV;
>> +}
>> +
>> static int kvm_device_ioctl_attr(struct kvm_device *dev,
>> int (*accessor)(struct kvm_device *dev,
>> struct kvm_device_attr *attr),
>> @@ -2927,6 +2937,7 @@ static const struct file_operations kvm_device_fops = {
>> .unlocked_ioctl = kvm_device_ioctl,
>> .release = kvm_device_release,
>> KVM_COMPAT(kvm_device_ioctl),
>> + .mmap = kvm_device_mmap,
>> };
>>
>> struct kvm_device *kvm_device_from_filp(struct file *filp)
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-02-25 10:57 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-25 10:57 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, Paolo Bonzini, linuxppc-dev
Hello Paolo,
On 2/25/19 4:33 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
>> Some KVM devices will want to handle special mappings related to the
>> underlying HW. For instance, the XIVE interrupt controller of the
>> POWER9 processor has MMIO pages for thread interrupt management and
>> for interrupt source control that need to be exposed to the guest when
>> the OS has the required support.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>
> Ah, when I suggested mmap() on the base device fd, I hadn't realized
> there wasn't a facility for that yet.
>
> Have you discussed this with Paolo?
Not yet.
> We'll need some core KVM buy in to merge this.
Here is an extension of the KVM device to allow special mappings.
Something we would need for the support of the POWER9 XIVE interrupt
controller.
There are two MMIOs we need to expose to the guest :
1. HW MMIO controlling of the interrupt presenter registers (TIMA)
2. HW MMIO of the interrupt sources for interrupt management (ESB)
The TIMA could have been exposed with a page offset in the vCPU mapping
but as it only makes sense when the XIVE interrupt mode is active, we
chose to use directly the KVM device fd for that. Is that ok ?
An alternate solution is to use a device ioctl to allocate an anon fd
and do the mapping, but that seems like extra fuss for the same result.
Thanks,
C.
>> ---
>> include/linux/kvm_host.h | 1 +
>> virt/kvm/kvm_main.c | 11 +++++++++++
>> 2 files changed, 12 insertions(+)
>>
>> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
>> index c38cc5eb7e73..cbf81487b69f 100644
>> --- a/include/linux/kvm_host.h
>> +++ b/include/linux/kvm_host.h
>> @@ -1223,6 +1223,7 @@ struct kvm_device_ops {
>> int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
>> long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
>> unsigned long arg);
>> + int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
>> };
>>
>> void kvm_device_get(struct kvm_device *dev);
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 585845203db8..84717d8cb5e4 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -2878,6 +2878,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
>> }
>> #endif
>>
>> +static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
>> +{
>> + struct kvm_device *dev = filp->private_data;
>> +
>> + if (dev->ops->mmap)
>> + return dev->ops->mmap(dev, vma);
>> +
>> + return -ENODEV;
>> +}
>> +
>> static int kvm_device_ioctl_attr(struct kvm_device *dev,
>> int (*accessor)(struct kvm_device *dev,
>> struct kvm_device_attr *attr),
>> @@ -2927,6 +2937,7 @@ static const struct file_operations kvm_device_fops = {
>> .unlocked_ioctl = kvm_device_ioctl,
>> .release = kvm_device_release,
>> KVM_COMPAT(kvm_device_ioctl),
>> + .mmap = kvm_device_mmap,
>> };
>>
>> struct kvm_device *kvm_device_from_filp(struct file *filp)
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
2019-02-25 10:57 ` Cédric Le Goater
@ 2019-02-26 12:52 ` Paolo Bonzini
-1 siblings, 0 replies; 142+ messages in thread
From: Paolo Bonzini @ 2019-02-26 12:52 UTC (permalink / raw)
To: Cédric Le Goater, David Gibson
Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 25/02/19 11:57, Cédric Le Goater wrote:
> Hello Paolo,
>
> On 2/25/19 4:33 AM, David Gibson wrote:
>> On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
>>> Some KVM devices will want to handle special mappings related to the
>>> underlying HW. For instance, the XIVE interrupt controller of the
>>> POWER9 processor has MMIO pages for thread interrupt management and
>>> for interrupt source control that need to be exposed to the guest when
>>> the OS has the required support.
>>>
>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>
>> Ah, when I suggested mmap() on the base device fd, I hadn't realized
>> there wasn't a facility for that yet.
>>
>> Have you discussed this with Paolo?
>
> Not yet.
>
>> We'll need some core KVM buy in to merge this.
>
> Here is an extension of the KVM device to allow special mappings.
> Something we would need for the support of the POWER9 XIVE interrupt
> controller.
>
> There are two MMIOs we need to expose to the guest :
>
> 1. HW MMIO controlling of the interrupt presenter registers (TIMA)
> 2. HW MMIO of the interrupt sources for interrupt management (ESB)
>
> The TIMA could have been exposed with a page offset in the vCPU mapping
> but as it only makes sense when the XIVE interrupt mode is active, we
> chose to use directly the KVM device fd for that. Is that ok ?
>
> An alternate solution is to use a device ioctl to allocate an anon fd
> and do the mapping, but that seems like extra fuss for the same result.
It's okay, it's a natural extension to dev_ops - but thanks for asking
anyway. :)
Paolo
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-02-26 12:52 ` Paolo Bonzini
0 siblings, 0 replies; 142+ messages in thread
From: Paolo Bonzini @ 2019-02-26 12:52 UTC (permalink / raw)
To: Cédric Le Goater, David Gibson
Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 25/02/19 11:57, Cédric Le Goater wrote:
> Hello Paolo,
>
> On 2/25/19 4:33 AM, David Gibson wrote:
>> On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
>>> Some KVM devices will want to handle special mappings related to the
>>> underlying HW. For instance, the XIVE interrupt controller of the
>>> POWER9 processor has MMIO pages for thread interrupt management and
>>> for interrupt source control that need to be exposed to the guest when
>>> the OS has the required support.
>>>
>>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>>
>> Ah, when I suggested mmap() on the base device fd, I hadn't realized
>> there wasn't a facility for that yet.
>>
>> Have you discussed this with Paolo?
>
> Not yet.
>
>> We'll need some core KVM buy in to merge this.
>
> Here is an extension of the KVM device to allow special mappings.
> Something we would need for the support of the POWER9 XIVE interrupt
> controller.
>
> There are two MMIOs we need to expose to the guest :
>
> 1. HW MMIO controlling of the interrupt presenter registers (TIMA)
> 2. HW MMIO of the interrupt sources for interrupt management (ESB)
>
> The TIMA could have been exposed with a page offset in the vCPU mapping
> but as it only makes sense when the XIVE interrupt mode is active, we
> chose to use directly the KVM device fd for that. Is that ok ?
>
> An alternate solution is to use a device ioctl to allocate an anon fd
> and do the mapping, but that seems like extra fuss for the same result.
It's okay, it's a natural extension to dev_ops - but thanks for asking
anyway. :)
Paolo
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
2019-02-26 12:52 ` Paolo Bonzini
@ 2019-02-26 23:22 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-26 23:22 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, kvm-ppc, Paul Mackerras, Cédric Le Goater, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]
On Tue, Feb 26, 2019 at 01:52:39PM +0100, Paolo Bonzini wrote:
> On 25/02/19 11:57, Cédric Le Goater wrote:
> > Hello Paolo,
> >
> > On 2/25/19 4:33 AM, David Gibson wrote:
> >> On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
> >>> Some KVM devices will want to handle special mappings related to the
> >>> underlying HW. For instance, the XIVE interrupt controller of the
> >>> POWER9 processor has MMIO pages for thread interrupt management and
> >>> for interrupt source control that need to be exposed to the guest when
> >>> the OS has the required support.
> >>>
> >>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >>
> >> Ah, when I suggested mmap() on the base device fd, I hadn't realized
> >> there wasn't a facility for that yet.
> >>
> >> Have you discussed this with Paolo?
> >
> > Not yet.
> >
> >> We'll need some core KVM buy in to merge this.
> >
> > Here is an extension of the KVM device to allow special mappings.
> > Something we would need for the support of the POWER9 XIVE interrupt
> > controller.
> >
> > There are two MMIOs we need to expose to the guest :
> >
> > 1. HW MMIO controlling of the interrupt presenter registers (TIMA)
> > 2. HW MMIO of the interrupt sources for interrupt management (ESB)
> >
> > The TIMA could have been exposed with a page offset in the vCPU mapping
> > but as it only makes sense when the XIVE interrupt mode is active, we
> > chose to use directly the KVM device fd for that. Is that ok ?
> >
> > An alternate solution is to use a device ioctl to allocate an anon fd
> > and do the mapping, but that seems like extra fuss for the same result.
>
> It's okay, it's a natural extension to dev_ops - but thanks for asking
> anyway. :)
Ok, cool.
Given that, do you want to merge directly - since this looks sound
enough, even though the rest of the series needs some polish? Or
would you prefer it to come in via Paulus' tree?
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-02-26 23:22 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-26 23:22 UTC (permalink / raw)
To: Paolo Bonzini
Cc: kvm, kvm-ppc, Paul Mackerras, Cédric Le Goater, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2169 bytes --]
On Tue, Feb 26, 2019 at 01:52:39PM +0100, Paolo Bonzini wrote:
> On 25/02/19 11:57, Cédric Le Goater wrote:
> > Hello Paolo,
> >
> > On 2/25/19 4:33 AM, David Gibson wrote:
> >> On Fri, Feb 22, 2019 at 12:28:35PM +0100, Cédric Le Goater wrote:
> >>> Some KVM devices will want to handle special mappings related to the
> >>> underlying HW. For instance, the XIVE interrupt controller of the
> >>> POWER9 processor has MMIO pages for thread interrupt management and
> >>> for interrupt source control that need to be exposed to the guest when
> >>> the OS has the required support.
> >>>
> >>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >>
> >> Ah, when I suggested mmap() on the base device fd, I hadn't realized
> >> there wasn't a facility for that yet.
> >>
> >> Have you discussed this with Paolo?
> >
> > Not yet.
> >
> >> We'll need some core KVM buy in to merge this.
> >
> > Here is an extension of the KVM device to allow special mappings.
> > Something we would need for the support of the POWER9 XIVE interrupt
> > controller.
> >
> > There are two MMIOs we need to expose to the guest :
> >
> > 1. HW MMIO controlling of the interrupt presenter registers (TIMA)
> > 2. HW MMIO of the interrupt sources for interrupt management (ESB)
> >
> > The TIMA could have been exposed with a page offset in the vCPU mapping
> > but as it only makes sense when the XIVE interrupt mode is active, we
> > chose to use directly the KVM device fd for that. Is that ok ?
> >
> > An alternate solution is to use a device ioctl to allocate an anon fd
> > and do the mapping, but that seems like extra fuss for the same result.
>
> It's okay, it's a natural extension to dev_ops - but thanks for asking
> anyway. :)
Ok, cool.
Given that, do you want to merge directly - since this looks sound
enough, even though the rest of the series needs some polish? Or
would you prefer it to come in via Paulus' tree?
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Each thread has an associated Thread Interrupt Management context
composed of a set of registers. These registers let the thread handle
priority management and interrupt acknowledgment. The most important
are :
- Interrupt Pending Buffer (IPB)
- Current Processor Priority (CPPR)
- Notification Source Register (NSR)
They are exposed to software in four different pages each proposing a
view with a different privilege. The first page is for the physical
thread context and the second for the hypervisor. Only the third
(operating system) and the fourth (user level) are exposed the guest.
A custom VM fault handler will populate the VMA with the appropriate
pages, which should only be the OS page for now.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/xive.h | 1 +
arch/powerpc/include/uapi/asm/kvm.h | 2 ++
arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 11 ++++++
Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
5 files changed, 76 insertions(+)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 46891f321606..eb6d302082da 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -23,6 +23,7 @@
* same offset regardless of where the code is executing
*/
extern void __iomem *xive_tima;
+extern unsigned long xive_tima_os;
/*
* Offset in the TM area of our current execution level (provided by
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 42d4ef93ec2d..be9b255e061d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
+#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 132bff52d70a..c6ac818a13b2 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -176,6 +176,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int xive_native_tima_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+
+ switch (vmf->pgoff - vma->vm_pgoff) {
+ case 0: /* HW - forbid access */
+ case 1: /* HV - forbid access */
+ return VM_FAULT_SIGBUS;
+ case 2: /* OS */
+ vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+ case 3: /* USER - TODO */
+ default:
+ return VM_FAULT_SIGBUS;
+ }
+}
+
+static const struct vm_operations_struct xive_native_tima_vmops = {
+ .fault = xive_native_tima_fault,
+};
+
+static int kvmppc_xive_native_mmap(struct kvm_device *dev,
+ struct vm_area_struct *vma)
+{
+ /* We only allow mappings at fixed offset for now */
+ if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
+ if (vma_pages(vma) > 4)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_tima_vmops;
+ } else {
+ return -EINVAL;
+ }
+
+ vma->vm_flags |= VM_IO | VM_PFNMAP;
+ vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+ return 0;
+}
+
static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
u64 addr)
{
@@ -1005,6 +1043,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
.has_attr = kvmppc_xive_native_has_attr,
+ .mmap = kvmppc_xive_native_mmap,
};
void kvmppc_xive_native_init_module(void)
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 0c037e933e55..7782201e5fe8 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
}
EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
+unsigned long xive_tima_os;
+EXPORT_SYMBOL_GPL(xive_tima_os);
+
bool __init xive_native_init(void)
{
struct device_node *np;
@@ -573,6 +576,14 @@ bool __init xive_native_init(void)
for_each_possible_cpu(cpu)
kvmppc_set_xive_tima(cpu, r.start, tima);
+ /* Resource 2 is OS window */
+ if (of_address_to_resource(np, 2, &r)) {
+ pr_err("Failed to get thread mgmnt area resource\n");
+ return false;
+ }
+
+ xive_tima_os = r.start;
+
/* Grab size of provisionning pages */
xive_parse_provisioning(np);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 1b8957c50c53..4d6b41609fd9 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
XIVE native exploitation interrupt mode. If not, it should run using
the legacy interrupt mode, referred as XICS (POWER7/8).
+* Device Mappings
+
+ The KVM device exposes different MMIO ranges of the XIVE HW which
+ are required for interrupt management. These are exposed to the
+ guest in VMAs populated with a custom VM fault handler.
+
+ 1. Thread Interrupt Management Area (TIMA)
+
+ Each thread has an associated Thread Interrupt Management context
+ composed of a set of registers. These registers let the thread
+ handle priority management and interrupt acknowledgment. The most
+ important are :
+
+ - Interrupt Pending Buffer (IPB)
+ - Current Processor Priority (CPPR)
+ - Notification Source Register (NSR)
+
+ They are exposed to software in four different pages each proposing
+ a view with a different privilege. The first page is for the
+ physical thread context and the second for the hypervisor. Only the
+ third (operating system) and the fourth (user level) are exposed the
+ guest.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Each thread has an associated Thread Interrupt Management context
composed of a set of registers. These registers let the thread handle
priority management and interrupt acknowledgment. The most important
are :
- Interrupt Pending Buffer (IPB)
- Current Processor Priority (CPPR)
- Notification Source Register (NSR)
They are exposed to software in four different pages each proposing a
view with a different privilege. The first page is for the physical
thread context and the second for the hypervisor. Only the third
(operating system) and the fourth (user level) are exposed the guest.
A custom VM fault handler will populate the VMA with the appropriate
pages, which should only be the OS page for now.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/xive.h | 1 +
arch/powerpc/include/uapi/asm/kvm.h | 2 ++
arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 11 ++++++
Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
5 files changed, 76 insertions(+)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 46891f321606..eb6d302082da 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -23,6 +23,7 @@
* same offset regardless of where the code is executing
*/
extern void __iomem *xive_tima;
+extern unsigned long xive_tima_os;
/*
* Offset in the TM area of our current execution level (provided by
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 42d4ef93ec2d..be9b255e061d 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
+#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 132bff52d70a..c6ac818a13b2 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -176,6 +176,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int xive_native_tima_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+
+ switch (vmf->pgoff - vma->vm_pgoff) {
+ case 0: /* HW - forbid access */
+ case 1: /* HV - forbid access */
+ return VM_FAULT_SIGBUS;
+ case 2: /* OS */
+ vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+ case 3: /* USER - TODO */
+ default:
+ return VM_FAULT_SIGBUS;
+ }
+}
+
+static const struct vm_operations_struct xive_native_tima_vmops = {
+ .fault = xive_native_tima_fault,
+};
+
+static int kvmppc_xive_native_mmap(struct kvm_device *dev,
+ struct vm_area_struct *vma)
+{
+ /* We only allow mappings at fixed offset for now */
+ if (vma->vm_pgoff = KVM_XIVE_TIMA_PAGE_OFFSET) {
+ if (vma_pages(vma) > 4)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_tima_vmops;
+ } else {
+ return -EINVAL;
+ }
+
+ vma->vm_flags |= VM_IO | VM_PFNMAP;
+ vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+ return 0;
+}
+
static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
u64 addr)
{
@@ -1005,6 +1043,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
.has_attr = kvmppc_xive_native_has_attr,
+ .mmap = kvmppc_xive_native_mmap,
};
void kvmppc_xive_native_init_module(void)
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 0c037e933e55..7782201e5fe8 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
}
EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
+unsigned long xive_tima_os;
+EXPORT_SYMBOL_GPL(xive_tima_os);
+
bool __init xive_native_init(void)
{
struct device_node *np;
@@ -573,6 +576,14 @@ bool __init xive_native_init(void)
for_each_possible_cpu(cpu)
kvmppc_set_xive_tima(cpu, r.start, tima);
+ /* Resource 2 is OS window */
+ if (of_address_to_resource(np, 2, &r)) {
+ pr_err("Failed to get thread mgmnt area resource\n");
+ return false;
+ }
+
+ xive_tima_os = r.start;
+
/* Grab size of provisionning pages */
xive_parse_provisioning(np);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 1b8957c50c53..4d6b41609fd9 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
XIVE native exploitation interrupt mode. If not, it should run using
the legacy interrupt mode, referred as XICS (POWER7/8).
+* Device Mappings
+
+ The KVM device exposes different MMIO ranges of the XIVE HW which
+ are required for interrupt management. These are exposed to the
+ guest in VMAs populated with a custom VM fault handler.
+
+ 1. Thread Interrupt Management Area (TIMA)
+
+ Each thread has an associated Thread Interrupt Management context
+ composed of a set of registers. These registers let the thread
+ handle priority management and interrupt acknowledgment. The most
+ important are :
+
+ - Interrupt Pending Buffer (IPB)
+ - Current Processor Priority (CPPR)
+ - Notification Source Register (NSR)
+
+ They are exposed to software in four different pages each proposing
+ a view with a different privilege. The first page is for the
+ physical thread context and the second for the hypervisor. Only the
+ third (operating system) and the fourth (user level) are exposed the
+ guest.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 3:42 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:42 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 6843 bytes --]
On Fri, Feb 22, 2019 at 12:28:36PM +0100, Cédric Le Goater wrote:
> Each thread has an associated Thread Interrupt Management context
> composed of a set of registers. These registers let the thread handle
> priority management and interrupt acknowledgment. The most important
> are :
>
> - Interrupt Pending Buffer (IPB)
> - Current Processor Priority (CPPR)
> - Notification Source Register (NSR)
>
> They are exposed to software in four different pages each proposing a
> view with a different privilege. The first page is for the physical
> thread context and the second for the hypervisor. Only the third
> (operating system) and the fourth (user level) are exposed the guest.
>
> A custom VM fault handler will populate the VMA with the appropriate
> pages, which should only be the OS page for now.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Subject to possible modification depending on whether we go with the
generic change to allow mmap() on kvm devices.
> ---
> arch/powerpc/include/asm/xive.h | 1 +
> arch/powerpc/include/uapi/asm/kvm.h | 2 ++
> arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
> arch/powerpc/sysdev/xive/native.c | 11 ++++++
> Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
> 5 files changed, 76 insertions(+)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index 46891f321606..eb6d302082da 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -23,6 +23,7 @@
> * same offset regardless of where the code is executing
> */
> extern void __iomem *xive_tima;
> +extern unsigned long xive_tima_os;
>
> /*
> * Offset in the TM area of our current execution level (provided by
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 42d4ef93ec2d..be9b255e061d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
> #define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
> #define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
>
> +#define KVM_XIVE_TIMA_PAGE_OFFSET 0
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 132bff52d70a..c6ac818a13b2 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -176,6 +176,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +static int xive_native_tima_fault(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *vma = vmf->vma;
> +
> + switch (vmf->pgoff - vma->vm_pgoff) {
> + case 0: /* HW - forbid access */
> + case 1: /* HV - forbid access */
> + return VM_FAULT_SIGBUS;
> + case 2: /* OS */
> + vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
> + return VM_FAULT_NOPAGE;
> + case 3: /* USER - TODO */
> + default:
> + return VM_FAULT_SIGBUS;
> + }
> +}
> +
> +static const struct vm_operations_struct xive_native_tima_vmops = {
> + .fault = xive_native_tima_fault,
> +};
> +
> +static int kvmppc_xive_native_mmap(struct kvm_device *dev,
> + struct vm_area_struct *vma)
> +{
> + /* We only allow mappings at fixed offset for now */
> + if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
> + if (vma_pages(vma) > 4)
> + return -EINVAL;
> + vma->vm_ops = &xive_native_tima_vmops;
> + } else {
> + return -EINVAL;
> + }
> +
> + vma->vm_flags |= VM_IO | VM_PFNMAP;
> + vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> u64 addr)
> {
> @@ -1005,6 +1043,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
> .set_attr = kvmppc_xive_native_set_attr,
> .get_attr = kvmppc_xive_native_get_attr,
> .has_attr = kvmppc_xive_native_has_attr,
> + .mmap = kvmppc_xive_native_mmap,
> };
>
> void kvmppc_xive_native_init_module(void)
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> index 0c037e933e55..7782201e5fe8 100644
> --- a/arch/powerpc/sysdev/xive/native.c
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
> }
> EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
>
> +unsigned long xive_tima_os;
> +EXPORT_SYMBOL_GPL(xive_tima_os);
> +
> bool __init xive_native_init(void)
> {
> struct device_node *np;
> @@ -573,6 +576,14 @@ bool __init xive_native_init(void)
> for_each_possible_cpu(cpu)
> kvmppc_set_xive_tima(cpu, r.start, tima);
>
> + /* Resource 2 is OS window */
> + if (of_address_to_resource(np, 2, &r)) {
> + pr_err("Failed to get thread mgmnt area resource\n");
> + return false;
> + }
> +
> + xive_tima_os = r.start;
> +
> /* Grab size of provisionning pages */
> xive_parse_provisioning(np);
>
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 1b8957c50c53..4d6b41609fd9 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
> XIVE native exploitation interrupt mode. If not, it should run using
> the legacy interrupt mode, referred as XICS (POWER7/8).
>
> +* Device Mappings
> +
> + The KVM device exposes different MMIO ranges of the XIVE HW which
> + are required for interrupt management. These are exposed to the
> + guest in VMAs populated with a custom VM fault handler.
> +
> + 1. Thread Interrupt Management Area (TIMA)
> +
> + Each thread has an associated Thread Interrupt Management context
> + composed of a set of registers. These registers let the thread
> + handle priority management and interrupt acknowledgment. The most
> + important are :
> +
> + - Interrupt Pending Buffer (IPB)
> + - Current Processor Priority (CPPR)
> + - Notification Source Register (NSR)
> +
> + They are exposed to software in four different pages each proposing
> + a view with a different privilege. The first page is for the
> + physical thread context and the second for the hypervisor. Only the
> + third (operating system) and the fourth (user level) are exposed the
> + guest.
> +
> * Groups:
>
> 1. KVM_DEV_XIVE_GRP_CTRL
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
@ 2019-02-25 3:42 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:42 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 6843 bytes --]
On Fri, Feb 22, 2019 at 12:28:36PM +0100, Cédric Le Goater wrote:
> Each thread has an associated Thread Interrupt Management context
> composed of a set of registers. These registers let the thread handle
> priority management and interrupt acknowledgment. The most important
> are :
>
> - Interrupt Pending Buffer (IPB)
> - Current Processor Priority (CPPR)
> - Notification Source Register (NSR)
>
> They are exposed to software in four different pages each proposing a
> view with a different privilege. The first page is for the physical
> thread context and the second for the hypervisor. Only the third
> (operating system) and the fourth (user level) are exposed the guest.
>
> A custom VM fault handler will populate the VMA with the appropriate
> pages, which should only be the OS page for now.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Subject to possible modification depending on whether we go with the
generic change to allow mmap() on kvm devices.
> ---
> arch/powerpc/include/asm/xive.h | 1 +
> arch/powerpc/include/uapi/asm/kvm.h | 2 ++
> arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
> arch/powerpc/sysdev/xive/native.c | 11 ++++++
> Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
> 5 files changed, 76 insertions(+)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index 46891f321606..eb6d302082da 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -23,6 +23,7 @@
> * same offset regardless of where the code is executing
> */
> extern void __iomem *xive_tima;
> +extern unsigned long xive_tima_os;
>
> /*
> * Offset in the TM area of our current execution level (provided by
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index 42d4ef93ec2d..be9b255e061d 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
> #define KVM_XIVE_EQ_FLAG_ALWAYS_NOTIFY 0x00000002
> #define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
>
> +#define KVM_XIVE_TIMA_PAGE_OFFSET 0
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 132bff52d70a..c6ac818a13b2 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -176,6 +176,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +static int xive_native_tima_fault(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *vma = vmf->vma;
> +
> + switch (vmf->pgoff - vma->vm_pgoff) {
> + case 0: /* HW - forbid access */
> + case 1: /* HV - forbid access */
> + return VM_FAULT_SIGBUS;
> + case 2: /* OS */
> + vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
> + return VM_FAULT_NOPAGE;
> + case 3: /* USER - TODO */
> + default:
> + return VM_FAULT_SIGBUS;
> + }
> +}
> +
> +static const struct vm_operations_struct xive_native_tima_vmops = {
> + .fault = xive_native_tima_fault,
> +};
> +
> +static int kvmppc_xive_native_mmap(struct kvm_device *dev,
> + struct vm_area_struct *vma)
> +{
> + /* We only allow mappings at fixed offset for now */
> + if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
> + if (vma_pages(vma) > 4)
> + return -EINVAL;
> + vma->vm_ops = &xive_native_tima_vmops;
> + } else {
> + return -EINVAL;
> + }
> +
> + vma->vm_flags |= VM_IO | VM_PFNMAP;
> + vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
> u64 addr)
> {
> @@ -1005,6 +1043,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
> .set_attr = kvmppc_xive_native_set_attr,
> .get_attr = kvmppc_xive_native_get_attr,
> .has_attr = kvmppc_xive_native_has_attr,
> + .mmap = kvmppc_xive_native_mmap,
> };
>
> void kvmppc_xive_native_init_module(void)
> diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
> index 0c037e933e55..7782201e5fe8 100644
> --- a/arch/powerpc/sysdev/xive/native.c
> +++ b/arch/powerpc/sysdev/xive/native.c
> @@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
> }
> EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
>
> +unsigned long xive_tima_os;
> +EXPORT_SYMBOL_GPL(xive_tima_os);
> +
> bool __init xive_native_init(void)
> {
> struct device_node *np;
> @@ -573,6 +576,14 @@ bool __init xive_native_init(void)
> for_each_possible_cpu(cpu)
> kvmppc_set_xive_tima(cpu, r.start, tima);
>
> + /* Resource 2 is OS window */
> + if (of_address_to_resource(np, 2, &r)) {
> + pr_err("Failed to get thread mgmnt area resource\n");
> + return false;
> + }
> +
> + xive_tima_os = r.start;
> +
> /* Grab size of provisionning pages */
> xive_parse_provisioning(np);
>
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 1b8957c50c53..4d6b41609fd9 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
> XIVE native exploitation interrupt mode. If not, it should run using
> the legacy interrupt mode, referred as XICS (POWER7/8).
>
> +* Device Mappings
> +
> + The KVM device exposes different MMIO ranges of the XIVE HW which
> + are required for interrupt management. These are exposed to the
> + guest in VMAs populated with a custom VM fault handler.
> +
> + 1. Thread Interrupt Management Area (TIMA)
> +
> + Each thread has an associated Thread Interrupt Management context
> + composed of a set of registers. These registers let the thread
> + handle priority management and interrupt acknowledgment. The most
> + important are :
> +
> + - Interrupt Pending Buffer (IPB)
> + - Current Processor Priority (CPPR)
> + - Notification Source Register (NSR)
> +
> + They are exposed to software in four different pages each proposing
> + a view with a different privilege. The first page is for the
> + physical thread context and the second for the hypervisor. Only the
> + third (operating system) and the fourth (user level) are exposed the
> + guest.
> +
> * Groups:
>
> 1. KVM_DEV_XIVE_GRP_CTRL
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Each source is associated with an Event State Buffer (ESB) with a
even/odd pair of pages which provides commands to manage the source:
to trigger, to EOI, to turn off the source for instance.
The custom VM fault handler will deduce the guest IRQ number from the
offset of the fault, and the ESB page of the associated XIVE interrupt
will be inserted into the VMA using the internal structure caching
information on the interrupts.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 7 +++
3 files changed, 65 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index be9b255e061d..d8990e9044e3 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+#define KVM_XIVE_ESB_PAGE_OFFSET 4
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index c6ac818a13b2..92cab6409e8e 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -176,6 +176,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int xive_native_esb_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct kvm_device *dev = vma->vm_file->private_data;
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ u64 page;
+ unsigned long irq;
+ u64 page_offset;
+
+ /*
+ * Linux/KVM uses a two pages ESB setting, one for trigger and
+ * one for EOI
+ */
+ page_offset = vmf->pgoff - vma->vm_pgoff;
+ irq = page_offset / 2;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb) {
+ pr_devel("%s: source %lx not found !\n", __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ state = &sb->irq_state[src];
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * first/even page is for trigger
+ * second/odd page is for EOI and management.
+ */
+ page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
+ arch_spin_unlock(&sb->lock);
+
+ if (WARN_ON(!page)) {
+ pr_err("%s: acessing invalid ESB page for source %lx !\n",
+ __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct xive_native_esb_vmops = {
+ .fault = xive_native_esb_fault,
+};
+
static int xive_native_tima_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -205,6 +258,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
if (vma_pages(vma) > 4)
return -EINVAL;
vma->vm_ops = &xive_native_tima_vmops;
+ } else if (vma->vm_pgoff == KVM_XIVE_ESB_PAGE_OFFSET) {
+ if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_esb_vmops;
} else {
return -EINVAL;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 4d6b41609fd9..be5000b2eb5a 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
third (operating system) and the fourth (user level) are exposed the
guest.
+ 2. Event State Buffer (ESB)
+
+ Each source is associated with an Event State Buffer (ESB) with
+ either a pair of even/odd pair of pages which provides commands to
+ manage the source: to trigger, to EOI, to turn off the source for
+ instance.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Each source is associated with an Event State Buffer (ESB) with a
even/odd pair of pages which provides commands to manage the source:
to trigger, to EOI, to turn off the source for instance.
The custom VM fault handler will deduce the guest IRQ number from the
offset of the fault, and the ESB page of the associated XIVE interrupt
will be inserted into the VMA using the internal structure caching
information on the interrupts.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 7 +++
3 files changed, 65 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index be9b255e061d..d8990e9044e3 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+#define KVM_XIVE_ESB_PAGE_OFFSET 4
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index c6ac818a13b2..92cab6409e8e 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -176,6 +176,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int xive_native_esb_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct kvm_device *dev = vma->vm_file->private_data;
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ u64 page;
+ unsigned long irq;
+ u64 page_offset;
+
+ /*
+ * Linux/KVM uses a two pages ESB setting, one for trigger and
+ * one for EOI
+ */
+ page_offset = vmf->pgoff - vma->vm_pgoff;
+ irq = page_offset / 2;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb) {
+ pr_devel("%s: source %lx not found !\n", __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ state = &sb->irq_state[src];
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * first/even page is for trigger
+ * second/odd page is for EOI and management.
+ */
+ page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
+ arch_spin_unlock(&sb->lock);
+
+ if (WARN_ON(!page)) {
+ pr_err("%s: acessing invalid ESB page for source %lx !\n",
+ __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct xive_native_esb_vmops = {
+ .fault = xive_native_esb_fault,
+};
+
static int xive_native_tima_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -205,6 +258,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
if (vma_pages(vma) > 4)
return -EINVAL;
vma->vm_ops = &xive_native_tima_vmops;
+ } else if (vma->vm_pgoff = KVM_XIVE_ESB_PAGE_OFFSET) {
+ if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_esb_vmops;
} else {
return -EINVAL;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 4d6b41609fd9..be5000b2eb5a 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
third (operating system) and the fourth (user level) are exposed the
guest.
+ 2. Event State Buffer (ESB)
+
+ Each source is associated with an Event State Buffer (ESB) with
+ either a pair of even/odd pair of pages which provides commands to
+ manage the source: to trigger, to EOI, to turn off the source for
+ instance.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 3:47 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:47 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4661 bytes --]
On Fri, Feb 22, 2019 at 12:28:37PM +0100, Cédric Le Goater wrote:
> Each source is associated with an Event State Buffer (ESB) with a
> even/odd pair of pages which provides commands to manage the source:
> to trigger, to EOI, to turn off the source for instance.
>
> The custom VM fault handler will deduce the guest IRQ number from the
> offset of the fault, and the ESB page of the associated XIVE interrupt
> will be inserted into the VMA using the internal structure caching
> information on the interrupts.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
With the same caveat as the previous patch.
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 7 +++
> 3 files changed, 65 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index be9b255e061d..d8990e9044e3 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
> #define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
>
> #define KVM_XIVE_TIMA_PAGE_OFFSET 0
> +#define KVM_XIVE_ESB_PAGE_OFFSET 4
>
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index c6ac818a13b2..92cab6409e8e 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -176,6 +176,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +static int xive_native_esb_fault(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *vma = vmf->vma;
> + struct kvm_device *dev = vma->vm_file->private_data;
> + struct kvmppc_xive *xive = dev->private;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + struct xive_irq_data *xd;
> + u32 hw_num;
> + u16 src;
> + u64 page;
> + unsigned long irq;
> + u64 page_offset;
> +
> + /*
> + * Linux/KVM uses a two pages ESB setting, one for trigger and
> + * one for EOI
> + */
> + page_offset = vmf->pgoff - vma->vm_pgoff;
> + irq = page_offset / 2;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb) {
> + pr_devel("%s: source %lx not found !\n", __func__, irq);
> + return VM_FAULT_SIGBUS;
> + }
> +
> + state = &sb->irq_state[src];
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> +
> + arch_spin_lock(&sb->lock);
> +
> + /*
> + * first/even page is for trigger
> + * second/odd page is for EOI and management.
> + */
> + page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
> + arch_spin_unlock(&sb->lock);
> +
> + if (WARN_ON(!page)) {
> + pr_err("%s: acessing invalid ESB page for source %lx !\n",
> + __func__, irq);
> + return VM_FAULT_SIGBUS;
> + }
> +
> + vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
> + return VM_FAULT_NOPAGE;
> +}
> +
> +static const struct vm_operations_struct xive_native_esb_vmops = {
> + .fault = xive_native_esb_fault,
> +};
> +
> static int xive_native_tima_fault(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> @@ -205,6 +258,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
> if (vma_pages(vma) > 4)
> return -EINVAL;
> vma->vm_ops = &xive_native_tima_vmops;
> + } else if (vma->vm_pgoff == KVM_XIVE_ESB_PAGE_OFFSET) {
> + if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
> + return -EINVAL;
> + vma->vm_ops = &xive_native_esb_vmops;
> } else {
> return -EINVAL;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 4d6b41609fd9..be5000b2eb5a 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> third (operating system) and the fourth (user level) are exposed the
> guest.
>
> + 2. Event State Buffer (ESB)
> +
> + Each source is associated with an Event State Buffer (ESB) with
> + either a pair of even/odd pair of pages which provides commands to
> + manage the source: to trigger, to EOI, to turn off the source for
> + instance.
> +
> * Groups:
>
> 1. KVM_DEV_XIVE_GRP_CTRL
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
@ 2019-02-25 3:47 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 3:47 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4661 bytes --]
On Fri, Feb 22, 2019 at 12:28:37PM +0100, Cédric Le Goater wrote:
> Each source is associated with an Event State Buffer (ESB) with a
> even/odd pair of pages which provides commands to manage the source:
> to trigger, to EOI, to turn off the source for instance.
>
> The custom VM fault handler will deduce the guest IRQ number from the
> offset of the fault, and the ESB page of the associated XIVE interrupt
> will be inserted into the VMA using the internal structure caching
> information on the interrupts.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
With the same caveat as the previous patch.
> ---
> arch/powerpc/include/uapi/asm/kvm.h | 1 +
> arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 7 +++
> 3 files changed, 65 insertions(+)
>
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index be9b255e061d..d8990e9044e3 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
> #define KVM_XIVE_EQ_FLAG_ESCALATE 0x00000004
>
> #define KVM_XIVE_TIMA_PAGE_OFFSET 0
> +#define KVM_XIVE_ESB_PAGE_OFFSET 4
>
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index c6ac818a13b2..92cab6409e8e 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -176,6 +176,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +static int xive_native_esb_fault(struct vm_fault *vmf)
> +{
> + struct vm_area_struct *vma = vmf->vma;
> + struct kvm_device *dev = vma->vm_file->private_data;
> + struct kvmppc_xive *xive = dev->private;
> + struct kvmppc_xive_src_block *sb;
> + struct kvmppc_xive_irq_state *state;
> + struct xive_irq_data *xd;
> + u32 hw_num;
> + u16 src;
> + u64 page;
> + unsigned long irq;
> + u64 page_offset;
> +
> + /*
> + * Linux/KVM uses a two pages ESB setting, one for trigger and
> + * one for EOI
> + */
> + page_offset = vmf->pgoff - vma->vm_pgoff;
> + irq = page_offset / 2;
> +
> + sb = kvmppc_xive_find_source(xive, irq, &src);
> + if (!sb) {
> + pr_devel("%s: source %lx not found !\n", __func__, irq);
> + return VM_FAULT_SIGBUS;
> + }
> +
> + state = &sb->irq_state[src];
> + kvmppc_xive_select_irq(state, &hw_num, &xd);
> +
> + arch_spin_lock(&sb->lock);
> +
> + /*
> + * first/even page is for trigger
> + * second/odd page is for EOI and management.
> + */
> + page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
> + arch_spin_unlock(&sb->lock);
> +
> + if (WARN_ON(!page)) {
> + pr_err("%s: acessing invalid ESB page for source %lx !\n",
> + __func__, irq);
> + return VM_FAULT_SIGBUS;
> + }
> +
> + vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
> + return VM_FAULT_NOPAGE;
> +}
> +
> +static const struct vm_operations_struct xive_native_esb_vmops = {
> + .fault = xive_native_esb_fault,
> +};
> +
> static int xive_native_tima_fault(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> @@ -205,6 +258,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
> if (vma_pages(vma) > 4)
> return -EINVAL;
> vma->vm_ops = &xive_native_tima_vmops;
> + } else if (vma->vm_pgoff == KVM_XIVE_ESB_PAGE_OFFSET) {
> + if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
> + return -EINVAL;
> + vma->vm_ops = &xive_native_esb_vmops;
> } else {
> return -EINVAL;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 4d6b41609fd9..be5000b2eb5a 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> third (operating system) and the fourth (user level) are exposed the
> guest.
>
> + 2. Event State Buffer (ESB)
> +
> + Each source is associated with an Event State Buffer (ESB) with
> + either a pair of even/odd pair of pages which provides commands to
> + manage the source: to trigger, to EOI, to turn off the source for
> + instance.
> +
> * Groups:
>
> 1. KVM_DEV_XIVE_GRP_CTRL
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
implement an IRQ space for the guest using the generic IPI interrupts
of the XIVE IC controller. These interrupts are allocated at the OPAL
level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
Interrupt management is performed in the XIVE way: using loads and
stores on the addresses of the XIVE IPI interrupt ESB pages.
Both KVM devices share the same internal structure caching information
on the interrupts, among which the xive_irq_data struct containing the
addresses of the IPI ESB pages and an extra one in case of passthrough.
The later contains the addresses of the ESB pages of the underlying HW
controller interrupts, PHB4 in all cases for now.
A guest, when running in the XICS legacy interrupt mode, lets the KVM
XICS-over-XIVE device "handle" interrupt management, that is to
perform the loads and stores on the addresses of the ESB pages of the
guest interrupts. However, when running in XIVE native exploitation
mode, the KVM XIVE native device exposes the interrupt ESB pages to
the guest and lets the guest perform directly the loads and stores.
The VMA exposing the ESB pages make use of a custom VM fault handler
which role is to populate the VMA with appropriate pages. When a fault
occurs, the guest IRQ number is deduced from the offset, and the ESB
pages of associated XIVE IPI interrupt are inserted in the VMA (using
the internal structure caching information on the interrupts).
Supporting device passthrough in the guest running in XIVE native
exploitation mode adds some extra refinements because the ESB pages
of a different HW controller (PHB4) need to be exposed to the guest
along with the initial IPI ESB pages of the XIVE IC controller. But
the overall mechanic is the same.
When the device HW irqs are mapped into or unmapped from the guest
IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
and kvmppc_xive_clr_mapped(), are called to record or clear the
passthrough interrupt information and to perform the switch.
The approach taken by this patch is to clear the ESB pages of the
guest IRQ number being mapped and let the VM fault handler repopulate.
The handler will insert the ESB page corresponding to the HW interrupt
of the device being passed-through or the initial IPI ESB page if the
device is being removed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/kvm/book3s_xive.h | 9 +++++
arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 ++++++++
4 files changed, 80 insertions(+)
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 6660d138c6b7..d1f832a53811 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
};
+struct kvmppc_xive;
+
+struct kvmppc_xive_ops {
+ int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
+};
struct kvmppc_xive {
struct kvm *kvm;
@@ -132,6 +137,10 @@ struct kvmppc_xive {
/* Flags */
u8 single_escalation;
+
+ struct kvmppc_xive_ops *ops;
+ struct address_space *mapping;
+ struct mutex mapping_lock;
};
#define KVMPPC_XIVE_Q_COUNT 8
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 7431e31bc541..7a14512b8944 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -942,6 +942,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Turn the IPI hard off */
xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped)
+ xive->ops->reset_mapped(kvm, guest_irq);
+
/* Grab info about irq */
state->pt_number = hw_irq;
state->pt_data = irq_data_get_irq_handler_data(host_data);
@@ -1027,6 +1034,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
state->pt_number = 0;
state->pt_data = NULL;
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped) {
+ xive->ops->reset_mapped(kvm, guest_irq);
+ }
+
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
xive_vp(xive, state->act_server),
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 92cab6409e8e..bf60870144f1 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -14,6 +14,7 @@
#include <linux/delay.h>
#include <linux/percpu.h>
#include <linux/cpumask.h>
+#include <linux/file.h>
#include <asm/uaccess.h>
#include <asm/kvm_book3s.h>
#include <asm/kvm_ppc.h>
@@ -176,6 +177,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+/*
+ * Device passthrough support
+ */
+static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+
+ if (irq >= KVMPPC_XIVE_NR_IRQS)
+ return -EINVAL;
+
+ /*
+ * Clear the ESB pages of the IRQ number being mapped (or
+ * unmapped) into the guest and let the the VM fault handler
+ * repopulate with the appropriate ESB pages (device or IC)
+ */
+ pr_debug("clearing esb pages for girq 0x%lx\n", irq);
+ mutex_lock(&xive->mapping_lock);
+ if (xive->mapping)
+ unmap_mapping_range(xive->mapping,
+ irq * (2ull << PAGE_SHIFT),
+ 2ull << PAGE_SHIFT, 1);
+ mutex_unlock(&xive->mapping_lock);
+ return 0;
+}
+
+static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
+ .reset_mapped = kvmppc_xive_native_reset_mapped,
+};
+
static int xive_native_esb_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -253,6 +283,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
static int kvmppc_xive_native_mmap(struct kvm_device *dev,
struct vm_area_struct *vma)
{
+ struct kvmppc_xive *xive = dev->private;
+
/* We only allow mappings at fixed offset for now */
if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
if (vma_pages(vma) > 4)
@@ -268,6 +300,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
vma->vm_flags |= VM_IO | VM_PFNMAP;
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+
+ /*
+ * Grab the KVM device file address_space to be able to clear
+ * the ESB pages mapping when a device is passed-through into
+ * the guest.
+ */
+ xive->mapping = vma->vm_file->f_mapping;
return 0;
}
@@ -913,6 +952,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
xive->dev = dev;
xive->kvm = kvm;
kvm->arch.xive = xive;
+ mutex_init(&xive->mapping_lock);
/* We use the default queue size set by the host */
xive->q_order = xive_native_default_eq_shift();
@@ -933,6 +973,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
ret = -ENOMEM;
xive->single_escalation = xive_native_has_single_escalation();
+ xive->ops = &kvmppc_xive_native_ops;
if (ret)
kfree(xive);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index be5000b2eb5a..7a242cb07e7c 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -43,6 +43,21 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
manage the source: to trigger, to EOI, to turn off the source for
instance.
+ 3. Device passthrough
+
+ When a device is passed-through into the guest, the source
+ interrupts are from a different HW controller (PHB4) and the ESB
+ pages exposed to the guest should accommadate this change.
+
+ The passthru_irq helpers, kvmppc_xive_set_mapped() and
+ kvmppc_xive_clr_mapped() are called when the device HW irqs are
+ mapped into or unmapped from the guest IRQ number space. The KVM
+ device extends these helpers to clear the ESB pages of the guest IRQ
+ number being mapped and then lets the VM fault handler repopulate.
+ The handler will insert the ESB page corresponding to the HW
+ interrupt of the device being passed-through or the initial IPI ESB
+ page if the device has being removed.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
implement an IRQ space for the guest using the generic IPI interrupts
of the XIVE IC controller. These interrupts are allocated at the OPAL
level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
Interrupt management is performed in the XIVE way: using loads and
stores on the addresses of the XIVE IPI interrupt ESB pages.
Both KVM devices share the same internal structure caching information
on the interrupts, among which the xive_irq_data struct containing the
addresses of the IPI ESB pages and an extra one in case of passthrough.
The later contains the addresses of the ESB pages of the underlying HW
controller interrupts, PHB4 in all cases for now.
A guest, when running in the XICS legacy interrupt mode, lets the KVM
XICS-over-XIVE device "handle" interrupt management, that is to
perform the loads and stores on the addresses of the ESB pages of the
guest interrupts. However, when running in XIVE native exploitation
mode, the KVM XIVE native device exposes the interrupt ESB pages to
the guest and lets the guest perform directly the loads and stores.
The VMA exposing the ESB pages make use of a custom VM fault handler
which role is to populate the VMA with appropriate pages. When a fault
occurs, the guest IRQ number is deduced from the offset, and the ESB
pages of associated XIVE IPI interrupt are inserted in the VMA (using
the internal structure caching information on the interrupts).
Supporting device passthrough in the guest running in XIVE native
exploitation mode adds some extra refinements because the ESB pages
of a different HW controller (PHB4) need to be exposed to the guest
along with the initial IPI ESB pages of the XIVE IC controller. But
the overall mechanic is the same.
When the device HW irqs are mapped into or unmapped from the guest
IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
and kvmppc_xive_clr_mapped(), are called to record or clear the
passthrough interrupt information and to perform the switch.
The approach taken by this patch is to clear the ESB pages of the
guest IRQ number being mapped and let the VM fault handler repopulate.
The handler will insert the ESB page corresponding to the HW interrupt
of the device being passed-through or the initial IPI ESB page if the
device is being removed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/kvm/book3s_xive.h | 9 +++++
arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 ++++++++
4 files changed, 80 insertions(+)
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 6660d138c6b7..d1f832a53811 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
};
+struct kvmppc_xive;
+
+struct kvmppc_xive_ops {
+ int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
+};
struct kvmppc_xive {
struct kvm *kvm;
@@ -132,6 +137,10 @@ struct kvmppc_xive {
/* Flags */
u8 single_escalation;
+
+ struct kvmppc_xive_ops *ops;
+ struct address_space *mapping;
+ struct mutex mapping_lock;
};
#define KVMPPC_XIVE_Q_COUNT 8
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 7431e31bc541..7a14512b8944 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -942,6 +942,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Turn the IPI hard off */
xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped)
+ xive->ops->reset_mapped(kvm, guest_irq);
+
/* Grab info about irq */
state->pt_number = hw_irq;
state->pt_data = irq_data_get_irq_handler_data(host_data);
@@ -1027,6 +1034,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
state->pt_number = 0;
state->pt_data = NULL;
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped) {
+ xive->ops->reset_mapped(kvm, guest_irq);
+ }
+
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
xive_vp(xive, state->act_server),
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 92cab6409e8e..bf60870144f1 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -14,6 +14,7 @@
#include <linux/delay.h>
#include <linux/percpu.h>
#include <linux/cpumask.h>
+#include <linux/file.h>
#include <asm/uaccess.h>
#include <asm/kvm_book3s.h>
#include <asm/kvm_ppc.h>
@@ -176,6 +177,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+/*
+ * Device passthrough support
+ */
+static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+
+ if (irq >= KVMPPC_XIVE_NR_IRQS)
+ return -EINVAL;
+
+ /*
+ * Clear the ESB pages of the IRQ number being mapped (or
+ * unmapped) into the guest and let the the VM fault handler
+ * repopulate with the appropriate ESB pages (device or IC)
+ */
+ pr_debug("clearing esb pages for girq 0x%lx\n", irq);
+ mutex_lock(&xive->mapping_lock);
+ if (xive->mapping)
+ unmap_mapping_range(xive->mapping,
+ irq * (2ull << PAGE_SHIFT),
+ 2ull << PAGE_SHIFT, 1);
+ mutex_unlock(&xive->mapping_lock);
+ return 0;
+}
+
+static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
+ .reset_mapped = kvmppc_xive_native_reset_mapped,
+};
+
static int xive_native_esb_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -253,6 +283,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
static int kvmppc_xive_native_mmap(struct kvm_device *dev,
struct vm_area_struct *vma)
{
+ struct kvmppc_xive *xive = dev->private;
+
/* We only allow mappings at fixed offset for now */
if (vma->vm_pgoff = KVM_XIVE_TIMA_PAGE_OFFSET) {
if (vma_pages(vma) > 4)
@@ -268,6 +300,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
vma->vm_flags |= VM_IO | VM_PFNMAP;
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+
+ /*
+ * Grab the KVM device file address_space to be able to clear
+ * the ESB pages mapping when a device is passed-through into
+ * the guest.
+ */
+ xive->mapping = vma->vm_file->f_mapping;
return 0;
}
@@ -913,6 +952,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
xive->dev = dev;
xive->kvm = kvm;
kvm->arch.xive = xive;
+ mutex_init(&xive->mapping_lock);
/* We use the default queue size set by the host */
xive->q_order = xive_native_default_eq_shift();
@@ -933,6 +973,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
ret = -ENOMEM;
xive->single_escalation = xive_native_has_single_escalation();
+ xive->ops = &kvmppc_xive_native_ops;
if (ret)
kfree(xive);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index be5000b2eb5a..7a242cb07e7c 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -43,6 +43,21 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
manage the source: to trigger, to EOI, to turn off the source for
instance.
+ 3. Device passthrough
+
+ When a device is passed-through into the guest, the source
+ interrupts are from a different HW controller (PHB4) and the ESB
+ pages exposed to the guest should accommadate this change.
+
+ The passthru_irq helpers, kvmppc_xive_set_mapped() and
+ kvmppc_xive_clr_mapped() are called when the device HW irqs are
+ mapped into or unmapped from the guest IRQ number space. The KVM
+ device extends these helpers to clear the ESB pages of the guest IRQ
+ number being mapped and then lets the VM fault handler repopulate.
+ The handler will insert the ESB page corresponding to the HW
+ interrupt of the device being passed-through or the initial IPI ESB
+ page if the device has being removed.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 4:13 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 4:13 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 9646 bytes --]
On Fri, Feb 22, 2019 at 12:28:38PM +0100, Cédric Le Goater wrote:
> The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
> implement an IRQ space for the guest using the generic IPI interrupts
> of the XIVE IC controller. These interrupts are allocated at the OPAL
> level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
> Interrupt management is performed in the XIVE way: using loads and
> stores on the addresses of the XIVE IPI interrupt ESB pages.
>
> Both KVM devices share the same internal structure caching information
> on the interrupts, among which the xive_irq_data struct containing the
> addresses of the IPI ESB pages and an extra one in case of passthrough.
> The later contains the addresses of the ESB pages of the underlying HW
> controller interrupts, PHB4 in all cases for now.
>
> A guest, when running in the XICS legacy interrupt mode, lets the KVM
> XICS-over-XIVE device "handle" interrupt management, that is to
> perform the loads and stores on the addresses of the ESB pages of the
> guest interrupts. However, when running in XIVE native exploitation
> mode, the KVM XIVE native device exposes the interrupt ESB pages to
> the guest and lets the guest perform directly the loads and stores.
>
> The VMA exposing the ESB pages make use of a custom VM fault handler
> which role is to populate the VMA with appropriate pages. When a fault
> occurs, the guest IRQ number is deduced from the offset, and the ESB
> pages of associated XIVE IPI interrupt are inserted in the VMA (using
> the internal structure caching information on the interrupts).
>
> Supporting device passthrough in the guest running in XIVE native
> exploitation mode adds some extra refinements because the ESB pages
> of a different HW controller (PHB4) need to be exposed to the guest
> along with the initial IPI ESB pages of the XIVE IC controller. But
> the overall mechanic is the same.
>
> When the device HW irqs are mapped into or unmapped from the guest
> IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
> and kvmppc_xive_clr_mapped(), are called to record or clear the
> passthrough interrupt information and to perform the switch.
>
> The approach taken by this patch is to clear the ESB pages of the
> guest IRQ number being mapped and let the VM fault handler repopulate.
> The handler will insert the ESB page corresponding to the HW interrupt
> of the device being passed-through or the initial IPI ESB page if the
> device is being removed.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/kvm/book3s_xive.h | 9 +++++
> arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
> arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 15 ++++++++
> 4 files changed, 80 insertions(+)
>
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index 6660d138c6b7..d1f832a53811 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
> struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
> };
>
> +struct kvmppc_xive;
> +
> +struct kvmppc_xive_ops {
> + int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
> +};
>
> struct kvmppc_xive {
> struct kvm *kvm;
> @@ -132,6 +137,10 @@ struct kvmppc_xive {
>
> /* Flags */
> u8 single_escalation;
> +
> + struct kvmppc_xive_ops *ops;
> + struct address_space *mapping;
> + struct mutex mapping_lock;
> };
>
> #define KVMPPC_XIVE_Q_COUNT 8
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 7431e31bc541..7a14512b8944 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -942,6 +942,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
> /* Turn the IPI hard off */
> xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
>
> + /*
> + * Reset ESB guest mapping. Needed when ESB pages are exposed
> + * to the guest in XIVE native mode
> + */
> + if (xive->ops && xive->ops->reset_mapped)
> + xive->ops->reset_mapped(kvm, guest_irq);
> +
> /* Grab info about irq */
> state->pt_number = hw_irq;
> state->pt_data = irq_data_get_irq_handler_data(host_data);
> @@ -1027,6 +1034,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
> state->pt_number = 0;
> state->pt_data = NULL;
>
> + /*
> + * Reset ESB guest mapping. Needed when ESB pages are exposed
> + * to the guest in XIVE native mode
> + */
> + if (xive->ops && xive->ops->reset_mapped) {
> + xive->ops->reset_mapped(kvm, guest_irq);
> + }
> +
> /* Reconfigure the IPI */
> xive_native_configure_irq(state->ipi_number,
> xive_vp(xive, state->act_server),
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 92cab6409e8e..bf60870144f1 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -14,6 +14,7 @@
> #include <linux/delay.h>
> #include <linux/percpu.h>
> #include <linux/cpumask.h>
> +#include <linux/file.h>
> #include <asm/uaccess.h>
> #include <asm/kvm_book3s.h>
> #include <asm/kvm_ppc.h>
> @@ -176,6 +177,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +/*
> + * Device passthrough support
> + */
> +static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> +
> + if (irq >= KVMPPC_XIVE_NR_IRQS)
> + return -EINVAL;
> +
> + /*
> + * Clear the ESB pages of the IRQ number being mapped (or
> + * unmapped) into the guest and let the the VM fault handler
> + * repopulate with the appropriate ESB pages (device or IC)
> + */
> + pr_debug("clearing esb pages for girq 0x%lx\n", irq);
> + mutex_lock(&xive->mapping_lock);
> + if (xive->mapping)
> + unmap_mapping_range(xive->mapping,
> + irq * (2ull << PAGE_SHIFT),
> + 2ull << PAGE_SHIFT, 1);
> + mutex_unlock(&xive->mapping_lock);
> + return 0;
> +}
> +
> +static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
> + .reset_mapped = kvmppc_xive_native_reset_mapped,
> +};
> +
> static int xive_native_esb_fault(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> @@ -253,6 +283,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
> static int kvmppc_xive_native_mmap(struct kvm_device *dev,
> struct vm_area_struct *vma)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> /* We only allow mappings at fixed offset for now */
> if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
> if (vma_pages(vma) > 4)
> @@ -268,6 +300,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
>
> vma->vm_flags |= VM_IO | VM_PFNMAP;
> vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
> +
> + /*
> + * Grab the KVM device file address_space to be able to clear
> + * the ESB pages mapping when a device is passed-through into
> + * the guest.
> + */
> + xive->mapping = vma->vm_file->f_mapping;
> return 0;
> }
>
> @@ -913,6 +952,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> xive->dev = dev;
> xive->kvm = kvm;
> kvm->arch.xive = xive;
> + mutex_init(&xive->mapping_lock);
>
> /* We use the default queue size set by the host */
> xive->q_order = xive_native_default_eq_shift();
> @@ -933,6 +973,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> ret = -ENOMEM;
>
> xive->single_escalation = xive_native_has_single_escalation();
> + xive->ops = &kvmppc_xive_native_ops;
>
> if (ret)
> kfree(xive);
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index be5000b2eb5a..7a242cb07e7c 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -43,6 +43,21 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> manage the source: to trigger, to EOI, to turn off the source for
> instance.
>
> + 3. Device passthrough
> +
> + When a device is passed-through into the guest, the source
> + interrupts are from a different HW controller (PHB4) and the ESB
> + pages exposed to the guest should accommadate this change.
> +
> + The passthru_irq helpers, kvmppc_xive_set_mapped() and
> + kvmppc_xive_clr_mapped() are called when the device HW irqs are
> + mapped into or unmapped from the guest IRQ number space. The KVM
> + device extends these helpers to clear the ESB pages of the guest IRQ
> + number being mapped and then lets the VM fault handler repopulate.
> + The handler will insert the ESB page corresponding to the HW
> + interrupt of the device being passed-through or the initial IPI ESB
> + page if the device has being removed.
I think it might be worth emphasizing that this all happens with KVM
and userspace / the guest doesn't need to do anything about this
remapping. Really this is an informational aside, not something a
user of the device actually needs to know.
> * Groups:
>
> 1. KVM_DEV_XIVE_GRP_CTRL
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
@ 2019-02-25 4:13 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 4:13 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 9646 bytes --]
On Fri, Feb 22, 2019 at 12:28:38PM +0100, Cédric Le Goater wrote:
> The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
> implement an IRQ space for the guest using the generic IPI interrupts
> of the XIVE IC controller. These interrupts are allocated at the OPAL
> level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
> Interrupt management is performed in the XIVE way: using loads and
> stores on the addresses of the XIVE IPI interrupt ESB pages.
>
> Both KVM devices share the same internal structure caching information
> on the interrupts, among which the xive_irq_data struct containing the
> addresses of the IPI ESB pages and an extra one in case of passthrough.
> The later contains the addresses of the ESB pages of the underlying HW
> controller interrupts, PHB4 in all cases for now.
>
> A guest, when running in the XICS legacy interrupt mode, lets the KVM
> XICS-over-XIVE device "handle" interrupt management, that is to
> perform the loads and stores on the addresses of the ESB pages of the
> guest interrupts. However, when running in XIVE native exploitation
> mode, the KVM XIVE native device exposes the interrupt ESB pages to
> the guest and lets the guest perform directly the loads and stores.
>
> The VMA exposing the ESB pages make use of a custom VM fault handler
> which role is to populate the VMA with appropriate pages. When a fault
> occurs, the guest IRQ number is deduced from the offset, and the ESB
> pages of associated XIVE IPI interrupt are inserted in the VMA (using
> the internal structure caching information on the interrupts).
>
> Supporting device passthrough in the guest running in XIVE native
> exploitation mode adds some extra refinements because the ESB pages
> of a different HW controller (PHB4) need to be exposed to the guest
> along with the initial IPI ESB pages of the XIVE IC controller. But
> the overall mechanic is the same.
>
> When the device HW irqs are mapped into or unmapped from the guest
> IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
> and kvmppc_xive_clr_mapped(), are called to record or clear the
> passthrough interrupt information and to perform the switch.
>
> The approach taken by this patch is to clear the ESB pages of the
> guest IRQ number being mapped and let the VM fault handler repopulate.
> The handler will insert the ESB page corresponding to the HW interrupt
> of the device being passed-through or the initial IPI ESB page if the
> device is being removed.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/kvm/book3s_xive.h | 9 +++++
> arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
> arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 15 ++++++++
> 4 files changed, 80 insertions(+)
>
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index 6660d138c6b7..d1f832a53811 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
> struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
> };
>
> +struct kvmppc_xive;
> +
> +struct kvmppc_xive_ops {
> + int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
> +};
>
> struct kvmppc_xive {
> struct kvm *kvm;
> @@ -132,6 +137,10 @@ struct kvmppc_xive {
>
> /* Flags */
> u8 single_escalation;
> +
> + struct kvmppc_xive_ops *ops;
> + struct address_space *mapping;
> + struct mutex mapping_lock;
> };
>
> #define KVMPPC_XIVE_Q_COUNT 8
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 7431e31bc541..7a14512b8944 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -942,6 +942,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
> /* Turn the IPI hard off */
> xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
>
> + /*
> + * Reset ESB guest mapping. Needed when ESB pages are exposed
> + * to the guest in XIVE native mode
> + */
> + if (xive->ops && xive->ops->reset_mapped)
> + xive->ops->reset_mapped(kvm, guest_irq);
> +
> /* Grab info about irq */
> state->pt_number = hw_irq;
> state->pt_data = irq_data_get_irq_handler_data(host_data);
> @@ -1027,6 +1034,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
> state->pt_number = 0;
> state->pt_data = NULL;
>
> + /*
> + * Reset ESB guest mapping. Needed when ESB pages are exposed
> + * to the guest in XIVE native mode
> + */
> + if (xive->ops && xive->ops->reset_mapped) {
> + xive->ops->reset_mapped(kvm, guest_irq);
> + }
> +
> /* Reconfigure the IPI */
> xive_native_configure_irq(state->ipi_number,
> xive_vp(xive, state->act_server),
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 92cab6409e8e..bf60870144f1 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -14,6 +14,7 @@
> #include <linux/delay.h>
> #include <linux/percpu.h>
> #include <linux/cpumask.h>
> +#include <linux/file.h>
> #include <asm/uaccess.h>
> #include <asm/kvm_book3s.h>
> #include <asm/kvm_ppc.h>
> @@ -176,6 +177,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
> return rc;
> }
>
> +/*
> + * Device passthrough support
> + */
> +static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
> +{
> + struct kvmppc_xive *xive = kvm->arch.xive;
> +
> + if (irq >= KVMPPC_XIVE_NR_IRQS)
> + return -EINVAL;
> +
> + /*
> + * Clear the ESB pages of the IRQ number being mapped (or
> + * unmapped) into the guest and let the the VM fault handler
> + * repopulate with the appropriate ESB pages (device or IC)
> + */
> + pr_debug("clearing esb pages for girq 0x%lx\n", irq);
> + mutex_lock(&xive->mapping_lock);
> + if (xive->mapping)
> + unmap_mapping_range(xive->mapping,
> + irq * (2ull << PAGE_SHIFT),
> + 2ull << PAGE_SHIFT, 1);
> + mutex_unlock(&xive->mapping_lock);
> + return 0;
> +}
> +
> +static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
> + .reset_mapped = kvmppc_xive_native_reset_mapped,
> +};
> +
> static int xive_native_esb_fault(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> @@ -253,6 +283,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
> static int kvmppc_xive_native_mmap(struct kvm_device *dev,
> struct vm_area_struct *vma)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> /* We only allow mappings at fixed offset for now */
> if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
> if (vma_pages(vma) > 4)
> @@ -268,6 +300,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
>
> vma->vm_flags |= VM_IO | VM_PFNMAP;
> vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
> +
> + /*
> + * Grab the KVM device file address_space to be able to clear
> + * the ESB pages mapping when a device is passed-through into
> + * the guest.
> + */
> + xive->mapping = vma->vm_file->f_mapping;
> return 0;
> }
>
> @@ -913,6 +952,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> xive->dev = dev;
> xive->kvm = kvm;
> kvm->arch.xive = xive;
> + mutex_init(&xive->mapping_lock);
>
> /* We use the default queue size set by the host */
> xive->q_order = xive_native_default_eq_shift();
> @@ -933,6 +973,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
> ret = -ENOMEM;
>
> xive->single_escalation = xive_native_has_single_escalation();
> + xive->ops = &kvmppc_xive_native_ops;
>
> if (ret)
> kfree(xive);
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index be5000b2eb5a..7a242cb07e7c 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -43,6 +43,21 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> manage the source: to trigger, to EOI, to turn off the source for
> instance.
>
> + 3. Device passthrough
> +
> + When a device is passed-through into the guest, the source
> + interrupts are from a different HW controller (PHB4) and the ESB
> + pages exposed to the guest should accommadate this change.
> +
> + The passthru_irq helpers, kvmppc_xive_set_mapped() and
> + kvmppc_xive_clr_mapped() are called when the device HW irqs are
> + mapped into or unmapped from the guest IRQ number space. The KVM
> + device extends these helpers to clear the ESB pages of the guest IRQ
> + number being mapped and then lets the VM fault handler repopulate.
> + The handler will insert the ESB page corresponding to the HW
> + interrupt of the device being passed-through or the initial IPI ESB
> + page if the device has being removed.
I think it might be worth emphasizing that this all happens with KVM
and userspace / the guest doesn't need to do anything about this
remapping. Really this is an informational aside, not something a
user of the device actually needs to know.
> * Groups:
>
> 1. KVM_DEV_XIVE_GRP_CTRL
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The 'destroy' method is currently used to destroy all devices when the
VM is destroyed after the vCPUs have been freed.
This new KVM ioctl exposes the same KVM device method. It acts as a
software reset of the VM to 'destroy' selected devices when necessary
and perform the required cleanups on the vCPUs. Called with the
kvm->lock.
The 'destroy' method could be improved by returning an error code.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
include/uapi/linux/kvm.h | 7 ++++++
virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++++++++
Documentation/virtual/kvm/api.txt | 19 ++++++++++++++++
3 files changed, 64 insertions(+)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 52bf74a1616e..d78fafa54274 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1183,6 +1183,11 @@ struct kvm_create_device {
__u32 flags; /* in: KVM_CREATE_DEVICE_xxx */
};
+struct kvm_destroy_device {
+ __u32 fd; /* in: device handle */
+ __u32 flags; /* in: unused */
+};
+
struct kvm_device_attr {
__u32 flags; /* no flags currently defined */
__u32 group; /* device-defined */
@@ -1331,6 +1336,8 @@ struct kvm_s390_ucas_mapping {
#define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)
#define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr)
+#define KVM_DESTROY_DEVICE _IOWR(KVMIO, 0xf0, struct kvm_destroy_device)
+
/*
* ioctls for vcpu fds
*/
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 84717d8cb5e4..983d5c37f710 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3026,6 +3026,30 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
return 0;
}
+static int kvm_ioctl_destroy_device(struct kvm *kvm,
+ struct kvm_destroy_device *dd)
+{
+ struct fd f;
+ struct kvm_device *dev;
+
+ f = fdget(dd->fd);
+ if (!f.file)
+ return -EBADF;
+
+ dev = kvm_device_from_filp(f.file);
+ fdput(f);
+
+ if (!dev)
+ return -ENODEV;
+
+ mutex_lock(&kvm->lock);
+ list_del(&dev->vm_node);
+ dev->ops->destroy(dev);
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
{
switch (arg) {
@@ -3270,6 +3294,20 @@ static long kvm_vm_ioctl(struct file *filp,
r = 0;
break;
}
+ case KVM_DESTROY_DEVICE: {
+ struct kvm_destroy_device dd;
+
+ r = -EFAULT;
+ if (copy_from_user(&dd, argp, sizeof(dd)))
+ goto out;
+
+ r = kvm_ioctl_destroy_device(kvm, &dd);
+ if (r)
+ goto out;
+
+ r = 0;
+ break;
+ }
case KVM_CHECK_EXTENSION:
r = kvm_vm_ioctl_check_extension_generic(kvm, arg);
break;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 1db1435769b4..c2ba18279298 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -3857,6 +3857,25 @@ number of valid entries in the 'entries' array, which is then filled.
'index' and 'flags' fields in 'struct kvm_cpuid_entry2' are currently reserved,
userspace should not expect to get any particular value there.
+4.119 KVM_DESTROY_DEVICE
+
+Capability: KVM_CAP_DEVICE_CTRL
+Type: vm ioctl
+Parameters: struct kvm_destroy_device (in)
+Returns: 0 on success, -1 on error
+Errors:
+ ENODEV: The device type is unknown or unsupported
+
+ Other error conditions may be defined by individual device types or
+ have their standard meanings.
+
+Destroys an emulated device in the kernel.
+
+struct kvm_destroy_device {
+ __u32 fd; /* in: device handle */
+ __u32 flags; /* unused */
+};
+
5. The kvm_run structure
------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The 'destroy' method is currently used to destroy all devices when the
VM is destroyed after the vCPUs have been freed.
This new KVM ioctl exposes the same KVM device method. It acts as a
software reset of the VM to 'destroy' selected devices when necessary
and perform the required cleanups on the vCPUs. Called with the
kvm->lock.
The 'destroy' method could be improved by returning an error code.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
include/uapi/linux/kvm.h | 7 ++++++
virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++++++++
Documentation/virtual/kvm/api.txt | 19 ++++++++++++++++
3 files changed, 64 insertions(+)
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 52bf74a1616e..d78fafa54274 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1183,6 +1183,11 @@ struct kvm_create_device {
__u32 flags; /* in: KVM_CREATE_DEVICE_xxx */
};
+struct kvm_destroy_device {
+ __u32 fd; /* in: device handle */
+ __u32 flags; /* in: unused */
+};
+
struct kvm_device_attr {
__u32 flags; /* no flags currently defined */
__u32 group; /* device-defined */
@@ -1331,6 +1336,8 @@ struct kvm_s390_ucas_mapping {
#define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)
#define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr)
+#define KVM_DESTROY_DEVICE _IOWR(KVMIO, 0xf0, struct kvm_destroy_device)
+
/*
* ioctls for vcpu fds
*/
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 84717d8cb5e4..983d5c37f710 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3026,6 +3026,30 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
return 0;
}
+static int kvm_ioctl_destroy_device(struct kvm *kvm,
+ struct kvm_destroy_device *dd)
+{
+ struct fd f;
+ struct kvm_device *dev;
+
+ f = fdget(dd->fd);
+ if (!f.file)
+ return -EBADF;
+
+ dev = kvm_device_from_filp(f.file);
+ fdput(f);
+
+ if (!dev)
+ return -ENODEV;
+
+ mutex_lock(&kvm->lock);
+ list_del(&dev->vm_node);
+ dev->ops->destroy(dev);
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
{
switch (arg) {
@@ -3270,6 +3294,20 @@ static long kvm_vm_ioctl(struct file *filp,
r = 0;
break;
}
+ case KVM_DESTROY_DEVICE: {
+ struct kvm_destroy_device dd;
+
+ r = -EFAULT;
+ if (copy_from_user(&dd, argp, sizeof(dd)))
+ goto out;
+
+ r = kvm_ioctl_destroy_device(kvm, &dd);
+ if (r)
+ goto out;
+
+ r = 0;
+ break;
+ }
case KVM_CHECK_EXTENSION:
r = kvm_vm_ioctl_check_extension_generic(kvm, arg);
break;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 1db1435769b4..c2ba18279298 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -3857,6 +3857,25 @@ number of valid entries in the 'entries' array, which is then filled.
'index' and 'flags' fields in 'struct kvm_cpuid_entry2' are currently reserved,
userspace should not expect to get any particular value there.
+4.119 KVM_DESTROY_DEVICE
+
+Capability: KVM_CAP_DEVICE_CTRL
+Type: vm ioctl
+Parameters: struct kvm_destroy_device (in)
+Returns: 0 on success, -1 on error
+Errors:
+ ENODEV: The device type is unknown or unsupported
+
+ Other error conditions may be defined by individual device types or
+ have their standard meanings.
+
+Destroys an emulated device in the kernel.
+
+struct kvm_destroy_device {
+ __u32 fd; /* in: device handle */
+ __u32 flags; /* unused */
+};
+
5. The kvm_run structure
------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 4:15 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 4:15 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4393 bytes --]
On Fri, Feb 22, 2019 at 12:28:39PM +0100, Cédric Le Goater wrote:
> The 'destroy' method is currently used to destroy all devices when the
> VM is destroyed after the vCPUs have been freed.
>
> This new KVM ioctl exposes the same KVM device method. It acts as a
> software reset of the VM to 'destroy' selected devices when necessary
> and perform the required cleanups on the vCPUs. Called with the
> kvm->lock.
>
> The 'destroy' method could be improved by returning an error code.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Again, has this been discussed with Paolo and/or other KVM core
people?
> ---
> include/uapi/linux/kvm.h | 7 ++++++
> virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++++++++
> Documentation/virtual/kvm/api.txt | 19 ++++++++++++++++
> 3 files changed, 64 insertions(+)
>
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 52bf74a1616e..d78fafa54274 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1183,6 +1183,11 @@ struct kvm_create_device {
> __u32 flags; /* in: KVM_CREATE_DEVICE_xxx */
> };
>
> +struct kvm_destroy_device {
> + __u32 fd; /* in: device handle */
> + __u32 flags; /* in: unused */
> +};
> +
> struct kvm_device_attr {
> __u32 flags; /* no flags currently defined */
> __u32 group; /* device-defined */
> @@ -1331,6 +1336,8 @@ struct kvm_s390_ucas_mapping {
> #define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)
> #define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr)
>
> +#define KVM_DESTROY_DEVICE _IOWR(KVMIO, 0xf0, struct kvm_destroy_device)
> +
> /*
> * ioctls for vcpu fds
> */
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 84717d8cb5e4..983d5c37f710 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -3026,6 +3026,30 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
> return 0;
> }
>
> +static int kvm_ioctl_destroy_device(struct kvm *kvm,
> + struct kvm_destroy_device *dd)
> +{
> + struct fd f;
> + struct kvm_device *dev;
> +
> + f = fdget(dd->fd);
> + if (!f.file)
> + return -EBADF;
> +
> + dev = kvm_device_from_filp(f.file);
> + fdput(f);
> +
> + if (!dev)
> + return -ENODEV;
Don't you need to verify that the device belongs to this KVM instance?
> + mutex_lock(&kvm->lock);
> + list_del(&dev->vm_node);
> + dev->ops->destroy(dev);
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +
> static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
> {
> switch (arg) {
> @@ -3270,6 +3294,20 @@ static long kvm_vm_ioctl(struct file *filp,
> r = 0;
> break;
> }
> + case KVM_DESTROY_DEVICE: {
> + struct kvm_destroy_device dd;
> +
> + r = -EFAULT;
> + if (copy_from_user(&dd, argp, sizeof(dd)))
> + goto out;
> +
> + r = kvm_ioctl_destroy_device(kvm, &dd);
> + if (r)
> + goto out;
> +
> + r = 0;
> + break;
> + }
> case KVM_CHECK_EXTENSION:
> r = kvm_vm_ioctl_check_extension_generic(kvm, arg);
> break;
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 1db1435769b4..c2ba18279298 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -3857,6 +3857,25 @@ number of valid entries in the 'entries' array, which is then filled.
> 'index' and 'flags' fields in 'struct kvm_cpuid_entry2' are currently reserved,
> userspace should not expect to get any particular value there.
>
> +4.119 KVM_DESTROY_DEVICE
> +
> +Capability: KVM_CAP_DEVICE_CTRL
> +Type: vm ioctl
> +Parameters: struct kvm_destroy_device (in)
> +Returns: 0 on success, -1 on error
> +Errors:
> + ENODEV: The device type is unknown or unsupported
> +
> + Other error conditions may be defined by individual device types or
> + have their standard meanings.
> +
> +Destroys an emulated device in the kernel.
> +
> +struct kvm_destroy_device {
> + __u32 fd; /* in: device handle */
> + __u32 flags; /* unused */
> +};
> +
> 5. The kvm_run structure
> ------------------------
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
@ 2019-02-25 4:15 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 4:15 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 4393 bytes --]
On Fri, Feb 22, 2019 at 12:28:39PM +0100, Cédric Le Goater wrote:
> The 'destroy' method is currently used to destroy all devices when the
> VM is destroyed after the vCPUs have been freed.
>
> This new KVM ioctl exposes the same KVM device method. It acts as a
> software reset of the VM to 'destroy' selected devices when necessary
> and perform the required cleanups on the vCPUs. Called with the
> kvm->lock.
>
> The 'destroy' method could be improved by returning an error code.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Again, has this been discussed with Paolo and/or other KVM core
people?
> ---
> include/uapi/linux/kvm.h | 7 ++++++
> virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++++++++
> Documentation/virtual/kvm/api.txt | 19 ++++++++++++++++
> 3 files changed, 64 insertions(+)
>
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 52bf74a1616e..d78fafa54274 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -1183,6 +1183,11 @@ struct kvm_create_device {
> __u32 flags; /* in: KVM_CREATE_DEVICE_xxx */
> };
>
> +struct kvm_destroy_device {
> + __u32 fd; /* in: device handle */
> + __u32 flags; /* in: unused */
> +};
> +
> struct kvm_device_attr {
> __u32 flags; /* no flags currently defined */
> __u32 group; /* device-defined */
> @@ -1331,6 +1336,8 @@ struct kvm_s390_ucas_mapping {
> #define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)
> #define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr)
>
> +#define KVM_DESTROY_DEVICE _IOWR(KVMIO, 0xf0, struct kvm_destroy_device)
> +
> /*
> * ioctls for vcpu fds
> */
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index 84717d8cb5e4..983d5c37f710 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -3026,6 +3026,30 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
> return 0;
> }
>
> +static int kvm_ioctl_destroy_device(struct kvm *kvm,
> + struct kvm_destroy_device *dd)
> +{
> + struct fd f;
> + struct kvm_device *dev;
> +
> + f = fdget(dd->fd);
> + if (!f.file)
> + return -EBADF;
> +
> + dev = kvm_device_from_filp(f.file);
> + fdput(f);
> +
> + if (!dev)
> + return -ENODEV;
Don't you need to verify that the device belongs to this KVM instance?
> + mutex_lock(&kvm->lock);
> + list_del(&dev->vm_node);
> + dev->ops->destroy(dev);
> + mutex_unlock(&kvm->lock);
> +
> + return 0;
> +}
> +
> static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
> {
> switch (arg) {
> @@ -3270,6 +3294,20 @@ static long kvm_vm_ioctl(struct file *filp,
> r = 0;
> break;
> }
> + case KVM_DESTROY_DEVICE: {
> + struct kvm_destroy_device dd;
> +
> + r = -EFAULT;
> + if (copy_from_user(&dd, argp, sizeof(dd)))
> + goto out;
> +
> + r = kvm_ioctl_destroy_device(kvm, &dd);
> + if (r)
> + goto out;
> +
> + r = 0;
> + break;
> + }
> case KVM_CHECK_EXTENSION:
> r = kvm_vm_ioctl_check_extension_generic(kvm, arg);
> break;
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 1db1435769b4..c2ba18279298 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -3857,6 +3857,25 @@ number of valid entries in the 'entries' array, which is then filled.
> 'index' and 'flags' fields in 'struct kvm_cpuid_entry2' are currently reserved,
> userspace should not expect to get any particular value there.
>
> +4.119 KVM_DESTROY_DEVICE
> +
> +Capability: KVM_CAP_DEVICE_CTRL
> +Type: vm ioctl
> +Parameters: struct kvm_destroy_device (in)
> +Returns: 0 on success, -1 on error
> +Errors:
> + ENODEV: The device type is unknown or unsupported
> +
> + Other error conditions may be defined by individual device types or
> + have their standard meanings.
> +
> +Destroys an emulated device in the kernel.
> +
> +struct kvm_destroy_device {
> + __u32 fd; /* in: device handle */
> + __u32 flags; /* unused */
> +};
> +
> 5. The kvm_run structure
> ------------------------
>
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
2019-02-25 4:15 ` David Gibson
@ 2019-03-13 8:02 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:02 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, Paolo Bonzini, linuxppc-dev
On 2/25/19 5:15 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:39PM +0100, Cédric Le Goater wrote:
>> The 'destroy' method is currently used to destroy all devices when the
>> VM is destroyed after the vCPUs have been freed.
>>
>> This new KVM ioctl exposes the same KVM device method. It acts as a
>> software reset of the VM to 'destroy' selected devices when necessary
>> and perform the required cleanups on the vCPUs. Called with the
>> kvm->lock.
>>
>> The 'destroy' method could be improved by returning an error code.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>
> Again, has this been discussed with Paolo and/or other KVM core
> people?
Not yet. Adding Paolo for feedback.
>> ---
>> include/uapi/linux/kvm.h | 7 ++++++
>> virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++++++++
>> Documentation/virtual/kvm/api.txt | 19 ++++++++++++++++
>> 3 files changed, 64 insertions(+)
>>
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 52bf74a1616e..d78fafa54274 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1183,6 +1183,11 @@ struct kvm_create_device {
>> __u32 flags; /* in: KVM_CREATE_DEVICE_xxx */
>> };
>>
>> +struct kvm_destroy_device {
>> + __u32 fd; /* in: device handle */
>> + __u32 flags; /* in: unused */
>> +};
>> +
>> struct kvm_device_attr {
>> __u32 flags; /* no flags currently defined */
>> __u32 group; /* device-defined */
>> @@ -1331,6 +1336,8 @@ struct kvm_s390_ucas_mapping {
>> #define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)
>> #define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr)
>>
>> +#define KVM_DESTROY_DEVICE _IOWR(KVMIO, 0xf0, struct kvm_destroy_device)
>> +
>> /*
>> * ioctls for vcpu fds
>> */
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 84717d8cb5e4..983d5c37f710 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -3026,6 +3026,30 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
>> return 0;
>> }
>>
>> +static int kvm_ioctl_destroy_device(struct kvm *kvm,
>> + struct kvm_destroy_device *dd)
>> +{
>> + struct fd f;
>> + struct kvm_device *dev;
>> +
>> + f = fdget(dd->fd);
>> + if (!f.file)
>> + return -EBADF;
>> +
>> + dev = kvm_device_from_filp(f.file);
>> + fdput(f);
>> +
>> + if (!dev)
>> + return -ENODEV;
>
> Don't you need to verify that the device belongs to this KVM instance?
ah yes. I should at least check : dev->kvm == kvm
>> + mutex_lock(&kvm->lock);
>> + list_del(&dev->vm_node);
>> + dev->ops->destroy(dev);
>> + mutex_unlock(&kvm->lock);
And there, I should problably drop a ref count on the VM. I am not sure
of that.
Thanks,
C.
>> + return 0;
>> +}
>> +
>> static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
>> {
>> switch (arg) {
>> @@ -3270,6 +3294,20 @@ static long kvm_vm_ioctl(struct file *filp,
>> r = 0;
>> break;
>> }
>> + case KVM_DESTROY_DEVICE: {
>> + struct kvm_destroy_device dd;
>> +
>> + r = -EFAULT;
>> + if (copy_from_user(&dd, argp, sizeof(dd)))
>> + goto out;
>> +
>> + r = kvm_ioctl_destroy_device(kvm, &dd);
>> + if (r)
>> + goto out;
>> +
>> + r = 0;
>> + break;
>> + }
>> case KVM_CHECK_EXTENSION:
>> r = kvm_vm_ioctl_check_extension_generic(kvm, arg);
>> break;
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 1db1435769b4..c2ba18279298 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -3857,6 +3857,25 @@ number of valid entries in the 'entries' array, which is then filled.
>> 'index' and 'flags' fields in 'struct kvm_cpuid_entry2' are currently reserved,
>> userspace should not expect to get any particular value there.
>>
>> +4.119 KVM_DESTROY_DEVICE
>> +
>> +Capability: KVM_CAP_DEVICE_CTRL
>> +Type: vm ioctl
>> +Parameters: struct kvm_destroy_device (in)
>> +Returns: 0 on success, -1 on error
>> +Errors:
>> + ENODEV: The device type is unknown or unsupported
>> +
>> + Other error conditions may be defined by individual device types or
>> + have their standard meanings.
>> +
>> +Destroys an emulated device in the kernel.
>> +
>> +struct kvm_destroy_device {
>> + __u32 fd; /* in: device handle */
>> + __u32 flags; /* unused */
>> +};
>> +
>> 5. The kvm_run structure
>> ------------------------
>>
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
@ 2019-03-13 8:02 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:02 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, Paolo Bonzini, linuxppc-dev
On 2/25/19 5:15 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:39PM +0100, Cédric Le Goater wrote:
>> The 'destroy' method is currently used to destroy all devices when the
>> VM is destroyed after the vCPUs have been freed.
>>
>> This new KVM ioctl exposes the same KVM device method. It acts as a
>> software reset of the VM to 'destroy' selected devices when necessary
>> and perform the required cleanups on the vCPUs. Called with the
>> kvm->lock.
>>
>> The 'destroy' method could be improved by returning an error code.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>
> Again, has this been discussed with Paolo and/or other KVM core
> people?
Not yet. Adding Paolo for feedback.
>> ---
>> include/uapi/linux/kvm.h | 7 ++++++
>> virt/kvm/kvm_main.c | 38 +++++++++++++++++++++++++++++++
>> Documentation/virtual/kvm/api.txt | 19 ++++++++++++++++
>> 3 files changed, 64 insertions(+)
>>
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 52bf74a1616e..d78fafa54274 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -1183,6 +1183,11 @@ struct kvm_create_device {
>> __u32 flags; /* in: KVM_CREATE_DEVICE_xxx */
>> };
>>
>> +struct kvm_destroy_device {
>> + __u32 fd; /* in: device handle */
>> + __u32 flags; /* in: unused */
>> +};
>> +
>> struct kvm_device_attr {
>> __u32 flags; /* no flags currently defined */
>> __u32 group; /* device-defined */
>> @@ -1331,6 +1336,8 @@ struct kvm_s390_ucas_mapping {
>> #define KVM_GET_DEVICE_ATTR _IOW(KVMIO, 0xe2, struct kvm_device_attr)
>> #define KVM_HAS_DEVICE_ATTR _IOW(KVMIO, 0xe3, struct kvm_device_attr)
>>
>> +#define KVM_DESTROY_DEVICE _IOWR(KVMIO, 0xf0, struct kvm_destroy_device)
>> +
>> /*
>> * ioctls for vcpu fds
>> */
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index 84717d8cb5e4..983d5c37f710 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -3026,6 +3026,30 @@ static int kvm_ioctl_create_device(struct kvm *kvm,
>> return 0;
>> }
>>
>> +static int kvm_ioctl_destroy_device(struct kvm *kvm,
>> + struct kvm_destroy_device *dd)
>> +{
>> + struct fd f;
>> + struct kvm_device *dev;
>> +
>> + f = fdget(dd->fd);
>> + if (!f.file)
>> + return -EBADF;
>> +
>> + dev = kvm_device_from_filp(f.file);
>> + fdput(f);
>> +
>> + if (!dev)
>> + return -ENODEV;
>
> Don't you need to verify that the device belongs to this KVM instance?
ah yes. I should at least check : dev->kvm = kvm
>> + mutex_lock(&kvm->lock);
>> + list_del(&dev->vm_node);
>> + dev->ops->destroy(dev);
>> + mutex_unlock(&kvm->lock);
And there, I should problably drop a ref count on the VM. I am not sure
of that.
Thanks,
C.
>> + return 0;
>> +}
>> +
>> static long kvm_vm_ioctl_check_extension_generic(struct kvm *kvm, long arg)
>> {
>> switch (arg) {
>> @@ -3270,6 +3294,20 @@ static long kvm_vm_ioctl(struct file *filp,
>> r = 0;
>> break;
>> }
>> + case KVM_DESTROY_DEVICE: {
>> + struct kvm_destroy_device dd;
>> +
>> + r = -EFAULT;
>> + if (copy_from_user(&dd, argp, sizeof(dd)))
>> + goto out;
>> +
>> + r = kvm_ioctl_destroy_device(kvm, &dd);
>> + if (r)
>> + goto out;
>> +
>> + r = 0;
>> + break;
>> + }
>> case KVM_CHECK_EXTENSION:
>> r = kvm_vm_ioctl_check_extension_generic(kvm, arg);
>> break;
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 1db1435769b4..c2ba18279298 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -3857,6 +3857,25 @@ number of valid entries in the 'entries' array, which is then filled.
>> 'index' and 'flags' fields in 'struct kvm_cpuid_entry2' are currently reserved,
>> userspace should not expect to get any particular value there.
>>
>> +4.119 KVM_DESTROY_DEVICE
>> +
>> +Capability: KVM_CAP_DEVICE_CTRL
>> +Type: vm ioctl
>> +Parameters: struct kvm_destroy_device (in)
>> +Returns: 0 on success, -1 on error
>> +Errors:
>> + ENODEV: The device type is unknown or unsupported
>> +
>> + Other error conditions may be defined by individual device types or
>> + have their standard meanings.
>> +
>> +Destroys an emulated device in the kernel.
>> +
>> +struct kvm_destroy_device {
>> + __u32 fd; /* in: device handle */
>> + __u32 flags; /* unused */
>> +};
>> +
>> 5. The kvm_run structure
>> ------------------------
>>
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
2019-03-13 8:02 ` Cédric Le Goater
@ 2019-03-15 17:57 ` Paolo Bonzini
-1 siblings, 0 replies; 142+ messages in thread
From: Paolo Bonzini @ 2019-03-15 17:57 UTC (permalink / raw)
To: Cédric Le Goater, David Gibson
Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 13/03/19 09:02, Cédric Le Goater wrote:
> The 'destroy' method is currently used to destroy all devices when the
> VM is destroyed after the vCPUs have been freed.
>
> This new KVM ioctl exposes the same KVM device method. It acts as a
> software reset of the VM to 'destroy' selected devices when necessary
> and perform the required cleanups on the vCPUs. Called with the
> kvm->lock.
>
> The 'destroy' method could be improved by returning an error code.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Looks good to me.
Paolo
^ permalink raw reply [flat|nested] 142+ messages in thread
* Re: [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl
@ 2019-03-15 17:57 ` Paolo Bonzini
0 siblings, 0 replies; 142+ messages in thread
From: Paolo Bonzini @ 2019-03-15 17:57 UTC (permalink / raw)
To: Cédric Le Goater, David Gibson
Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 13/03/19 09:02, Cédric Le Goater wrote:
> The 'destroy' method is currently used to destroy all devices when the
> VM is destroyed after the vCPUs have been freed.
>
> This new KVM ioctl exposes the same KVM device method. It acts as a
> software reset of the VM to 'destroy' selected devices when necessary
> and perform the required cleanups on the vCPUs. Called with the
> kvm->lock.
>
> The 'destroy' method could be improved by returning an error code.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Looks good to me.
Paolo
^ permalink raw reply [flat|nested] 142+ messages in thread
* [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-22 11:28 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
When the VM boots, the CAS negotiation process determines which
interrupt mode to use and invokes a machine reset. At that time, the
previous KVM interrupt device is 'destroyed' before the chosen one is
created. Upon destruction, the vCPU interrupt presenters using the KVM
device should be cleared first, the machine will reconnect them later
to the new device after it is created.
When using the KVM device, there is still a race window with the early
checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
3 files changed, 72 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
index f27ee57ab46e..81cdabf4295f 100644
--- a/arch/powerpc/kvm/book3s_xics.c
+++ b/arch/powerpc/kvm/book3s_xics.c
@@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
struct kvmppc_xics *xics = dev->private;
int i;
struct kvm *kvm = xics->kvm;
+ struct kvm_vcpu *vcpu;
+
+ /*
+ * When destroying the VM, the vCPUs are destroyed first and
+ * the vCPU list should be empty. If this is not the case,
+ * then we are simply destroying the device and we should
+ * clean up the vCPU interrupt presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xics_free_icp(vcpu);
+ }
debugfs_remove(xics->dentry);
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 7a14512b8944..0a1c11d6881c 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- struct kvmppc_xive *xive = xc->xive;
+ struct kvmppc_xive *xive;
int i;
+ if (!kvmppc_xics_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
+ xive = xc->xive;
+
/* Ensure no interrupt is still routed to that VP */
xc->valid = false;
kvmppc_xive_disable_vcpu_interrupts(vcpu);
@@ -1146,6 +1154,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
}
/* Free the VP */
kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
}
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
@@ -1163,7 +1175,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
if (xive->kvm != vcpu->kvm)
return -EPERM;
- if (vcpu->arch.irq_type)
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
return -EBUSY;
if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
pr_devel("Duplicate !\n");
@@ -1833,8 +1845,31 @@ static void kvmppc_xive_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
int i;
+ /*
+ * When destroying the VM, the vCPUs are destroyed first and
+ * the vCPU list should be empty. If this is not the case,
+ * then we are simply destroying the device and we should
+ * clean up the vCPU interrupt presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ }
+
debugfs_remove(xive->dentry);
if (kvm)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index bf60870144f1..c0655164d9af 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -909,8 +909,24 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
int i;
+ /*
+ * When destroying the VM, the vCPUs are destroyed first and
+ * the vCPU list should be empty. If this is not the case,
+ * then we are simply destroying the device and we should
+ * clean up the vCPU interrupt presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_xive_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ }
+
debugfs_remove(xive->dentry);
pr_devel("Destroying xive native device\n");
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
@ 2019-02-22 11:28 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-02-22 11:28 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
When the VM boots, the CAS negotiation process determines which
interrupt mode to use and invokes a machine reset. At that time, the
previous KVM interrupt device is 'destroyed' before the chosen one is
created. Upon destruction, the vCPU interrupt presenters using the KVM
device should be cleared first, the machine will reconnect them later
to the new device after it is created.
When using the KVM device, there is still a race window with the early
checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
3 files changed, 72 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
index f27ee57ab46e..81cdabf4295f 100644
--- a/arch/powerpc/kvm/book3s_xics.c
+++ b/arch/powerpc/kvm/book3s_xics.c
@@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
struct kvmppc_xics *xics = dev->private;
int i;
struct kvm *kvm = xics->kvm;
+ struct kvm_vcpu *vcpu;
+
+ /*
+ * When destroying the VM, the vCPUs are destroyed first and
+ * the vCPU list should be empty. If this is not the case,
+ * then we are simply destroying the device and we should
+ * clean up the vCPU interrupt presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xics_free_icp(vcpu);
+ }
debugfs_remove(xics->dentry);
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 7a14512b8944..0a1c11d6881c 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- struct kvmppc_xive *xive = xc->xive;
+ struct kvmppc_xive *xive;
int i;
+ if (!kvmppc_xics_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
+ xive = xc->xive;
+
/* Ensure no interrupt is still routed to that VP */
xc->valid = false;
kvmppc_xive_disable_vcpu_interrupts(vcpu);
@@ -1146,6 +1154,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
}
/* Free the VP */
kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
}
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
@@ -1163,7 +1175,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
if (xive->kvm != vcpu->kvm)
return -EPERM;
- if (vcpu->arch.irq_type)
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
return -EBUSY;
if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
pr_devel("Duplicate !\n");
@@ -1833,8 +1845,31 @@ static void kvmppc_xive_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
int i;
+ /*
+ * When destroying the VM, the vCPUs are destroyed first and
+ * the vCPU list should be empty. If this is not the case,
+ * then we are simply destroying the device and we should
+ * clean up the vCPU interrupt presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ }
+
debugfs_remove(xive->dentry);
if (kvm)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index bf60870144f1..c0655164d9af 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -909,8 +909,24 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
int i;
+ /*
+ * When destroying the VM, the vCPUs are destroyed first and
+ * the vCPU list should be empty. If this is not the case,
+ * then we are simply destroying the device and we should
+ * clean up the vCPU interrupt presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_xive_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ }
+
debugfs_remove(xive->dentry);
pr_devel("Destroying xive native device\n");
--
2.20.1
^ permalink raw reply related [flat|nested] 142+ messages in thread* Re: [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
2019-02-22 11:28 ` Cédric Le Goater
@ 2019-02-25 4:18 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 4:18 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 5712 bytes --]
On Fri, Feb 22, 2019 at 12:28:40PM +0100, Cédric Le Goater wrote:
> When the VM boots, the CAS negotiation process determines which
> interrupt mode to use and invokes a machine reset. At that time, the
> previous KVM interrupt device is 'destroyed' before the chosen one is
> created. Upon destruction, the vCPU interrupt presenters using the KVM
> device should be cleared first, the machine will reconnect them later
> to the new device after it is created.
>
> When using the KVM device, there is still a race window with the early
> checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
> arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
> arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
> 3 files changed, 72 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
> index f27ee57ab46e..81cdabf4295f 100644
> --- a/arch/powerpc/kvm/book3s_xics.c
> +++ b/arch/powerpc/kvm/book3s_xics.c
> @@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
> struct kvmppc_xics *xics = dev->private;
> int i;
> struct kvm *kvm = xics->kvm;
> + struct kvm_vcpu *vcpu;
> +
> + /*
> + * When destroying the VM, the vCPUs are destroyed first and
> + * the vCPU list should be empty. If this is not the case,
> + * then we are simply destroying the device and we should
> + * clean up the vCPU interrupt presenters first.
> + */
> + if (atomic_read(&kvm->online_vcpus) != 0) {
> + /*
> + * call kick_all_cpus_sync() to ensure that all CPUs
> + * have executed any pending interrupts
> + */
> + if (is_kvmppc_hv_enabled(kvm))
> + kick_all_cpus_sync();
> +
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvmppc_xics_free_icp(vcpu);
> + }
>
> debugfs_remove(xics->dentry);
>
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 7a14512b8944..0a1c11d6881c 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - struct kvmppc_xive *xive = xc->xive;
> + struct kvmppc_xive *xive;
> int i;
>
> + if (!kvmppc_xics_enabled(vcpu))
This should be kvmppc_xive_enabled(), no?
> + return;
> +
> + if (!xc)
> + return;
> +
> pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
>
> + xive = xc->xive;
> +
> /* Ensure no interrupt is still routed to that VP */
> xc->valid = false;
> kvmppc_xive_disable_vcpu_interrupts(vcpu);
> @@ -1146,6 +1154,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> }
> /* Free the VP */
> kfree(xc);
> +
> + /* Cleanup the vcpu */
> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
> + vcpu->arch.xive_vcpu = NULL;
> }
>
> int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> @@ -1163,7 +1175,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
> if (xive->kvm != vcpu->kvm)
> return -EPERM;
> - if (vcpu->arch.irq_type)
> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
> return -EBUSY;
> if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
> pr_devel("Duplicate !\n");
> @@ -1833,8 +1845,31 @@ static void kvmppc_xive_free(struct kvm_device *dev)
> {
> struct kvmppc_xive *xive = dev->private;
> struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> int i;
>
> + /*
> + * When destroying the VM, the vCPUs are destroyed first and
> + * the vCPU list should be empty. If this is not the case,
> + * then we are simply destroying the device and we should
> + * clean up the vCPU interrupt presenters first.
> + */
> + if (atomic_read(&kvm->online_vcpus) != 0) {
> + /*
> + * call kick_all_cpus_sync() to ensure that all CPUs
> + * have executed any pending interrupts
> + */
> + if (is_kvmppc_hv_enabled(kvm))
> + kick_all_cpus_sync();
> +
> + /*
> + * TODO: There is still a race window with the early
> + * checks in kvmppc_native_connect_vcpu()
> + */
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvmppc_xive_cleanup_vcpu(vcpu);
> + }
> +
> debugfs_remove(xive->dentry);
>
> if (kvm)
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index bf60870144f1..c0655164d9af 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -909,8 +909,24 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> {
> struct kvmppc_xive *xive = dev->private;
> struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> int i;
>
> + /*
> + * When destroying the VM, the vCPUs are destroyed first and
> + * the vCPU list should be empty. If this is not the case,
> + * then we are simply destroying the device and we should
> + * clean up the vCPU interrupt presenters first.
> + */
> + if (atomic_read(&kvm->online_vcpus) != 0) {
> + /*
> + * TODO: There is still a race window with the early
> + * checks in kvmppc_xive_native_connect_vcpu()
> + */
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvmppc_xive_native_cleanup_vcpu(vcpu);
> + }
> +
> debugfs_remove(xive->dentry);
>
> pr_devel("Destroying xive native device\n");
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
@ 2019-02-25 4:18 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-02-25 4:18 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 5712 bytes --]
On Fri, Feb 22, 2019 at 12:28:40PM +0100, Cédric Le Goater wrote:
> When the VM boots, the CAS negotiation process determines which
> interrupt mode to use and invokes a machine reset. At that time, the
> previous KVM interrupt device is 'destroyed' before the chosen one is
> created. Upon destruction, the vCPU interrupt presenters using the KVM
> device should be cleared first, the machine will reconnect them later
> to the new device after it is created.
>
> When using the KVM device, there is still a race window with the early
> checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> ---
> arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
> arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
> arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
> 3 files changed, 72 insertions(+), 2 deletions(-)
>
> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
> index f27ee57ab46e..81cdabf4295f 100644
> --- a/arch/powerpc/kvm/book3s_xics.c
> +++ b/arch/powerpc/kvm/book3s_xics.c
> @@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
> struct kvmppc_xics *xics = dev->private;
> int i;
> struct kvm *kvm = xics->kvm;
> + struct kvm_vcpu *vcpu;
> +
> + /*
> + * When destroying the VM, the vCPUs are destroyed first and
> + * the vCPU list should be empty. If this is not the case,
> + * then we are simply destroying the device and we should
> + * clean up the vCPU interrupt presenters first.
> + */
> + if (atomic_read(&kvm->online_vcpus) != 0) {
> + /*
> + * call kick_all_cpus_sync() to ensure that all CPUs
> + * have executed any pending interrupts
> + */
> + if (is_kvmppc_hv_enabled(kvm))
> + kick_all_cpus_sync();
> +
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvmppc_xics_free_icp(vcpu);
> + }
>
> debugfs_remove(xics->dentry);
>
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 7a14512b8944..0a1c11d6881c 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - struct kvmppc_xive *xive = xc->xive;
> + struct kvmppc_xive *xive;
> int i;
>
> + if (!kvmppc_xics_enabled(vcpu))
This should be kvmppc_xive_enabled(), no?
> + return;
> +
> + if (!xc)
> + return;
> +
> pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
>
> + xive = xc->xive;
> +
> /* Ensure no interrupt is still routed to that VP */
> xc->valid = false;
> kvmppc_xive_disable_vcpu_interrupts(vcpu);
> @@ -1146,6 +1154,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> }
> /* Free the VP */
> kfree(xc);
> +
> + /* Cleanup the vcpu */
> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
> + vcpu->arch.xive_vcpu = NULL;
> }
>
> int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> @@ -1163,7 +1175,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
> if (xive->kvm != vcpu->kvm)
> return -EPERM;
> - if (vcpu->arch.irq_type)
> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
> return -EBUSY;
> if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
> pr_devel("Duplicate !\n");
> @@ -1833,8 +1845,31 @@ static void kvmppc_xive_free(struct kvm_device *dev)
> {
> struct kvmppc_xive *xive = dev->private;
> struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> int i;
>
> + /*
> + * When destroying the VM, the vCPUs are destroyed first and
> + * the vCPU list should be empty. If this is not the case,
> + * then we are simply destroying the device and we should
> + * clean up the vCPU interrupt presenters first.
> + */
> + if (atomic_read(&kvm->online_vcpus) != 0) {
> + /*
> + * call kick_all_cpus_sync() to ensure that all CPUs
> + * have executed any pending interrupts
> + */
> + if (is_kvmppc_hv_enabled(kvm))
> + kick_all_cpus_sync();
> +
> + /*
> + * TODO: There is still a race window with the early
> + * checks in kvmppc_native_connect_vcpu()
> + */
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvmppc_xive_cleanup_vcpu(vcpu);
> + }
> +
> debugfs_remove(xive->dentry);
>
> if (kvm)
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index bf60870144f1..c0655164d9af 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -909,8 +909,24 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
> {
> struct kvmppc_xive *xive = dev->private;
> struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> int i;
>
> + /*
> + * When destroying the VM, the vCPUs are destroyed first and
> + * the vCPU list should be empty. If this is not the case,
> + * then we are simply destroying the device and we should
> + * clean up the vCPU interrupt presenters first.
> + */
> + if (atomic_read(&kvm->online_vcpus) != 0) {
> + /*
> + * TODO: There is still a race window with the early
> + * checks in kvmppc_xive_native_connect_vcpu()
> + */
> + kvm_for_each_vcpu(i, vcpu, kvm)
> + kvmppc_xive_native_cleanup_vcpu(vcpu);
> + }
> +
> debugfs_remove(xive->dentry);
>
> pr_devel("Destroying xive native device\n");
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
2019-02-25 4:18 ` David Gibson
@ 2019-03-13 8:17 ` Cédric Le Goater
-1 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:17 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 5:18 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:40PM +0100, Cédric Le Goater wrote:
>> When the VM boots, the CAS negotiation process determines which
>> interrupt mode to use and invokes a machine reset. At that time, the
>> previous KVM interrupt device is 'destroyed' before the chosen one is
>> created. Upon destruction, the vCPU interrupt presenters using the KVM
>> device should be cleared first, the machine will reconnect them later
>> to the new device after it is created.
>>
>> When using the KVM device, there is still a race window with the early
>> checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
>> arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
>> arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
>> 3 files changed, 72 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
>> index f27ee57ab46e..81cdabf4295f 100644
>> --- a/arch/powerpc/kvm/book3s_xics.c
>> +++ b/arch/powerpc/kvm/book3s_xics.c
>> @@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
>> struct kvmppc_xics *xics = dev->private;
>> int i;
>> struct kvm *kvm = xics->kvm;
>> + struct kvm_vcpu *vcpu;
>> +
>> + /*
>> + * When destroying the VM, the vCPUs are destroyed first and
>> + * the vCPU list should be empty. If this is not the case,
>> + * then we are simply destroying the device and we should
>> + * clean up the vCPU interrupt presenters first.
>> + */
>> + if (atomic_read(&kvm->online_vcpus) != 0) {
>> + /*
>> + * call kick_all_cpus_sync() to ensure that all CPUs
>> + * have executed any pending interrupts
>> + */
>> + if (is_kvmppc_hv_enabled(kvm))
>> + kick_all_cpus_sync();
>> +
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvmppc_xics_free_icp(vcpu);
>> + }
>>
>> debugfs_remove(xics->dentry);
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index 7a14512b8944..0a1c11d6881c 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> - struct kvmppc_xive *xive = xc->xive;
>> + struct kvmppc_xive *xive;
>> int i;
>>
>> + if (!kvmppc_xics_enabled(vcpu))
>
> This should be kvmppc_xive_enabled(), no?
This is the KVM XICS-on-XIVE device and its IRQ type is KVMPPC_IRQ_XICS.
So this is correct :/
May be we should introduce a KVMPPC_IRQ_XICS_ON_XIVE macro to clarify.
>
>> + return;
>> +
>> + if (!xc)
>> + return;
>> +
>> pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
>>
>> + xive = xc->xive;
>> +
>> /* Ensure no interrupt is still routed to that VP */
>> xc->valid = false;
>> kvmppc_xive_disable_vcpu_interrupts(vcpu);
>> @@ -1146,6 +1154,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>> }
>> /* Free the VP */
>> kfree(xc);
>> +
>> + /* Cleanup the vcpu */
>> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
>> + vcpu->arch.xive_vcpu = NULL;
>> }
>>
>> int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> @@ -1163,7 +1175,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> }
>> if (xive->kvm != vcpu->kvm)
>> return -EPERM;
>> - if (vcpu->arch.irq_type)
>> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
>> return -EBUSY;
>> if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
>> pr_devel("Duplicate !\n");
>> @@ -1833,8 +1845,31 @@ static void kvmppc_xive_free(struct kvm_device *dev)
>> {
>> struct kvmppc_xive *xive = dev->private;
>> struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> int i;
>>
>> + /*
>> + * When destroying the VM, the vCPUs are destroyed first and
>> + * the vCPU list should be empty. If this is not the case,
>> + * then we are simply destroying the device and we should
>> + * clean up the vCPU interrupt presenters first.
>> + */
>> + if (atomic_read(&kvm->online_vcpus) != 0) {
>> + /*
>> + * call kick_all_cpus_sync() to ensure that all CPUs
>> + * have executed any pending interrupts
>> + */
>> + if (is_kvmppc_hv_enabled(kvm))
>> + kick_all_cpus_sync();
>> +
>> + /*
>> + * TODO: There is still a race window with the early
>> + * checks in kvmppc_native_connect_vcpu()
>> + */
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvmppc_xive_cleanup_vcpu(vcpu);
>> + }
>> +
>> debugfs_remove(xive->dentry);
>>
>> if (kvm)
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index bf60870144f1..c0655164d9af 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -909,8 +909,24 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
>> {
>> struct kvmppc_xive *xive = dev->private;
>> struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> int i;
>>
>> + /*
>> + * When destroying the VM, the vCPUs are destroyed first and
>> + * the vCPU list should be empty. If this is not the case,
>> + * then we are simply destroying the device and we should
>> + * clean up the vCPU interrupt presenters first.
>> + */
>> + if (atomic_read(&kvm->online_vcpus) != 0) {
>> + /*
>> + * TODO: There is still a race window with the early
>> + * checks in kvmppc_xive_native_connect_vcpu()
>> + */
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvmppc_xive_native_cleanup_vcpu(vcpu);
>> + }
>> +
>> debugfs_remove(xive->dentry);
>>
>> pr_devel("Destroying xive native device\n");
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
@ 2019-03-13 8:17 ` Cédric Le Goater
0 siblings, 0 replies; 142+ messages in thread
From: Cédric Le Goater @ 2019-03-13 8:17 UTC (permalink / raw)
To: David Gibson; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
On 2/25/19 5:18 AM, David Gibson wrote:
> On Fri, Feb 22, 2019 at 12:28:40PM +0100, Cédric Le Goater wrote:
>> When the VM boots, the CAS negotiation process determines which
>> interrupt mode to use and invokes a machine reset. At that time, the
>> previous KVM interrupt device is 'destroyed' before the chosen one is
>> created. Upon destruction, the vCPU interrupt presenters using the KVM
>> device should be cleared first, the machine will reconnect them later
>> to the new device after it is created.
>>
>> When using the KVM device, there is still a race window with the early
>> checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
>>
>> Signed-off-by: Cédric Le Goater <clg@kaod.org>
>> ---
>> arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
>> arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
>> arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
>> 3 files changed, 72 insertions(+), 2 deletions(-)
>>
>> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
>> index f27ee57ab46e..81cdabf4295f 100644
>> --- a/arch/powerpc/kvm/book3s_xics.c
>> +++ b/arch/powerpc/kvm/book3s_xics.c
>> @@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
>> struct kvmppc_xics *xics = dev->private;
>> int i;
>> struct kvm *kvm = xics->kvm;
>> + struct kvm_vcpu *vcpu;
>> +
>> + /*
>> + * When destroying the VM, the vCPUs are destroyed first and
>> + * the vCPU list should be empty. If this is not the case,
>> + * then we are simply destroying the device and we should
>> + * clean up the vCPU interrupt presenters first.
>> + */
>> + if (atomic_read(&kvm->online_vcpus) != 0) {
>> + /*
>> + * call kick_all_cpus_sync() to ensure that all CPUs
>> + * have executed any pending interrupts
>> + */
>> + if (is_kvmppc_hv_enabled(kvm))
>> + kick_all_cpus_sync();
>> +
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvmppc_xics_free_icp(vcpu);
>> + }
>>
>> debugfs_remove(xics->dentry);
>>
>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>> index 7a14512b8944..0a1c11d6881c 100644
>> --- a/arch/powerpc/kvm/book3s_xive.c
>> +++ b/arch/powerpc/kvm/book3s_xive.c
>> @@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>> {
>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>> - struct kvmppc_xive *xive = xc->xive;
>> + struct kvmppc_xive *xive;
>> int i;
>>
>> + if (!kvmppc_xics_enabled(vcpu))
>
> This should be kvmppc_xive_enabled(), no?
This is the KVM XICS-on-XIVE device and its IRQ type is KVMPPC_IRQ_XICS.
So this is correct :/
May be we should introduce a KVMPPC_IRQ_XICS_ON_XIVE macro to clarify.
>
>> + return;
>> +
>> + if (!xc)
>> + return;
>> +
>> pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
>>
>> + xive = xc->xive;
>> +
>> /* Ensure no interrupt is still routed to that VP */
>> xc->valid = false;
>> kvmppc_xive_disable_vcpu_interrupts(vcpu);
>> @@ -1146,6 +1154,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>> }
>> /* Free the VP */
>> kfree(xc);
>> +
>> + /* Cleanup the vcpu */
>> + vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
>> + vcpu->arch.xive_vcpu = NULL;
>> }
>>
>> int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> @@ -1163,7 +1175,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
>> }
>> if (xive->kvm != vcpu->kvm)
>> return -EPERM;
>> - if (vcpu->arch.irq_type)
>> + if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
>> return -EBUSY;
>> if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
>> pr_devel("Duplicate !\n");
>> @@ -1833,8 +1845,31 @@ static void kvmppc_xive_free(struct kvm_device *dev)
>> {
>> struct kvmppc_xive *xive = dev->private;
>> struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> int i;
>>
>> + /*
>> + * When destroying the VM, the vCPUs are destroyed first and
>> + * the vCPU list should be empty. If this is not the case,
>> + * then we are simply destroying the device and we should
>> + * clean up the vCPU interrupt presenters first.
>> + */
>> + if (atomic_read(&kvm->online_vcpus) != 0) {
>> + /*
>> + * call kick_all_cpus_sync() to ensure that all CPUs
>> + * have executed any pending interrupts
>> + */
>> + if (is_kvmppc_hv_enabled(kvm))
>> + kick_all_cpus_sync();
>> +
>> + /*
>> + * TODO: There is still a race window with the early
>> + * checks in kvmppc_native_connect_vcpu()
>> + */
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvmppc_xive_cleanup_vcpu(vcpu);
>> + }
>> +
>> debugfs_remove(xive->dentry);
>>
>> if (kvm)
>> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
>> index bf60870144f1..c0655164d9af 100644
>> --- a/arch/powerpc/kvm/book3s_xive_native.c
>> +++ b/arch/powerpc/kvm/book3s_xive_native.c
>> @@ -909,8 +909,24 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
>> {
>> struct kvmppc_xive *xive = dev->private;
>> struct kvm *kvm = xive->kvm;
>> + struct kvm_vcpu *vcpu;
>> int i;
>>
>> + /*
>> + * When destroying the VM, the vCPUs are destroyed first and
>> + * the vCPU list should be empty. If this is not the case,
>> + * then we are simply destroying the device and we should
>> + * clean up the vCPU interrupt presenters first.
>> + */
>> + if (atomic_read(&kvm->online_vcpus) != 0) {
>> + /*
>> + * TODO: There is still a race window with the early
>> + * checks in kvmppc_xive_native_connect_vcpu()
>> + */
>> + kvm_for_each_vcpu(i, vcpu, kvm)
>> + kvmppc_xive_native_cleanup_vcpu(vcpu);
>> + }
>> +
>> debugfs_remove(xive->dentry);
>>
>> pr_devel("Destroying xive native device\n");
>
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
2019-03-13 8:17 ` Cédric Le Goater
@ 2019-03-14 2:26 ` David Gibson
-1 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:26 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3263 bytes --]
On Wed, Mar 13, 2019 at 09:17:17AM +0100, Cédric Le Goater wrote:
> On 2/25/19 5:18 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:40PM +0100, Cédric Le Goater wrote:
> >> When the VM boots, the CAS negotiation process determines which
> >> interrupt mode to use and invokes a machine reset. At that time, the
> >> previous KVM interrupt device is 'destroyed' before the chosen one is
> >> created. Upon destruction, the vCPU interrupt presenters using the KVM
> >> device should be cleared first, the machine will reconnect them later
> >> to the new device after it is created.
> >>
> >> When using the KVM device, there is still a race window with the early
> >> checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
> >> arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
> >> arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
> >> 3 files changed, 72 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
> >> index f27ee57ab46e..81cdabf4295f 100644
> >> --- a/arch/powerpc/kvm/book3s_xics.c
> >> +++ b/arch/powerpc/kvm/book3s_xics.c
> >> @@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
> >> struct kvmppc_xics *xics = dev->private;
> >> int i;
> >> struct kvm *kvm = xics->kvm;
> >> + struct kvm_vcpu *vcpu;
> >> +
> >> + /*
> >> + * When destroying the VM, the vCPUs are destroyed first and
> >> + * the vCPU list should be empty. If this is not the case,
> >> + * then we are simply destroying the device and we should
> >> + * clean up the vCPU interrupt presenters first.
> >> + */
> >> + if (atomic_read(&kvm->online_vcpus) != 0) {
> >> + /*
> >> + * call kick_all_cpus_sync() to ensure that all CPUs
> >> + * have executed any pending interrupts
> >> + */
> >> + if (is_kvmppc_hv_enabled(kvm))
> >> + kick_all_cpus_sync();
> >> +
> >> + kvm_for_each_vcpu(i, vcpu, kvm)
> >> + kvmppc_xics_free_icp(vcpu);
> >> + }
> >>
> >> debugfs_remove(xics->dentry);
> >>
> >> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> >> index 7a14512b8944..0a1c11d6881c 100644
> >> --- a/arch/powerpc/kvm/book3s_xive.c
> >> +++ b/arch/powerpc/kvm/book3s_xive.c
> >> @@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> >> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> >> {
> >> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> - struct kvmppc_xive *xive = xc->xive;
> >> + struct kvmppc_xive *xive;
> >> int i;
> >>
> >> + if (!kvmppc_xics_enabled(vcpu))
> >
> > This should be kvmppc_xive_enabled(), no?
>
> This is the KVM XICS-on-XIVE device and its IRQ type is KVMPPC_IRQ_XICS.
> So this is correct :/
Ah, right, sorry.
> May be we should introduce a KVMPPC_IRQ_XICS_ON_XIVE macro to
> clarify.
Yeah, maybe.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread* Re: [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters
@ 2019-03-14 2:26 ` David Gibson
0 siblings, 0 replies; 142+ messages in thread
From: David Gibson @ 2019-03-14 2:26 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: kvm, kvm-ppc, Paul Mackerras, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 3263 bytes --]
On Wed, Mar 13, 2019 at 09:17:17AM +0100, Cédric Le Goater wrote:
> On 2/25/19 5:18 AM, David Gibson wrote:
> > On Fri, Feb 22, 2019 at 12:28:40PM +0100, Cédric Le Goater wrote:
> >> When the VM boots, the CAS negotiation process determines which
> >> interrupt mode to use and invokes a machine reset. At that time, the
> >> previous KVM interrupt device is 'destroyed' before the chosen one is
> >> created. Upon destruction, the vCPU interrupt presenters using the KVM
> >> device should be cleared first, the machine will reconnect them later
> >> to the new device after it is created.
> >>
> >> When using the KVM device, there is still a race window with the early
> >> checks in kvmppc_native_connect_vcpu(). Yet to be fixed.
> >>
> >> Signed-off-by: Cédric Le Goater <clg@kaod.org>
> >> ---
> >> arch/powerpc/kvm/book3s_xics.c | 19 +++++++++++++
> >> arch/powerpc/kvm/book3s_xive.c | 39 +++++++++++++++++++++++++--
> >> arch/powerpc/kvm/book3s_xive_native.c | 16 +++++++++++
> >> 3 files changed, 72 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/arch/powerpc/kvm/book3s_xics.c b/arch/powerpc/kvm/book3s_xics.c
> >> index f27ee57ab46e..81cdabf4295f 100644
> >> --- a/arch/powerpc/kvm/book3s_xics.c
> >> +++ b/arch/powerpc/kvm/book3s_xics.c
> >> @@ -1342,6 +1342,25 @@ static void kvmppc_xics_free(struct kvm_device *dev)
> >> struct kvmppc_xics *xics = dev->private;
> >> int i;
> >> struct kvm *kvm = xics->kvm;
> >> + struct kvm_vcpu *vcpu;
> >> +
> >> + /*
> >> + * When destroying the VM, the vCPUs are destroyed first and
> >> + * the vCPU list should be empty. If this is not the case,
> >> + * then we are simply destroying the device and we should
> >> + * clean up the vCPU interrupt presenters first.
> >> + */
> >> + if (atomic_read(&kvm->online_vcpus) != 0) {
> >> + /*
> >> + * call kick_all_cpus_sync() to ensure that all CPUs
> >> + * have executed any pending interrupts
> >> + */
> >> + if (is_kvmppc_hv_enabled(kvm))
> >> + kick_all_cpus_sync();
> >> +
> >> + kvm_for_each_vcpu(i, vcpu, kvm)
> >> + kvmppc_xics_free_icp(vcpu);
> >> + }
> >>
> >> debugfs_remove(xics->dentry);
> >>
> >> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> >> index 7a14512b8944..0a1c11d6881c 100644
> >> --- a/arch/powerpc/kvm/book3s_xive.c
> >> +++ b/arch/powerpc/kvm/book3s_xive.c
> >> @@ -1105,11 +1105,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> >> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> >> {
> >> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> >> - struct kvmppc_xive *xive = xc->xive;
> >> + struct kvmppc_xive *xive;
> >> int i;
> >>
> >> + if (!kvmppc_xics_enabled(vcpu))
> >
> > This should be kvmppc_xive_enabled(), no?
>
> This is the KVM XICS-on-XIVE device and its IRQ type is KVMPPC_IRQ_XICS.
> So this is correct :/
Ah, right, sorry.
> May be we should introduce a KVMPPC_IRQ_XICS_ON_XIVE macro to
> clarify.
Yeah, maybe.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 142+ messages in thread