* [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Hello,
On the POWER9 processor, the XIVE interrupt controller can control
interrupt sources using MMIOs to trigger events, to EOI or to turn off
the sources. Priority management and interrupt acknowledgment is also
controlled by MMIO in the CPU presenter sub-engine.
PowerNV/baremetal Linux runs natively under XIVE but sPAPR guests need
special support from the hypervisor to do the same. This is called the
XIVE native exploitation mode and today, it can be activated under the
PowerPC Hypervisor, pHyp. However, Linux/KVM lacks XIVE native support
and still offers the old interrupt mode interface using a KVM device
implementing the XICS hcalls over XIVE.
The following series is proposal to add the same support under KVM.
A new KVM device is introduced for the XIVE native exploitation
mode. It reuses most of the XICS-over-XIVE glue implementation
structures which are internal to KVM but has a completely different
interface. A set of KVM device ioctls provide support for the
hypervisor calls, all handled in QEMU, to configure the sources and
the event queues. From there, all interrupt control is transferred to
the guest which can use MMIOs.
These MMIO regions (ESB and TIMA) are exposed to guests in QEMU,
similarly to VFIO, and the associated VMAs are populated dynamically
with the appropriate pages using a fault handler. These are now
implemented using mmap()s of the KVM device fd.
Migration has its own specific needs regarding memory. The patchset
provides a specific control to quiesce XIVE before capturing the
memory. The save and restore of the internal state is based on the
same ioctls used for the hcalls.
On a POWER9 sPAPR machine, the Client Architecture Support (CAS)
negotiation process determines whether the guest operates with a
interrupt controller using the XICS legacy model, as found on POWER8,
or in XIVE exploitation mode. Which means that the KVM interrupt
device should be created at run-time, after the machine has started.
This requires extra support from KVM to destroy KVM devices. It is
introduced at the end of the patchset and requires some attention.
This is based on Linux 5.1-rc4 and is a candidate for 5.2. The OPAL
patches have been merged now.
GitHub trees available here :
QEMU sPAPR:
https://github.com/legoater/qemu/commits/xive-next
Linux/KVM:
https://github.com/legoater/linux/commits/xive-5.1
Thanks,
C.
Caveats :
- We should introduce a set of definitions common to XIVE and XICS
- The XICS-over-XIVE device file book3s_xive.c could be renamed to
book3s_xics_on_xive.c or book3s_xics_p9.c
- The XICS-over-XIVE device still has locking issues in the setup.
- xc->valid is not useful
- xc->xive is not very useful either
Changes since v4:
- add check on EQ page alignment
- add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
- add documentation in Documentation/virtual/kvm/api.txt
- remove 'destroy' method
- introduce a 'release' device operation called when device fd is
closed.
- introduce a 'xive_devices' array under the VM to store kvmppc_xive
objects until VM is destroyed.
Changes since v3:
- removed a couple of useless includes
- fix the test ont the initial setting of the EQ toggle bit : 0 -> 1
- renamed qsize to qshift
- renamed qpage to qaddr
- checked host page size
- limited flags to KVM_XIVE_EQ_ALWAYS_NOTIFY to fit sPAPR specs
- Fixed xive_timaval description in documentation
Changes since v2:
- removed extra OPAL call definitions
- removed ->q_order setting. Only useful in the XICS-on-XIVE KVM
device which allocates the EQs on behalf of the guest.
- returned -ENXIO when VP base is invalid
- made use of the xive_vp() macro to compute VP identifiers
- reworked locking in kvmppc_xive_native_connect_vcpu() to fix races
- stop advertising KVM_CAP_PPC_IRQ_XIVE as support is not fully
available yet
- fixed comment on XIVE IRQ number space
- removed usage of the __x_* macros
- fixed locking on source block
- fixed comments on the KVM device attribute definitions
- handled MASKED EAS configuration
- fixed check on supported EQ size to restrict to 64K pages
- checked kvm_eq.flags that need to be zero
- removed the OPAL call when EQ qtoggle bit and index are zero.
- reduced the size of kvmppc_one_reg timaval attribute to two u64s
- stopped returning of the OS CAM line value
Changes since v1:
- Better documentation (was missing)
- Nested support. XIVE not advertised on non PowerNV platforms. This
is a good way to test the fallback on QEMU emulated devices.
- ESB and TIMA special mapping done using the KVM device fd
- All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
- Reworked of the KVM device ioctl controls to support hcalls and
migration needs to capture/save states
- Merged the control syncing XIVE and marking the EQ page dirty
- Fixed passthrough support using the KVM device file address_space
to clear the ESB pages from the mapping
- Misc enhancements and fixes
Cédric Le Goater (16):
powerpc/xive: add OPAL extensions for the XIVE native exploitation
support
KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a new capability
KVM_CAP_PPC_IRQ_XIVE
KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
KVM: PPC: Book3S HV: XIVE: add a control to configure a source
KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
KVM: PPC: Book3S HV: XIVE: add a global reset control
KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
KVM: introduce a 'mmap' method for KVM devices
KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
KVM: PPC: Book3S HV: XIVE: add passthrough support
KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
arch/powerpc/include/asm/kvm_host.h | 3 +
arch/powerpc/include/asm/kvm_ppc.h | 32 +
arch/powerpc/include/asm/opal-api.h | 7 +-
arch/powerpc/include/asm/opal.h | 7 +
arch/powerpc/include/asm/xive.h | 17 +
arch/powerpc/include/uapi/asm/kvm.h | 46 +
arch/powerpc/kvm/book3s_xive.h | 37 +
include/linux/kvm_host.h | 2 +
include/uapi/linux/kvm.h | 3 +
arch/powerpc/kvm/book3s.c | 31 +-
arch/powerpc/kvm/book3s_xive.c | 230 +++-
arch/powerpc/kvm/book3s_xive_native.c | 1243 ++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 37 +
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/sysdev/xive/native.c | 110 ++
virt/kvm/kvm_main.c | 24 +
Documentation/virtual/kvm/api.txt | 10 +
Documentation/virtual/kvm/devices/xive.txt | 197 ++++
arch/powerpc/kvm/Makefile | 2 +-
19 files changed, 1980 insertions(+), 61 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
--
2.20.1
^ permalink raw reply [flat|nested] 81+ messages in thread
* [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Hello,
On the POWER9 processor, the XIVE interrupt controller can control
interrupt sources using MMIOs to trigger events, to EOI or to turn off
the sources. Priority management and interrupt acknowledgment is also
controlled by MMIO in the CPU presenter sub-engine.
PowerNV/baremetal Linux runs natively under XIVE but sPAPR guests need
special support from the hypervisor to do the same. This is called the
XIVE native exploitation mode and today, it can be activated under the
PowerPC Hypervisor, pHyp. However, Linux/KVM lacks XIVE native support
and still offers the old interrupt mode interface using a KVM device
implementing the XICS hcalls over XIVE.
The following series is proposal to add the same support under KVM.
A new KVM device is introduced for the XIVE native exploitation
mode. It reuses most of the XICS-over-XIVE glue implementation
structures which are internal to KVM but has a completely different
interface. A set of KVM device ioctls provide support for the
hypervisor calls, all handled in QEMU, to configure the sources and
the event queues. From there, all interrupt control is transferred to
the guest which can use MMIOs.
These MMIO regions (ESB and TIMA) are exposed to guests in QEMU,
similarly to VFIO, and the associated VMAs are populated dynamically
with the appropriate pages using a fault handler. These are now
implemented using mmap()s of the KVM device fd.
Migration has its own specific needs regarding memory. The patchset
provides a specific control to quiesce XIVE before capturing the
memory. The save and restore of the internal state is based on the
same ioctls used for the hcalls.
On a POWER9 sPAPR machine, the Client Architecture Support (CAS)
negotiation process determines whether the guest operates with a
interrupt controller using the XICS legacy model, as found on POWER8,
or in XIVE exploitation mode. Which means that the KVM interrupt
device should be created at run-time, after the machine has started.
This requires extra support from KVM to destroy KVM devices. It is
introduced at the end of the patchset and requires some attention.
This is based on Linux 5.1-rc4 and is a candidate for 5.2. The OPAL
patches have been merged now.
GitHub trees available here :
QEMU sPAPR:
https://github.com/legoater/qemu/commits/xive-next
Linux/KVM:
https://github.com/legoater/linux/commits/xive-5.1
Thanks,
C.
Caveats :
- We should introduce a set of definitions common to XIVE and XICS
- The XICS-over-XIVE device file book3s_xive.c could be renamed to
book3s_xics_on_xive.c or book3s_xics_p9.c
- The XICS-over-XIVE device still has locking issues in the setup.
- xc->valid is not useful
- xc->xive is not very useful either
Changes since v4:
- add check on EQ page alignment
- add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
- add documentation in Documentation/virtual/kvm/api.txt
- remove 'destroy' method
- introduce a 'release' device operation called when device fd is
closed.
- introduce a 'xive_devices' array under the VM to store kvmppc_xive
objects until VM is destroyed.
Changes since v3:
- removed a couple of useless includes
- fix the test ont the initial setting of the EQ toggle bit : 0 -> 1
- renamed qsize to qshift
- renamed qpage to qaddr
- checked host page size
- limited flags to KVM_XIVE_EQ_ALWAYS_NOTIFY to fit sPAPR specs
- Fixed xive_timaval description in documentation
Changes since v2:
- removed extra OPAL call definitions
- removed ->q_order setting. Only useful in the XICS-on-XIVE KVM
device which allocates the EQs on behalf of the guest.
- returned -ENXIO when VP base is invalid
- made use of the xive_vp() macro to compute VP identifiers
- reworked locking in kvmppc_xive_native_connect_vcpu() to fix races
- stop advertising KVM_CAP_PPC_IRQ_XIVE as support is not fully
available yet
- fixed comment on XIVE IRQ number space
- removed usage of the __x_* macros
- fixed locking on source block
- fixed comments on the KVM device attribute definitions
- handled MASKED EAS configuration
- fixed check on supported EQ size to restrict to 64K pages
- checked kvm_eq.flags that need to be zero
- removed the OPAL call when EQ qtoggle bit and index are zero.
- reduced the size of kvmppc_one_reg timaval attribute to two u64s
- stopped returning of the OS CAM line value
Changes since v1:
- Better documentation (was missing)
- Nested support. XIVE not advertised on non PowerNV platforms. This
is a good way to test the fallback on QEMU emulated devices.
- ESB and TIMA special mapping done using the KVM device fd
- All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
- Reworked of the KVM device ioctl controls to support hcalls and
migration needs to capture/save states
- Merged the control syncing XIVE and marking the EQ page dirty
- Fixed passthrough support using the KVM device file address_space
to clear the ESB pages from the mapping
- Misc enhancements and fixes
Cédric Le Goater (16):
powerpc/xive: add OPAL extensions for the XIVE native exploitation
support
KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a new capability
KVM_CAP_PPC_IRQ_XIVE
KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
KVM: PPC: Book3S HV: XIVE: add a control to configure a source
KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
KVM: PPC: Book3S HV: XIVE: add a global reset control
KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
KVM: introduce a 'mmap' method for KVM devices
KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
KVM: PPC: Book3S HV: XIVE: add passthrough support
KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
arch/powerpc/include/asm/kvm_host.h | 3 +
arch/powerpc/include/asm/kvm_ppc.h | 32 +
arch/powerpc/include/asm/opal-api.h | 7 +-
arch/powerpc/include/asm/opal.h | 7 +
arch/powerpc/include/asm/xive.h | 17 +
arch/powerpc/include/uapi/asm/kvm.h | 46 +
arch/powerpc/kvm/book3s_xive.h | 37 +
include/linux/kvm_host.h | 2 +
include/uapi/linux/kvm.h | 3 +
arch/powerpc/kvm/book3s.c | 31 +-
arch/powerpc/kvm/book3s_xive.c | 230 +++-
arch/powerpc/kvm/book3s_xive_native.c | 1243 ++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 37 +
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/sysdev/xive/native.c | 110 ++
virt/kvm/kvm_main.c | 24 +
Documentation/virtual/kvm/api.txt | 10 +
Documentation/virtual/kvm/devices/xive.txt | 197 ++++
arch/powerpc/kvm/Makefile | 2 +-
19 files changed, 1980 insertions(+), 61 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
--
2.20.1
^ permalink raw reply [flat|nested] 81+ messages in thread
* [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Hello,
On the POWER9 processor, the XIVE interrupt controller can control
interrupt sources using MMIOs to trigger events, to EOI or to turn off
the sources. Priority management and interrupt acknowledgment is also
controlled by MMIO in the CPU presenter sub-engine.
PowerNV/baremetal Linux runs natively under XIVE but sPAPR guests need
special support from the hypervisor to do the same. This is called the
XIVE native exploitation mode and today, it can be activated under the
PowerPC Hypervisor, pHyp. However, Linux/KVM lacks XIVE native support
and still offers the old interrupt mode interface using a KVM device
implementing the XICS hcalls over XIVE.
The following series is proposal to add the same support under KVM.
A new KVM device is introduced for the XIVE native exploitation
mode. It reuses most of the XICS-over-XIVE glue implementation
structures which are internal to KVM but has a completely different
interface. A set of KVM device ioctls provide support for the
hypervisor calls, all handled in QEMU, to configure the sources and
the event queues. From there, all interrupt control is transferred to
the guest which can use MMIOs.
These MMIO regions (ESB and TIMA) are exposed to guests in QEMU,
similarly to VFIO, and the associated VMAs are populated dynamically
with the appropriate pages using a fault handler. These are now
implemented using mmap()s of the KVM device fd.
Migration has its own specific needs regarding memory. The patchset
provides a specific control to quiesce XIVE before capturing the
memory. The save and restore of the internal state is based on the
same ioctls used for the hcalls.
On a POWER9 sPAPR machine, the Client Architecture Support (CAS)
negotiation process determines whether the guest operates with a
interrupt controller using the XICS legacy model, as found on POWER8,
or in XIVE exploitation mode. Which means that the KVM interrupt
device should be created at run-time, after the machine has started.
This requires extra support from KVM to destroy KVM devices. It is
introduced at the end of the patchset and requires some attention.
This is based on Linux 5.1-rc4 and is a candidate for 5.2. The OPAL
patches have been merged now.
GitHub trees available here :
QEMU sPAPR:
https://github.com/legoater/qemu/commits/xive-next
Linux/KVM:
https://github.com/legoater/linux/commits/xive-5.1
Thanks,
C.
Caveats :
- We should introduce a set of definitions common to XIVE and XICS
- The XICS-over-XIVE device file book3s_xive.c could be renamed to
book3s_xics_on_xive.c or book3s_xics_p9.c
- The XICS-over-XIVE device still has locking issues in the setup.
- xc->valid is not useful
- xc->xive is not very useful either
Changes since v4:
- add check on EQ page alignment
- add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
- add documentation in Documentation/virtual/kvm/api.txt
- remove 'destroy' method
- introduce a 'release' device operation called when device fd is
closed.
- introduce a 'xive_devices' array under the VM to store kvmppc_xive
objects until VM is destroyed.
Changes since v3:
- removed a couple of useless includes
- fix the test ont the initial setting of the EQ toggle bit : 0 -> 1
- renamed qsize to qshift
- renamed qpage to qaddr
- checked host page size
- limited flags to KVM_XIVE_EQ_ALWAYS_NOTIFY to fit sPAPR specs
- Fixed xive_timaval description in documentation
Changes since v2:
- removed extra OPAL call definitions
- removed ->q_order setting. Only useful in the XICS-on-XIVE KVM
device which allocates the EQs on behalf of the guest.
- returned -ENXIO when VP base is invalid
- made use of the xive_vp() macro to compute VP identifiers
- reworked locking in kvmppc_xive_native_connect_vcpu() to fix races
- stop advertising KVM_CAP_PPC_IRQ_XIVE as support is not fully
available yet
- fixed comment on XIVE IRQ number space
- removed usage of the __x_* macros
- fixed locking on source block
- fixed comments on the KVM device attribute definitions
- handled MASKED EAS configuration
- fixed check on supported EQ size to restrict to 64K pages
- checked kvm_eq.flags that need to be zero
- removed the OPAL call when EQ qtoggle bit and index are zero.
- reduced the size of kvmppc_one_reg timaval attribute to two u64s
- stopped returning of the OS CAM line value
Changes since v1:
- Better documentation (was missing)
- Nested support. XIVE not advertised on non PowerNV platforms. This
is a good way to test the fallback on QEMU emulated devices.
- ESB and TIMA special mapping done using the KVM device fd
- All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
- Reworked of the KVM device ioctl controls to support hcalls and
migration needs to capture/save states
- Merged the control syncing XIVE and marking the EQ page dirty
- Fixed passthrough support using the KVM device file address_space
to clear the ESB pages from the mapping
- Misc enhancements and fixes
Cédric Le Goater (16):
powerpc/xive: add OPAL extensions for the XIVE native exploitation
support
KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a new capability
KVM_CAP_PPC_IRQ_XIVE
KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
KVM: PPC: Book3S HV: XIVE: add a control to configure a source
KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
KVM: PPC: Book3S HV: XIVE: add a global reset control
KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
KVM: introduce a 'mmap' method for KVM devices
KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
KVM: PPC: Book3S HV: XIVE: add passthrough support
KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
arch/powerpc/include/asm/kvm_host.h | 3 +
arch/powerpc/include/asm/kvm_ppc.h | 32 +
arch/powerpc/include/asm/opal-api.h | 7 +-
arch/powerpc/include/asm/opal.h | 7 +
arch/powerpc/include/asm/xive.h | 17 +
arch/powerpc/include/uapi/asm/kvm.h | 46 +
arch/powerpc/kvm/book3s_xive.h | 37 +
include/linux/kvm_host.h | 2 +
include/uapi/linux/kvm.h | 3 +
arch/powerpc/kvm/book3s.c | 31 +-
arch/powerpc/kvm/book3s_xive.c | 230 +++-
arch/powerpc/kvm/book3s_xive_native.c | 1243 ++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 37 +
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/sysdev/xive/native.c | 110 ++
virt/kvm/kvm_main.c | 24 +
Documentation/virtual/kvm/api.txt | 10 +
Documentation/virtual/kvm/devices/xive.txt | 197 ++++
arch/powerpc/kvm/Makefile | 2 +-
19 files changed, 1980 insertions(+), 61 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
--
2.20.1
^ permalink raw reply [flat|nested] 81+ messages in thread
* [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The support for XIVE native exploitation mode in Linux/KVM needs a
couple more OPAL calls to get and set the state of the XIVE internal
structures being used by a sPAPR guest.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/opal-api.h | 7 +-
arch/powerpc/include/asm/opal.h | 7 ++
arch/powerpc/include/asm/xive.h | 14 +++
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/sysdev/xive/native.c | 99 ++++++++++++++++++++++
5 files changed, 127 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 870fb7b239ea..e1d118ac61dc 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -186,8 +186,8 @@
#define OPAL_XIVE_FREE_IRQ 140
#define OPAL_XIVE_SYNC 141
#define OPAL_XIVE_DUMP 142
-#define OPAL_XIVE_RESERVED3 143
-#define OPAL_XIVE_RESERVED4 144
+#define OPAL_XIVE_GET_QUEUE_STATE 143
+#define OPAL_XIVE_SET_QUEUE_STATE 144
#define OPAL_SIGNAL_SYSTEM_RESET 145
#define OPAL_NPU_INIT_CONTEXT 146
#define OPAL_NPU_DESTROY_CONTEXT 147
@@ -210,7 +210,8 @@
#define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
#define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
#define OPAL_NX_COPROC_INIT 167
-#define OPAL_LAST 167
+#define OPAL_XIVE_GET_VP_STATE 170
+#define OPAL_LAST 170
#define QUIESCE_HOLD 1 /* Spin all calls at entry */
#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index a55b01c90bb1..4e978d4dea5c 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
int64_t opal_xive_free_irq(uint32_t girq);
int64_t opal_xive_sync(uint32_t type, uint32_t id);
int64_t opal_xive_dump(uint32_t type, uint32_t id);
+int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
+ __be32 *out_qtoggle,
+ __be32 *out_qindex);
+int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
+ uint32_t qtoggle,
+ uint32_t qindex);
+int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
uint64_t desc, uint16_t pe_number);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 3c704f5dd3ae..b579a943407b 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
extern void xive_native_sync_source(u32 hw_irq);
+extern void xive_native_sync_queue(u32 hw_irq);
extern bool is_xive_irq(struct irq_chip *chip);
extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
extern int xive_native_disable_vp(u32 vp_id);
extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
extern bool xive_native_has_single_escalation(void);
+extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags);
+
+extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
+ u32 *qindex);
+extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
+ u32 qindex);
+extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
+
#else
static inline bool xive_enabled(void) { return false; }
diff --git a/arch/powerpc/platforms/powernv/opal-call.c b/arch/powerpc/platforms/powernv/opal-call.c
index daad8c45c8e7..7472244e7f30 100644
--- a/arch/powerpc/platforms/powernv/opal-call.c
+++ b/arch/powerpc/platforms/powernv/opal-call.c
@@ -260,6 +260,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
+OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
+OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
+OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 1ca127d052a6..0c037e933e55 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
}
EXPORT_SYMBOL_GPL(xive_native_sync_source);
+void xive_native_sync_queue(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
+}
+EXPORT_SYMBOL_GPL(xive_native_sync_queue);
+
static const struct xive_ops xive_native_ops = {
.populate_irq_data = xive_native_populate_irq_data,
.configure_irq = xive_native_configure_irq,
@@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
return xive_has_single_esc;
}
EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
+
+int xive_native_get_queue_info(u32 vp_id, u32 prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags)
+{
+ __be64 qpage;
+ __be64 qsize;
+ __be64 qeoi_page;
+ __be32 escalate_irq;
+ __be64 qflags;
+ s64 rc;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc) {
+ pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (out_qpage)
+ *out_qpage = be64_to_cpu(qpage);
+ if (out_qsize)
+ *out_qsize = be32_to_cpu(qsize);
+ if (out_qeoi_page)
+ *out_qeoi_page = be64_to_cpu(qeoi_page);
+ if (out_escalate_irq)
+ *out_escalate_irq = be32_to_cpu(escalate_irq);
+ if (out_qflags)
+ *out_qflags = be64_to_cpu(qflags);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
+
+int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
+{
+ __be32 opal_qtoggle;
+ __be32 opal_qindex;
+ s64 rc;
+
+ rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
+ &opal_qindex);
+ if (rc) {
+ pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (qtoggle)
+ *qtoggle = be32_to_cpu(opal_qtoggle);
+ if (qindex)
+ *qindex = be32_to_cpu(opal_qindex);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
+
+int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
+{
+ s64 rc;
+
+ rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
+ if (rc) {
+ pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
+
+int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
+{
+ __be64 state;
+ s64 rc;
+
+ rc = opal_xive_get_vp_state(vp_id, &state);
+ if (rc) {
+ pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
+ vp_id, rc);
+ return -EIO;
+ }
+
+ if (out_state)
+ *out_state = be64_to_cpu(state);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The support for XIVE native exploitation mode in Linux/KVM needs a
couple more OPAL calls to get and set the state of the XIVE internal
structures being used by a sPAPR guest.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/opal-api.h | 7 +-
arch/powerpc/include/asm/opal.h | 7 ++
arch/powerpc/include/asm/xive.h | 14 +++
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/sysdev/xive/native.c | 99 ++++++++++++++++++++++
5 files changed, 127 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 870fb7b239ea..e1d118ac61dc 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -186,8 +186,8 @@
#define OPAL_XIVE_FREE_IRQ 140
#define OPAL_XIVE_SYNC 141
#define OPAL_XIVE_DUMP 142
-#define OPAL_XIVE_RESERVED3 143
-#define OPAL_XIVE_RESERVED4 144
+#define OPAL_XIVE_GET_QUEUE_STATE 143
+#define OPAL_XIVE_SET_QUEUE_STATE 144
#define OPAL_SIGNAL_SYSTEM_RESET 145
#define OPAL_NPU_INIT_CONTEXT 146
#define OPAL_NPU_DESTROY_CONTEXT 147
@@ -210,7 +210,8 @@
#define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
#define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
#define OPAL_NX_COPROC_INIT 167
-#define OPAL_LAST 167
+#define OPAL_XIVE_GET_VP_STATE 170
+#define OPAL_LAST 170
#define QUIESCE_HOLD 1 /* Spin all calls at entry */
#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index a55b01c90bb1..4e978d4dea5c 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
int64_t opal_xive_free_irq(uint32_t girq);
int64_t opal_xive_sync(uint32_t type, uint32_t id);
int64_t opal_xive_dump(uint32_t type, uint32_t id);
+int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
+ __be32 *out_qtoggle,
+ __be32 *out_qindex);
+int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
+ uint32_t qtoggle,
+ uint32_t qindex);
+int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
uint64_t desc, uint16_t pe_number);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 3c704f5dd3ae..b579a943407b 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
extern void xive_native_sync_source(u32 hw_irq);
+extern void xive_native_sync_queue(u32 hw_irq);
extern bool is_xive_irq(struct irq_chip *chip);
extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
extern int xive_native_disable_vp(u32 vp_id);
extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
extern bool xive_native_has_single_escalation(void);
+extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags);
+
+extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
+ u32 *qindex);
+extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
+ u32 qindex);
+extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
+
#else
static inline bool xive_enabled(void) { return false; }
diff --git a/arch/powerpc/platforms/powernv/opal-call.c b/arch/powerpc/platforms/powernv/opal-call.c
index daad8c45c8e7..7472244e7f30 100644
--- a/arch/powerpc/platforms/powernv/opal-call.c
+++ b/arch/powerpc/platforms/powernv/opal-call.c
@@ -260,6 +260,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
+OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
+OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
+OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 1ca127d052a6..0c037e933e55 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
}
EXPORT_SYMBOL_GPL(xive_native_sync_source);
+void xive_native_sync_queue(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
+}
+EXPORT_SYMBOL_GPL(xive_native_sync_queue);
+
static const struct xive_ops xive_native_ops = {
.populate_irq_data = xive_native_populate_irq_data,
.configure_irq = xive_native_configure_irq,
@@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
return xive_has_single_esc;
}
EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
+
+int xive_native_get_queue_info(u32 vp_id, u32 prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags)
+{
+ __be64 qpage;
+ __be64 qsize;
+ __be64 qeoi_page;
+ __be32 escalate_irq;
+ __be64 qflags;
+ s64 rc;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc) {
+ pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (out_qpage)
+ *out_qpage = be64_to_cpu(qpage);
+ if (out_qsize)
+ *out_qsize = be32_to_cpu(qsize);
+ if (out_qeoi_page)
+ *out_qeoi_page = be64_to_cpu(qeoi_page);
+ if (out_escalate_irq)
+ *out_escalate_irq = be32_to_cpu(escalate_irq);
+ if (out_qflags)
+ *out_qflags = be64_to_cpu(qflags);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
+
+int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
+{
+ __be32 opal_qtoggle;
+ __be32 opal_qindex;
+ s64 rc;
+
+ rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
+ &opal_qindex);
+ if (rc) {
+ pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (qtoggle)
+ *qtoggle = be32_to_cpu(opal_qtoggle);
+ if (qindex)
+ *qindex = be32_to_cpu(opal_qindex);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
+
+int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
+{
+ s64 rc;
+
+ rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
+ if (rc) {
+ pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
+
+int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
+{
+ __be64 state;
+ s64 rc;
+
+ rc = opal_xive_get_vp_state(vp_id, &state);
+ if (rc) {
+ pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
+ vp_id, rc);
+ return -EIO;
+ }
+
+ if (out_state)
+ *out_state = be64_to_cpu(state);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The support for XIVE native exploitation mode in Linux/KVM needs a
couple more OPAL calls to get and set the state of the XIVE internal
structures being used by a sPAPR guest.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/opal-api.h | 7 +-
arch/powerpc/include/asm/opal.h | 7 ++
arch/powerpc/include/asm/xive.h | 14 +++
arch/powerpc/platforms/powernv/opal-call.c | 3 +
arch/powerpc/sysdev/xive/native.c | 99 ++++++++++++++++++++++
5 files changed, 127 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/include/asm/opal-api.h b/arch/powerpc/include/asm/opal-api.h
index 870fb7b239ea..e1d118ac61dc 100644
--- a/arch/powerpc/include/asm/opal-api.h
+++ b/arch/powerpc/include/asm/opal-api.h
@@ -186,8 +186,8 @@
#define OPAL_XIVE_FREE_IRQ 140
#define OPAL_XIVE_SYNC 141
#define OPAL_XIVE_DUMP 142
-#define OPAL_XIVE_RESERVED3 143
-#define OPAL_XIVE_RESERVED4 144
+#define OPAL_XIVE_GET_QUEUE_STATE 143
+#define OPAL_XIVE_SET_QUEUE_STATE 144
#define OPAL_SIGNAL_SYSTEM_RESET 145
#define OPAL_NPU_INIT_CONTEXT 146
#define OPAL_NPU_DESTROY_CONTEXT 147
@@ -210,7 +210,8 @@
#define OPAL_PCI_GET_PBCQ_TUNNEL_BAR 164
#define OPAL_PCI_SET_PBCQ_TUNNEL_BAR 165
#define OPAL_NX_COPROC_INIT 167
-#define OPAL_LAST 167
+#define OPAL_XIVE_GET_VP_STATE 170
+#define OPAL_LAST 170
#define QUIESCE_HOLD 1 /* Spin all calls at entry */
#define QUIESCE_REJECT 2 /* Fail all calls with OPAL_BUSY */
diff --git a/arch/powerpc/include/asm/opal.h b/arch/powerpc/include/asm/opal.h
index a55b01c90bb1..4e978d4dea5c 100644
--- a/arch/powerpc/include/asm/opal.h
+++ b/arch/powerpc/include/asm/opal.h
@@ -279,6 +279,13 @@ int64_t opal_xive_allocate_irq(uint32_t chip_id);
int64_t opal_xive_free_irq(uint32_t girq);
int64_t opal_xive_sync(uint32_t type, uint32_t id);
int64_t opal_xive_dump(uint32_t type, uint32_t id);
+int64_t opal_xive_get_queue_state(uint64_t vp, uint32_t prio,
+ __be32 *out_qtoggle,
+ __be32 *out_qindex);
+int64_t opal_xive_set_queue_state(uint64_t vp, uint32_t prio,
+ uint32_t qtoggle,
+ uint32_t qindex);
+int64_t opal_xive_get_vp_state(uint64_t vp, __be64 *out_w01);
int64_t opal_pci_set_p2p(uint64_t phb_init, uint64_t phb_target,
uint64_t desc, uint16_t pe_number);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 3c704f5dd3ae..b579a943407b 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -109,12 +109,26 @@ extern int xive_native_configure_queue(u32 vp_id, struct xive_q *q, u8 prio,
extern void xive_native_disable_queue(u32 vp_id, struct xive_q *q, u8 prio);
extern void xive_native_sync_source(u32 hw_irq);
+extern void xive_native_sync_queue(u32 hw_irq);
extern bool is_xive_irq(struct irq_chip *chip);
extern int xive_native_enable_vp(u32 vp_id, bool single_escalation);
extern int xive_native_disable_vp(u32 vp_id);
extern int xive_native_get_vp_info(u32 vp_id, u32 *out_cam_id, u32 *out_chip_id);
extern bool xive_native_has_single_escalation(void);
+extern int xive_native_get_queue_info(u32 vp_id, uint32_t prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags);
+
+extern int xive_native_get_queue_state(u32 vp_id, uint32_t prio, u32 *qtoggle,
+ u32 *qindex);
+extern int xive_native_set_queue_state(u32 vp_id, uint32_t prio, u32 qtoggle,
+ u32 qindex);
+extern int xive_native_get_vp_state(u32 vp_id, u64 *out_state);
+
#else
static inline bool xive_enabled(void) { return false; }
diff --git a/arch/powerpc/platforms/powernv/opal-call.c b/arch/powerpc/platforms/powernv/opal-call.c
index daad8c45c8e7..7472244e7f30 100644
--- a/arch/powerpc/platforms/powernv/opal-call.c
+++ b/arch/powerpc/platforms/powernv/opal-call.c
@@ -260,6 +260,9 @@ OPAL_CALL(opal_xive_get_vp_info, OPAL_XIVE_GET_VP_INFO);
OPAL_CALL(opal_xive_set_vp_info, OPAL_XIVE_SET_VP_INFO);
OPAL_CALL(opal_xive_sync, OPAL_XIVE_SYNC);
OPAL_CALL(opal_xive_dump, OPAL_XIVE_DUMP);
+OPAL_CALL(opal_xive_get_queue_state, OPAL_XIVE_GET_QUEUE_STATE);
+OPAL_CALL(opal_xive_set_queue_state, OPAL_XIVE_SET_QUEUE_STATE);
+OPAL_CALL(opal_xive_get_vp_state, OPAL_XIVE_GET_VP_STATE);
OPAL_CALL(opal_signal_system_reset, OPAL_SIGNAL_SYSTEM_RESET);
OPAL_CALL(opal_npu_init_context, OPAL_NPU_INIT_CONTEXT);
OPAL_CALL(opal_npu_destroy_context, OPAL_NPU_DESTROY_CONTEXT);
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 1ca127d052a6..0c037e933e55 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -437,6 +437,12 @@ void xive_native_sync_source(u32 hw_irq)
}
EXPORT_SYMBOL_GPL(xive_native_sync_source);
+void xive_native_sync_queue(u32 hw_irq)
+{
+ opal_xive_sync(XIVE_SYNC_QUEUE, hw_irq);
+}
+EXPORT_SYMBOL_GPL(xive_native_sync_queue);
+
static const struct xive_ops xive_native_ops = {
.populate_irq_data = xive_native_populate_irq_data,
.configure_irq = xive_native_configure_irq,
@@ -711,3 +717,96 @@ bool xive_native_has_single_escalation(void)
return xive_has_single_esc;
}
EXPORT_SYMBOL_GPL(xive_native_has_single_escalation);
+
+int xive_native_get_queue_info(u32 vp_id, u32 prio,
+ u64 *out_qpage,
+ u64 *out_qsize,
+ u64 *out_qeoi_page,
+ u32 *out_escalate_irq,
+ u64 *out_qflags)
+{
+ __be64 qpage;
+ __be64 qsize;
+ __be64 qeoi_page;
+ __be32 escalate_irq;
+ __be64 qflags;
+ s64 rc;
+
+ rc = opal_xive_get_queue_info(vp_id, prio, &qpage, &qsize,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc) {
+ pr_err("OPAL failed to get queue info for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (out_qpage)
+ *out_qpage = be64_to_cpu(qpage);
+ if (out_qsize)
+ *out_qsize = be32_to_cpu(qsize);
+ if (out_qeoi_page)
+ *out_qeoi_page = be64_to_cpu(qeoi_page);
+ if (out_escalate_irq)
+ *out_escalate_irq = be32_to_cpu(escalate_irq);
+ if (out_qflags)
+ *out_qflags = be64_to_cpu(qflags);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_info);
+
+int xive_native_get_queue_state(u32 vp_id, u32 prio, u32 *qtoggle, u32 *qindex)
+{
+ __be32 opal_qtoggle;
+ __be32 opal_qindex;
+ s64 rc;
+
+ rc = opal_xive_get_queue_state(vp_id, prio, &opal_qtoggle,
+ &opal_qindex);
+ if (rc) {
+ pr_err("OPAL failed to get queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ if (qtoggle)
+ *qtoggle = be32_to_cpu(opal_qtoggle);
+ if (qindex)
+ *qindex = be32_to_cpu(opal_qindex);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_queue_state);
+
+int xive_native_set_queue_state(u32 vp_id, u32 prio, u32 qtoggle, u32 qindex)
+{
+ s64 rc;
+
+ rc = opal_xive_set_queue_state(vp_id, prio, qtoggle, qindex);
+ if (rc) {
+ pr_err("OPAL failed to set queue state for VCPU %d/%d : %lld\n",
+ vp_id, prio, rc);
+ return -EIO;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_set_queue_state);
+
+int xive_native_get_vp_state(u32 vp_id, u64 *out_state)
+{
+ __be64 state;
+ s64 rc;
+
+ rc = opal_xive_get_vp_state(vp_id, &state);
+ if (rc) {
+ pr_err("OPAL failed to get vp state for VCPU %d : %lld\n",
+ vp_id, rc);
+ return -EIO;
+ }
+
+ if (out_state)
+ *out_state = be64_to_cpu(state);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(xive_native_get_vp_state);
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This is the basic framework for the new KVM device supporting the XIVE
native exploitation mode. The user interface exposes a new KVM device
to be created by QEMU, only available when running on a L0 hypervisor.
Support for nested guests is not available yet.
The XIVE device reuses the device structure of the XICS-on-XIVE device
as they have a lot in common. That could possibly change in the future
if the need arise.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 8 +
arch/powerpc/include/uapi/asm/kvm.h | 3 +
include/uapi/linux/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 7 +-
arch/powerpc/kvm/book3s_xive_native.c | 179 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 +++
arch/powerpc/kvm/Makefile | 2 +-
8 files changed, 219 insertions(+), 2 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index e6b5bb012ccb..008523224e7a 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -222,6 +222,7 @@ extern struct kvm_device_ops kvm_xics_ops;
struct kvmppc_xive;
struct kvmppc_xive_vcpu;
extern struct kvm_device_ops kvm_xive_ops;
+extern struct kvm_device_ops kvm_xive_native_ops;
struct kvmppc_passthru_irqmap;
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index ac22b28ae78d..f3383e76017a 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -594,6 +594,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+
+extern void kvmppc_xive_native_init_module(void);
+extern void kvmppc_xive_native_exit_module(void);
+
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
u32 priority) { return -1; }
@@ -617,6 +621,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+
+static inline void kvmppc_xive_native_init_module(void) { }
+static inline void kvmppc_xive_native_exit_module(void) { }
+
#endif /* CONFIG_KVM_XIVE */
#if defined(CONFIG_PPC_POWERNV) && defined(CONFIG_KVM_BOOK3S_64_HANDLER)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 26ca425f4c2c..be0ce1f17625 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,4 +677,7 @@ struct kvm_ppc_cpu_char {
#define KVM_XICS_PRESENTED (1ULL << 43)
#define KVM_XICS_QUEUED (1ULL << 44)
+/* POWER9 XIVE Native Interrupt Controller */
+#define KVM_DEV_XIVE_GRP_CTRL 1
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6d4ea4b6c922..e6368163d3a0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1211,6 +1211,8 @@ enum kvm_device_type {
#define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
KVM_DEV_TYPE_ARM_VGIC_ITS,
#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
+ KVM_DEV_TYPE_XIVE,
+#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
KVM_DEV_TYPE_MAX,
};
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 10c5579d20ce..7c3348fa27e1 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -1050,6 +1050,9 @@ static int kvmppc_book3s_init(void)
if (xics_on_xive()) {
kvmppc_xive_init_module();
kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
+ kvmppc_xive_native_init_module();
+ kvm_register_device_ops(&kvm_xive_native_ops,
+ KVM_DEV_TYPE_XIVE);
} else
#endif
kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
@@ -1060,8 +1063,10 @@ static int kvmppc_book3s_init(void)
static void kvmppc_book3s_exit(void)
{
#ifdef CONFIG_KVM_XICS
- if (xics_on_xive())
+ if (xics_on_xive()) {
kvmppc_xive_exit_module();
+ kvmppc_xive_native_exit_module();
+ }
#endif
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
kvmppc_book3s_exit_pr();
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
new file mode 100644
index 000000000000..751259394150
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2017-2019, IBM Corporation.
+ */
+
+#define pr_fmt(fmt) "xive-kvm: " fmt
+
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <asm/uaccess.h>
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xive.h>
+#include <asm/xive-regs.h>
+#include <asm/debug.h>
+#include <asm/debugfs.h>
+#include <asm/opal.h>
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+#include "book3s_xive.h"
+
+static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static void kvmppc_xive_native_free(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+
+ debugfs_remove(xive->dentry);
+
+ pr_devel("Destroying xive native device\n");
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ kfree(xive);
+ kfree(dev);
+}
+
+static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
+{
+ struct kvmppc_xive *xive;
+ struct kvm *kvm = dev->kvm;
+ int ret = 0;
+
+ pr_devel("Creating xive native device\n");
+
+ if (kvm->arch.xive)
+ return -EEXIST;
+
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ if (!xive)
+ return -ENOMEM;
+
+ dev->private = xive;
+ xive->dev = dev;
+ xive->kvm = kvm;
+ kvm->arch.xive = xive;
+
+ /*
+ * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
+ * a default. Getting the max number of CPUs the VM was
+ * configured with would improve our usage of the XIVE VP space.
+ */
+ xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
+ pr_devel("VP_Base=%x\n", xive->vp_base);
+
+ if (xive->vp_base == XIVE_INVALID_VP)
+ ret = -ENXIO;
+
+ xive->single_escalation = xive_native_has_single_escalation();
+
+ if (ret)
+ kfree(xive);
+
+ return ret;
+}
+
+static int xive_native_debug_show(struct seq_file *m, void *private)
+{
+ struct kvmppc_xive *xive = m->private;
+ struct kvm *kvm = xive->kvm;
+
+ if (!kvm)
+ return 0;
+
+ return 0;
+}
+
+static int xive_native_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, xive_native_debug_show, inode->i_private);
+}
+
+static const struct file_operations xive_native_debug_fops = {
+ .open = xive_native_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void xive_native_debugfs_init(struct kvmppc_xive *xive)
+{
+ char *name;
+
+ name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
+ if (!name) {
+ pr_err("%s: no memory for name\n", __func__);
+ return;
+ }
+
+ xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
+ xive, &xive_native_debug_fops);
+
+ pr_debug("%s: created %s\n", __func__, name);
+ kfree(name);
+}
+
+static void kvmppc_xive_native_init(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
+
+ /* Register some debug interfaces */
+ xive_native_debugfs_init(xive);
+}
+
+struct kvm_device_ops kvm_xive_native_ops = {
+ .name = "kvm-xive-native",
+ .create = kvmppc_xive_native_create,
+ .init = kvmppc_xive_native_init,
+ .destroy = kvmppc_xive_native_free,
+ .set_attr = kvmppc_xive_native_set_attr,
+ .get_attr = kvmppc_xive_native_get_attr,
+ .has_attr = kvmppc_xive_native_has_attr,
+};
+
+void kvmppc_xive_native_init_module(void)
+{
+ ;
+}
+
+void kvmppc_xive_native_exit_module(void)
+{
+ ;
+}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
new file mode 100644
index 000000000000..fdbd2ff92a88
--- /dev/null
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -0,0 +1,19 @@
+POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
+==========================================================
+
+Device types supported:
+ KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
+
+This device acts as a VM interrupt controller. It provides the KVM
+interface to configure the interrupt sources of a VM in the underlying
+POWER9 XIVE interrupt controller.
+
+Only one XIVE instance may be instantiated. A guest XIVE device
+requires a POWER9 host and the guest OS should have support for the
+XIVE native exploitation interrupt mode. If not, it should run using
+the legacy interrupt mode, referred as XICS (POWER7/8).
+
+* Groups:
+
+ 1. KVM_DEV_XIVE_GRP_CTRL
+ Provides global controls on the device
diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
index 3223aec88b2c..4c67cc79de7c 100644
--- a/arch/powerpc/kvm/Makefile
+++ b/arch/powerpc/kvm/Makefile
@@ -94,7 +94,7 @@ endif
kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
book3s_xics.o
-kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
+kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
kvm-book3s_64-module-objs := \
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This is the basic framework for the new KVM device supporting the XIVE
native exploitation mode. The user interface exposes a new KVM device
to be created by QEMU, only available when running on a L0 hypervisor.
Support for nested guests is not available yet.
The XIVE device reuses the device structure of the XICS-on-XIVE device
as they have a lot in common. That could possibly change in the future
if the need arise.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 8 +
arch/powerpc/include/uapi/asm/kvm.h | 3 +
include/uapi/linux/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 7 +-
arch/powerpc/kvm/book3s_xive_native.c | 179 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 +++
arch/powerpc/kvm/Makefile | 2 +-
8 files changed, 219 insertions(+), 2 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index e6b5bb012ccb..008523224e7a 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -222,6 +222,7 @@ extern struct kvm_device_ops kvm_xics_ops;
struct kvmppc_xive;
struct kvmppc_xive_vcpu;
extern struct kvm_device_ops kvm_xive_ops;
+extern struct kvm_device_ops kvm_xive_native_ops;
struct kvmppc_passthru_irqmap;
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index ac22b28ae78d..f3383e76017a 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -594,6 +594,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+
+extern void kvmppc_xive_native_init_module(void);
+extern void kvmppc_xive_native_exit_module(void);
+
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
u32 priority) { return -1; }
@@ -617,6 +621,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+
+static inline void kvmppc_xive_native_init_module(void) { }
+static inline void kvmppc_xive_native_exit_module(void) { }
+
#endif /* CONFIG_KVM_XIVE */
#if defined(CONFIG_PPC_POWERNV) && defined(CONFIG_KVM_BOOK3S_64_HANDLER)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 26ca425f4c2c..be0ce1f17625 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,4 +677,7 @@ struct kvm_ppc_cpu_char {
#define KVM_XICS_PRESENTED (1ULL << 43)
#define KVM_XICS_QUEUED (1ULL << 44)
+/* POWER9 XIVE Native Interrupt Controller */
+#define KVM_DEV_XIVE_GRP_CTRL 1
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6d4ea4b6c922..e6368163d3a0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1211,6 +1211,8 @@ enum kvm_device_type {
#define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
KVM_DEV_TYPE_ARM_VGIC_ITS,
#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
+ KVM_DEV_TYPE_XIVE,
+#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
KVM_DEV_TYPE_MAX,
};
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 10c5579d20ce..7c3348fa27e1 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -1050,6 +1050,9 @@ static int kvmppc_book3s_init(void)
if (xics_on_xive()) {
kvmppc_xive_init_module();
kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
+ kvmppc_xive_native_init_module();
+ kvm_register_device_ops(&kvm_xive_native_ops,
+ KVM_DEV_TYPE_XIVE);
} else
#endif
kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
@@ -1060,8 +1063,10 @@ static int kvmppc_book3s_init(void)
static void kvmppc_book3s_exit(void)
{
#ifdef CONFIG_KVM_XICS
- if (xics_on_xive())
+ if (xics_on_xive()) {
kvmppc_xive_exit_module();
+ kvmppc_xive_native_exit_module();
+ }
#endif
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
kvmppc_book3s_exit_pr();
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
new file mode 100644
index 000000000000..751259394150
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2017-2019, IBM Corporation.
+ */
+
+#define pr_fmt(fmt) "xive-kvm: " fmt
+
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <asm/uaccess.h>
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xive.h>
+#include <asm/xive-regs.h>
+#include <asm/debug.h>
+#include <asm/debugfs.h>
+#include <asm/opal.h>
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+#include "book3s_xive.h"
+
+static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static void kvmppc_xive_native_free(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+
+ debugfs_remove(xive->dentry);
+
+ pr_devel("Destroying xive native device\n");
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ kfree(xive);
+ kfree(dev);
+}
+
+static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
+{
+ struct kvmppc_xive *xive;
+ struct kvm *kvm = dev->kvm;
+ int ret = 0;
+
+ pr_devel("Creating xive native device\n");
+
+ if (kvm->arch.xive)
+ return -EEXIST;
+
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ if (!xive)
+ return -ENOMEM;
+
+ dev->private = xive;
+ xive->dev = dev;
+ xive->kvm = kvm;
+ kvm->arch.xive = xive;
+
+ /*
+ * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
+ * a default. Getting the max number of CPUs the VM was
+ * configured with would improve our usage of the XIVE VP space.
+ */
+ xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
+ pr_devel("VP_Base=%x\n", xive->vp_base);
+
+ if (xive->vp_base == XIVE_INVALID_VP)
+ ret = -ENXIO;
+
+ xive->single_escalation = xive_native_has_single_escalation();
+
+ if (ret)
+ kfree(xive);
+
+ return ret;
+}
+
+static int xive_native_debug_show(struct seq_file *m, void *private)
+{
+ struct kvmppc_xive *xive = m->private;
+ struct kvm *kvm = xive->kvm;
+
+ if (!kvm)
+ return 0;
+
+ return 0;
+}
+
+static int xive_native_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, xive_native_debug_show, inode->i_private);
+}
+
+static const struct file_operations xive_native_debug_fops = {
+ .open = xive_native_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void xive_native_debugfs_init(struct kvmppc_xive *xive)
+{
+ char *name;
+
+ name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
+ if (!name) {
+ pr_err("%s: no memory for name\n", __func__);
+ return;
+ }
+
+ xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
+ xive, &xive_native_debug_fops);
+
+ pr_debug("%s: created %s\n", __func__, name);
+ kfree(name);
+}
+
+static void kvmppc_xive_native_init(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
+
+ /* Register some debug interfaces */
+ xive_native_debugfs_init(xive);
+}
+
+struct kvm_device_ops kvm_xive_native_ops = {
+ .name = "kvm-xive-native",
+ .create = kvmppc_xive_native_create,
+ .init = kvmppc_xive_native_init,
+ .destroy = kvmppc_xive_native_free,
+ .set_attr = kvmppc_xive_native_set_attr,
+ .get_attr = kvmppc_xive_native_get_attr,
+ .has_attr = kvmppc_xive_native_has_attr,
+};
+
+void kvmppc_xive_native_init_module(void)
+{
+ ;
+}
+
+void kvmppc_xive_native_exit_module(void)
+{
+ ;
+}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
new file mode 100644
index 000000000000..fdbd2ff92a88
--- /dev/null
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -0,0 +1,19 @@
+POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
+==========================================================
+
+Device types supported:
+ KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
+
+This device acts as a VM interrupt controller. It provides the KVM
+interface to configure the interrupt sources of a VM in the underlying
+POWER9 XIVE interrupt controller.
+
+Only one XIVE instance may be instantiated. A guest XIVE device
+requires a POWER9 host and the guest OS should have support for the
+XIVE native exploitation interrupt mode. If not, it should run using
+the legacy interrupt mode, referred as XICS (POWER7/8).
+
+* Groups:
+
+ 1. KVM_DEV_XIVE_GRP_CTRL
+ Provides global controls on the device
diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
index 3223aec88b2c..4c67cc79de7c 100644
--- a/arch/powerpc/kvm/Makefile
+++ b/arch/powerpc/kvm/Makefile
@@ -94,7 +94,7 @@ endif
kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
book3s_xics.o
-kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
+kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
kvm-book3s_64-module-objs := \
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This is the basic framework for the new KVM device supporting the XIVE
native exploitation mode. The user interface exposes a new KVM device
to be created by QEMU, only available when running on a L0 hypervisor.
Support for nested guests is not available yet.
The XIVE device reuses the device structure of the XICS-on-XIVE device
as they have a lot in common. That could possibly change in the future
if the need arise.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 8 +
arch/powerpc/include/uapi/asm/kvm.h | 3 +
include/uapi/linux/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 7 +-
arch/powerpc/kvm/book3s_xive_native.c | 179 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 +++
arch/powerpc/kvm/Makefile | 2 +-
8 files changed, 219 insertions(+), 2 deletions(-)
create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
create mode 100644 Documentation/virtual/kvm/devices/xive.txt
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index e6b5bb012ccb..008523224e7a 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -222,6 +222,7 @@ extern struct kvm_device_ops kvm_xics_ops;
struct kvmppc_xive;
struct kvmppc_xive_vcpu;
extern struct kvm_device_ops kvm_xive_ops;
+extern struct kvm_device_ops kvm_xive_native_ops;
struct kvmppc_passthru_irqmap;
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index ac22b28ae78d..f3383e76017a 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -594,6 +594,10 @@ extern int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval);
extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+
+extern void kvmppc_xive_native_init_module(void);
+extern void kvmppc_xive_native_exit_module(void);
+
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
u32 priority) { return -1; }
@@ -617,6 +621,10 @@ static inline int kvmppc_xive_set_icp(struct kvm_vcpu *vcpu, u64 icpval) { retur
static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+
+static inline void kvmppc_xive_native_init_module(void) { }
+static inline void kvmppc_xive_native_exit_module(void) { }
+
#endif /* CONFIG_KVM_XIVE */
#if defined(CONFIG_PPC_POWERNV) && defined(CONFIG_KVM_BOOK3S_64_HANDLER)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 26ca425f4c2c..be0ce1f17625 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -677,4 +677,7 @@ struct kvm_ppc_cpu_char {
#define KVM_XICS_PRESENTED (1ULL << 43)
#define KVM_XICS_QUEUED (1ULL << 44)
+/* POWER9 XIVE Native Interrupt Controller */
+#define KVM_DEV_XIVE_GRP_CTRL 1
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 6d4ea4b6c922..e6368163d3a0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1211,6 +1211,8 @@ enum kvm_device_type {
#define KVM_DEV_TYPE_ARM_VGIC_V3 KVM_DEV_TYPE_ARM_VGIC_V3
KVM_DEV_TYPE_ARM_VGIC_ITS,
#define KVM_DEV_TYPE_ARM_VGIC_ITS KVM_DEV_TYPE_ARM_VGIC_ITS
+ KVM_DEV_TYPE_XIVE,
+#define KVM_DEV_TYPE_XIVE KVM_DEV_TYPE_XIVE
KVM_DEV_TYPE_MAX,
};
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 10c5579d20ce..7c3348fa27e1 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -1050,6 +1050,9 @@ static int kvmppc_book3s_init(void)
if (xics_on_xive()) {
kvmppc_xive_init_module();
kvm_register_device_ops(&kvm_xive_ops, KVM_DEV_TYPE_XICS);
+ kvmppc_xive_native_init_module();
+ kvm_register_device_ops(&kvm_xive_native_ops,
+ KVM_DEV_TYPE_XIVE);
} else
#endif
kvm_register_device_ops(&kvm_xics_ops, KVM_DEV_TYPE_XICS);
@@ -1060,8 +1063,10 @@ static int kvmppc_book3s_init(void)
static void kvmppc_book3s_exit(void)
{
#ifdef CONFIG_KVM_XICS
- if (xics_on_xive())
+ if (xics_on_xive()) {
kvmppc_xive_exit_module();
+ kvmppc_xive_native_exit_module();
+ }
#endif
#ifdef CONFIG_KVM_BOOK3S_32_HANDLER
kvmppc_book3s_exit_pr();
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
new file mode 100644
index 000000000000..751259394150
--- /dev/null
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -0,0 +1,179 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (c) 2017-2019, IBM Corporation.
+ */
+
+#define pr_fmt(fmt) "xive-kvm: " fmt
+
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/err.h>
+#include <linux/gfp.h>
+#include <linux/spinlock.h>
+#include <linux/delay.h>
+#include <asm/uaccess.h>
+#include <asm/kvm_book3s.h>
+#include <asm/kvm_ppc.h>
+#include <asm/hvcall.h>
+#include <asm/xive.h>
+#include <asm/xive-regs.h>
+#include <asm/debug.h>
+#include <asm/debugfs.h>
+#include <asm/opal.h>
+
+#include <linux/debugfs.h>
+#include <linux/seq_file.h>
+
+#include "book3s_xive.h"
+
+static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ return -ENXIO;
+}
+
+static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
+ struct kvm_device_attr *attr)
+{
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_CTRL:
+ break;
+ }
+ return -ENXIO;
+}
+
+static void kvmppc_xive_native_free(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+
+ debugfs_remove(xive->dentry);
+
+ pr_devel("Destroying xive native device\n");
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ kfree(xive);
+ kfree(dev);
+}
+
+static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
+{
+ struct kvmppc_xive *xive;
+ struct kvm *kvm = dev->kvm;
+ int ret = 0;
+
+ pr_devel("Creating xive native device\n");
+
+ if (kvm->arch.xive)
+ return -EEXIST;
+
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ if (!xive)
+ return -ENOMEM;
+
+ dev->private = xive;
+ xive->dev = dev;
+ xive->kvm = kvm;
+ kvm->arch.xive = xive;
+
+ /*
+ * Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
+ * a default. Getting the max number of CPUs the VM was
+ * configured with would improve our usage of the XIVE VP space.
+ */
+ xive->vp_base = xive_native_alloc_vp_block(KVM_MAX_VCPUS);
+ pr_devel("VP_Base=%x\n", xive->vp_base);
+
+ if (xive->vp_base = XIVE_INVALID_VP)
+ ret = -ENXIO;
+
+ xive->single_escalation = xive_native_has_single_escalation();
+
+ if (ret)
+ kfree(xive);
+
+ return ret;
+}
+
+static int xive_native_debug_show(struct seq_file *m, void *private)
+{
+ struct kvmppc_xive *xive = m->private;
+ struct kvm *kvm = xive->kvm;
+
+ if (!kvm)
+ return 0;
+
+ return 0;
+}
+
+static int xive_native_debug_open(struct inode *inode, struct file *file)
+{
+ return single_open(file, xive_native_debug_show, inode->i_private);
+}
+
+static const struct file_operations xive_native_debug_fops = {
+ .open = xive_native_debug_open,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+
+static void xive_native_debugfs_init(struct kvmppc_xive *xive)
+{
+ char *name;
+
+ name = kasprintf(GFP_KERNEL, "kvm-xive-%p", xive);
+ if (!name) {
+ pr_err("%s: no memory for name\n", __func__);
+ return;
+ }
+
+ xive->dentry = debugfs_create_file(name, 0444, powerpc_debugfs_root,
+ xive, &xive_native_debug_fops);
+
+ pr_debug("%s: created %s\n", __func__, name);
+ kfree(name);
+}
+
+static void kvmppc_xive_native_init(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = (struct kvmppc_xive *)dev->private;
+
+ /* Register some debug interfaces */
+ xive_native_debugfs_init(xive);
+}
+
+struct kvm_device_ops kvm_xive_native_ops = {
+ .name = "kvm-xive-native",
+ .create = kvmppc_xive_native_create,
+ .init = kvmppc_xive_native_init,
+ .destroy = kvmppc_xive_native_free,
+ .set_attr = kvmppc_xive_native_set_attr,
+ .get_attr = kvmppc_xive_native_get_attr,
+ .has_attr = kvmppc_xive_native_has_attr,
+};
+
+void kvmppc_xive_native_init_module(void)
+{
+ ;
+}
+
+void kvmppc_xive_native_exit_module(void)
+{
+ ;
+}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
new file mode 100644
index 000000000000..fdbd2ff92a88
--- /dev/null
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -0,0 +1,19 @@
+POWER9 eXternal Interrupt Virtualization Engine (XIVE Gen1)
+=============================
+
+Device types supported:
+ KVM_DEV_TYPE_XIVE POWER9 XIVE Interrupt Controller generation 1
+
+This device acts as a VM interrupt controller. It provides the KVM
+interface to configure the interrupt sources of a VM in the underlying
+POWER9 XIVE interrupt controller.
+
+Only one XIVE instance may be instantiated. A guest XIVE device
+requires a POWER9 host and the guest OS should have support for the
+XIVE native exploitation interrupt mode. If not, it should run using
+the legacy interrupt mode, referred as XICS (POWER7/8).
+
+* Groups:
+
+ 1. KVM_DEV_XIVE_GRP_CTRL
+ Provides global controls on the device
diff --git a/arch/powerpc/kvm/Makefile b/arch/powerpc/kvm/Makefile
index 3223aec88b2c..4c67cc79de7c 100644
--- a/arch/powerpc/kvm/Makefile
+++ b/arch/powerpc/kvm/Makefile
@@ -94,7 +94,7 @@ endif
kvm-book3s_64-objs-$(CONFIG_KVM_XICS) += \
book3s_xics.o
-kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o
+kvm-book3s_64-objs-$(CONFIG_KVM_XIVE) += book3s_xive.o book3s_xive_native.o
kvm-book3s_64-objs-$(CONFIG_SPAPR_TCE_IOMMU) += book3s_64_vio.o
kvm-book3s_64-module-objs := \
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The user interface exposes a new capability KVM_CAP_PPC_IRQ_XIVE to
let QEMU connect the vCPU presenters to the XIVE KVM device if
required. The capability is not advertised for now as the full support
for the XIVE native exploitation mode is not yet available. When this
is case, the capability will be advertised on PowerNV Hypervisors
only. Nested guests (pseries KVM Hypervisor) are not supported.
Internally, the interface to the new KVM device is protected with a
new interrupt mode: KVMPPC_IRQ_XIVE.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 13 +++
arch/powerpc/kvm/book3s_xive.h | 11 ++
include/uapi/linux/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 88 ++++++++-------
arch/powerpc/kvm/book3s_xive_native.c | 150 ++++++++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 36 +++++++
Documentation/virtual/kvm/api.txt | 9 ++
8 files changed, 268 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 008523224e7a..9cc6abdce1b9 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -450,6 +450,7 @@ struct kvmppc_passthru_irqmap {
#define KVMPPC_IRQ_DEFAULT 0
#define KVMPPC_IRQ_MPIC 1
#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
+#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
#define MMIO_HPTE_CACHE_SIZE 4
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index f3383e76017a..6928a35ac3c7 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -595,6 +595,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+{
+ return vcpu->arch.irq_type == KVMPPC_IRQ_XIVE;
+}
+
+extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu);
+extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
@@ -622,6 +630,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+ { return 0; }
+static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
+static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index a08ae6fd4c51..d366df69b9cb 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -198,6 +198,11 @@ static inline struct kvmppc_xive_src_block *kvmppc_xive_find_source(struct kvmpp
return xive->src_blocks[bid];
}
+static inline u32 kvmppc_xive_vp(struct kvmppc_xive *xive, u32 server)
+{
+ return xive->vp_base + kvmppc_pack_vcpu_id(xive->kvm, server);
+}
+
/*
* Mapping between guest priorities and host priorities
* is as follow.
@@ -248,5 +253,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+/*
+ * Common Xive routines for XICS-over-XIVE and XIVE native
+ */
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e6368163d3a0..52bf74a1616e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_ARM_VM_IPA_SIZE 165
#define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
#define KVM_CAP_HYPERV_CPUID 167
+#define KVM_CAP_PPC_IRQ_XIVE 168
#ifdef KVM_CAP_IRQ_ROUTING
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index f78d002f0fe0..e7f1ada1c3de 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -380,11 +380,6 @@ static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
return -EBUSY;
}
-static u32 xive_vp(struct kvmppc_xive *xive, u32 server)
-{
- return xive->vp_base + kvmppc_pack_vcpu_id(xive->kvm, server);
-}
-
static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
struct kvmppc_xive_src_block *sb,
struct kvmppc_xive_irq_state *state)
@@ -430,8 +425,8 @@ static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
*/
if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
xive_native_configure_irq(hw_num,
- xive_vp(xive, state->act_server),
- MASKED, state->number);
+ kvmppc_xive_vp(xive, state->act_server),
+ MASKED, state->number);
/* set old_p so we can track if an H_EOI was done */
state->old_p = true;
state->old_q = false;
@@ -486,8 +481,8 @@ static void xive_finish_unmask(struct kvmppc_xive *xive,
*/
if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
xive_native_configure_irq(hw_num,
- xive_vp(xive, state->act_server),
- state->act_priority, state->number);
+ kvmppc_xive_vp(xive, state->act_server),
+ state->act_priority, state->number);
/* If an EOI is needed, do it here */
if (!state->old_p)
xive_vm_source_eoi(hw_num, xd);
@@ -563,7 +558,7 @@ static int xive_target_interrupt(struct kvm *kvm,
kvmppc_xive_select_irq(state, &hw_num, NULL);
return xive_native_configure_irq(hw_num,
- xive_vp(xive, server),
+ kvmppc_xive_vp(xive, server),
prio, state->number);
}
@@ -951,7 +946,7 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
* which is fine for a never started interrupt.
*/
xive_native_configure_irq(hw_irq,
- xive_vp(xive, state->act_server),
+ kvmppc_xive_vp(xive, state->act_server),
state->act_priority, state->number);
/*
@@ -1027,7 +1022,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
- xive_vp(xive, state->act_server),
+ kvmppc_xive_vp(xive, state->act_server),
state->act_priority, state->number);
/*
@@ -1049,7 +1044,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
}
EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
-static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct kvm *kvm = vcpu->kvm;
@@ -1166,7 +1161,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
xc->xive = xive;
xc->vcpu = vcpu;
xc->server_num = cpu;
- xc->vp_id = xive_vp(xive, cpu);
+ xc->vp_id = kvmppc_xive_vp(xive, cpu);
xc->mfrr = 0xff;
xc->valid = true;
@@ -1883,6 +1878,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
return 0;
}
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int i;
+
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+ u32 i0, i1, idx;
+
+ if (!q->qpage && !xc->esc_virq[i])
+ continue;
+
+ seq_printf(m, " [q%d]: ", i);
+
+ if (q->qpage) {
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
+ i0, i1);
+ }
+ if (xc->esc_virq[i]) {
+ struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
+ struct xive_irq_data *xd =
+ irq_data_get_irq_handler_data(d);
+ u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
+
+ seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
+ (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
+ (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
+ xc->esc_virq[i], pq, xd->eoi_page);
+ seq_puts(m, "\n");
+ }
+ }
+ return 0;
+}
static int xive_debug_show(struct seq_file *m, void *private)
{
@@ -1908,7 +1940,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
kvm_for_each_vcpu(i, vcpu, kvm) {
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- unsigned int i;
if (!xc)
continue;
@@ -1918,33 +1949,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
xc->server_num, xc->cppr, xc->hw_cppr,
xc->mfrr, xc->pending,
xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
- for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
- struct xive_q *q = &xc->queues[i];
- u32 i0, i1, idx;
- if (!q->qpage && !xc->esc_virq[i])
- continue;
-
- seq_printf(m, " [q%d]: ", i);
-
- if (q->qpage) {
- idx = q->idx;
- i0 = be32_to_cpup(q->qpage + idx);
- idx = (idx + 1) & q->msk;
- i1 = be32_to_cpup(q->qpage + idx);
- seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
- }
- if (xc->esc_virq[i]) {
- struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
- struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
- u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
- seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
- (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
- (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
- xc->esc_virq[i], pq, xd->eoi_page);
- seq_printf(m, "\n");
- }
- }
+ kvmppc_xive_debug_show_queues(m, vcpu);
t_rm_h_xirr += xc->stat_rm_h_xirr;
t_rm_h_ipoll += xc->stat_rm_h_ipoll;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 751259394150..6fa73cfd9d9c 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -26,6 +26,134 @@
#include "book3s_xive.h"
+static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q = &xc->queues[prio];
+
+ xive_native_disable_queue(xc->vp_id, q, prio);
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+}
+
+void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ int i;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
+ pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
+
+ /* Ensure no interrupt is still routed to that VP */
+ xc->valid = false;
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ /* Disable the VP */
+ xive_native_disable_vp(xc->vp_id);
+
+ /* Free the queues & associated interrupts */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ /* Free the escalation irq */
+ if (xc->esc_virq[i]) {
+ free_irq(xc->esc_virq[i], vcpu);
+ irq_dispose_mapping(xc->esc_virq[i]);
+ kfree(xc->esc_virq_names[i]);
+ xc->esc_virq[i] = 0;
+ }
+
+ /* Free the queue */
+ kvmppc_xive_native_cleanup_queue(vcpu, i);
+ }
+
+ /* Free the VP */
+ kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
+}
+
+int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 server_num)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_vcpu *xc = NULL;
+ int rc;
+
+ pr_devel("native_connect_vcpu(server=%d)\n", server_num);
+
+ if (dev->ops != &kvm_xive_native_ops) {
+ pr_devel("Wrong ops !\n");
+ return -EPERM;
+ }
+ if (xive->kvm != vcpu->kvm)
+ return -EPERM;
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
+ return -EBUSY;
+ if (server_num >= KVM_MAX_VCPUS) {
+ pr_devel("Out of bounds !\n");
+ return -EINVAL;
+ }
+
+ mutex_lock(&vcpu->kvm->lock);
+
+ if (kvmppc_xive_find_server(vcpu->kvm, server_num)) {
+ pr_devel("Duplicate !\n");
+ rc = -EEXIST;
+ goto bail;
+ }
+
+ xc = kzalloc(sizeof(*xc), GFP_KERNEL);
+ if (!xc) {
+ rc = -ENOMEM;
+ goto bail;
+ }
+
+ vcpu->arch.xive_vcpu = xc;
+ xc->xive = xive;
+ xc->vcpu = vcpu;
+ xc->server_num = server_num;
+
+ xc->vp_id = kvmppc_xive_vp(xive, server_num);
+ xc->valid = true;
+ vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+
+ rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
+ if (rc) {
+ pr_err("Failed to get VP info from OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /*
+ * Enable the VP first as the single escalation mode will
+ * affect escalation interrupts numbering
+ */
+ rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
+ if (rc) {
+ pr_err("Failed to enable VP in OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /* Configure VCPU fields for use by assembly push/pull */
+ vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
+ vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
+
+ /* TODO: reset all queues to a clean state ? */
+bail:
+ mutex_unlock(&vcpu->kvm->lock);
+ if (rc)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -114,10 +242,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
if (!kvm)
return 0;
+ seq_puts(m, "=========\nVCPU state\n=========\n");
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
+ xc->server_num,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word);
+
+ kvmppc_xive_debug_show_queues(m, vcpu);
+ }
+
return 0;
}
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 8885377ec3e0..b0858ee61460 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -570,6 +570,15 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_PPC_GET_CPU_CHAR:
r = 1;
break;
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE:
+ /*
+ * Return false until all the XIVE infrastructure is
+ * in place including support for migration.
+ */
+ r = 0;
+ break;
+#endif
case KVM_CAP_PPC_ALLOC_HTAB:
r = hv_enabled;
@@ -753,6 +762,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
else
kvmppc_xics_free_icp(vcpu);
break;
+ case KVMPPC_IRQ_XIVE:
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ break;
}
kvmppc_core_vcpu_free(vcpu);
@@ -1941,6 +1953,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
break;
}
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE: {
+ struct fd f;
+ struct kvm_device *dev;
+
+ r = -EBADF;
+ f = fdget(cap->args[0]);
+ if (!f.file)
+ break;
+
+ r = -ENXIO;
+ if (!xive_enabled())
+ break;
+
+ r = -EPERM;
+ dev = kvm_device_from_filp(f.file);
+ if (dev)
+ r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
+ cap->args[1]);
+
+ fdput(f);
+ break;
+ }
+#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
case KVM_CAP_PPC_FWNMI:
r = -EINVAL;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 67068c47c591..e38eb17b7be6 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -4504,6 +4504,15 @@ struct kvm_sync_regs {
struct kvm_vcpu_events events;
};
+6.75 KVM_CAP_PPC_IRQ_XIVE
+
+Architectures: ppc
+Target: vcpu
+Parameters: args[0] is the XIVE device fd
+ args[1] is the XIVE CPU number (server ID) for this vcpu
+
+This capability connects the vcpu to an in-kernel XIVE device.
+
7. Capabilities that can be enabled on VMs
------------------------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The user interface exposes a new capability KVM_CAP_PPC_IRQ_XIVE to
let QEMU connect the vCPU presenters to the XIVE KVM device if
required. The capability is not advertised for now as the full support
for the XIVE native exploitation mode is not yet available. When this
is case, the capability will be advertised on PowerNV Hypervisors
only. Nested guests (pseries KVM Hypervisor) are not supported.
Internally, the interface to the new KVM device is protected with a
new interrupt mode: KVMPPC_IRQ_XIVE.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 13 +++
arch/powerpc/kvm/book3s_xive.h | 11 ++
include/uapi/linux/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 88 ++++++++-------
arch/powerpc/kvm/book3s_xive_native.c | 150 ++++++++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 36 +++++++
Documentation/virtual/kvm/api.txt | 9 ++
8 files changed, 268 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 008523224e7a..9cc6abdce1b9 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -450,6 +450,7 @@ struct kvmppc_passthru_irqmap {
#define KVMPPC_IRQ_DEFAULT 0
#define KVMPPC_IRQ_MPIC 1
#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
+#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
#define MMIO_HPTE_CACHE_SIZE 4
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index f3383e76017a..6928a35ac3c7 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -595,6 +595,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+{
+ return vcpu->arch.irq_type == KVMPPC_IRQ_XIVE;
+}
+
+extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu);
+extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
@@ -622,6 +630,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+ { return 0; }
+static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
+static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index a08ae6fd4c51..d366df69b9cb 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -198,6 +198,11 @@ static inline struct kvmppc_xive_src_block *kvmppc_xive_find_source(struct kvmpp
return xive->src_blocks[bid];
}
+static inline u32 kvmppc_xive_vp(struct kvmppc_xive *xive, u32 server)
+{
+ return xive->vp_base + kvmppc_pack_vcpu_id(xive->kvm, server);
+}
+
/*
* Mapping between guest priorities and host priorities
* is as follow.
@@ -248,5 +253,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+/*
+ * Common Xive routines for XICS-over-XIVE and XIVE native
+ */
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e6368163d3a0..52bf74a1616e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_ARM_VM_IPA_SIZE 165
#define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
#define KVM_CAP_HYPERV_CPUID 167
+#define KVM_CAP_PPC_IRQ_XIVE 168
#ifdef KVM_CAP_IRQ_ROUTING
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index f78d002f0fe0..e7f1ada1c3de 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -380,11 +380,6 @@ static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
return -EBUSY;
}
-static u32 xive_vp(struct kvmppc_xive *xive, u32 server)
-{
- return xive->vp_base + kvmppc_pack_vcpu_id(xive->kvm, server);
-}
-
static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
struct kvmppc_xive_src_block *sb,
struct kvmppc_xive_irq_state *state)
@@ -430,8 +425,8 @@ static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
*/
if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
xive_native_configure_irq(hw_num,
- xive_vp(xive, state->act_server),
- MASKED, state->number);
+ kvmppc_xive_vp(xive, state->act_server),
+ MASKED, state->number);
/* set old_p so we can track if an H_EOI was done */
state->old_p = true;
state->old_q = false;
@@ -486,8 +481,8 @@ static void xive_finish_unmask(struct kvmppc_xive *xive,
*/
if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
xive_native_configure_irq(hw_num,
- xive_vp(xive, state->act_server),
- state->act_priority, state->number);
+ kvmppc_xive_vp(xive, state->act_server),
+ state->act_priority, state->number);
/* If an EOI is needed, do it here */
if (!state->old_p)
xive_vm_source_eoi(hw_num, xd);
@@ -563,7 +558,7 @@ static int xive_target_interrupt(struct kvm *kvm,
kvmppc_xive_select_irq(state, &hw_num, NULL);
return xive_native_configure_irq(hw_num,
- xive_vp(xive, server),
+ kvmppc_xive_vp(xive, server),
prio, state->number);
}
@@ -951,7 +946,7 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
* which is fine for a never started interrupt.
*/
xive_native_configure_irq(hw_irq,
- xive_vp(xive, state->act_server),
+ kvmppc_xive_vp(xive, state->act_server),
state->act_priority, state->number);
/*
@@ -1027,7 +1022,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
- xive_vp(xive, state->act_server),
+ kvmppc_xive_vp(xive, state->act_server),
state->act_priority, state->number);
/*
@@ -1049,7 +1044,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
}
EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
-static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct kvm *kvm = vcpu->kvm;
@@ -1166,7 +1161,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
xc->xive = xive;
xc->vcpu = vcpu;
xc->server_num = cpu;
- xc->vp_id = xive_vp(xive, cpu);
+ xc->vp_id = kvmppc_xive_vp(xive, cpu);
xc->mfrr = 0xff;
xc->valid = true;
@@ -1883,6 +1878,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
return 0;
}
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int i;
+
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+ u32 i0, i1, idx;
+
+ if (!q->qpage && !xc->esc_virq[i])
+ continue;
+
+ seq_printf(m, " [q%d]: ", i);
+
+ if (q->qpage) {
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
+ i0, i1);
+ }
+ if (xc->esc_virq[i]) {
+ struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
+ struct xive_irq_data *xd =
+ irq_data_get_irq_handler_data(d);
+ u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
+
+ seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
+ (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
+ (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
+ xc->esc_virq[i], pq, xd->eoi_page);
+ seq_puts(m, "\n");
+ }
+ }
+ return 0;
+}
static int xive_debug_show(struct seq_file *m, void *private)
{
@@ -1908,7 +1940,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
kvm_for_each_vcpu(i, vcpu, kvm) {
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- unsigned int i;
if (!xc)
continue;
@@ -1918,33 +1949,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
xc->server_num, xc->cppr, xc->hw_cppr,
xc->mfrr, xc->pending,
xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
- for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
- struct xive_q *q = &xc->queues[i];
- u32 i0, i1, idx;
- if (!q->qpage && !xc->esc_virq[i])
- continue;
-
- seq_printf(m, " [q%d]: ", i);
-
- if (q->qpage) {
- idx = q->idx;
- i0 = be32_to_cpup(q->qpage + idx);
- idx = (idx + 1) & q->msk;
- i1 = be32_to_cpup(q->qpage + idx);
- seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
- }
- if (xc->esc_virq[i]) {
- struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
- struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
- u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
- seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
- (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
- (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
- xc->esc_virq[i], pq, xd->eoi_page);
- seq_printf(m, "\n");
- }
- }
+ kvmppc_xive_debug_show_queues(m, vcpu);
t_rm_h_xirr += xc->stat_rm_h_xirr;
t_rm_h_ipoll += xc->stat_rm_h_ipoll;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 751259394150..6fa73cfd9d9c 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -26,6 +26,134 @@
#include "book3s_xive.h"
+static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q = &xc->queues[prio];
+
+ xive_native_disable_queue(xc->vp_id, q, prio);
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+}
+
+void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ int i;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
+ pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
+
+ /* Ensure no interrupt is still routed to that VP */
+ xc->valid = false;
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ /* Disable the VP */
+ xive_native_disable_vp(xc->vp_id);
+
+ /* Free the queues & associated interrupts */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ /* Free the escalation irq */
+ if (xc->esc_virq[i]) {
+ free_irq(xc->esc_virq[i], vcpu);
+ irq_dispose_mapping(xc->esc_virq[i]);
+ kfree(xc->esc_virq_names[i]);
+ xc->esc_virq[i] = 0;
+ }
+
+ /* Free the queue */
+ kvmppc_xive_native_cleanup_queue(vcpu, i);
+ }
+
+ /* Free the VP */
+ kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
+}
+
+int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 server_num)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_vcpu *xc = NULL;
+ int rc;
+
+ pr_devel("native_connect_vcpu(server=%d)\n", server_num);
+
+ if (dev->ops != &kvm_xive_native_ops) {
+ pr_devel("Wrong ops !\n");
+ return -EPERM;
+ }
+ if (xive->kvm != vcpu->kvm)
+ return -EPERM;
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
+ return -EBUSY;
+ if (server_num >= KVM_MAX_VCPUS) {
+ pr_devel("Out of bounds !\n");
+ return -EINVAL;
+ }
+
+ mutex_lock(&vcpu->kvm->lock);
+
+ if (kvmppc_xive_find_server(vcpu->kvm, server_num)) {
+ pr_devel("Duplicate !\n");
+ rc = -EEXIST;
+ goto bail;
+ }
+
+ xc = kzalloc(sizeof(*xc), GFP_KERNEL);
+ if (!xc) {
+ rc = -ENOMEM;
+ goto bail;
+ }
+
+ vcpu->arch.xive_vcpu = xc;
+ xc->xive = xive;
+ xc->vcpu = vcpu;
+ xc->server_num = server_num;
+
+ xc->vp_id = kvmppc_xive_vp(xive, server_num);
+ xc->valid = true;
+ vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+
+ rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
+ if (rc) {
+ pr_err("Failed to get VP info from OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /*
+ * Enable the VP first as the single escalation mode will
+ * affect escalation interrupts numbering
+ */
+ rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
+ if (rc) {
+ pr_err("Failed to enable VP in OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /* Configure VCPU fields for use by assembly push/pull */
+ vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
+ vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
+
+ /* TODO: reset all queues to a clean state ? */
+bail:
+ mutex_unlock(&vcpu->kvm->lock);
+ if (rc)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -114,10 +242,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
if (!kvm)
return 0;
+ seq_puts(m, "=========\nVCPU state\n=========\n");
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
+ xc->server_num,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word);
+
+ kvmppc_xive_debug_show_queues(m, vcpu);
+ }
+
return 0;
}
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 8885377ec3e0..b0858ee61460 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -570,6 +570,15 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_PPC_GET_CPU_CHAR:
r = 1;
break;
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE:
+ /*
+ * Return false until all the XIVE infrastructure is
+ * in place including support for migration.
+ */
+ r = 0;
+ break;
+#endif
case KVM_CAP_PPC_ALLOC_HTAB:
r = hv_enabled;
@@ -753,6 +762,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
else
kvmppc_xics_free_icp(vcpu);
break;
+ case KVMPPC_IRQ_XIVE:
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ break;
}
kvmppc_core_vcpu_free(vcpu);
@@ -1941,6 +1953,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
break;
}
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE: {
+ struct fd f;
+ struct kvm_device *dev;
+
+ r = -EBADF;
+ f = fdget(cap->args[0]);
+ if (!f.file)
+ break;
+
+ r = -ENXIO;
+ if (!xive_enabled())
+ break;
+
+ r = -EPERM;
+ dev = kvm_device_from_filp(f.file);
+ if (dev)
+ r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
+ cap->args[1]);
+
+ fdput(f);
+ break;
+ }
+#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
case KVM_CAP_PPC_FWNMI:
r = -EINVAL;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 67068c47c591..e38eb17b7be6 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -4504,6 +4504,15 @@ struct kvm_sync_regs {
struct kvm_vcpu_events events;
};
+6.75 KVM_CAP_PPC_IRQ_XIVE
+
+Architectures: ppc
+Target: vcpu
+Parameters: args[0] is the XIVE device fd
+ args[1] is the XIVE CPU number (server ID) for this vcpu
+
+This capability connects the vcpu to an in-kernel XIVE device.
+
7. Capabilities that can be enabled on VMs
------------------------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The user interface exposes a new capability KVM_CAP_PPC_IRQ_XIVE to
let QEMU connect the vCPU presenters to the XIVE KVM device if
required. The capability is not advertised for now as the full support
for the XIVE native exploitation mode is not yet available. When this
is case, the capability will be advertised on PowerNV Hypervisors
only. Nested guests (pseries KVM Hypervisor) are not supported.
Internally, the interface to the new KVM device is protected with a
new interrupt mode: KVMPPC_IRQ_XIVE.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/include/asm/kvm_ppc.h | 13 +++
arch/powerpc/kvm/book3s_xive.h | 11 ++
include/uapi/linux/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 88 ++++++++-------
arch/powerpc/kvm/book3s_xive_native.c | 150 ++++++++++++++++++++++++++
arch/powerpc/kvm/powerpc.c | 36 +++++++
Documentation/virtual/kvm/api.txt | 9 ++
8 files changed, 268 insertions(+), 41 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 008523224e7a..9cc6abdce1b9 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -450,6 +450,7 @@ struct kvmppc_passthru_irqmap {
#define KVMPPC_IRQ_DEFAULT 0
#define KVMPPC_IRQ_MPIC 1
#define KVMPPC_IRQ_XICS 2 /* Includes a XIVE option */
+#define KVMPPC_IRQ_XIVE 3 /* XIVE native exploitation mode */
#define MMIO_HPTE_CACHE_SIZE 4
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index f3383e76017a..6928a35ac3c7 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -595,6 +595,14 @@ extern int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 irq,
int level, bool line_status);
extern void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu);
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+{
+ return vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+}
+
+extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu);
+extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
@@ -622,6 +630,11 @@ static inline int kvmppc_xive_set_irq(struct kvm *kvm, int irq_source_id, u32 ir
int level, bool line_status) { return -ENODEV; }
static inline void kvmppc_xive_push_vcpu(struct kvm_vcpu *vcpu) { }
+static inline int kvmppc_xive_enabled(struct kvm_vcpu *vcpu)
+ { return 0; }
+static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 cpu) { return -EBUSY; }
+static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index a08ae6fd4c51..d366df69b9cb 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -198,6 +198,11 @@ static inline struct kvmppc_xive_src_block *kvmppc_xive_find_source(struct kvmpp
return xive->src_blocks[bid];
}
+static inline u32 kvmppc_xive_vp(struct kvmppc_xive *xive, u32 server)
+{
+ return xive->vp_base + kvmppc_pack_vcpu_id(xive->kvm, server);
+}
+
/*
* Mapping between guest priorities and host priorities
* is as follow.
@@ -248,5 +253,11 @@ extern int (*__xive_vm_h_ipi)(struct kvm_vcpu *vcpu, unsigned long server,
extern int (*__xive_vm_h_cppr)(struct kvm_vcpu *vcpu, unsigned long cppr);
extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
+/*
+ * Common Xive routines for XICS-over-XIVE and XIVE native
+ */
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index e6368163d3a0..52bf74a1616e 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -988,6 +988,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_ARM_VM_IPA_SIZE 165
#define KVM_CAP_MANUAL_DIRTY_LOG_PROTECT 166
#define KVM_CAP_HYPERV_CPUID 167
+#define KVM_CAP_PPC_IRQ_XIVE 168
#ifdef KVM_CAP_IRQ_ROUTING
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index f78d002f0fe0..e7f1ada1c3de 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -380,11 +380,6 @@ static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
return -EBUSY;
}
-static u32 xive_vp(struct kvmppc_xive *xive, u32 server)
-{
- return xive->vp_base + kvmppc_pack_vcpu_id(xive->kvm, server);
-}
-
static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
struct kvmppc_xive_src_block *sb,
struct kvmppc_xive_irq_state *state)
@@ -430,8 +425,8 @@ static u8 xive_lock_and_mask(struct kvmppc_xive *xive,
*/
if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
xive_native_configure_irq(hw_num,
- xive_vp(xive, state->act_server),
- MASKED, state->number);
+ kvmppc_xive_vp(xive, state->act_server),
+ MASKED, state->number);
/* set old_p so we can track if an H_EOI was done */
state->old_p = true;
state->old_q = false;
@@ -486,8 +481,8 @@ static void xive_finish_unmask(struct kvmppc_xive *xive,
*/
if (xd->flags & OPAL_XIVE_IRQ_MASK_VIA_FW) {
xive_native_configure_irq(hw_num,
- xive_vp(xive, state->act_server),
- state->act_priority, state->number);
+ kvmppc_xive_vp(xive, state->act_server),
+ state->act_priority, state->number);
/* If an EOI is needed, do it here */
if (!state->old_p)
xive_vm_source_eoi(hw_num, xd);
@@ -563,7 +558,7 @@ static int xive_target_interrupt(struct kvm *kvm,
kvmppc_xive_select_irq(state, &hw_num, NULL);
return xive_native_configure_irq(hw_num,
- xive_vp(xive, server),
+ kvmppc_xive_vp(xive, server),
prio, state->number);
}
@@ -951,7 +946,7 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
* which is fine for a never started interrupt.
*/
xive_native_configure_irq(hw_irq,
- xive_vp(xive, state->act_server),
+ kvmppc_xive_vp(xive, state->act_server),
state->act_priority, state->number);
/*
@@ -1027,7 +1022,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
- xive_vp(xive, state->act_server),
+ kvmppc_xive_vp(xive, state->act_server),
state->act_priority, state->number);
/*
@@ -1049,7 +1044,7 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
}
EXPORT_SYMBOL_GPL(kvmppc_xive_clr_mapped);
-static void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
+void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct kvm *kvm = vcpu->kvm;
@@ -1166,7 +1161,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
xc->xive = xive;
xc->vcpu = vcpu;
xc->server_num = cpu;
- xc->vp_id = xive_vp(xive, cpu);
+ xc->vp_id = kvmppc_xive_vp(xive, cpu);
xc->mfrr = 0xff;
xc->valid = true;
@@ -1883,6 +1878,43 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
return 0;
}
+int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int i;
+
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ struct xive_q *q = &xc->queues[i];
+ u32 i0, i1, idx;
+
+ if (!q->qpage && !xc->esc_virq[i])
+ continue;
+
+ seq_printf(m, " [q%d]: ", i);
+
+ if (q->qpage) {
+ idx = q->idx;
+ i0 = be32_to_cpup(q->qpage + idx);
+ idx = (idx + 1) & q->msk;
+ i1 = be32_to_cpup(q->qpage + idx);
+ seq_printf(m, "T=%d %08x %08x...\n", q->toggle,
+ i0, i1);
+ }
+ if (xc->esc_virq[i]) {
+ struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
+ struct xive_irq_data *xd + irq_data_get_irq_handler_data(d);
+ u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
+
+ seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
+ (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
+ (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
+ xc->esc_virq[i], pq, xd->eoi_page);
+ seq_puts(m, "\n");
+ }
+ }
+ return 0;
+}
static int xive_debug_show(struct seq_file *m, void *private)
{
@@ -1908,7 +1940,6 @@ static int xive_debug_show(struct seq_file *m, void *private)
kvm_for_each_vcpu(i, vcpu, kvm) {
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- unsigned int i;
if (!xc)
continue;
@@ -1918,33 +1949,8 @@ static int xive_debug_show(struct seq_file *m, void *private)
xc->server_num, xc->cppr, xc->hw_cppr,
xc->mfrr, xc->pending,
xc->stat_rm_h_xirr, xc->stat_vm_h_xirr);
- for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
- struct xive_q *q = &xc->queues[i];
- u32 i0, i1, idx;
- if (!q->qpage && !xc->esc_virq[i])
- continue;
-
- seq_printf(m, " [q%d]: ", i);
-
- if (q->qpage) {
- idx = q->idx;
- i0 = be32_to_cpup(q->qpage + idx);
- idx = (idx + 1) & q->msk;
- i1 = be32_to_cpup(q->qpage + idx);
- seq_printf(m, "T=%d %08x %08x... \n", q->toggle, i0, i1);
- }
- if (xc->esc_virq[i]) {
- struct irq_data *d = irq_get_irq_data(xc->esc_virq[i]);
- struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
- u64 pq = xive_vm_esb_load(xd, XIVE_ESB_GET);
- seq_printf(m, "E:%c%c I(%d:%llx:%llx)",
- (pq & XIVE_ESB_VAL_P) ? 'P' : 'p',
- (pq & XIVE_ESB_VAL_Q) ? 'Q' : 'q',
- xc->esc_virq[i], pq, xd->eoi_page);
- seq_printf(m, "\n");
- }
- }
+ kvmppc_xive_debug_show_queues(m, vcpu);
t_rm_h_xirr += xc->stat_rm_h_xirr;
t_rm_h_ipoll += xc->stat_rm_h_ipoll;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 751259394150..6fa73cfd9d9c 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -26,6 +26,134 @@
#include "book3s_xive.h"
+static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct xive_q *q = &xc->queues[prio];
+
+ xive_native_disable_queue(xc->vp_id, q, prio);
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+}
+
+void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ int i;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
+ pr_devel("native_cleanup_vcpu(cpu=%d)\n", xc->server_num);
+
+ /* Ensure no interrupt is still routed to that VP */
+ xc->valid = false;
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ /* Disable the VP */
+ xive_native_disable_vp(xc->vp_id);
+
+ /* Free the queues & associated interrupts */
+ for (i = 0; i < KVMPPC_XIVE_Q_COUNT; i++) {
+ /* Free the escalation irq */
+ if (xc->esc_virq[i]) {
+ free_irq(xc->esc_virq[i], vcpu);
+ irq_dispose_mapping(xc->esc_virq[i]);
+ kfree(xc->esc_virq_names[i]);
+ xc->esc_virq[i] = 0;
+ }
+
+ /* Free the queue */
+ kvmppc_xive_native_cleanup_queue(vcpu, i);
+ }
+
+ /* Free the VP */
+ kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
+}
+
+int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
+ struct kvm_vcpu *vcpu, u32 server_num)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_vcpu *xc = NULL;
+ int rc;
+
+ pr_devel("native_connect_vcpu(server=%d)\n", server_num);
+
+ if (dev->ops != &kvm_xive_native_ops) {
+ pr_devel("Wrong ops !\n");
+ return -EPERM;
+ }
+ if (xive->kvm != vcpu->kvm)
+ return -EPERM;
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
+ return -EBUSY;
+ if (server_num >= KVM_MAX_VCPUS) {
+ pr_devel("Out of bounds !\n");
+ return -EINVAL;
+ }
+
+ mutex_lock(&vcpu->kvm->lock);
+
+ if (kvmppc_xive_find_server(vcpu->kvm, server_num)) {
+ pr_devel("Duplicate !\n");
+ rc = -EEXIST;
+ goto bail;
+ }
+
+ xc = kzalloc(sizeof(*xc), GFP_KERNEL);
+ if (!xc) {
+ rc = -ENOMEM;
+ goto bail;
+ }
+
+ vcpu->arch.xive_vcpu = xc;
+ xc->xive = xive;
+ xc->vcpu = vcpu;
+ xc->server_num = server_num;
+
+ xc->vp_id = kvmppc_xive_vp(xive, server_num);
+ xc->valid = true;
+ vcpu->arch.irq_type = KVMPPC_IRQ_XIVE;
+
+ rc = xive_native_get_vp_info(xc->vp_id, &xc->vp_cam, &xc->vp_chip_id);
+ if (rc) {
+ pr_err("Failed to get VP info from OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /*
+ * Enable the VP first as the single escalation mode will
+ * affect escalation interrupts numbering
+ */
+ rc = xive_native_enable_vp(xc->vp_id, xive->single_escalation);
+ if (rc) {
+ pr_err("Failed to enable VP in OPAL: %d\n", rc);
+ goto bail;
+ }
+
+ /* Configure VCPU fields for use by assembly push/pull */
+ vcpu->arch.xive_saved_state.w01 = cpu_to_be64(0xff000000);
+ vcpu->arch.xive_cam_word = cpu_to_be32(xc->vp_cam | TM_QW1W2_VO);
+
+ /* TODO: reset all queues to a clean state ? */
+bail:
+ mutex_unlock(&vcpu->kvm->lock);
+ if (rc)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -114,10 +242,32 @@ static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
if (!kvm)
return 0;
+ seq_puts(m, "=====\nVCPU state\n=====\n");
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+
+ if (!xc)
+ continue;
+
+ seq_printf(m, "cpu server %#x NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x\n",
+ xc->server_num,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word);
+
+ kvmppc_xive_debug_show_queues(m, vcpu);
+ }
+
return 0;
}
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index 8885377ec3e0..b0858ee61460 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -570,6 +570,15 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_PPC_GET_CPU_CHAR:
r = 1;
break;
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE:
+ /*
+ * Return false until all the XIVE infrastructure is
+ * in place including support for migration.
+ */
+ r = 0;
+ break;
+#endif
case KVM_CAP_PPC_ALLOC_HTAB:
r = hv_enabled;
@@ -753,6 +762,9 @@ void kvm_arch_vcpu_free(struct kvm_vcpu *vcpu)
else
kvmppc_xics_free_icp(vcpu);
break;
+ case KVMPPC_IRQ_XIVE:
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ break;
}
kvmppc_core_vcpu_free(vcpu);
@@ -1941,6 +1953,30 @@ static int kvm_vcpu_ioctl_enable_cap(struct kvm_vcpu *vcpu,
break;
}
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_CAP_PPC_IRQ_XIVE: {
+ struct fd f;
+ struct kvm_device *dev;
+
+ r = -EBADF;
+ f = fdget(cap->args[0]);
+ if (!f.file)
+ break;
+
+ r = -ENXIO;
+ if (!xive_enabled())
+ break;
+
+ r = -EPERM;
+ dev = kvm_device_from_filp(f.file);
+ if (dev)
+ r = kvmppc_xive_native_connect_vcpu(dev, vcpu,
+ cap->args[1]);
+
+ fdput(f);
+ break;
+ }
+#endif /* CONFIG_KVM_XIVE */
#ifdef CONFIG_KVM_BOOK3S_HV_POSSIBLE
case KVM_CAP_PPC_FWNMI:
r = -EINVAL;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 67068c47c591..e38eb17b7be6 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -4504,6 +4504,15 @@ struct kvm_sync_regs {
struct kvm_vcpu_events events;
};
+6.75 KVM_CAP_PPC_IRQ_XIVE
+
+Architectures: ppc
+Target: vcpu
+Parameters: args[0] is the XIVE device fd
+ args[1] is the XIVE CPU number (server ID) for this vcpu
+
+This capability connects the vcpu to an in-kernel XIVE device.
+
7. Capabilities that can be enabled on VMs
------------------------------------------
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The XIVE KVM device maintains a list of interrupt sources for the VM
which are allocated in the pool of generic interrupts (IPIs) of the
main XIVE IC controller. These are used for the CPU IPIs as well as
for virtual device interrupts. The IRQ number space is defined by
QEMU.
The XIVE device reuses the source structures of the XICS-on-XIVE
device for the source blocks (2-level tree) and for the source
interrupts. Under XIVE native, the source interrupt caches mostly
configuration information and is less used than under the XICS-on-XIVE
device in which hcalls are still necessary at run-time.
When a source is initialized in KVM, an IPI interrupt source is simply
allocated at the OPAL level and then MASKED. KVM only needs to know
about its type: LSI or MSI.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 5 +
arch/powerpc/kvm/book3s_xive.h | 10 ++
arch/powerpc/kvm/book3s_xive.c | 8 +-
arch/powerpc/kvm/book3s_xive_native.c | 106 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 +++
5 files changed, 140 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index be0ce1f17625..d468294c2a67 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,5 +679,10 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
+
+/* Layout of 64-bit XIVE source attribute values */
+#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
+#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index d366df69b9cb..1be921cb5dcb 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -12,6 +12,13 @@
#ifdef CONFIG_KVM_XICS
#include "book3s_xics.h"
+/*
+ * The XIVE Interrupt source numbers are within the range 0 to
+ * KVMPPC_XICS_NR_IRQS.
+ */
+#define KVMPPC_XIVE_FIRST_IRQ 0
+#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
+
/*
* State for one guest irq source.
*
@@ -258,6 +265,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
*/
void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq);
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index e7f1ada1c3de..6c9f9fd0855f 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1480,8 +1480,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
return 0;
}
-static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
- int irq)
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq)
{
struct kvm *kvm = xive->kvm;
struct kvmppc_xive_src_block *sb;
@@ -1560,7 +1560,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
sb = kvmppc_xive_find_source(xive, irq, &idx);
if (!sb) {
pr_devel("No source, creating source block...\n");
- sb = xive_create_src_block(xive, irq);
+ sb = kvmppc_xive_create_src_block(xive, irq);
if (!sb) {
pr_devel("Failed to create block...\n");
return -ENOMEM;
@@ -1784,7 +1784,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
xive_cleanup_irq_data(xd);
}
-static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
{
int i;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 6fa73cfd9d9c..5f2bd6c137b7 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -26,6 +26,17 @@
#include "book3s_xive.h"
+static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = in_be64(xd->eoi_mmio + offset);
+ return (u8)val;
+}
+
static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
@@ -154,12 +165,94 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
+ u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u64 val;
+ u16 idx;
+ int rc;
+
+ pr_devel("%s irq=0x%lx\n", __func__, irq);
+
+ if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
+ return -E2BIG;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb) {
+ pr_debug("No source, creating source block...\n");
+ sb = kvmppc_xive_create_src_block(xive, irq);
+ if (!sb) {
+ pr_err("Failed to create block...\n");
+ return -ENOMEM;
+ }
+ }
+ state = &sb->irq_state[idx];
+
+ if (get_user(val, ubufp)) {
+ pr_err("fault getting user info !\n");
+ return -EFAULT;
+ }
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * If the source doesn't already have an IPI, allocate
+ * one and get the corresponding data
+ */
+ if (!state->ipi_number) {
+ state->ipi_number = xive_native_alloc_irq();
+ if (state->ipi_number == 0) {
+ pr_err("Failed to allocate IRQ !\n");
+ rc = -ENXIO;
+ goto unlock;
+ }
+ xive_native_populate_irq_data(state->ipi_number,
+ &state->ipi_data);
+ pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
+ state->ipi_number, irq);
+ }
+
+ /* Restore LSI state */
+ if (val & KVM_XIVE_LEVEL_SENSITIVE) {
+ state->lsi = true;
+ if (val & KVM_XIVE_LEVEL_ASSERTED)
+ state->asserted = true;
+ pr_devel(" LSI ! Asserted=%d\n", state->asserted);
+ }
+
+ /* Mask IRQ to start with */
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+
+ /* Increment the number of valid sources and mark this one valid */
+ if (!state->valid)
+ xive->src_count++;
+ state->valid = true;
+
+ rc = 0;
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ return kvmppc_xive_native_set_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -176,6 +269,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
+ attr->attr < KVMPPC_XIVE_NR_IRQS)
+ return 0;
+ break;
}
return -ENXIO;
}
@@ -184,6 +282,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ int i;
debugfs_remove(xive->dentry);
@@ -192,6 +291,13 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
if (kvm)
kvm->arch.xive = NULL;
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
if (xive->vp_base != XIVE_INVALID_VP)
xive_native_free_vp_block(xive->vp_base);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index fdbd2ff92a88..cd8bfc37b72e 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+
+ 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
+ Initializes a new source in the XIVE device and mask it.
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 2 | 1 | 0
+ values: | unused | level | type
+ - type: 0:MSI 1:LSI
+ - level: assertion level in case of an LSI.
+ Errors:
+ -E2BIG: Interrupt source number is out of range
+ -ENOMEM: Could not create a new source block
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: Could not allocate underlying HW interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The XIVE KVM device maintains a list of interrupt sources for the VM
which are allocated in the pool of generic interrupts (IPIs) of the
main XIVE IC controller. These are used for the CPU IPIs as well as
for virtual device interrupts. The IRQ number space is defined by
QEMU.
The XIVE device reuses the source structures of the XICS-on-XIVE
device for the source blocks (2-level tree) and for the source
interrupts. Under XIVE native, the source interrupt caches mostly
configuration information and is less used than under the XICS-on-XIVE
device in which hcalls are still necessary at run-time.
When a source is initialized in KVM, an IPI interrupt source is simply
allocated at the OPAL level and then MASKED. KVM only needs to know
about its type: LSI or MSI.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 5 +
arch/powerpc/kvm/book3s_xive.h | 10 ++
arch/powerpc/kvm/book3s_xive.c | 8 +-
arch/powerpc/kvm/book3s_xive_native.c | 106 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 +++
5 files changed, 140 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index be0ce1f17625..d468294c2a67 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,5 +679,10 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
+
+/* Layout of 64-bit XIVE source attribute values */
+#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
+#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index d366df69b9cb..1be921cb5dcb 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -12,6 +12,13 @@
#ifdef CONFIG_KVM_XICS
#include "book3s_xics.h"
+/*
+ * The XIVE Interrupt source numbers are within the range 0 to
+ * KVMPPC_XICS_NR_IRQS.
+ */
+#define KVMPPC_XIVE_FIRST_IRQ 0
+#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
+
/*
* State for one guest irq source.
*
@@ -258,6 +265,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
*/
void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq);
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index e7f1ada1c3de..6c9f9fd0855f 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1480,8 +1480,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
return 0;
}
-static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
- int irq)
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq)
{
struct kvm *kvm = xive->kvm;
struct kvmppc_xive_src_block *sb;
@@ -1560,7 +1560,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
sb = kvmppc_xive_find_source(xive, irq, &idx);
if (!sb) {
pr_devel("No source, creating source block...\n");
- sb = xive_create_src_block(xive, irq);
+ sb = kvmppc_xive_create_src_block(xive, irq);
if (!sb) {
pr_devel("Failed to create block...\n");
return -ENOMEM;
@@ -1784,7 +1784,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
xive_cleanup_irq_data(xd);
}
-static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
{
int i;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 6fa73cfd9d9c..5f2bd6c137b7 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -26,6 +26,17 @@
#include "book3s_xive.h"
+static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = in_be64(xd->eoi_mmio + offset);
+ return (u8)val;
+}
+
static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
@@ -154,12 +165,94 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
+ u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u64 val;
+ u16 idx;
+ int rc;
+
+ pr_devel("%s irq=0x%lx\n", __func__, irq);
+
+ if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
+ return -E2BIG;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb) {
+ pr_debug("No source, creating source block...\n");
+ sb = kvmppc_xive_create_src_block(xive, irq);
+ if (!sb) {
+ pr_err("Failed to create block...\n");
+ return -ENOMEM;
+ }
+ }
+ state = &sb->irq_state[idx];
+
+ if (get_user(val, ubufp)) {
+ pr_err("fault getting user info !\n");
+ return -EFAULT;
+ }
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * If the source doesn't already have an IPI, allocate
+ * one and get the corresponding data
+ */
+ if (!state->ipi_number) {
+ state->ipi_number = xive_native_alloc_irq();
+ if (state->ipi_number == 0) {
+ pr_err("Failed to allocate IRQ !\n");
+ rc = -ENXIO;
+ goto unlock;
+ }
+ xive_native_populate_irq_data(state->ipi_number,
+ &state->ipi_data);
+ pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
+ state->ipi_number, irq);
+ }
+
+ /* Restore LSI state */
+ if (val & KVM_XIVE_LEVEL_SENSITIVE) {
+ state->lsi = true;
+ if (val & KVM_XIVE_LEVEL_ASSERTED)
+ state->asserted = true;
+ pr_devel(" LSI ! Asserted=%d\n", state->asserted);
+ }
+
+ /* Mask IRQ to start with */
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+
+ /* Increment the number of valid sources and mark this one valid */
+ if (!state->valid)
+ xive->src_count++;
+ state->valid = true;
+
+ rc = 0;
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ return kvmppc_xive_native_set_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -176,6 +269,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
+ attr->attr < KVMPPC_XIVE_NR_IRQS)
+ return 0;
+ break;
}
return -ENXIO;
}
@@ -184,6 +282,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ int i;
debugfs_remove(xive->dentry);
@@ -192,6 +291,13 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
if (kvm)
kvm->arch.xive = NULL;
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
if (xive->vp_base != XIVE_INVALID_VP)
xive_native_free_vp_block(xive->vp_base);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index fdbd2ff92a88..cd8bfc37b72e 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+
+ 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
+ Initializes a new source in the XIVE device and mask it.
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 2 | 1 | 0
+ values: | unused | level | type
+ - type: 0:MSI 1:LSI
+ - level: assertion level in case of an LSI.
+ Errors:
+ -E2BIG: Interrupt source number is out of range
+ -ENOMEM: Could not create a new source block
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: Could not allocate underlying HW interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The XIVE KVM device maintains a list of interrupt sources for the VM
which are allocated in the pool of generic interrupts (IPIs) of the
main XIVE IC controller. These are used for the CPU IPIs as well as
for virtual device interrupts. The IRQ number space is defined by
QEMU.
The XIVE device reuses the source structures of the XICS-on-XIVE
device for the source blocks (2-level tree) and for the source
interrupts. Under XIVE native, the source interrupt caches mostly
configuration information and is less used than under the XICS-on-XIVE
device in which hcalls are still necessary at run-time.
When a source is initialized in KVM, an IPI interrupt source is simply
allocated at the OPAL level and then MASKED. KVM only needs to know
about its type: LSI or MSI.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 5 +
arch/powerpc/kvm/book3s_xive.h | 10 ++
arch/powerpc/kvm/book3s_xive.c | 8 +-
arch/powerpc/kvm/book3s_xive_native.c | 106 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 15 +++
5 files changed, 140 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index be0ce1f17625..d468294c2a67 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,5 +679,10 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
+
+/* Layout of 64-bit XIVE source attribute values */
+#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
+#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index d366df69b9cb..1be921cb5dcb 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -12,6 +12,13 @@
#ifdef CONFIG_KVM_XICS
#include "book3s_xics.h"
+/*
+ * The XIVE Interrupt source numbers are within the range 0 to
+ * KVMPPC_XICS_NR_IRQS.
+ */
+#define KVMPPC_XIVE_FIRST_IRQ 0
+#define KVMPPC_XIVE_NR_IRQS KVMPPC_XICS_NR_IRQS
+
/*
* State for one guest irq source.
*
@@ -258,6 +265,9 @@ extern int (*__xive_vm_h_eoi)(struct kvm_vcpu *vcpu, unsigned long xirr);
*/
void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu);
int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq);
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index e7f1ada1c3de..6c9f9fd0855f 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1480,8 +1480,8 @@ static int xive_get_source(struct kvmppc_xive *xive, long irq, u64 addr)
return 0;
}
-static struct kvmppc_xive_src_block *xive_create_src_block(struct kvmppc_xive *xive,
- int irq)
+struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
+ struct kvmppc_xive *xive, int irq)
{
struct kvm *kvm = xive->kvm;
struct kvmppc_xive_src_block *sb;
@@ -1560,7 +1560,7 @@ static int xive_set_source(struct kvmppc_xive *xive, long irq, u64 addr)
sb = kvmppc_xive_find_source(xive, irq, &idx);
if (!sb) {
pr_devel("No source, creating source block...\n");
- sb = xive_create_src_block(xive, irq);
+ sb = kvmppc_xive_create_src_block(xive, irq);
if (!sb) {
pr_devel("Failed to create block...\n");
return -ENOMEM;
@@ -1784,7 +1784,7 @@ static void kvmppc_xive_cleanup_irq(u32 hw_num, struct xive_irq_data *xd)
xive_cleanup_irq_data(xd);
}
-static void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
+void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
{
int i;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 6fa73cfd9d9c..5f2bd6c137b7 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -26,6 +26,17 @@
#include "book3s_xive.h"
+static u8 xive_vm_esb_load(struct xive_irq_data *xd, u32 offset)
+{
+ u64 val;
+
+ if (xd->flags & XIVE_IRQ_FLAG_SHIFT_BUG)
+ offset |= offset << 4;
+
+ val = in_be64(xd->eoi_mmio + offset);
+ return (u8)val;
+}
+
static void kvmppc_xive_native_cleanup_queue(struct kvm_vcpu *vcpu, int prio)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
@@ -154,12 +165,94 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
+ u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u64 val;
+ u16 idx;
+ int rc;
+
+ pr_devel("%s irq=0x%lx\n", __func__, irq);
+
+ if (irq < KVMPPC_XIVE_FIRST_IRQ || irq >= KVMPPC_XIVE_NR_IRQS)
+ return -E2BIG;
+
+ sb = kvmppc_xive_find_source(xive, irq, &idx);
+ if (!sb) {
+ pr_debug("No source, creating source block...\n");
+ sb = kvmppc_xive_create_src_block(xive, irq);
+ if (!sb) {
+ pr_err("Failed to create block...\n");
+ return -ENOMEM;
+ }
+ }
+ state = &sb->irq_state[idx];
+
+ if (get_user(val, ubufp)) {
+ pr_err("fault getting user info !\n");
+ return -EFAULT;
+ }
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * If the source doesn't already have an IPI, allocate
+ * one and get the corresponding data
+ */
+ if (!state->ipi_number) {
+ state->ipi_number = xive_native_alloc_irq();
+ if (state->ipi_number = 0) {
+ pr_err("Failed to allocate IRQ !\n");
+ rc = -ENXIO;
+ goto unlock;
+ }
+ xive_native_populate_irq_data(state->ipi_number,
+ &state->ipi_data);
+ pr_debug("%s allocated hw_irq=0x%x for irq=0x%lx\n", __func__,
+ state->ipi_number, irq);
+ }
+
+ /* Restore LSI state */
+ if (val & KVM_XIVE_LEVEL_SENSITIVE) {
+ state->lsi = true;
+ if (val & KVM_XIVE_LEVEL_ASSERTED)
+ state->asserted = true;
+ pr_devel(" LSI ! Asserted=%d\n", state->asserted);
+ }
+
+ /* Mask IRQ to start with */
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+
+ /* Increment the number of valid sources and mark this one valid */
+ if (!state->valid)
+ xive->src_count++;
+ state->valid = true;
+
+ rc = 0;
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+
+ return rc;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ return kvmppc_xive_native_set_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -176,6 +269,11 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
break;
+ case KVM_DEV_XIVE_GRP_SOURCE:
+ if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
+ attr->attr < KVMPPC_XIVE_NR_IRQS)
+ return 0;
+ break;
}
return -ENXIO;
}
@@ -184,6 +282,7 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
struct kvm *kvm = xive->kvm;
+ int i;
debugfs_remove(xive->dentry);
@@ -192,6 +291,13 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
if (kvm)
kvm->arch.xive = NULL;
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
if (xive->vp_base != XIVE_INVALID_VP)
xive_native_free_vp_block(xive->vp_base);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index fdbd2ff92a88..cd8bfc37b72e 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,3 +17,18 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+
+ 2. KVM_DEV_XIVE_GRP_SOURCE (write only)
+ Initializes a new source in the XIVE device and mask it.
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 2 | 1 | 0
+ values: | unused | level | type
+ - type: 0:MSI 1:LSI
+ - level: assertion level in case of an LSI.
+ Errors:
+ -E2BIG: Interrupt source number is out of range
+ -ENOMEM: Could not create a new source block
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: Could not allocate underlying HW interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
QEMU to configure the target of a source and also to restore the
configuration of a source when migrating the VM.
The XIVE source interrupt structure is extended with the value of the
Effective Interrupt Source Number. The EISN is the interrupt number
pushed in the event queue that the guest OS will use to dispatch
events internally. Caching the EISN value in KVM eases the test when
checking if a reconfiguration is indeed needed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 11 +++
arch/powerpc/kvm/book3s_xive.h | 4 +
arch/powerpc/kvm/book3s_xive.c | 5 +-
arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 21 +++++
5 files changed, 136 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index d468294c2a67..e8161e21629b 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -680,9 +680,20 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
+#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
+/* Layout of 64-bit XIVE source configuration attribute values */
+#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
+#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
+#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
+#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
+#define KVM_XIVE_SOURCE_MASKED_SHIFT 32
+#define KVM_XIVE_SOURCE_MASKED_MASK 0x100000000ULL
+#define KVM_XIVE_SOURCE_EISN_SHIFT 33
+#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 1be921cb5dcb..ae26fe653d98 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
bool saved_p;
bool saved_q;
u8 saved_scan_prio;
+
+ /* Xive native */
+ u32 eisn; /* Guest Effective IRQ number */
};
/* Select the "right" interrupt (IPI vs. passthrough) */
@@ -268,6 +271,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 6c9f9fd0855f..e09f3addffe5 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
}
-static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
{
struct kvm_vcpu *vcpu;
int i, rc;
@@ -530,7 +530,7 @@ static int xive_target_interrupt(struct kvm *kvm,
* priority. The count for that new target will have
* already been incremented.
*/
- rc = xive_select_target(kvm, &server, prio);
+ rc = kvmppc_xive_select_target(kvm, &server, prio);
/*
* We failed to find a target ? Not much we can do
@@ -1504,6 +1504,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
+ sb->irq_state[i].eisn = 0;
sb->irq_state[i].guest_priority = MASKED;
sb->irq_state[i].saved_priority = MASKED;
sb->irq_state[i].act_priority = MASKED;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 5f2bd6c137b7..492825a35958 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -242,6 +242,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
return rc;
}
+static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state,
+ u32 server, u8 priority, bool masked,
+ u32 eisn)
+{
+ struct kvm *kvm = xive->kvm;
+ u32 hw_num;
+ int rc = 0;
+
+ arch_spin_lock(&sb->lock);
+
+ if (state->act_server == server && state->act_priority == priority &&
+ state->eisn == eisn)
+ goto unlock;
+
+ pr_devel("new_act_prio=%d new_act_server=%d mask=%d act_server=%d act_prio=%d\n",
+ priority, server, masked, state->act_server,
+ state->act_priority);
+
+ kvmppc_xive_select_irq(state, &hw_num, NULL);
+
+ if (priority != MASKED && !masked) {
+ rc = kvmppc_xive_select_target(kvm, &server, priority);
+ if (rc)
+ goto unlock;
+
+ state->act_priority = priority;
+ state->act_server = server;
+ state->eisn = eisn;
+
+ rc = xive_native_configure_irq(hw_num,
+ kvmppc_xive_vp(xive, server),
+ priority, eisn);
+ } else {
+ state->act_priority = MASKED;
+ state->act_server = 0;
+ state->eisn = 0;
+
+ rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
+ }
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
+static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u16 src;
+ u64 kvm_cfg;
+ u32 server;
+ u8 priority;
+ bool masked;
+ u32 eisn;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ if (get_user(kvm_cfg, ubufp))
+ return -EFAULT;
+
+ pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
+
+ priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
+ KVM_XIVE_SOURCE_PRIORITY_SHIFT;
+ server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
+ KVM_XIVE_SOURCE_SERVER_SHIFT;
+ masked = (kvm_cfg & KVM_XIVE_SOURCE_MASKED_MASK) >>
+ KVM_XIVE_SOURCE_MASKED_SHIFT;
+ eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
+ KVM_XIVE_SOURCE_EISN_SHIFT;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+
+ return kvmppc_xive_native_update_source_config(xive, sb, state, server,
+ priority, masked, eisn);
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -253,6 +346,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ return kvmppc_xive_native_set_source_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -270,6 +366,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
break;
case KVM_DEV_XIVE_GRP_SOURCE:
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cd8bfc37b72e..33c64b2cdbe8 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -32,3 +32,24 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOMEM: Could not create a new source block
-EFAULT: Invalid user pointer for attr->addr.
-ENXIO: Could not allocate underlying HW interrupt
+
+ 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
+ Configures source targeting
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
+ values: | eisn | mask | server | priority
+ - priority: 0-7 interrupt priority level
+ - server: CPU number chosen to handle the interrupt
+ - mask: mask flag (unused)
+ - eisn: Effective Interrupt Source Number
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number
+ -EINVAL: Invalid priority
+ -EINVAL: Invalid CPU number.
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: CPU event queues not configured or configuration of the
+ underlying HW interrupt failed
+ -EBUSY: No CPU available to serve interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
QEMU to configure the target of a source and also to restore the
configuration of a source when migrating the VM.
The XIVE source interrupt structure is extended with the value of the
Effective Interrupt Source Number. The EISN is the interrupt number
pushed in the event queue that the guest OS will use to dispatch
events internally. Caching the EISN value in KVM eases the test when
checking if a reconfiguration is indeed needed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 11 +++
arch/powerpc/kvm/book3s_xive.h | 4 +
arch/powerpc/kvm/book3s_xive.c | 5 +-
arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 21 +++++
5 files changed, 136 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index d468294c2a67..e8161e21629b 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -680,9 +680,20 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
+#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
+/* Layout of 64-bit XIVE source configuration attribute values */
+#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
+#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
+#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
+#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
+#define KVM_XIVE_SOURCE_MASKED_SHIFT 32
+#define KVM_XIVE_SOURCE_MASKED_MASK 0x100000000ULL
+#define KVM_XIVE_SOURCE_EISN_SHIFT 33
+#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 1be921cb5dcb..ae26fe653d98 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
bool saved_p;
bool saved_q;
u8 saved_scan_prio;
+
+ /* Xive native */
+ u32 eisn; /* Guest Effective IRQ number */
};
/* Select the "right" interrupt (IPI vs. passthrough) */
@@ -268,6 +271,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 6c9f9fd0855f..e09f3addffe5 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
}
-static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
{
struct kvm_vcpu *vcpu;
int i, rc;
@@ -530,7 +530,7 @@ static int xive_target_interrupt(struct kvm *kvm,
* priority. The count for that new target will have
* already been incremented.
*/
- rc = xive_select_target(kvm, &server, prio);
+ rc = kvmppc_xive_select_target(kvm, &server, prio);
/*
* We failed to find a target ? Not much we can do
@@ -1504,6 +1504,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
+ sb->irq_state[i].eisn = 0;
sb->irq_state[i].guest_priority = MASKED;
sb->irq_state[i].saved_priority = MASKED;
sb->irq_state[i].act_priority = MASKED;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 5f2bd6c137b7..492825a35958 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -242,6 +242,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
return rc;
}
+static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state,
+ u32 server, u8 priority, bool masked,
+ u32 eisn)
+{
+ struct kvm *kvm = xive->kvm;
+ u32 hw_num;
+ int rc = 0;
+
+ arch_spin_lock(&sb->lock);
+
+ if (state->act_server == server && state->act_priority == priority &&
+ state->eisn == eisn)
+ goto unlock;
+
+ pr_devel("new_act_prio=%d new_act_server=%d mask=%d act_server=%d act_prio=%d\n",
+ priority, server, masked, state->act_server,
+ state->act_priority);
+
+ kvmppc_xive_select_irq(state, &hw_num, NULL);
+
+ if (priority != MASKED && !masked) {
+ rc = kvmppc_xive_select_target(kvm, &server, priority);
+ if (rc)
+ goto unlock;
+
+ state->act_priority = priority;
+ state->act_server = server;
+ state->eisn = eisn;
+
+ rc = xive_native_configure_irq(hw_num,
+ kvmppc_xive_vp(xive, server),
+ priority, eisn);
+ } else {
+ state->act_priority = MASKED;
+ state->act_server = 0;
+ state->eisn = 0;
+
+ rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
+ }
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
+static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u16 src;
+ u64 kvm_cfg;
+ u32 server;
+ u8 priority;
+ bool masked;
+ u32 eisn;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ if (get_user(kvm_cfg, ubufp))
+ return -EFAULT;
+
+ pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
+
+ priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
+ KVM_XIVE_SOURCE_PRIORITY_SHIFT;
+ server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
+ KVM_XIVE_SOURCE_SERVER_SHIFT;
+ masked = (kvm_cfg & KVM_XIVE_SOURCE_MASKED_MASK) >>
+ KVM_XIVE_SOURCE_MASKED_SHIFT;
+ eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
+ KVM_XIVE_SOURCE_EISN_SHIFT;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+
+ return kvmppc_xive_native_update_source_config(xive, sb, state, server,
+ priority, masked, eisn);
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -253,6 +346,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ return kvmppc_xive_native_set_source_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -270,6 +366,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
break;
case KVM_DEV_XIVE_GRP_SOURCE:
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cd8bfc37b72e..33c64b2cdbe8 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -32,3 +32,24 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOMEM: Could not create a new source block
-EFAULT: Invalid user pointer for attr->addr.
-ENXIO: Could not allocate underlying HW interrupt
+
+ 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
+ Configures source targeting
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
+ values: | eisn | mask | server | priority
+ - priority: 0-7 interrupt priority level
+ - server: CPU number chosen to handle the interrupt
+ - mask: mask flag (unused)
+ - eisn: Effective Interrupt Source Number
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number
+ -EINVAL: Invalid priority
+ -EINVAL: Invalid CPU number.
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: CPU event queues not configured or configuration of the
+ underlying HW interrupt failed
+ -EBUSY: No CPU available to serve interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure a source
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This control will be used by the H_INT_SET_SOURCE_CONFIG hcall from
QEMU to configure the target of a source and also to restore the
configuration of a source when migrating the VM.
The XIVE source interrupt structure is extended with the value of the
Effective Interrupt Source Number. The EISN is the interrupt number
pushed in the event queue that the guest OS will use to dispatch
events internally. Caching the EISN value in KVM eases the test when
checking if a reconfiguration is indeed needed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 11 +++
arch/powerpc/kvm/book3s_xive.h | 4 +
arch/powerpc/kvm/book3s_xive.c | 5 +-
arch/powerpc/kvm/book3s_xive_native.c | 97 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 21 +++++
5 files changed, 136 insertions(+), 2 deletions(-)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index d468294c2a67..e8161e21629b 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -680,9 +680,20 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
+#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
#define KVM_XIVE_LEVEL_ASSERTED (1ULL << 1)
+/* Layout of 64-bit XIVE source configuration attribute values */
+#define KVM_XIVE_SOURCE_PRIORITY_SHIFT 0
+#define KVM_XIVE_SOURCE_PRIORITY_MASK 0x7
+#define KVM_XIVE_SOURCE_SERVER_SHIFT 3
+#define KVM_XIVE_SOURCE_SERVER_MASK 0xfffffff8ULL
+#define KVM_XIVE_SOURCE_MASKED_SHIFT 32
+#define KVM_XIVE_SOURCE_MASKED_MASK 0x100000000ULL
+#define KVM_XIVE_SOURCE_EISN_SHIFT 33
+#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 1be921cb5dcb..ae26fe653d98 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -61,6 +61,9 @@ struct kvmppc_xive_irq_state {
bool saved_p;
bool saved_q;
u8 saved_scan_prio;
+
+ /* Xive native */
+ u32 eisn; /* Guest Effective IRQ number */
};
/* Select the "right" interrupt (IPI vs. passthrough) */
@@ -268,6 +271,7 @@ int kvmppc_xive_debug_show_queues(struct seq_file *m, struct kvm_vcpu *vcpu);
struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 6c9f9fd0855f..e09f3addffe5 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -342,7 +342,7 @@ static int xive_try_pick_queue(struct kvm_vcpu *vcpu, u8 prio)
return atomic_add_unless(&q->count, 1, max) ? 0 : -EBUSY;
}
-static int xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
+int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio)
{
struct kvm_vcpu *vcpu;
int i, rc;
@@ -530,7 +530,7 @@ static int xive_target_interrupt(struct kvm *kvm,
* priority. The count for that new target will have
* already been incremented.
*/
- rc = xive_select_target(kvm, &server, prio);
+ rc = kvmppc_xive_select_target(kvm, &server, prio);
/*
* We failed to find a target ? Not much we can do
@@ -1504,6 +1504,7 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
sb->irq_state[i].number = (bid << KVMPPC_XICS_ICS_SHIFT) | i;
+ sb->irq_state[i].eisn = 0;
sb->irq_state[i].guest_priority = MASKED;
sb->irq_state[i].saved_priority = MASKED;
sb->irq_state[i].act_priority = MASKED;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 5f2bd6c137b7..492825a35958 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -242,6 +242,99 @@ static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
return rc;
}
+static int kvmppc_xive_native_update_source_config(struct kvmppc_xive *xive,
+ struct kvmppc_xive_src_block *sb,
+ struct kvmppc_xive_irq_state *state,
+ u32 server, u8 priority, bool masked,
+ u32 eisn)
+{
+ struct kvm *kvm = xive->kvm;
+ u32 hw_num;
+ int rc = 0;
+
+ arch_spin_lock(&sb->lock);
+
+ if (state->act_server = server && state->act_priority = priority &&
+ state->eisn = eisn)
+ goto unlock;
+
+ pr_devel("new_act_prio=%d new_act_server=%d mask=%d act_server=%d act_prio=%d\n",
+ priority, server, masked, state->act_server,
+ state->act_priority);
+
+ kvmppc_xive_select_irq(state, &hw_num, NULL);
+
+ if (priority != MASKED && !masked) {
+ rc = kvmppc_xive_select_target(kvm, &server, priority);
+ if (rc)
+ goto unlock;
+
+ state->act_priority = priority;
+ state->act_server = server;
+ state->eisn = eisn;
+
+ rc = xive_native_configure_irq(hw_num,
+ kvmppc_xive_vp(xive, server),
+ priority, eisn);
+ } else {
+ state->act_priority = MASKED;
+ state->act_server = 0;
+ state->eisn = 0;
+
+ rc = xive_native_configure_irq(hw_num, 0, MASKED, 0);
+ }
+
+unlock:
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
+static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ u64 __user *ubufp = (u64 __user *) addr;
+ u16 src;
+ u64 kvm_cfg;
+ u32 server;
+ u8 priority;
+ bool masked;
+ u32 eisn;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ if (!state->valid)
+ return -EINVAL;
+
+ if (get_user(kvm_cfg, ubufp))
+ return -EFAULT;
+
+ pr_devel("%s irq=0x%lx cfg=%016llx\n", __func__, irq, kvm_cfg);
+
+ priority = (kvm_cfg & KVM_XIVE_SOURCE_PRIORITY_MASK) >>
+ KVM_XIVE_SOURCE_PRIORITY_SHIFT;
+ server = (kvm_cfg & KVM_XIVE_SOURCE_SERVER_MASK) >>
+ KVM_XIVE_SOURCE_SERVER_SHIFT;
+ masked = (kvm_cfg & KVM_XIVE_SOURCE_MASKED_MASK) >>
+ KVM_XIVE_SOURCE_MASKED_SHIFT;
+ eisn = (kvm_cfg & KVM_XIVE_SOURCE_EISN_MASK) >>
+ KVM_XIVE_SOURCE_EISN_SHIFT;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+
+ return kvmppc_xive_native_update_source_config(xive, sb, state, server,
+ priority, masked, eisn);
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -253,6 +346,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ return kvmppc_xive_native_set_source_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -270,6 +366,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
break;
case KVM_DEV_XIVE_GRP_SOURCE:
+ case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cd8bfc37b72e..33c64b2cdbe8 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -32,3 +32,24 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOMEM: Could not create a new source block
-EFAULT: Invalid user pointer for attr->addr.
-ENXIO: Could not allocate underlying HW interrupt
+
+ 3. KVM_DEV_XIVE_GRP_SOURCE_CONFIG (write only)
+ Configures source targeting
+ Attributes:
+ Interrupt source number (64-bit)
+ The kvm_device_attr.addr points to a __u64 value:
+ bits: | 63 .... 33 | 32 | 31 .. 3 | 2 .. 0
+ values: | eisn | mask | server | priority
+ - priority: 0-7 interrupt priority level
+ - server: CPU number chosen to handle the interrupt
+ - mask: mask flag (unused)
+ - eisn: Effective Interrupt Source Number
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number
+ -EINVAL: Invalid priority
+ -EINVAL: Invalid CPU number.
+ -EFAULT: Invalid user pointer for attr->addr.
+ -ENXIO: CPU event queues not configured or configuration of the
+ underlying HW interrupt failed
+ -EBUSY: No CPU available to serve interrupt
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
These controls will be used by the H_INT_SET_QUEUE_CONFIG and
H_INT_GET_QUEUE_CONFIG hcalls from QEMU to configure the underlying
Event Queue in the XIVE IC. They will also be used to restore the
configuration of the XIVE EQs and to capture the internal run-time
state of the EQs. Both 'get' and 'set' rely on an OPAL call to access
the EQ toggle bit and EQ index which are updated by the XIVE IC when
event notifications are enqueued in the EQ.
The value of the guest physical address of the event queue is saved in
the XIVE internal xive_q structure for later use. That is when
migration needs to mark the EQ pages dirty to capture a consistent
memory state of the VM.
To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
but restoring the EQ state will.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
Changes since v4 :
- add check on EQ page alignment
- add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
arch/powerpc/include/asm/xive.h | 2 +
arch/powerpc/include/uapi/asm/kvm.h | 19 ++
arch/powerpc/kvm/book3s_xive.h | 2 +
arch/powerpc/kvm/book3s_xive.c | 15 +-
arch/powerpc/kvm/book3s_xive_native.c | 249 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 34 +++
6 files changed, 315 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index b579a943407b..c4e88abd3b67 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -73,6 +73,8 @@ struct xive_q {
u32 esc_irq;
atomic_t count;
atomic_t pending_count;
+ u64 guest_qaddr;
+ u32 guest_qshift;
};
/* Global enable flags for the XIVE support */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index e8161e21629b..85005400fd86 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
+#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
@@ -696,4 +697,22 @@ struct kvm_ppc_cpu_char {
#define KVM_XIVE_SOURCE_EISN_SHIFT 33
#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+/* Layout of 64-bit EQ identifier */
+#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
+#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
+#define KVM_XIVE_EQ_SERVER_SHIFT 3
+#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
+
+/* Layout of EQ configuration values (64 bytes) */
+struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qshift;
+ __u64 qaddr;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+};
+
+#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index ae26fe653d98..622f594d93e1 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -272,6 +272,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index e09f3addffe5..c1b7aa7dbc28 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
return IRQ_HANDLED;
}
-static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct xive_q *q = &xc->queues[prio];
@@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
return -EIO;
}
- if (xc->xive->single_escalation)
+ if (single_escalation)
name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
vcpu->kvm->arch.lpid, xc->server_num);
else
@@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
* interrupt, thus leaving it effectively masked after
* it fires once.
*/
- if (xc->xive->single_escalation) {
+ if (single_escalation) {
struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
@@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
continue;
rc = xive_provision_queue(vcpu, prio);
if (rc == 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, prio);
+ kvmppc_xive_attach_escalation(vcpu, prio,
+ xive->single_escalation);
if (rc)
return rc;
}
@@ -1214,7 +1216,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
if (xive->qmap & (1 << i)) {
r = xive_provision_queue(vcpu, i);
if (r == 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, i);
+ kvmppc_xive_attach_escalation(
+ vcpu, i, xive->single_escalation);
if (r)
goto bail;
} else {
@@ -1229,7 +1232,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
/* If not done above, attach priority 0 escalation */
- r = xive_attach_escalation(vcpu, 0);
+ r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
if (r)
goto bail;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 492825a35958..3e7cdcacc932 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -335,6 +335,243 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, masked, eisn);
}
+static int xive_native_validate_queue_size(u32 qshift)
+{
+ /*
+ * We only support 64K pages for the moment. This is also
+ * advertised in the DT property "ibm,xive-eq-sizes"
+ */
+ switch (qshift) {
+ case 0: /* EQ reset */
+ case 16:
+ return 0;
+ case 12:
+ case 21:
+ case 24:
+ default:
+ return -EINVAL;
+ }
+}
+
+static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ void __user *ubufp = (void __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ int rc;
+ __be32 *qaddr = 0;
+ struct page *page;
+ struct xive_q *q;
+ gfn_t gfn;
+ unsigned long page_size;
+
+ /*
+ * Demangle priority/server tuple from the EQ identifier
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("Trying to restore invalid queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ /*
+ * sPAPR specifies a "Unconditional Notify (n) flag" for the
+ * H_INT_SET_QUEUE_CONFIG hcall which forces notification
+ * without using the coalescing mechanisms provided by the
+ * XIVE END ESBs. This is required on KVM as notification
+ * using the END ESBs is not supported.
+ */
+ if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
+ pr_err("invalid flags %d\n", kvm_eq.flags);
+ return -EINVAL;
+ }
+
+ rc = xive_native_validate_queue_size(kvm_eq.qshift);
+ if (rc) {
+ pr_err("invalid queue size %d\n", kvm_eq.qshift);
+ return rc;
+ }
+
+ /* reset queue and disable queueing */
+ if (!kvm_eq.qshift) {
+ q->guest_qaddr = 0;
+ q->guest_qshift = 0;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ NULL, 0, true);
+ if (rc) {
+ pr_err("Failed to reset queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ return rc;
+ }
+
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+
+ return 0;
+ }
+
+ if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
+ pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
+ 1ull << kvm_eq.qshift);
+ return -EINVAL;
+ }
+
+ gfn = gpa_to_gfn(kvm_eq.qaddr);
+ page = gfn_to_page(kvm, gfn);
+ if (is_error_page(page)) {
+ pr_err("Couldn't get queue page %llx!\n", kvm_eq.qaddr);
+ return -EINVAL;
+ }
+
+ page_size = kvm_host_page_size(kvm, gfn);
+ if (1ull << kvm_eq.qshift > page_size) {
+ pr_warn("Incompatible host page size %lx!\n", page_size);
+ return -EINVAL;
+ }
+
+ qaddr = page_to_virt(page) + (kvm_eq.qaddr & ~PAGE_MASK);
+
+ /*
+ * Backup the queue page guest address to the mark EQ page
+ * dirty for migration.
+ */
+ q->guest_qaddr = kvm_eq.qaddr;
+ q->guest_qshift = kvm_eq.qshift;
+
+ /*
+ * Unconditional Notification is forced by default at the
+ * OPAL level because the use of END ESBs is not supported by
+ * Linux.
+ */
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ (__be32 *) qaddr, kvm_eq.qshift, true);
+ if (rc) {
+ pr_err("Failed to configure queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ put_page(page);
+ return rc;
+ }
+
+ /*
+ * Only restore the queue state when needed. When doing the
+ * H_INT_SET_SOURCE_CONFIG hcall, it should not.
+ */
+ if (kvm_eq.qtoggle != 1 || kvm_eq.qindex != 0) {
+ rc = xive_native_set_queue_state(xc->vp_id, priority,
+ kvm_eq.qtoggle,
+ kvm_eq.qindex);
+ if (rc)
+ goto error;
+ }
+
+ rc = kvmppc_xive_attach_escalation(vcpu, priority,
+ xive->single_escalation);
+error:
+ if (rc)
+ kvmppc_xive_native_cleanup_queue(vcpu, priority);
+ return rc;
+}
+
+static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ struct xive_q *q;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ u64 qaddr;
+ u64 qshift;
+ u64 qeoi_page;
+ u32 escalate_irq;
+ u64 qflags;
+ int rc;
+
+ /*
+ * Demangle priority/server tuple from the EQ identifier
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ memset(&kvm_eq, 0, sizeof(kvm_eq));
+
+ if (!q->qpage)
+ return 0;
+
+ rc = xive_native_get_queue_info(xc->vp_id, priority, &qaddr, &qshift,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc)
+ return rc;
+
+ kvm_eq.flags = 0;
+ if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
+ kvm_eq.flags |= KVM_XIVE_EQ_ALWAYS_NOTIFY;
+
+ kvm_eq.qshift = q->guest_qshift;
+ kvm_eq.qaddr = q->guest_qaddr;
+
+ rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
+ &kvm_eq.qindex);
+ if (rc)
+ return rc;
+
+ pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -349,6 +586,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
return kvmppc_xive_native_set_source_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_set_queue_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -356,6 +596,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_get_queue_config(xive, attr->attr,
+ attr->addr);
+ }
return -ENXIO;
}
@@ -371,6 +618,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
break;
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return 0;
}
return -ENXIO;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 33c64b2cdbe8..cc13bfd5cf53 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -53,3 +53,37 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENXIO: CPU event queues not configured or configuration of the
underlying HW interrupt failed
-EBUSY: No CPU available to serve interrupt
+
+ 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
+ Configures an event queue of a CPU
+ Attributes:
+ EQ descriptor identifier (64-bit)
+ The EQ descriptor identifier is a tuple (server, priority) :
+ bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
+ values: | unused | server | priority
+ The kvm_device_attr.addr points to :
+ struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qshift;
+ __u64 qaddr;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+ };
+ - flags: queue flags
+ KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
+ forces notification without using the coalescing mechanism
+ provided by the XIVE END ESBs.
+ - qshift: queue size (power of 2)
+ - qaddr: real address of queue
+ - qtoggle: current queue toggle bit
+ - qindex: current queue index
+ - pad: reserved for future use
+ Errors:
+ -ENOENT: Invalid CPU number
+ -EINVAL: Invalid priority
+ -EINVAL: Invalid flags
+ -EINVAL: Invalid queue size
+ -EINVAL: Invalid queue address
+ -EFAULT: Invalid user pointer for attr->addr.
+ -EIO: Configuration of the underlying HW failed
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
These controls will be used by the H_INT_SET_QUEUE_CONFIG and
H_INT_GET_QUEUE_CONFIG hcalls from QEMU to configure the underlying
Event Queue in the XIVE IC. They will also be used to restore the
configuration of the XIVE EQs and to capture the internal run-time
state of the EQs. Both 'get' and 'set' rely on an OPAL call to access
the EQ toggle bit and EQ index which are updated by the XIVE IC when
event notifications are enqueued in the EQ.
The value of the guest physical address of the event queue is saved in
the XIVE internal xive_q structure for later use. That is when
migration needs to mark the EQ pages dirty to capture a consistent
memory state of the VM.
To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
but restoring the EQ state will.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
Changes since v4 :
- add check on EQ page alignment
- add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
arch/powerpc/include/asm/xive.h | 2 +
arch/powerpc/include/uapi/asm/kvm.h | 19 ++
arch/powerpc/kvm/book3s_xive.h | 2 +
arch/powerpc/kvm/book3s_xive.c | 15 +-
arch/powerpc/kvm/book3s_xive_native.c | 249 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 34 +++
6 files changed, 315 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index b579a943407b..c4e88abd3b67 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -73,6 +73,8 @@ struct xive_q {
u32 esc_irq;
atomic_t count;
atomic_t pending_count;
+ u64 guest_qaddr;
+ u32 guest_qshift;
};
/* Global enable flags for the XIVE support */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index e8161e21629b..85005400fd86 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
+#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
@@ -696,4 +697,22 @@ struct kvm_ppc_cpu_char {
#define KVM_XIVE_SOURCE_EISN_SHIFT 33
#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+/* Layout of 64-bit EQ identifier */
+#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
+#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
+#define KVM_XIVE_EQ_SERVER_SHIFT 3
+#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
+
+/* Layout of EQ configuration values (64 bytes) */
+struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qshift;
+ __u64 qaddr;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+};
+
+#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index ae26fe653d98..622f594d93e1 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -272,6 +272,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index e09f3addffe5..c1b7aa7dbc28 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
return IRQ_HANDLED;
}
-static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct xive_q *q = &xc->queues[prio];
@@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
return -EIO;
}
- if (xc->xive->single_escalation)
+ if (single_escalation)
name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
vcpu->kvm->arch.lpid, xc->server_num);
else
@@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
* interrupt, thus leaving it effectively masked after
* it fires once.
*/
- if (xc->xive->single_escalation) {
+ if (single_escalation) {
struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
@@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
continue;
rc = xive_provision_queue(vcpu, prio);
if (rc == 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, prio);
+ kvmppc_xive_attach_escalation(vcpu, prio,
+ xive->single_escalation);
if (rc)
return rc;
}
@@ -1214,7 +1216,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
if (xive->qmap & (1 << i)) {
r = xive_provision_queue(vcpu, i);
if (r == 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, i);
+ kvmppc_xive_attach_escalation(
+ vcpu, i, xive->single_escalation);
if (r)
goto bail;
} else {
@@ -1229,7 +1232,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
/* If not done above, attach priority 0 escalation */
- r = xive_attach_escalation(vcpu, 0);
+ r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
if (r)
goto bail;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 492825a35958..3e7cdcacc932 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -335,6 +335,243 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, masked, eisn);
}
+static int xive_native_validate_queue_size(u32 qshift)
+{
+ /*
+ * We only support 64K pages for the moment. This is also
+ * advertised in the DT property "ibm,xive-eq-sizes"
+ */
+ switch (qshift) {
+ case 0: /* EQ reset */
+ case 16:
+ return 0;
+ case 12:
+ case 21:
+ case 24:
+ default:
+ return -EINVAL;
+ }
+}
+
+static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ void __user *ubufp = (void __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ int rc;
+ __be32 *qaddr = 0;
+ struct page *page;
+ struct xive_q *q;
+ gfn_t gfn;
+ unsigned long page_size;
+
+ /*
+ * Demangle priority/server tuple from the EQ identifier
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("Trying to restore invalid queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ /*
+ * sPAPR specifies a "Unconditional Notify (n) flag" for the
+ * H_INT_SET_QUEUE_CONFIG hcall which forces notification
+ * without using the coalescing mechanisms provided by the
+ * XIVE END ESBs. This is required on KVM as notification
+ * using the END ESBs is not supported.
+ */
+ if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
+ pr_err("invalid flags %d\n", kvm_eq.flags);
+ return -EINVAL;
+ }
+
+ rc = xive_native_validate_queue_size(kvm_eq.qshift);
+ if (rc) {
+ pr_err("invalid queue size %d\n", kvm_eq.qshift);
+ return rc;
+ }
+
+ /* reset queue and disable queueing */
+ if (!kvm_eq.qshift) {
+ q->guest_qaddr = 0;
+ q->guest_qshift = 0;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ NULL, 0, true);
+ if (rc) {
+ pr_err("Failed to reset queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ return rc;
+ }
+
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+
+ return 0;
+ }
+
+ if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
+ pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
+ 1ull << kvm_eq.qshift);
+ return -EINVAL;
+ }
+
+ gfn = gpa_to_gfn(kvm_eq.qaddr);
+ page = gfn_to_page(kvm, gfn);
+ if (is_error_page(page)) {
+ pr_err("Couldn't get queue page %llx!\n", kvm_eq.qaddr);
+ return -EINVAL;
+ }
+
+ page_size = kvm_host_page_size(kvm, gfn);
+ if (1ull << kvm_eq.qshift > page_size) {
+ pr_warn("Incompatible host page size %lx!\n", page_size);
+ return -EINVAL;
+ }
+
+ qaddr = page_to_virt(page) + (kvm_eq.qaddr & ~PAGE_MASK);
+
+ /*
+ * Backup the queue page guest address to the mark EQ page
+ * dirty for migration.
+ */
+ q->guest_qaddr = kvm_eq.qaddr;
+ q->guest_qshift = kvm_eq.qshift;
+
+ /*
+ * Unconditional Notification is forced by default at the
+ * OPAL level because the use of END ESBs is not supported by
+ * Linux.
+ */
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ (__be32 *) qaddr, kvm_eq.qshift, true);
+ if (rc) {
+ pr_err("Failed to configure queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ put_page(page);
+ return rc;
+ }
+
+ /*
+ * Only restore the queue state when needed. When doing the
+ * H_INT_SET_SOURCE_CONFIG hcall, it should not.
+ */
+ if (kvm_eq.qtoggle != 1 || kvm_eq.qindex != 0) {
+ rc = xive_native_set_queue_state(xc->vp_id, priority,
+ kvm_eq.qtoggle,
+ kvm_eq.qindex);
+ if (rc)
+ goto error;
+ }
+
+ rc = kvmppc_xive_attach_escalation(vcpu, priority,
+ xive->single_escalation);
+error:
+ if (rc)
+ kvmppc_xive_native_cleanup_queue(vcpu, priority);
+ return rc;
+}
+
+static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ struct xive_q *q;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ u64 qaddr;
+ u64 qshift;
+ u64 qeoi_page;
+ u32 escalate_irq;
+ u64 qflags;
+ int rc;
+
+ /*
+ * Demangle priority/server tuple from the EQ identifier
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ memset(&kvm_eq, 0, sizeof(kvm_eq));
+
+ if (!q->qpage)
+ return 0;
+
+ rc = xive_native_get_queue_info(xc->vp_id, priority, &qaddr, &qshift,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc)
+ return rc;
+
+ kvm_eq.flags = 0;
+ if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
+ kvm_eq.flags |= KVM_XIVE_EQ_ALWAYS_NOTIFY;
+
+ kvm_eq.qshift = q->guest_qshift;
+ kvm_eq.qaddr = q->guest_qaddr;
+
+ rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
+ &kvm_eq.qindex);
+ if (rc)
+ return rc;
+
+ pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -349,6 +586,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
return kvmppc_xive_native_set_source_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_set_queue_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -356,6 +596,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_get_queue_config(xive, attr->attr,
+ attr->addr);
+ }
return -ENXIO;
}
@@ -371,6 +618,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
break;
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return 0;
}
return -ENXIO;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 33c64b2cdbe8..cc13bfd5cf53 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -53,3 +53,37 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENXIO: CPU event queues not configured or configuration of the
underlying HW interrupt failed
-EBUSY: No CPU available to serve interrupt
+
+ 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
+ Configures an event queue of a CPU
+ Attributes:
+ EQ descriptor identifier (64-bit)
+ The EQ descriptor identifier is a tuple (server, priority) :
+ bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
+ values: | unused | server | priority
+ The kvm_device_attr.addr points to :
+ struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qshift;
+ __u64 qaddr;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+ };
+ - flags: queue flags
+ KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
+ forces notification without using the coalescing mechanism
+ provided by the XIVE END ESBs.
+ - qshift: queue size (power of 2)
+ - qaddr: real address of queue
+ - qtoggle: current queue toggle bit
+ - qindex: current queue index
+ - pad: reserved for future use
+ Errors:
+ -ENOENT: Invalid CPU number
+ -EINVAL: Invalid priority
+ -EINVAL: Invalid flags
+ -EINVAL: Invalid queue size
+ -EINVAL: Invalid queue address
+ -EFAULT: Invalid user pointer for attr->addr.
+ -EIO: Configuration of the underlying HW failed
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
These controls will be used by the H_INT_SET_QUEUE_CONFIG and
H_INT_GET_QUEUE_CONFIG hcalls from QEMU to configure the underlying
Event Queue in the XIVE IC. They will also be used to restore the
configuration of the XIVE EQs and to capture the internal run-time
state of the EQs. Both 'get' and 'set' rely on an OPAL call to access
the EQ toggle bit and EQ index which are updated by the XIVE IC when
event notifications are enqueued in the EQ.
The value of the guest physical address of the event queue is saved in
the XIVE internal xive_q structure for later use. That is when
migration needs to mark the EQ pages dirty to capture a consistent
memory state of the VM.
To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
but restoring the EQ state will.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
Changes since v4 :
- add check on EQ page alignment
- add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
arch/powerpc/include/asm/xive.h | 2 +
arch/powerpc/include/uapi/asm/kvm.h | 19 ++
arch/powerpc/kvm/book3s_xive.h | 2 +
arch/powerpc/kvm/book3s_xive.c | 15 +-
arch/powerpc/kvm/book3s_xive_native.c | 249 +++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 34 +++
6 files changed, 315 insertions(+), 6 deletions(-)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index b579a943407b..c4e88abd3b67 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -73,6 +73,8 @@ struct xive_q {
u32 esc_irq;
atomic_t count;
atomic_t pending_count;
+ u64 guest_qaddr;
+ u32 guest_qshift;
};
/* Global enable flags for the XIVE support */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index e8161e21629b..85005400fd86 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
+#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
@@ -696,4 +697,22 @@ struct kvm_ppc_cpu_char {
#define KVM_XIVE_SOURCE_EISN_SHIFT 33
#define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
+/* Layout of 64-bit EQ identifier */
+#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
+#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
+#define KVM_XIVE_EQ_SERVER_SHIFT 3
+#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
+
+/* Layout of EQ configuration values (64 bytes) */
+struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qshift;
+ __u64 qaddr;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+};
+
+#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index ae26fe653d98..622f594d93e1 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -272,6 +272,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
struct kvmppc_xive *xive, int irq);
void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index e09f3addffe5..c1b7aa7dbc28 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
return IRQ_HANDLED;
}
-static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
+int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
+ bool single_escalation)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
struct xive_q *q = &xc->queues[prio];
@@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
return -EIO;
}
- if (xc->xive->single_escalation)
+ if (single_escalation)
name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
vcpu->kvm->arch.lpid, xc->server_num);
else
@@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
* interrupt, thus leaving it effectively masked after
* it fires once.
*/
- if (xc->xive->single_escalation) {
+ if (single_escalation) {
struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
@@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
continue;
rc = xive_provision_queue(vcpu, prio);
if (rc = 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, prio);
+ kvmppc_xive_attach_escalation(vcpu, prio,
+ xive->single_escalation);
if (rc)
return rc;
}
@@ -1214,7 +1216,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
if (xive->qmap & (1 << i)) {
r = xive_provision_queue(vcpu, i);
if (r = 0 && !xive->single_escalation)
- xive_attach_escalation(vcpu, i);
+ kvmppc_xive_attach_escalation(
+ vcpu, i, xive->single_escalation);
if (r)
goto bail;
} else {
@@ -1229,7 +1232,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
/* If not done above, attach priority 0 escalation */
- r = xive_attach_escalation(vcpu, 0);
+ r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
if (r)
goto bail;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 492825a35958..3e7cdcacc932 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -335,6 +335,243 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, masked, eisn);
}
+static int xive_native_validate_queue_size(u32 qshift)
+{
+ /*
+ * We only support 64K pages for the moment. This is also
+ * advertised in the DT property "ibm,xive-eq-sizes"
+ */
+ switch (qshift) {
+ case 0: /* EQ reset */
+ case 16:
+ return 0;
+ case 12:
+ case 21:
+ case 24:
+ default:
+ return -EINVAL;
+ }
+}
+
+static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ void __user *ubufp = (void __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ int rc;
+ __be32 *qaddr = 0;
+ struct page *page;
+ struct xive_q *q;
+ gfn_t gfn;
+ unsigned long page_size;
+
+ /*
+ * Demangle priority/server tuple from the EQ identifier
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("Trying to restore invalid queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ /*
+ * sPAPR specifies a "Unconditional Notify (n) flag" for the
+ * H_INT_SET_QUEUE_CONFIG hcall which forces notification
+ * without using the coalescing mechanisms provided by the
+ * XIVE END ESBs. This is required on KVM as notification
+ * using the END ESBs is not supported.
+ */
+ if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
+ pr_err("invalid flags %d\n", kvm_eq.flags);
+ return -EINVAL;
+ }
+
+ rc = xive_native_validate_queue_size(kvm_eq.qshift);
+ if (rc) {
+ pr_err("invalid queue size %d\n", kvm_eq.qshift);
+ return rc;
+ }
+
+ /* reset queue and disable queueing */
+ if (!kvm_eq.qshift) {
+ q->guest_qaddr = 0;
+ q->guest_qshift = 0;
+
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ NULL, 0, true);
+ if (rc) {
+ pr_err("Failed to reset queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ return rc;
+ }
+
+ if (q->qpage) {
+ put_page(virt_to_page(q->qpage));
+ q->qpage = NULL;
+ }
+
+ return 0;
+ }
+
+ if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
+ pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
+ 1ull << kvm_eq.qshift);
+ return -EINVAL;
+ }
+
+ gfn = gpa_to_gfn(kvm_eq.qaddr);
+ page = gfn_to_page(kvm, gfn);
+ if (is_error_page(page)) {
+ pr_err("Couldn't get queue page %llx!\n", kvm_eq.qaddr);
+ return -EINVAL;
+ }
+
+ page_size = kvm_host_page_size(kvm, gfn);
+ if (1ull << kvm_eq.qshift > page_size) {
+ pr_warn("Incompatible host page size %lx!\n", page_size);
+ return -EINVAL;
+ }
+
+ qaddr = page_to_virt(page) + (kvm_eq.qaddr & ~PAGE_MASK);
+
+ /*
+ * Backup the queue page guest address to the mark EQ page
+ * dirty for migration.
+ */
+ q->guest_qaddr = kvm_eq.qaddr;
+ q->guest_qshift = kvm_eq.qshift;
+
+ /*
+ * Unconditional Notification is forced by default at the
+ * OPAL level because the use of END ESBs is not supported by
+ * Linux.
+ */
+ rc = xive_native_configure_queue(xc->vp_id, q, priority,
+ (__be32 *) qaddr, kvm_eq.qshift, true);
+ if (rc) {
+ pr_err("Failed to configure queue %d for VCPU %d: %d\n",
+ priority, xc->server_num, rc);
+ put_page(page);
+ return rc;
+ }
+
+ /*
+ * Only restore the queue state when needed. When doing the
+ * H_INT_SET_SOURCE_CONFIG hcall, it should not.
+ */
+ if (kvm_eq.qtoggle != 1 || kvm_eq.qindex != 0) {
+ rc = xive_native_set_queue_state(xc->vp_id, priority,
+ kvm_eq.qtoggle,
+ kvm_eq.qindex);
+ if (rc)
+ goto error;
+ }
+
+ rc = kvmppc_xive_attach_escalation(vcpu, priority,
+ xive->single_escalation);
+error:
+ if (rc)
+ kvmppc_xive_native_cleanup_queue(vcpu, priority);
+ return rc;
+}
+
+static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
+ long eq_idx, u64 addr)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ struct kvmppc_xive_vcpu *xc;
+ struct xive_q *q;
+ void __user *ubufp = (u64 __user *) addr;
+ u32 server;
+ u8 priority;
+ struct kvm_ppc_xive_eq kvm_eq;
+ u64 qaddr;
+ u64 qshift;
+ u64 qeoi_page;
+ u32 escalate_irq;
+ u64 qflags;
+ int rc;
+
+ /*
+ * Demangle priority/server tuple from the EQ identifier
+ */
+ priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
+ KVM_XIVE_EQ_PRIORITY_SHIFT;
+ server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
+ KVM_XIVE_EQ_SERVER_SHIFT;
+
+ vcpu = kvmppc_xive_find_server(kvm, server);
+ if (!vcpu) {
+ pr_err("Can't find server %d\n", server);
+ return -ENOENT;
+ }
+ xc = vcpu->arch.xive_vcpu;
+
+ if (priority != xive_prio_from_guest(priority)) {
+ pr_err("invalid priority for queue %d for VCPU %d\n",
+ priority, server);
+ return -EINVAL;
+ }
+ q = &xc->queues[priority];
+
+ memset(&kvm_eq, 0, sizeof(kvm_eq));
+
+ if (!q->qpage)
+ return 0;
+
+ rc = xive_native_get_queue_info(xc->vp_id, priority, &qaddr, &qshift,
+ &qeoi_page, &escalate_irq, &qflags);
+ if (rc)
+ return rc;
+
+ kvm_eq.flags = 0;
+ if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
+ kvm_eq.flags |= KVM_XIVE_EQ_ALWAYS_NOTIFY;
+
+ kvm_eq.qshift = q->guest_qshift;
+ kvm_eq.qaddr = q->guest_qaddr;
+
+ rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
+ &kvm_eq.qindex);
+ if (rc)
+ return rc;
+
+ pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
+ __func__, server, priority, kvm_eq.flags,
+ kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
+
+ if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
+ return -EFAULT;
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -349,6 +586,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
return kvmppc_xive_native_set_source_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_set_queue_config(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -356,6 +596,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
+ struct kvmppc_xive *xive = dev->private;
+
+ switch (attr->group) {
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return kvmppc_xive_native_get_queue_config(xive, attr->attr,
+ attr->addr);
+ }
return -ENXIO;
}
@@ -371,6 +618,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
break;
+ case KVM_DEV_XIVE_GRP_EQ_CONFIG:
+ return 0;
}
return -ENXIO;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 33c64b2cdbe8..cc13bfd5cf53 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -53,3 +53,37 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENXIO: CPU event queues not configured or configuration of the
underlying HW interrupt failed
-EBUSY: No CPU available to serve interrupt
+
+ 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
+ Configures an event queue of a CPU
+ Attributes:
+ EQ descriptor identifier (64-bit)
+ The EQ descriptor identifier is a tuple (server, priority) :
+ bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
+ values: | unused | server | priority
+ The kvm_device_attr.addr points to :
+ struct kvm_ppc_xive_eq {
+ __u32 flags;
+ __u32 qshift;
+ __u64 qaddr;
+ __u32 qtoggle;
+ __u32 qindex;
+ __u8 pad[40];
+ };
+ - flags: queue flags
+ KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
+ forces notification without using the coalescing mechanism
+ provided by the XIVE END ESBs.
+ - qshift: queue size (power of 2)
+ - qaddr: real address of queue
+ - qtoggle: current queue toggle bit
+ - qindex: current queue index
+ - pad: reserved for future use
+ Errors:
+ -ENOENT: Invalid CPU number
+ -EINVAL: Invalid priority
+ -EINVAL: Invalid flags
+ -EINVAL: Invalid queue size
+ -EINVAL: Invalid queue address
+ -EFAULT: Invalid user pointer for attr->addr.
+ -EIO: Configuration of the underlying HW failed
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This control is to be used by the H_INT_RESET hcall from QEMU. Its
purpose is to clear all configuration of the sources and EQs. This is
necessary in case of a kexec (for a kdump kernel for instance) to make
sure that no remaining configuration is left from the previous boot
setup so that the new kernel can start safely from a clean state.
The queue 7 is ignored when the XIVE device is configured to run in
single escalation mode. Prio 7 is used by escalations.
The XIVE VP is kept enabled as the vCPU is still active and connected
to the XIVE device.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 85 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 5 ++
3 files changed, 91 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 85005400fd86..f045f9dee42e 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_RESET 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 3e7cdcacc932..b9597d80c95a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -572,6 +572,83 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
return 0;
}
+static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
+{
+ int i;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
+
+ if (!state->valid)
+ continue;
+
+ if (state->act_priority == MASKED)
+ continue;
+
+ state->eisn = 0;
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+ if (state->pt_number) {
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->pt_number,
+ 0, MASKED, 0);
+ }
+ }
+}
+
+static int kvmppc_xive_reset(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ continue;
+
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+
+ /* Single escalation, no queue 7 */
+ if (prio == 7 && xive->single_escalation)
+ break;
+
+ if (xc->esc_virq[prio]) {
+ free_irq(xc->esc_virq[prio], vcpu);
+ irq_dispose_mapping(xc->esc_virq[prio]);
+ kfree(xc->esc_virq_names[prio]);
+ xc->esc_virq[prio] = 0;
+ }
+
+ kvmppc_xive_native_cleanup_queue(vcpu, prio);
+ }
+ }
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (sb) {
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_reset_sources(sb);
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -579,6 +656,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return kvmppc_xive_reset(xive);
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
@@ -611,6 +692,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
{
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return 0;
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cc13bfd5cf53..429cbc4cf960 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+ Attributes:
+ 1.1 KVM_DEV_XIVE_RESET (write only)
+ Resets the interrupt controller configuration for sources and event
+ queues. To be used by kexec and kdump.
+ Errors: none
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control is to be used by the H_INT_RESET hcall from QEMU. Its
purpose is to clear all configuration of the sources and EQs. This is
necessary in case of a kexec (for a kdump kernel for instance) to make
sure that no remaining configuration is left from the previous boot
setup so that the new kernel can start safely from a clean state.
The queue 7 is ignored when the XIVE device is configured to run in
single escalation mode. Prio 7 is used by escalations.
The XIVE VP is kept enabled as the vCPU is still active and connected
to the XIVE device.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 85 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 5 ++
3 files changed, 91 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 85005400fd86..f045f9dee42e 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_RESET 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 3e7cdcacc932..b9597d80c95a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -572,6 +572,83 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
return 0;
}
+static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
+{
+ int i;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
+
+ if (!state->valid)
+ continue;
+
+ if (state->act_priority == MASKED)
+ continue;
+
+ state->eisn = 0;
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+ if (state->pt_number) {
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->pt_number,
+ 0, MASKED, 0);
+ }
+ }
+}
+
+static int kvmppc_xive_reset(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ continue;
+
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+
+ /* Single escalation, no queue 7 */
+ if (prio == 7 && xive->single_escalation)
+ break;
+
+ if (xc->esc_virq[prio]) {
+ free_irq(xc->esc_virq[prio], vcpu);
+ irq_dispose_mapping(xc->esc_virq[prio]);
+ kfree(xc->esc_virq_names[prio]);
+ xc->esc_virq[prio] = 0;
+ }
+
+ kvmppc_xive_native_cleanup_queue(vcpu, prio);
+ }
+ }
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (sb) {
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_reset_sources(sb);
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -579,6 +656,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return kvmppc_xive_reset(xive);
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
@@ -611,6 +692,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
{
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return 0;
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cc13bfd5cf53..429cbc4cf960 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+ Attributes:
+ 1.1 KVM_DEV_XIVE_RESET (write only)
+ Resets the interrupt controller configuration for sources and event
+ queues. To be used by kexec and kdump.
+ Errors: none
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This control is to be used by the H_INT_RESET hcall from QEMU. Its
purpose is to clear all configuration of the sources and EQs. This is
necessary in case of a kexec (for a kdump kernel for instance) to make
sure that no remaining configuration is left from the previous boot
setup so that the new kernel can start safely from a clean state.
The queue 7 is ignored when the XIVE device is configured to run in
single escalation mode. Prio 7 is used by escalations.
The XIVE VP is kept enabled as the vCPU is still active and connected
to the XIVE device.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 85 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 5 ++
3 files changed, 91 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 85005400fd86..f045f9dee42e 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -679,6 +679,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
+#define KVM_DEV_XIVE_RESET 1
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 3e7cdcacc932..b9597d80c95a 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -572,6 +572,83 @@ static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
return 0;
}
+static void kvmppc_xive_reset_sources(struct kvmppc_xive_src_block *sb)
+{
+ int i;
+
+ for (i = 0; i < KVMPPC_XICS_IRQ_PER_ICS; i++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[i];
+
+ if (!state->valid)
+ continue;
+
+ if (state->act_priority = MASKED)
+ continue;
+
+ state->eisn = 0;
+ state->act_server = 0;
+ state->act_priority = MASKED;
+ xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->ipi_number, 0, MASKED, 0);
+ if (state->pt_number) {
+ xive_vm_esb_load(state->pt_data, XIVE_ESB_SET_PQ_01);
+ xive_native_configure_irq(state->pt_number,
+ 0, MASKED, 0);
+ }
+ }
+}
+
+static int kvmppc_xive_reset(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ continue;
+
+ kvmppc_xive_disable_vcpu_interrupts(vcpu);
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+
+ /* Single escalation, no queue 7 */
+ if (prio = 7 && xive->single_escalation)
+ break;
+
+ if (xc->esc_virq[prio]) {
+ free_irq(xc->esc_virq[prio], vcpu);
+ irq_dispose_mapping(xc->esc_virq[prio]);
+ kfree(xc->esc_virq_names[prio]);
+ xc->esc_virq[prio] = 0;
+ }
+
+ kvmppc_xive_native_cleanup_queue(vcpu, prio);
+ }
+ }
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (sb) {
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_reset_sources(sb);
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -579,6 +656,10 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return kvmppc_xive_reset(xive);
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
return kvmppc_xive_native_set_source(xive, attr->attr,
@@ -611,6 +692,10 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
{
switch (attr->group) {
case KVM_DEV_XIVE_GRP_CTRL:
+ switch (attr->attr) {
+ case KVM_DEV_XIVE_RESET:
+ return 0;
+ }
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index cc13bfd5cf53..429cbc4cf960 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -17,6 +17,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
1. KVM_DEV_XIVE_GRP_CTRL
Provides global controls on the device
+ Attributes:
+ 1.1 KVM_DEV_XIVE_RESET (write only)
+ Resets the interrupt controller configuration for sources and event
+ queues. To be used by kexec and kdump.
+ Errors: none
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This control will be used by the H_INT_SYNC hcall from QEMU to flush
event notifications on the XIVE IC owning the source.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 36 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 8 +++++
3 files changed, 45 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index f045f9dee42e..e4abe30f6fc6 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -683,6 +683,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
+#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index b9597d80c95a..65380416d101 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -335,6 +335,38 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, masked, eisn);
}
+static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ int rc = 0;
+
+ pr_devel("%s irq=0x%lx", __func__, irq);
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ rc = -EINVAL;
+
+ arch_spin_lock(&sb->lock);
+
+ if (state->valid) {
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ rc = 0;
+ }
+
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
static int xive_native_validate_queue_size(u32 qshift)
{
/*
@@ -670,6 +702,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_EQ_CONFIG:
return kvmppc_xive_native_set_queue_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
+ return kvmppc_xive_native_sync_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -699,6 +734,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 429cbc4cf960..1e7f19d7594b 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -92,3 +92,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EINVAL: Invalid queue address
-EFAULT: Invalid user pointer for attr->addr.
-EIO: Configuration of the underlying HW failed
+
+ 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
+ Synchronize the source to flush event notifications
+ Attributes:
+ Interrupt source number (64-bit)
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
This control will be used by the H_INT_SYNC hcall from QEMU to flush
event notifications on the XIVE IC owning the source.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 36 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 8 +++++
3 files changed, 45 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index f045f9dee42e..e4abe30f6fc6 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -683,6 +683,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
+#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index b9597d80c95a..65380416d101 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -335,6 +335,38 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, masked, eisn);
}
+static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ int rc = 0;
+
+ pr_devel("%s irq=0x%lx", __func__, irq);
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ rc = -EINVAL;
+
+ arch_spin_lock(&sb->lock);
+
+ if (state->valid) {
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ rc = 0;
+ }
+
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
static int xive_native_validate_queue_size(u32 qshift)
{
/*
@@ -670,6 +702,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_EQ_CONFIG:
return kvmppc_xive_native_set_queue_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
+ return kvmppc_xive_native_sync_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -699,6 +734,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 429cbc4cf960..1e7f19d7594b 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -92,3 +92,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EINVAL: Invalid queue address
-EFAULT: Invalid user pointer for attr->addr.
-EIO: Configuration of the underlying HW failed
+
+ 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
+ Synchronize the source to flush event notifications
+ Attributes:
+ Interrupt source number (64-bit)
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
This control will be used by the H_INT_SYNC hcall from QEMU to flush
event notifications on the XIVE IC owning the source.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 36 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 8 +++++
3 files changed, 45 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index f045f9dee42e..e4abe30f6fc6 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -683,6 +683,7 @@ struct kvm_ppc_cpu_char {
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
+#define KVM_DEV_XIVE_GRP_SOURCE_SYNC 5 /* 64-bit source identifier */
/* Layout of 64-bit XIVE source attribute values */
#define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index b9597d80c95a..65380416d101 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -335,6 +335,38 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
priority, masked, eisn);
}
+static int kvmppc_xive_native_sync_source(struct kvmppc_xive *xive,
+ long irq, u64 addr)
+{
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ int rc = 0;
+
+ pr_devel("%s irq=0x%lx", __func__, irq);
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb)
+ return -ENOENT;
+
+ state = &sb->irq_state[src];
+
+ rc = -EINVAL;
+
+ arch_spin_lock(&sb->lock);
+
+ if (state->valid) {
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ rc = 0;
+ }
+
+ arch_spin_unlock(&sb->lock);
+ return rc;
+}
+
static int xive_native_validate_queue_size(u32 qshift)
{
/*
@@ -670,6 +702,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_EQ_CONFIG:
return kvmppc_xive_native_set_queue_config(xive, attr->attr,
attr->addr);
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
+ return kvmppc_xive_native_sync_source(xive, attr->attr,
+ attr->addr);
}
return -ENXIO;
}
@@ -699,6 +734,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
break;
case KVM_DEV_XIVE_GRP_SOURCE:
case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
+ case KVM_DEV_XIVE_GRP_SOURCE_SYNC:
if (attr->attr >= KVMPPC_XIVE_FIRST_IRQ &&
attr->attr < KVMPPC_XIVE_NR_IRQS)
return 0;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 429cbc4cf960..1e7f19d7594b 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -92,3 +92,11 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-EINVAL: Invalid queue address
-EFAULT: Invalid user pointer for attr->addr.
-EIO: Configuration of the underlying HW failed
+
+ 5. KVM_DEV_XIVE_GRP_SOURCE_SYNC (write only)
+ Synchronize the source to flush event notifications
+ Attributes:
+ Interrupt source number (64-bit)
+ Errors:
+ -ENOENT: Unknown source number
+ -EINVAL: Not initialized source number
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
When migration of a VM is initiated, a first copy of the RAM is
transferred to the destination before the VM is stopped, but there is
no guarantee that the EQ pages in which the event notifications are
queued have not been modified.
To make sure migration will capture a consistent memory state, the
XIVE device should perform a XIVE quiesce sequence to stop the flow of
event notifications and stabilize the EQs. This is the purpose of the
KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
to force their transfer.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 85 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++
3 files changed, 115 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index e4abe30f6fc6..12744608a61c 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -680,6 +680,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_RESET 1
+#define KVM_DEV_XIVE_EQ_SYNC 2
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 65380416d101..16d23ef3bd39 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -681,6 +681,88 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
return 0;
}
+static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
+{
+ int j;
+
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
+ struct xive_irq_data *xd;
+ u32 hw_num;
+
+ if (!state->valid)
+ continue;
+
+ /*
+ * The struct kvmppc_xive_irq_state reflects the state
+ * of the EAS configuration and not the state of the
+ * source. The source is masked setting the PQ bits to
+ * '-Q', which is what is being done before calling
+ * the KVM_DEV_XIVE_EQ_SYNC control.
+ *
+ * If a source EAS is configured, OPAL syncs the XIVE
+ * IC of the source and the XIVE IC of the previous
+ * target if any.
+ *
+ * So it should be fine ignoring MASKED sources as
+ * they have been synced already.
+ */
+ if (state->act_priority == MASKED)
+ continue;
+
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ xive_native_sync_queue(hw_num);
+ }
+}
+
+static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ return -ENOENT;
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+ struct xive_q *q = &xc->queues[prio];
+
+ if (!q->qpage)
+ continue;
+
+ /* Mark EQ page dirty for migration */
+ mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qaddr));
+ }
+ return 0;
+}
+
+static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (sb) {
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_native_sync_sources(sb);
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ kvmppc_xive_native_vcpu_eq_sync(vcpu);
+ }
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -691,6 +773,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
return kvmppc_xive_reset(xive);
+ case KVM_DEV_XIVE_EQ_SYNC:
+ return kvmppc_xive_native_eq_sync(xive);
}
break;
case KVM_DEV_XIVE_GRP_SOURCE:
@@ -729,6 +813,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
+ case KVM_DEV_XIVE_EQ_SYNC:
return 0;
}
break;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 1e7f19d7594b..7ffd4c7be7b5 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
queues. To be used by kexec and kdump.
Errors: none
+ 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
+ Sync all the sources and queues and mark the EQ pages dirty. This
+ to make sure that a consistent memory state is captured when
+ migrating the VM.
+ Errors: none
+
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
Attributes:
@@ -100,3 +106,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
Errors:
-ENOENT: Unknown source number
-EINVAL: Not initialized source number
+
+* Migration:
+
+ Saving the state of a VM using the XIVE native exploitation mode
+ should follow a specific sequence. When the VM is stopped :
+
+ 1. Mask all sources (PQ=01) to stop the flow of events.
+
+ 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
+ flush any in-flight event notification and to stabilize the EQs. At
+ this stage, the EQ pages are marked dirty to make sure they are
+ transferred in the migration sequence.
+
+ 3. Capture the state of the source targeting, the EQs configuration
+ and the state of thread interrupt context registers.
+
+ Restore is similar :
+
+ 1. Restore the EQ configuration. As targeting depends on it.
+ 2. Restore targeting
+ 3. Restore the thread interrupt contexts
+ 4. Restore the source states
+ 5. Let the vCPU run
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
When migration of a VM is initiated, a first copy of the RAM is
transferred to the destination before the VM is stopped, but there is
no guarantee that the EQ pages in which the event notifications are
queued have not been modified.
To make sure migration will capture a consistent memory state, the
XIVE device should perform a XIVE quiesce sequence to stop the flow of
event notifications and stabilize the EQs. This is the purpose of the
KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
to force their transfer.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 85 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++
3 files changed, 115 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index e4abe30f6fc6..12744608a61c 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -680,6 +680,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_RESET 1
+#define KVM_DEV_XIVE_EQ_SYNC 2
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 65380416d101..16d23ef3bd39 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -681,6 +681,88 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
return 0;
}
+static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
+{
+ int j;
+
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
+ struct xive_irq_data *xd;
+ u32 hw_num;
+
+ if (!state->valid)
+ continue;
+
+ /*
+ * The struct kvmppc_xive_irq_state reflects the state
+ * of the EAS configuration and not the state of the
+ * source. The source is masked setting the PQ bits to
+ * '-Q', which is what is being done before calling
+ * the KVM_DEV_XIVE_EQ_SYNC control.
+ *
+ * If a source EAS is configured, OPAL syncs the XIVE
+ * IC of the source and the XIVE IC of the previous
+ * target if any.
+ *
+ * So it should be fine ignoring MASKED sources as
+ * they have been synced already.
+ */
+ if (state->act_priority == MASKED)
+ continue;
+
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ xive_native_sync_queue(hw_num);
+ }
+}
+
+static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ return -ENOENT;
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+ struct xive_q *q = &xc->queues[prio];
+
+ if (!q->qpage)
+ continue;
+
+ /* Mark EQ page dirty for migration */
+ mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qaddr));
+ }
+ return 0;
+}
+
+static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (sb) {
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_native_sync_sources(sb);
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ kvmppc_xive_native_vcpu_eq_sync(vcpu);
+ }
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -691,6 +773,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
return kvmppc_xive_reset(xive);
+ case KVM_DEV_XIVE_EQ_SYNC:
+ return kvmppc_xive_native_eq_sync(xive);
}
break;
case KVM_DEV_XIVE_GRP_SOURCE:
@@ -729,6 +813,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
+ case KVM_DEV_XIVE_EQ_SYNC:
return 0;
}
break;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 1e7f19d7594b..7ffd4c7be7b5 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
queues. To be used by kexec and kdump.
Errors: none
+ 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
+ Sync all the sources and queues and mark the EQ pages dirty. This
+ to make sure that a consistent memory state is captured when
+ migrating the VM.
+ Errors: none
+
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
Attributes:
@@ -100,3 +106,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
Errors:
-ENOENT: Unknown source number
-EINVAL: Not initialized source number
+
+* Migration:
+
+ Saving the state of a VM using the XIVE native exploitation mode
+ should follow a specific sequence. When the VM is stopped :
+
+ 1. Mask all sources (PQ=01) to stop the flow of events.
+
+ 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
+ flush any in-flight event notification and to stabilize the EQs. At
+ this stage, the EQ pages are marked dirty to make sure they are
+ transferred in the migration sequence.
+
+ 3. Capture the state of the source targeting, the EQs configuration
+ and the state of thread interrupt context registers.
+
+ Restore is similar :
+
+ 1. Restore the EQ configuration. As targeting depends on it.
+ 2. Restore targeting
+ 3. Restore the thread interrupt contexts
+ 4. Restore the source states
+ 5. Let the vCPU run
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
When migration of a VM is initiated, a first copy of the RAM is
transferred to the destination before the VM is stopped, but there is
no guarantee that the EQ pages in which the event notifications are
queued have not been modified.
To make sure migration will capture a consistent memory state, the
XIVE device should perform a XIVE quiesce sequence to stop the flow of
event notifications and stabilize the EQs. This is the purpose of the
KVM_DEV_XIVE_EQ_SYNC control which will also marks the EQ pages dirty
to force their transfer.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 85 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 29 ++++++++
3 files changed, 115 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index e4abe30f6fc6..12744608a61c 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -680,6 +680,7 @@ struct kvm_ppc_cpu_char {
/* POWER9 XIVE Native Interrupt Controller */
#define KVM_DEV_XIVE_GRP_CTRL 1
#define KVM_DEV_XIVE_RESET 1
+#define KVM_DEV_XIVE_EQ_SYNC 2
#define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 65380416d101..16d23ef3bd39 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -681,6 +681,88 @@ static int kvmppc_xive_reset(struct kvmppc_xive *xive)
return 0;
}
+static void kvmppc_xive_native_sync_sources(struct kvmppc_xive_src_block *sb)
+{
+ int j;
+
+ for (j = 0; j < KVMPPC_XICS_IRQ_PER_ICS; j++) {
+ struct kvmppc_xive_irq_state *state = &sb->irq_state[j];
+ struct xive_irq_data *xd;
+ u32 hw_num;
+
+ if (!state->valid)
+ continue;
+
+ /*
+ * The struct kvmppc_xive_irq_state reflects the state
+ * of the EAS configuration and not the state of the
+ * source. The source is masked setting the PQ bits to
+ * '-Q', which is what is being done before calling
+ * the KVM_DEV_XIVE_EQ_SYNC control.
+ *
+ * If a source EAS is configured, OPAL syncs the XIVE
+ * IC of the source and the XIVE IC of the previous
+ * target if any.
+ *
+ * So it should be fine ignoring MASKED sources as
+ * they have been synced already.
+ */
+ if (state->act_priority = MASKED)
+ continue;
+
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+ xive_native_sync_source(hw_num);
+ xive_native_sync_queue(hw_num);
+ }
+}
+
+static int kvmppc_xive_native_vcpu_eq_sync(struct kvm_vcpu *vcpu)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ unsigned int prio;
+
+ if (!xc)
+ return -ENOENT;
+
+ for (prio = 0; prio < KVMPPC_XIVE_Q_COUNT; prio++) {
+ struct xive_q *q = &xc->queues[prio];
+
+ if (!q->qpage)
+ continue;
+
+ /* Mark EQ page dirty for migration */
+ mark_page_dirty(vcpu->kvm, gpa_to_gfn(q->guest_qaddr));
+ }
+ return 0;
+}
+
+static int kvmppc_xive_native_eq_sync(struct kvmppc_xive *xive)
+{
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ unsigned int i;
+
+ pr_devel("%s\n", __func__);
+
+ mutex_lock(&kvm->lock);
+ for (i = 0; i <= xive->max_sbid; i++) {
+ struct kvmppc_xive_src_block *sb = xive->src_blocks[i];
+
+ if (sb) {
+ arch_spin_lock(&sb->lock);
+ kvmppc_xive_native_sync_sources(sb);
+ arch_spin_unlock(&sb->lock);
+ }
+ }
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ kvmppc_xive_native_vcpu_eq_sync(vcpu);
+ }
+ mutex_unlock(&kvm->lock);
+
+ return 0;
+}
+
static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
struct kvm_device_attr *attr)
{
@@ -691,6 +773,8 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
return kvmppc_xive_reset(xive);
+ case KVM_DEV_XIVE_EQ_SYNC:
+ return kvmppc_xive_native_eq_sync(xive);
}
break;
case KVM_DEV_XIVE_GRP_SOURCE:
@@ -729,6 +813,7 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
case KVM_DEV_XIVE_GRP_CTRL:
switch (attr->attr) {
case KVM_DEV_XIVE_RESET:
+ case KVM_DEV_XIVE_EQ_SYNC:
return 0;
}
break;
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 1e7f19d7594b..7ffd4c7be7b5 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -23,6 +23,12 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
queues. To be used by kexec and kdump.
Errors: none
+ 1.2 KVM_DEV_XIVE_EQ_SYNC (write only)
+ Sync all the sources and queues and mark the EQ pages dirty. This
+ to make sure that a consistent memory state is captured when
+ migrating the VM.
+ Errors: none
+
2. KVM_DEV_XIVE_GRP_SOURCE (write only)
Initializes a new source in the XIVE device and mask it.
Attributes:
@@ -100,3 +106,26 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
Errors:
-ENOENT: Unknown source number
-EINVAL: Not initialized source number
+
+* Migration:
+
+ Saving the state of a VM using the XIVE native exploitation mode
+ should follow a specific sequence. When the VM is stopped :
+
+ 1. Mask all sources (PQ\x01) to stop the flow of events.
+
+ 2. Sync the XIVE device with the KVM control KVM_DEV_XIVE_EQ_SYNC to
+ flush any in-flight event notification and to stabilize the EQs. At
+ this stage, the EQ pages are marked dirty to make sure they are
+ transferred in the migration sequence.
+
+ 3. Capture the state of the source targeting, the EQs configuration
+ and the state of thread interrupt context registers.
+
+ Restore is similar :
+
+ 1. Restore the EQ configuration. As targeting depends on it.
+ 2. Restore targeting
+ 3. Restore the thread interrupt contexts
+ 4. Restore the source states
+ 5. Let the vCPU run
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The state of the thread interrupt management registers needs to be
collected for migration. These registers are cached under the
'xive_saved_state.w01' field of the VCPU when the VPCU context is
pulled from the HW thread. An OPAL call retrieves the backup of the
IPB register in the underlying XIVE NVT structure and merges it in the
KVM state.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
Changes since v4:
- add documentation in Documentation/virtual/kvm/api.txt
arch/powerpc/include/asm/kvm_ppc.h | 11 ++++
arch/powerpc/include/uapi/asm/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 24 +++++++
arch/powerpc/kvm/book3s_xive_native.c | 76 ++++++++++++++++++++++
Documentation/virtual/kvm/api.txt | 1 +
Documentation/virtual/kvm/devices/xive.txt | 17 +++++
6 files changed, 131 insertions(+)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 6928a35ac3c7..0579c9b253db 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -273,6 +273,7 @@ union kvmppc_one_reg {
u64 addr;
u64 length;
} vpaval;
+ u64 xive_timaval[2];
};
struct kvmppc_ops {
@@ -605,6 +606,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
+extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
+extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
@@ -637,6 +642,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
+static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return 0; }
+static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return -ENOENT; }
#endif /* CONFIG_KVM_XIVE */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 12744608a61c..cd3f16b70a2e 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -482,6 +482,8 @@ struct kvm_ppc_cpu_char {
#define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
#define KVM_REG_PPC_ICP_PPRI_MASK 0xff
+#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U128 | 0x8d)
+
/* Device control API: PPC-specific devices */
#define KVM_DEV_MPIC_GRP_MISC 1
#define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 7c3348fa27e1..efd15101eef0 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -651,6 +651,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
*val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_get_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
*val = get_reg_val(id, vcpu->arch.fscr);
break;
@@ -724,6 +736,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_set_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
vcpu->arch.fscr = set_reg_val(id, *val);
break;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 16d23ef3bd39..2f9d5e9439a6 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -896,6 +896,82 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
return ret;
}
+/*
+ * Interrupt Pending Buffer (IPB) offset
+ */
+#define TM_IPB_SHIFT 40
+#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
+
+int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u64 opal_state;
+ int rc;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc)
+ return -ENOENT;
+
+ /* Thread context registers. We only care about IPB and CPPR */
+ val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
+
+ /* Get the VP state from OPAL */
+ rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
+ if (rc)
+ return rc;
+
+ /*
+ * Capture the backup of IPB register in the NVT structure and
+ * merge it in our KVM VP state.
+ */
+ val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
+
+ pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
+ __func__,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word, opal_state);
+
+ return 0;
+}
+
+int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+
+ pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
+ val->xive_timaval[0], val->xive_timaval[1]);
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc || !xive)
+ return -ENOENT;
+
+ /* We can't update the state of a "pushed" VCPU */
+ if (WARN_ON(vcpu->arch.xive_pushed))
+ return -EBUSY;
+
+ /*
+ * Restore the thread context registers. IPB and CPPR should
+ * be the only ones that matter.
+ */
+ vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
+
+ /*
+ * There is no need to restore the XIVE internal state (IPB
+ * stored in the NVT) as the IPB register was merged in KVM VP
+ * state when captured.
+ */
+ return 0;
+}
+
static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index e38eb17b7be6..5b505520a616 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1985,6 +1985,7 @@ registers, find a list below:
PPC | KVM_REG_PPC_TLB3PS | 32
PPC | KVM_REG_PPC_EPTCFG | 32
PPC | KVM_REG_PPC_ICP_STATE | 64
+ PPC | KVM_REG_PPC_VP_STATE | 128
PPC | KVM_REG_PPC_TB_OFFSET | 64
PPC | KVM_REG_PPC_SPMC1 | 32
PPC | KVM_REG_PPC_SPMC2 | 32
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 7ffd4c7be7b5..525d1eebcf34 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -107,6 +107,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOENT: Unknown source number
-EINVAL: Not initialized source number
+* VCPU state
+
+ The XIVE IC maintains VP interrupt state in an internal structure
+ called the NVT. When a VP is not dispatched on a HW processor
+ thread, this structure can be updated by HW if the VP is the target
+ of an event notification.
+
+ It is important for migration to capture the cached IPB from the NVT
+ as it synthesizes the priorities of the pending interrupts. We
+ capture a bit more to report debug information.
+
+ KVM_REG_PPC_VP_STATE (2 * 64bits)
+ bits: | 63 .... 32 | 31 .... 0 |
+ values: | TIMA word0 | TIMA word1 |
+ bits: | 127 .......... 64 |
+ values: | unused |
+
* Migration:
Saving the state of a VM using the XIVE native exploitation mode
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The state of the thread interrupt management registers needs to be
collected for migration. These registers are cached under the
'xive_saved_state.w01' field of the VCPU when the VPCU context is
pulled from the HW thread. An OPAL call retrieves the backup of the
IPB register in the underlying XIVE NVT structure and merges it in the
KVM state.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
Changes since v4:
- add documentation in Documentation/virtual/kvm/api.txt
arch/powerpc/include/asm/kvm_ppc.h | 11 ++++
arch/powerpc/include/uapi/asm/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 24 +++++++
arch/powerpc/kvm/book3s_xive_native.c | 76 ++++++++++++++++++++++
Documentation/virtual/kvm/api.txt | 1 +
Documentation/virtual/kvm/devices/xive.txt | 17 +++++
6 files changed, 131 insertions(+)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 6928a35ac3c7..0579c9b253db 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -273,6 +273,7 @@ union kvmppc_one_reg {
u64 addr;
u64 length;
} vpaval;
+ u64 xive_timaval[2];
};
struct kvmppc_ops {
@@ -605,6 +606,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
+extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
+extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
@@ -637,6 +642,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
+static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return 0; }
+static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return -ENOENT; }
#endif /* CONFIG_KVM_XIVE */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 12744608a61c..cd3f16b70a2e 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -482,6 +482,8 @@ struct kvm_ppc_cpu_char {
#define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
#define KVM_REG_PPC_ICP_PPRI_MASK 0xff
+#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U128 | 0x8d)
+
/* Device control API: PPC-specific devices */
#define KVM_DEV_MPIC_GRP_MISC 1
#define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 7c3348fa27e1..efd15101eef0 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -651,6 +651,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
*val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_get_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
*val = get_reg_val(id, vcpu->arch.fscr);
break;
@@ -724,6 +736,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_set_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
vcpu->arch.fscr = set_reg_val(id, *val);
break;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 16d23ef3bd39..2f9d5e9439a6 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -896,6 +896,82 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
return ret;
}
+/*
+ * Interrupt Pending Buffer (IPB) offset
+ */
+#define TM_IPB_SHIFT 40
+#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
+
+int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u64 opal_state;
+ int rc;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc)
+ return -ENOENT;
+
+ /* Thread context registers. We only care about IPB and CPPR */
+ val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
+
+ /* Get the VP state from OPAL */
+ rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
+ if (rc)
+ return rc;
+
+ /*
+ * Capture the backup of IPB register in the NVT structure and
+ * merge it in our KVM VP state.
+ */
+ val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
+
+ pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
+ __func__,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word, opal_state);
+
+ return 0;
+}
+
+int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+
+ pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
+ val->xive_timaval[0], val->xive_timaval[1]);
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc || !xive)
+ return -ENOENT;
+
+ /* We can't update the state of a "pushed" VCPU */
+ if (WARN_ON(vcpu->arch.xive_pushed))
+ return -EBUSY;
+
+ /*
+ * Restore the thread context registers. IPB and CPPR should
+ * be the only ones that matter.
+ */
+ vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
+
+ /*
+ * There is no need to restore the XIVE internal state (IPB
+ * stored in the NVT) as the IPB register was merged in KVM VP
+ * state when captured.
+ */
+ return 0;
+}
+
static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index e38eb17b7be6..5b505520a616 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1985,6 +1985,7 @@ registers, find a list below:
PPC | KVM_REG_PPC_TLB3PS | 32
PPC | KVM_REG_PPC_EPTCFG | 32
PPC | KVM_REG_PPC_ICP_STATE | 64
+ PPC | KVM_REG_PPC_VP_STATE | 128
PPC | KVM_REG_PPC_TB_OFFSET | 64
PPC | KVM_REG_PPC_SPMC1 | 32
PPC | KVM_REG_PPC_SPMC2 | 32
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 7ffd4c7be7b5..525d1eebcf34 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -107,6 +107,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOENT: Unknown source number
-EINVAL: Not initialized source number
+* VCPU state
+
+ The XIVE IC maintains VP interrupt state in an internal structure
+ called the NVT. When a VP is not dispatched on a HW processor
+ thread, this structure can be updated by HW if the VP is the target
+ of an event notification.
+
+ It is important for migration to capture the cached IPB from the NVT
+ as it synthesizes the priorities of the pending interrupts. We
+ capture a bit more to report debug information.
+
+ KVM_REG_PPC_VP_STATE (2 * 64bits)
+ bits: | 63 .... 32 | 31 .... 0 |
+ values: | TIMA word0 | TIMA word1 |
+ bits: | 127 .......... 64 |
+ values: | unused |
+
* Migration:
Saving the state of a VM using the XIVE native exploitation mode
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The state of the thread interrupt management registers needs to be
collected for migration. These registers are cached under the
'xive_saved_state.w01' field of the VCPU when the VPCU context is
pulled from the HW thread. An OPAL call retrieves the backup of the
IPB register in the underlying XIVE NVT structure and merges it in the
KVM state.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
Changes since v4:
- add documentation in Documentation/virtual/kvm/api.txt
arch/powerpc/include/asm/kvm_ppc.h | 11 ++++
arch/powerpc/include/uapi/asm/kvm.h | 2 +
arch/powerpc/kvm/book3s.c | 24 +++++++
arch/powerpc/kvm/book3s_xive_native.c | 76 ++++++++++++++++++++++
Documentation/virtual/kvm/api.txt | 1 +
Documentation/virtual/kvm/devices/xive.txt | 17 +++++
6 files changed, 131 insertions(+)
diff --git a/arch/powerpc/include/asm/kvm_ppc.h b/arch/powerpc/include/asm/kvm_ppc.h
index 6928a35ac3c7..0579c9b253db 100644
--- a/arch/powerpc/include/asm/kvm_ppc.h
+++ b/arch/powerpc/include/asm/kvm_ppc.h
@@ -273,6 +273,7 @@ union kvmppc_one_reg {
u64 addr;
u64 length;
} vpaval;
+ u64 xive_timaval[2];
};
struct kvmppc_ops {
@@ -605,6 +606,10 @@ extern int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
extern void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu);
extern void kvmppc_xive_native_init_module(void);
extern void kvmppc_xive_native_exit_module(void);
+extern int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
+extern int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val);
#else
static inline int kvmppc_xive_set_xive(struct kvm *kvm, u32 irq, u32 server,
@@ -637,6 +642,12 @@ static inline int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
static inline void kvmppc_xive_native_cleanup_vcpu(struct kvm_vcpu *vcpu) { }
static inline void kvmppc_xive_native_init_module(void) { }
static inline void kvmppc_xive_native_exit_module(void) { }
+static inline int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return 0; }
+static inline int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu,
+ union kvmppc_one_reg *val)
+{ return -ENOENT; }
#endif /* CONFIG_KVM_XIVE */
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 12744608a61c..cd3f16b70a2e 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -482,6 +482,8 @@ struct kvm_ppc_cpu_char {
#define KVM_REG_PPC_ICP_PPRI_SHIFT 16 /* pending irq priority */
#define KVM_REG_PPC_ICP_PPRI_MASK 0xff
+#define KVM_REG_PPC_VP_STATE (KVM_REG_PPC | KVM_REG_SIZE_U128 | 0x8d)
+
/* Device control API: PPC-specific devices */
#define KVM_DEV_MPIC_GRP_MISC 1
#define KVM_DEV_MPIC_BASE_ADDR 0 /* 64-bit */
diff --git a/arch/powerpc/kvm/book3s.c b/arch/powerpc/kvm/book3s.c
index 7c3348fa27e1..efd15101eef0 100644
--- a/arch/powerpc/kvm/book3s.c
+++ b/arch/powerpc/kvm/book3s.c
@@ -651,6 +651,18 @@ int kvmppc_get_one_reg(struct kvm_vcpu *vcpu, u64 id,
*val = get_reg_val(id, kvmppc_xics_get_icp(vcpu));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_get_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
*val = get_reg_val(id, vcpu->arch.fscr);
break;
@@ -724,6 +736,18 @@ int kvmppc_set_one_reg(struct kvm_vcpu *vcpu, u64 id,
r = kvmppc_xics_set_icp(vcpu, set_reg_val(id, *val));
break;
#endif /* CONFIG_KVM_XICS */
+#ifdef CONFIG_KVM_XIVE
+ case KVM_REG_PPC_VP_STATE:
+ if (!vcpu->arch.xive_vcpu) {
+ r = -ENXIO;
+ break;
+ }
+ if (xive_enabled())
+ r = kvmppc_xive_native_set_vp(vcpu, val);
+ else
+ r = -ENXIO;
+ break;
+#endif /* CONFIG_KVM_XIVE */
case KVM_REG_PPC_FSCR:
vcpu->arch.fscr = set_reg_val(id, *val);
break;
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 16d23ef3bd39..2f9d5e9439a6 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -896,6 +896,82 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
return ret;
}
+/*
+ * Interrupt Pending Buffer (IPB) offset
+ */
+#define TM_IPB_SHIFT 40
+#define TM_IPB_MASK (((u64) 0xFF) << TM_IPB_SHIFT)
+
+int kvmppc_xive_native_get_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ u64 opal_state;
+ int rc;
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc)
+ return -ENOENT;
+
+ /* Thread context registers. We only care about IPB and CPPR */
+ val->xive_timaval[0] = vcpu->arch.xive_saved_state.w01;
+
+ /* Get the VP state from OPAL */
+ rc = xive_native_get_vp_state(xc->vp_id, &opal_state);
+ if (rc)
+ return rc;
+
+ /*
+ * Capture the backup of IPB register in the NVT structure and
+ * merge it in our KVM VP state.
+ */
+ val->xive_timaval[0] |= cpu_to_be64(opal_state & TM_IPB_MASK);
+
+ pr_devel("%s NSR=%02x CPPR=%02x IBP=%02x PIPR=%02x w01=%016llx w2=%08x opal=%016llx\n",
+ __func__,
+ vcpu->arch.xive_saved_state.nsr,
+ vcpu->arch.xive_saved_state.cppr,
+ vcpu->arch.xive_saved_state.ipb,
+ vcpu->arch.xive_saved_state.pipr,
+ vcpu->arch.xive_saved_state.w01,
+ (u32) vcpu->arch.xive_cam_word, opal_state);
+
+ return 0;
+}
+
+int kvmppc_xive_native_set_vp(struct kvm_vcpu *vcpu, union kvmppc_one_reg *val)
+{
+ struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
+ struct kvmppc_xive *xive = vcpu->kvm->arch.xive;
+
+ pr_devel("%s w01=%016llx vp=%016llx\n", __func__,
+ val->xive_timaval[0], val->xive_timaval[1]);
+
+ if (!kvmppc_xive_enabled(vcpu))
+ return -EPERM;
+
+ if (!xc || !xive)
+ return -ENOENT;
+
+ /* We can't update the state of a "pushed" VCPU */
+ if (WARN_ON(vcpu->arch.xive_pushed))
+ return -EBUSY;
+
+ /*
+ * Restore the thread context registers. IPB and CPPR should
+ * be the only ones that matter.
+ */
+ vcpu->arch.xive_saved_state.w01 = val->xive_timaval[0];
+
+ /*
+ * There is no need to restore the XIVE internal state (IPB
+ * stored in the NVT) as the IPB register was merged in KVM VP
+ * state when captured.
+ */
+ return 0;
+}
+
static int xive_native_debug_show(struct seq_file *m, void *private)
{
struct kvmppc_xive *xive = m->private;
diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index e38eb17b7be6..5b505520a616 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1985,6 +1985,7 @@ registers, find a list below:
PPC | KVM_REG_PPC_TLB3PS | 32
PPC | KVM_REG_PPC_EPTCFG | 32
PPC | KVM_REG_PPC_ICP_STATE | 64
+ PPC | KVM_REG_PPC_VP_STATE | 128
PPC | KVM_REG_PPC_TB_OFFSET | 64
PPC | KVM_REG_PPC_SPMC1 | 32
PPC | KVM_REG_PPC_SPMC2 | 32
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 7ffd4c7be7b5..525d1eebcf34 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -107,6 +107,23 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
-ENOENT: Unknown source number
-EINVAL: Not initialized source number
+* VCPU state
+
+ The XIVE IC maintains VP interrupt state in an internal structure
+ called the NVT. When a VP is not dispatched on a HW processor
+ thread, this structure can be updated by HW if the VP is the target
+ of an event notification.
+
+ It is important for migration to capture the cached IPB from the NVT
+ as it synthesizes the priorities of the pending interrupts. We
+ capture a bit more to report debug information.
+
+ KVM_REG_PPC_VP_STATE (2 * 64bits)
+ bits: | 63 .... 32 | 31 .... 0 |
+ values: | TIMA word0 | TIMA word1 |
+ bits: | 127 .......... 64 |
+ values: | unused |
+
* Migration:
Saving the state of a VM using the XIVE native exploitation mode
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 11/16] KVM: introduce a 'mmap' method for KVM devices
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater, Paolo Bonzini
Some KVM devices will want to handle special mappings related to the
underlying HW. For instance, the XIVE interrupt controller of the
POWER9 processor has MMIO pages for thread interrupt management and
for interrupt source control that need to be exposed to the guest when
the OS has the required support.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
include/linux/kvm_host.h | 1 +
virt/kvm/kvm_main.c | 11 +++++++++++
2 files changed, 12 insertions(+)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 9d55c63db09b..831d963451d8 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1245,6 +1245,7 @@ struct kvm_device_ops {
int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
+ int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 55fe8e20d8fd..ea2018ae1cd7 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2884,6 +2884,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
}
#endif
+static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct kvm_device *dev = filp->private_data;
+
+ if (dev->ops->mmap)
+ return dev->ops->mmap(dev, vma);
+
+ return -ENODEV;
+}
+
static int kvm_device_ioctl_attr(struct kvm_device *dev,
int (*accessor)(struct kvm_device *dev,
struct kvm_device_attr *attr),
@@ -2936,6 +2946,7 @@ static const struct file_operations kvm_device_fops = {
.unlocked_ioctl = kvm_device_ioctl,
.release = kvm_device_release,
KVM_COMPAT(kvm_device_ioctl),
+ .mmap = kvm_device_mmap,
};
struct kvm_device *kvm_device_from_filp(struct file *filp)
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, Paolo Bonzini,
linuxppc-dev, David Gibson
Some KVM devices will want to handle special mappings related to the
underlying HW. For instance, the XIVE interrupt controller of the
POWER9 processor has MMIO pages for thread interrupt management and
for interrupt source control that need to be exposed to the guest when
the OS has the required support.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
include/linux/kvm_host.h | 1 +
virt/kvm/kvm_main.c | 11 +++++++++++
2 files changed, 12 insertions(+)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 9d55c63db09b..831d963451d8 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1245,6 +1245,7 @@ struct kvm_device_ops {
int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
+ int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 55fe8e20d8fd..ea2018ae1cd7 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2884,6 +2884,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
}
#endif
+static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct kvm_device *dev = filp->private_data;
+
+ if (dev->ops->mmap)
+ return dev->ops->mmap(dev, vma);
+
+ return -ENODEV;
+}
+
static int kvm_device_ioctl_attr(struct kvm_device *dev,
int (*accessor)(struct kvm_device *dev,
struct kvm_device_attr *attr),
@@ -2936,6 +2946,7 @@ static const struct file_operations kvm_device_fops = {
.unlocked_ioctl = kvm_device_ioctl,
.release = kvm_device_release,
KVM_COMPAT(kvm_device_ioctl),
+ .mmap = kvm_device_mmap,
};
struct kvm_device *kvm_device_from_filp(struct file *filp)
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 11/16] KVM: introduce a 'mmap' method for KVM devices
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater, Paolo Bonzini
Some KVM devices will want to handle special mappings related to the
underlying HW. For instance, the XIVE interrupt controller of the
POWER9 processor has MMIO pages for thread interrupt management and
for interrupt source control that need to be exposed to the guest when
the OS has the required support.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
include/linux/kvm_host.h | 1 +
virt/kvm/kvm_main.c | 11 +++++++++++
2 files changed, 12 insertions(+)
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 9d55c63db09b..831d963451d8 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1245,6 +1245,7 @@ struct kvm_device_ops {
int (*has_attr)(struct kvm_device *dev, struct kvm_device_attr *attr);
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
+ int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index 55fe8e20d8fd..ea2018ae1cd7 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2884,6 +2884,16 @@ static long kvm_vcpu_compat_ioctl(struct file *filp,
}
#endif
+static int kvm_device_mmap(struct file *filp, struct vm_area_struct *vma)
+{
+ struct kvm_device *dev = filp->private_data;
+
+ if (dev->ops->mmap)
+ return dev->ops->mmap(dev, vma);
+
+ return -ENODEV;
+}
+
static int kvm_device_ioctl_attr(struct kvm_device *dev,
int (*accessor)(struct kvm_device *dev,
struct kvm_device_attr *attr),
@@ -2936,6 +2946,7 @@ static const struct file_operations kvm_device_fops = {
.unlocked_ioctl = kvm_device_ioctl,
.release = kvm_device_release,
KVM_COMPAT(kvm_device_ioctl),
+ .mmap = kvm_device_mmap,
};
struct kvm_device *kvm_device_from_filp(struct file *filp)
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Each thread has an associated Thread Interrupt Management context
composed of a set of registers. These registers let the thread handle
priority management and interrupt acknowledgment. The most important
are :
- Interrupt Pending Buffer (IPB)
- Current Processor Priority (CPPR)
- Notification Source Register (NSR)
They are exposed to software in four different pages each proposing a
view with a different privilege. The first page is for the physical
thread context and the second for the hypervisor. Only the third
(operating system) and the fourth (user level) are exposed the guest.
A custom VM fault handler will populate the VMA with the appropriate
pages, which should only be the OS page for now.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/xive.h | 1 +
arch/powerpc/include/uapi/asm/kvm.h | 2 ++
arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 11 ++++++
Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
5 files changed, 76 insertions(+)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index c4e88abd3b67..eaf76f57023a 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -23,6 +23,7 @@
* same offset regardless of where the code is executing
*/
extern void __iomem *xive_tima;
+extern unsigned long xive_tima_os;
/*
* Offset in the TM area of our current execution level (provided by
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index cd3f16b70a2e..0998e8edc91a 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
+#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 2f9d5e9439a6..f5314da0cb45 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -165,6 +165,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static vm_fault_t xive_native_tima_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+
+ switch (vmf->pgoff - vma->vm_pgoff) {
+ case 0: /* HW - forbid access */
+ case 1: /* HV - forbid access */
+ return VM_FAULT_SIGBUS;
+ case 2: /* OS */
+ vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+ case 3: /* USER - TODO */
+ default:
+ return VM_FAULT_SIGBUS;
+ }
+}
+
+static const struct vm_operations_struct xive_native_tima_vmops = {
+ .fault = xive_native_tima_fault,
+};
+
+static int kvmppc_xive_native_mmap(struct kvm_device *dev,
+ struct vm_area_struct *vma)
+{
+ /* We only allow mappings at fixed offset for now */
+ if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
+ if (vma_pages(vma) > 4)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_tima_vmops;
+ } else {
+ return -EINVAL;
+ }
+
+ vma->vm_flags |= VM_IO | VM_PFNMAP;
+ vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+ return 0;
+}
+
static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
u64 addr)
{
@@ -1050,6 +1088,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
.has_attr = kvmppc_xive_native_has_attr,
+ .mmap = kvmppc_xive_native_mmap,
};
void kvmppc_xive_native_init_module(void)
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 0c037e933e55..7782201e5fe8 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
}
EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
+unsigned long xive_tima_os;
+EXPORT_SYMBOL_GPL(xive_tima_os);
+
bool __init xive_native_init(void)
{
struct device_node *np;
@@ -573,6 +576,14 @@ bool __init xive_native_init(void)
for_each_possible_cpu(cpu)
kvmppc_set_xive_tima(cpu, r.start, tima);
+ /* Resource 2 is OS window */
+ if (of_address_to_resource(np, 2, &r)) {
+ pr_err("Failed to get thread mgmnt area resource\n");
+ return false;
+ }
+
+ xive_tima_os = r.start;
+
/* Grab size of provisionning pages */
xive_parse_provisioning(np);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 525d1eebcf34..0cd7847ec38a 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
XIVE native exploitation interrupt mode. If not, it should run using
the legacy interrupt mode, referred as XICS (POWER7/8).
+* Device Mappings
+
+ The KVM device exposes different MMIO ranges of the XIVE HW which
+ are required for interrupt management. These are exposed to the
+ guest in VMAs populated with a custom VM fault handler.
+
+ 1. Thread Interrupt Management Area (TIMA)
+
+ Each thread has an associated Thread Interrupt Management context
+ composed of a set of registers. These registers let the thread
+ handle priority management and interrupt acknowledgment. The most
+ important are :
+
+ - Interrupt Pending Buffer (IPB)
+ - Current Processor Priority (CPPR)
+ - Notification Source Register (NSR)
+
+ They are exposed to software in four different pages each proposing
+ a view with a different privilege. The first page is for the
+ physical thread context and the second for the hypervisor. Only the
+ third (operating system) and the fourth (user level) are exposed the
+ guest.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Each thread has an associated Thread Interrupt Management context
composed of a set of registers. These registers let the thread handle
priority management and interrupt acknowledgment. The most important
are :
- Interrupt Pending Buffer (IPB)
- Current Processor Priority (CPPR)
- Notification Source Register (NSR)
They are exposed to software in four different pages each proposing a
view with a different privilege. The first page is for the physical
thread context and the second for the hypervisor. Only the third
(operating system) and the fourth (user level) are exposed the guest.
A custom VM fault handler will populate the VMA with the appropriate
pages, which should only be the OS page for now.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/xive.h | 1 +
arch/powerpc/include/uapi/asm/kvm.h | 2 ++
arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 11 ++++++
Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
5 files changed, 76 insertions(+)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index c4e88abd3b67..eaf76f57023a 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -23,6 +23,7 @@
* same offset regardless of where the code is executing
*/
extern void __iomem *xive_tima;
+extern unsigned long xive_tima_os;
/*
* Offset in the TM area of our current execution level (provided by
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index cd3f16b70a2e..0998e8edc91a 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
+#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 2f9d5e9439a6..f5314da0cb45 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -165,6 +165,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static vm_fault_t xive_native_tima_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+
+ switch (vmf->pgoff - vma->vm_pgoff) {
+ case 0: /* HW - forbid access */
+ case 1: /* HV - forbid access */
+ return VM_FAULT_SIGBUS;
+ case 2: /* OS */
+ vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+ case 3: /* USER - TODO */
+ default:
+ return VM_FAULT_SIGBUS;
+ }
+}
+
+static const struct vm_operations_struct xive_native_tima_vmops = {
+ .fault = xive_native_tima_fault,
+};
+
+static int kvmppc_xive_native_mmap(struct kvm_device *dev,
+ struct vm_area_struct *vma)
+{
+ /* We only allow mappings at fixed offset for now */
+ if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
+ if (vma_pages(vma) > 4)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_tima_vmops;
+ } else {
+ return -EINVAL;
+ }
+
+ vma->vm_flags |= VM_IO | VM_PFNMAP;
+ vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+ return 0;
+}
+
static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
u64 addr)
{
@@ -1050,6 +1088,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
.has_attr = kvmppc_xive_native_has_attr,
+ .mmap = kvmppc_xive_native_mmap,
};
void kvmppc_xive_native_init_module(void)
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 0c037e933e55..7782201e5fe8 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
}
EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
+unsigned long xive_tima_os;
+EXPORT_SYMBOL_GPL(xive_tima_os);
+
bool __init xive_native_init(void)
{
struct device_node *np;
@@ -573,6 +576,14 @@ bool __init xive_native_init(void)
for_each_possible_cpu(cpu)
kvmppc_set_xive_tima(cpu, r.start, tima);
+ /* Resource 2 is OS window */
+ if (of_address_to_resource(np, 2, &r)) {
+ pr_err("Failed to get thread mgmnt area resource\n");
+ return false;
+ }
+
+ xive_tima_os = r.start;
+
/* Grab size of provisionning pages */
xive_parse_provisioning(np);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 525d1eebcf34..0cd7847ec38a 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
XIVE native exploitation interrupt mode. If not, it should run using
the legacy interrupt mode, referred as XICS (POWER7/8).
+* Device Mappings
+
+ The KVM device exposes different MMIO ranges of the XIVE HW which
+ are required for interrupt management. These are exposed to the
+ guest in VMAs populated with a custom VM fault handler.
+
+ 1. Thread Interrupt Management Area (TIMA)
+
+ Each thread has an associated Thread Interrupt Management context
+ composed of a set of registers. These registers let the thread
+ handle priority management and interrupt acknowledgment. The most
+ important are :
+
+ - Interrupt Pending Buffer (IPB)
+ - Current Processor Priority (CPPR)
+ - Notification Source Register (NSR)
+
+ They are exposed to software in four different pages each proposing
+ a view with a different privilege. The first page is for the
+ physical thread context and the second for the hypervisor. Only the
+ third (operating system) and the fourth (user level) are exposed the
+ guest.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Each thread has an associated Thread Interrupt Management context
composed of a set of registers. These registers let the thread handle
priority management and interrupt acknowledgment. The most important
are :
- Interrupt Pending Buffer (IPB)
- Current Processor Priority (CPPR)
- Notification Source Register (NSR)
They are exposed to software in four different pages each proposing a
view with a different privilege. The first page is for the physical
thread context and the second for the hypervisor. Only the third
(operating system) and the fourth (user level) are exposed the guest.
A custom VM fault handler will populate the VMA with the appropriate
pages, which should only be the OS page for now.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/asm/xive.h | 1 +
arch/powerpc/include/uapi/asm/kvm.h | 2 ++
arch/powerpc/kvm/book3s_xive_native.c | 39 ++++++++++++++++++++++
arch/powerpc/sysdev/xive/native.c | 11 ++++++
Documentation/virtual/kvm/devices/xive.txt | 23 +++++++++++++
5 files changed, 76 insertions(+)
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index c4e88abd3b67..eaf76f57023a 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -23,6 +23,7 @@
* same offset regardless of where the code is executing
*/
extern void __iomem *xive_tima;
+extern unsigned long xive_tima_os;
/*
* Offset in the TM area of our current execution level (provided by
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index cd3f16b70a2e..0998e8edc91a 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -720,4 +720,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
+#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 2f9d5e9439a6..f5314da0cb45 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -165,6 +165,44 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static vm_fault_t xive_native_tima_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+
+ switch (vmf->pgoff - vma->vm_pgoff) {
+ case 0: /* HW - forbid access */
+ case 1: /* HV - forbid access */
+ return VM_FAULT_SIGBUS;
+ case 2: /* OS */
+ vmf_insert_pfn(vma, vmf->address, xive_tima_os >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+ case 3: /* USER - TODO */
+ default:
+ return VM_FAULT_SIGBUS;
+ }
+}
+
+static const struct vm_operations_struct xive_native_tima_vmops = {
+ .fault = xive_native_tima_fault,
+};
+
+static int kvmppc_xive_native_mmap(struct kvm_device *dev,
+ struct vm_area_struct *vma)
+{
+ /* We only allow mappings at fixed offset for now */
+ if (vma->vm_pgoff = KVM_XIVE_TIMA_PAGE_OFFSET) {
+ if (vma_pages(vma) > 4)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_tima_vmops;
+ } else {
+ return -EINVAL;
+ }
+
+ vma->vm_flags |= VM_IO | VM_PFNMAP;
+ vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+ return 0;
+}
+
static int kvmppc_xive_native_set_source(struct kvmppc_xive *xive, long irq,
u64 addr)
{
@@ -1050,6 +1088,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
.has_attr = kvmppc_xive_native_has_attr,
+ .mmap = kvmppc_xive_native_mmap,
};
void kvmppc_xive_native_init_module(void)
diff --git a/arch/powerpc/sysdev/xive/native.c b/arch/powerpc/sysdev/xive/native.c
index 0c037e933e55..7782201e5fe8 100644
--- a/arch/powerpc/sysdev/xive/native.c
+++ b/arch/powerpc/sysdev/xive/native.c
@@ -521,6 +521,9 @@ u32 xive_native_default_eq_shift(void)
}
EXPORT_SYMBOL_GPL(xive_native_default_eq_shift);
+unsigned long xive_tima_os;
+EXPORT_SYMBOL_GPL(xive_tima_os);
+
bool __init xive_native_init(void)
{
struct device_node *np;
@@ -573,6 +576,14 @@ bool __init xive_native_init(void)
for_each_possible_cpu(cpu)
kvmppc_set_xive_tima(cpu, r.start, tima);
+ /* Resource 2 is OS window */
+ if (of_address_to_resource(np, 2, &r)) {
+ pr_err("Failed to get thread mgmnt area resource\n");
+ return false;
+ }
+
+ xive_tima_os = r.start;
+
/* Grab size of provisionning pages */
xive_parse_provisioning(np);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 525d1eebcf34..0cd7847ec38a 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -13,6 +13,29 @@ requires a POWER9 host and the guest OS should have support for the
XIVE native exploitation interrupt mode. If not, it should run using
the legacy interrupt mode, referred as XICS (POWER7/8).
+* Device Mappings
+
+ The KVM device exposes different MMIO ranges of the XIVE HW which
+ are required for interrupt management. These are exposed to the
+ guest in VMAs populated with a custom VM fault handler.
+
+ 1. Thread Interrupt Management Area (TIMA)
+
+ Each thread has an associated Thread Interrupt Management context
+ composed of a set of registers. These registers let the thread
+ handle priority management and interrupt acknowledgment. The most
+ important are :
+
+ - Interrupt Pending Buffer (IPB)
+ - Current Processor Priority (CPPR)
+ - Notification Source Register (NSR)
+
+ They are exposed to software in four different pages each proposing
+ a view with a different privilege. The first page is for the
+ physical thread context and the second for the hypervisor. Only the
+ third (operating system) and the fourth (user level) are exposed the
+ guest.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Each source is associated with an Event State Buffer (ESB) with a
even/odd pair of pages which provides commands to manage the source:
to trigger, to EOI, to turn off the source for instance.
The custom VM fault handler will deduce the guest IRQ number from the
offset of the fault, and the ESB page of the associated XIVE interrupt
will be inserted into the VMA using the internal structure caching
information on the interrupts.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 7 +++
3 files changed, 65 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 0998e8edc91a..b0f72dea8b11 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+#define KVM_XIVE_ESB_PAGE_OFFSET 4
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index f5314da0cb45..465eb90ff23e 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -165,6 +165,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct kvm_device *dev = vma->vm_file->private_data;
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ u64 page;
+ unsigned long irq;
+ u64 page_offset;
+
+ /*
+ * Linux/KVM uses a two pages ESB setting, one for trigger and
+ * one for EOI
+ */
+ page_offset = vmf->pgoff - vma->vm_pgoff;
+ irq = page_offset / 2;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb) {
+ pr_devel("%s: source %lx not found !\n", __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ state = &sb->irq_state[src];
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * first/even page is for trigger
+ * second/odd page is for EOI and management.
+ */
+ page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
+ arch_spin_unlock(&sb->lock);
+
+ if (WARN_ON(!page)) {
+ pr_err("%s: acessing invalid ESB page for source %lx !\n",
+ __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct xive_native_esb_vmops = {
+ .fault = xive_native_esb_fault,
+};
+
static vm_fault_t xive_native_tima_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -194,6 +247,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
if (vma_pages(vma) > 4)
return -EINVAL;
vma->vm_ops = &xive_native_tima_vmops;
+ } else if (vma->vm_pgoff == KVM_XIVE_ESB_PAGE_OFFSET) {
+ if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_esb_vmops;
} else {
return -EINVAL;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 0cd7847ec38a..69ee62d3d4dc 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
third (operating system) and the fourth (user level) are exposed the
guest.
+ 2. Event State Buffer (ESB)
+
+ Each source is associated with an Event State Buffer (ESB) with
+ either a pair of even/odd pair of pages which provides commands to
+ manage the source: to trigger, to EOI, to turn off the source for
+ instance.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Each source is associated with an Event State Buffer (ESB) with a
even/odd pair of pages which provides commands to manage the source:
to trigger, to EOI, to turn off the source for instance.
The custom VM fault handler will deduce the guest IRQ number from the
offset of the fault, and the ESB page of the associated XIVE interrupt
will be inserted into the VMA using the internal structure caching
information on the interrupts.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 7 +++
3 files changed, 65 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 0998e8edc91a..b0f72dea8b11 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+#define KVM_XIVE_ESB_PAGE_OFFSET 4
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index f5314da0cb45..465eb90ff23e 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -165,6 +165,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct kvm_device *dev = vma->vm_file->private_data;
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ u64 page;
+ unsigned long irq;
+ u64 page_offset;
+
+ /*
+ * Linux/KVM uses a two pages ESB setting, one for trigger and
+ * one for EOI
+ */
+ page_offset = vmf->pgoff - vma->vm_pgoff;
+ irq = page_offset / 2;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb) {
+ pr_devel("%s: source %lx not found !\n", __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ state = &sb->irq_state[src];
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * first/even page is for trigger
+ * second/odd page is for EOI and management.
+ */
+ page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
+ arch_spin_unlock(&sb->lock);
+
+ if (WARN_ON(!page)) {
+ pr_err("%s: acessing invalid ESB page for source %lx !\n",
+ __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct xive_native_esb_vmops = {
+ .fault = xive_native_esb_fault,
+};
+
static vm_fault_t xive_native_tima_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -194,6 +247,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
if (vma_pages(vma) > 4)
return -EINVAL;
vma->vm_ops = &xive_native_tima_vmops;
+ } else if (vma->vm_pgoff == KVM_XIVE_ESB_PAGE_OFFSET) {
+ if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_esb_vmops;
} else {
return -EINVAL;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 0cd7847ec38a..69ee62d3d4dc 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
third (operating system) and the fourth (user level) are exposed the
guest.
+ 2. Event State Buffer (ESB)
+
+ Each source is associated with an Event State Buffer (ESB) with
+ either a pair of even/odd pair of pages which provides commands to
+ manage the source: to trigger, to EOI, to turn off the source for
+ instance.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Each source is associated with an Event State Buffer (ESB) with a
even/odd pair of pages which provides commands to manage the source:
to trigger, to EOI, to turn off the source for instance.
The custom VM fault handler will deduce the guest IRQ number from the
offset of the fault, and the ESB page of the associated XIVE interrupt
will be inserted into the VMA using the internal structure caching
information on the interrupts.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/include/uapi/asm/kvm.h | 1 +
arch/powerpc/kvm/book3s_xive_native.c | 57 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 7 +++
3 files changed, 65 insertions(+)
diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
index 0998e8edc91a..b0f72dea8b11 100644
--- a/arch/powerpc/include/uapi/asm/kvm.h
+++ b/arch/powerpc/include/uapi/asm/kvm.h
@@ -721,5 +721,6 @@ struct kvm_ppc_xive_eq {
#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
#define KVM_XIVE_TIMA_PAGE_OFFSET 0
+#define KVM_XIVE_ESB_PAGE_OFFSET 4
#endif /* __LINUX_KVM_POWERPC_H */
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index f5314da0cb45..465eb90ff23e 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -165,6 +165,59 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)
+{
+ struct vm_area_struct *vma = vmf->vma;
+ struct kvm_device *dev = vma->vm_file->private_data;
+ struct kvmppc_xive *xive = dev->private;
+ struct kvmppc_xive_src_block *sb;
+ struct kvmppc_xive_irq_state *state;
+ struct xive_irq_data *xd;
+ u32 hw_num;
+ u16 src;
+ u64 page;
+ unsigned long irq;
+ u64 page_offset;
+
+ /*
+ * Linux/KVM uses a two pages ESB setting, one for trigger and
+ * one for EOI
+ */
+ page_offset = vmf->pgoff - vma->vm_pgoff;
+ irq = page_offset / 2;
+
+ sb = kvmppc_xive_find_source(xive, irq, &src);
+ if (!sb) {
+ pr_devel("%s: source %lx not found !\n", __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ state = &sb->irq_state[src];
+ kvmppc_xive_select_irq(state, &hw_num, &xd);
+
+ arch_spin_lock(&sb->lock);
+
+ /*
+ * first/even page is for trigger
+ * second/odd page is for EOI and management.
+ */
+ page = page_offset % 2 ? xd->eoi_page : xd->trig_page;
+ arch_spin_unlock(&sb->lock);
+
+ if (WARN_ON(!page)) {
+ pr_err("%s: acessing invalid ESB page for source %lx !\n",
+ __func__, irq);
+ return VM_FAULT_SIGBUS;
+ }
+
+ vmf_insert_pfn(vma, vmf->address, page >> PAGE_SHIFT);
+ return VM_FAULT_NOPAGE;
+}
+
+static const struct vm_operations_struct xive_native_esb_vmops = {
+ .fault = xive_native_esb_fault,
+};
+
static vm_fault_t xive_native_tima_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -194,6 +247,10 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
if (vma_pages(vma) > 4)
return -EINVAL;
vma->vm_ops = &xive_native_tima_vmops;
+ } else if (vma->vm_pgoff = KVM_XIVE_ESB_PAGE_OFFSET) {
+ if (vma_pages(vma) > KVMPPC_XIVE_NR_IRQS * 2)
+ return -EINVAL;
+ vma->vm_ops = &xive_native_esb_vmops;
} else {
return -EINVAL;
}
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 0cd7847ec38a..69ee62d3d4dc 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -36,6 +36,13 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
third (operating system) and the fourth (user level) are exposed the
guest.
+ 2. Event State Buffer (ESB)
+
+ Each source is associated with an Event State Buffer (ESB) with
+ either a pair of even/odd pair of pages which provides commands to
+ manage the source: to trigger, to EOI, to turn off the source for
+ instance.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
implement an IRQ space for the guest using the generic IPI interrupts
of the XIVE IC controller. These interrupts are allocated at the OPAL
level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
Interrupt management is performed in the XIVE way: using loads and
stores on the addresses of the XIVE IPI interrupt ESB pages.
Both KVM devices share the same internal structure caching information
on the interrupts, among which the xive_irq_data struct containing the
addresses of the IPI ESB pages and an extra one in case of pass-through.
The later contains the addresses of the ESB pages of the underlying HW
controller interrupts, PHB4 in all cases for now.
A guest, when running in the XICS legacy interrupt mode, lets the KVM
XICS-over-XIVE device "handle" interrupt management, that is to
perform the loads and stores on the addresses of the ESB pages of the
guest interrupts. However, when running in XIVE native exploitation
mode, the KVM XIVE native device exposes the interrupt ESB pages to
the guest and lets the guest perform directly the loads and stores.
The VMA exposing the ESB pages make use of a custom VM fault handler
which role is to populate the VMA with appropriate pages. When a fault
occurs, the guest IRQ number is deduced from the offset, and the ESB
pages of associated XIVE IPI interrupt are inserted in the VMA (using
the internal structure caching information on the interrupts).
Supporting device passthrough in the guest running in XIVE native
exploitation mode adds some extra refinements because the ESB pages
of a different HW controller (PHB4) need to be exposed to the guest
along with the initial IPI ESB pages of the XIVE IC controller. But
the overall mechanic is the same.
When the device HW irqs are mapped into or unmapped from the guest
IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
and kvmppc_xive_clr_mapped(), are called to record or clear the
passthrough interrupt information and to perform the switch.
The approach taken by this patch is to clear the ESB pages of the
guest IRQ number being mapped and let the VM fault handler repopulate.
The handler will insert the ESB page corresponding to the HW interrupt
of the device being passed-through or the initial IPI ESB page if the
device is being removed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/kvm/book3s_xive.h | 9 +++++
arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 ++++++++++
4 files changed, 84 insertions(+)
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 622f594d93e1..e011622dc038 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
};
+struct kvmppc_xive;
+
+struct kvmppc_xive_ops {
+ int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
+};
struct kvmppc_xive {
struct kvm *kvm;
@@ -132,6 +137,10 @@ struct kvmppc_xive {
/* Flags */
u8 single_escalation;
+
+ struct kvmppc_xive_ops *ops;
+ struct address_space *mapping;
+ struct mutex mapping_lock;
};
#define KVMPPC_XIVE_Q_COUNT 8
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index c1b7aa7dbc28..480a3fc6b9fd 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -937,6 +937,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Turn the IPI hard off */
xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped)
+ xive->ops->reset_mapped(kvm, guest_irq);
+
/* Grab info about irq */
state->pt_number = hw_irq;
state->pt_data = irq_data_get_irq_handler_data(host_data);
@@ -1022,6 +1029,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
state->pt_number = 0;
state->pt_data = NULL;
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped) {
+ xive->ops->reset_mapped(kvm, guest_irq);
+ }
+
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
kvmppc_xive_vp(xive, state->act_server),
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 465eb90ff23e..62648f833adf 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -11,6 +11,7 @@
#include <linux/gfp.h>
#include <linux/spinlock.h>
#include <linux/delay.h>
+#include <linux/file.h>
#include <asm/uaccess.h>
#include <asm/kvm_book3s.h>
#include <asm/kvm_ppc.h>
@@ -165,6 +166,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+/*
+ * Device passthrough support
+ */
+static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+
+ if (irq >= KVMPPC_XIVE_NR_IRQS)
+ return -EINVAL;
+
+ /*
+ * Clear the ESB pages of the IRQ number being mapped (or
+ * unmapped) into the guest and let the the VM fault handler
+ * repopulate with the appropriate ESB pages (device or IC)
+ */
+ pr_debug("clearing esb pages for girq 0x%lx\n", irq);
+ mutex_lock(&xive->mapping_lock);
+ if (xive->mapping)
+ unmap_mapping_range(xive->mapping,
+ irq * (2ull << PAGE_SHIFT),
+ 2ull << PAGE_SHIFT, 1);
+ mutex_unlock(&xive->mapping_lock);
+ return 0;
+}
+
+static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
+ .reset_mapped = kvmppc_xive_native_reset_mapped,
+};
+
static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -242,6 +272,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
static int kvmppc_xive_native_mmap(struct kvm_device *dev,
struct vm_area_struct *vma)
{
+ struct kvmppc_xive *xive = dev->private;
+
/* We only allow mappings at fixed offset for now */
if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
if (vma_pages(vma) > 4)
@@ -257,6 +289,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
vma->vm_flags |= VM_IO | VM_PFNMAP;
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+
+ /*
+ * Grab the KVM device file address_space to be able to clear
+ * the ESB pages mapping when a device is passed-through into
+ * the guest.
+ */
+ xive->mapping = vma->vm_file->f_mapping;
return 0;
}
@@ -971,6 +1010,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
xive->dev = dev;
xive->kvm = kvm;
kvm->arch.xive = xive;
+ mutex_init(&xive->mapping_lock);
/*
* Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
@@ -984,6 +1024,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
ret = -ENXIO;
xive->single_escalation = xive_native_has_single_escalation();
+ xive->ops = &kvmppc_xive_native_ops;
if (ret)
kfree(xive);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 69ee62d3d4dc..9a24a4525253 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -43,6 +43,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
manage the source: to trigger, to EOI, to turn off the source for
instance.
+ 3. Device pass-through
+
+ When a device is passed-through into the guest, the source
+ interrupts are from a different HW controller (PHB4) and the ESB
+ pages exposed to the guest should accommadate this change.
+
+ The passthru_irq helpers, kvmppc_xive_set_mapped() and
+ kvmppc_xive_clr_mapped() are called when the device HW irqs are
+ mapped into or unmapped from the guest IRQ number space. The KVM
+ device extends these helpers to clear the ESB pages of the guest IRQ
+ number being mapped and then lets the VM fault handler repopulate.
+ The handler will insert the ESB page corresponding to the HW
+ interrupt of the device being passed-through or the initial IPI ESB
+ page if the device has being removed.
+
+ The ESB remapping is fully transparent to the guest and the OS
+ device driver. All handling is done within VFIO and the above
+ helpers in KVM-PPC.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
implement an IRQ space for the guest using the generic IPI interrupts
of the XIVE IC controller. These interrupts are allocated at the OPAL
level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
Interrupt management is performed in the XIVE way: using loads and
stores on the addresses of the XIVE IPI interrupt ESB pages.
Both KVM devices share the same internal structure caching information
on the interrupts, among which the xive_irq_data struct containing the
addresses of the IPI ESB pages and an extra one in case of pass-through.
The later contains the addresses of the ESB pages of the underlying HW
controller interrupts, PHB4 in all cases for now.
A guest, when running in the XICS legacy interrupt mode, lets the KVM
XICS-over-XIVE device "handle" interrupt management, that is to
perform the loads and stores on the addresses of the ESB pages of the
guest interrupts. However, when running in XIVE native exploitation
mode, the KVM XIVE native device exposes the interrupt ESB pages to
the guest and lets the guest perform directly the loads and stores.
The VMA exposing the ESB pages make use of a custom VM fault handler
which role is to populate the VMA with appropriate pages. When a fault
occurs, the guest IRQ number is deduced from the offset, and the ESB
pages of associated XIVE IPI interrupt are inserted in the VMA (using
the internal structure caching information on the interrupts).
Supporting device passthrough in the guest running in XIVE native
exploitation mode adds some extra refinements because the ESB pages
of a different HW controller (PHB4) need to be exposed to the guest
along with the initial IPI ESB pages of the XIVE IC controller. But
the overall mechanic is the same.
When the device HW irqs are mapped into or unmapped from the guest
IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
and kvmppc_xive_clr_mapped(), are called to record or clear the
passthrough interrupt information and to perform the switch.
The approach taken by this patch is to clear the ESB pages of the
guest IRQ number being mapped and let the VM fault handler repopulate.
The handler will insert the ESB page corresponding to the HW interrupt
of the device being passed-through or the initial IPI ESB page if the
device is being removed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/kvm/book3s_xive.h | 9 +++++
arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 ++++++++++
4 files changed, 84 insertions(+)
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 622f594d93e1..e011622dc038 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
};
+struct kvmppc_xive;
+
+struct kvmppc_xive_ops {
+ int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
+};
struct kvmppc_xive {
struct kvm *kvm;
@@ -132,6 +137,10 @@ struct kvmppc_xive {
/* Flags */
u8 single_escalation;
+
+ struct kvmppc_xive_ops *ops;
+ struct address_space *mapping;
+ struct mutex mapping_lock;
};
#define KVMPPC_XIVE_Q_COUNT 8
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index c1b7aa7dbc28..480a3fc6b9fd 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -937,6 +937,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Turn the IPI hard off */
xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped)
+ xive->ops->reset_mapped(kvm, guest_irq);
+
/* Grab info about irq */
state->pt_number = hw_irq;
state->pt_data = irq_data_get_irq_handler_data(host_data);
@@ -1022,6 +1029,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
state->pt_number = 0;
state->pt_data = NULL;
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped) {
+ xive->ops->reset_mapped(kvm, guest_irq);
+ }
+
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
kvmppc_xive_vp(xive, state->act_server),
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 465eb90ff23e..62648f833adf 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -11,6 +11,7 @@
#include <linux/gfp.h>
#include <linux/spinlock.h>
#include <linux/delay.h>
+#include <linux/file.h>
#include <asm/uaccess.h>
#include <asm/kvm_book3s.h>
#include <asm/kvm_ppc.h>
@@ -165,6 +166,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+/*
+ * Device passthrough support
+ */
+static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+
+ if (irq >= KVMPPC_XIVE_NR_IRQS)
+ return -EINVAL;
+
+ /*
+ * Clear the ESB pages of the IRQ number being mapped (or
+ * unmapped) into the guest and let the the VM fault handler
+ * repopulate with the appropriate ESB pages (device or IC)
+ */
+ pr_debug("clearing esb pages for girq 0x%lx\n", irq);
+ mutex_lock(&xive->mapping_lock);
+ if (xive->mapping)
+ unmap_mapping_range(xive->mapping,
+ irq * (2ull << PAGE_SHIFT),
+ 2ull << PAGE_SHIFT, 1);
+ mutex_unlock(&xive->mapping_lock);
+ return 0;
+}
+
+static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
+ .reset_mapped = kvmppc_xive_native_reset_mapped,
+};
+
static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -242,6 +272,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
static int kvmppc_xive_native_mmap(struct kvm_device *dev,
struct vm_area_struct *vma)
{
+ struct kvmppc_xive *xive = dev->private;
+
/* We only allow mappings at fixed offset for now */
if (vma->vm_pgoff == KVM_XIVE_TIMA_PAGE_OFFSET) {
if (vma_pages(vma) > 4)
@@ -257,6 +289,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
vma->vm_flags |= VM_IO | VM_PFNMAP;
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+
+ /*
+ * Grab the KVM device file address_space to be able to clear
+ * the ESB pages mapping when a device is passed-through into
+ * the guest.
+ */
+ xive->mapping = vma->vm_file->f_mapping;
return 0;
}
@@ -971,6 +1010,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
xive->dev = dev;
xive->kvm = kvm;
kvm->arch.xive = xive;
+ mutex_init(&xive->mapping_lock);
/*
* Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
@@ -984,6 +1024,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
ret = -ENXIO;
xive->single_escalation = xive_native_has_single_escalation();
+ xive->ops = &kvmppc_xive_native_ops;
if (ret)
kfree(xive);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 69ee62d3d4dc..9a24a4525253 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -43,6 +43,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
manage the source: to trigger, to EOI, to turn off the source for
instance.
+ 3. Device pass-through
+
+ When a device is passed-through into the guest, the source
+ interrupts are from a different HW controller (PHB4) and the ESB
+ pages exposed to the guest should accommadate this change.
+
+ The passthru_irq helpers, kvmppc_xive_set_mapped() and
+ kvmppc_xive_clr_mapped() are called when the device HW irqs are
+ mapped into or unmapped from the guest IRQ number space. The KVM
+ device extends these helpers to clear the ESB pages of the guest IRQ
+ number being mapped and then lets the VM fault handler repopulate.
+ The handler will insert the ESB page corresponding to the HW
+ interrupt of the device being passed-through or the initial IPI ESB
+ page if the device has being removed.
+
+ The ESB remapping is fully transparent to the guest and the OS
+ device driver. All handling is done within VFIO and the above
+ helpers in KVM-PPC.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
The KVM XICS-over-XIVE device and the proposed KVM XIVE native device
implement an IRQ space for the guest using the generic IPI interrupts
of the XIVE IC controller. These interrupts are allocated at the OPAL
level and "mapped" into the guest IRQ number space in the range 0-0x1FFF.
Interrupt management is performed in the XIVE way: using loads and
stores on the addresses of the XIVE IPI interrupt ESB pages.
Both KVM devices share the same internal structure caching information
on the interrupts, among which the xive_irq_data struct containing the
addresses of the IPI ESB pages and an extra one in case of pass-through.
The later contains the addresses of the ESB pages of the underlying HW
controller interrupts, PHB4 in all cases for now.
A guest, when running in the XICS legacy interrupt mode, lets the KVM
XICS-over-XIVE device "handle" interrupt management, that is to
perform the loads and stores on the addresses of the ESB pages of the
guest interrupts. However, when running in XIVE native exploitation
mode, the KVM XIVE native device exposes the interrupt ESB pages to
the guest and lets the guest perform directly the loads and stores.
The VMA exposing the ESB pages make use of a custom VM fault handler
which role is to populate the VMA with appropriate pages. When a fault
occurs, the guest IRQ number is deduced from the offset, and the ESB
pages of associated XIVE IPI interrupt are inserted in the VMA (using
the internal structure caching information on the interrupts).
Supporting device passthrough in the guest running in XIVE native
exploitation mode adds some extra refinements because the ESB pages
of a different HW controller (PHB4) need to be exposed to the guest
along with the initial IPI ESB pages of the XIVE IC controller. But
the overall mechanic is the same.
When the device HW irqs are mapped into or unmapped from the guest
IRQ number space, the passthru_irq helpers, kvmppc_xive_set_mapped()
and kvmppc_xive_clr_mapped(), are called to record or clear the
passthrough interrupt information and to perform the switch.
The approach taken by this patch is to clear the ESB pages of the
guest IRQ number being mapped and let the VM fault handler repopulate.
The handler will insert the ESB page corresponding to the HW interrupt
of the device being passed-through or the initial IPI ESB page if the
device is being removed.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/kvm/book3s_xive.h | 9 +++++
arch/powerpc/kvm/book3s_xive.c | 15 ++++++++
arch/powerpc/kvm/book3s_xive_native.c | 41 ++++++++++++++++++++++
Documentation/virtual/kvm/devices/xive.txt | 19 ++++++++++
4 files changed, 84 insertions(+)
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index 622f594d93e1..e011622dc038 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -94,6 +94,11 @@ struct kvmppc_xive_src_block {
struct kvmppc_xive_irq_state irq_state[KVMPPC_XICS_IRQ_PER_ICS];
};
+struct kvmppc_xive;
+
+struct kvmppc_xive_ops {
+ int (*reset_mapped)(struct kvm *kvm, unsigned long guest_irq);
+};
struct kvmppc_xive {
struct kvm *kvm;
@@ -132,6 +137,10 @@ struct kvmppc_xive {
/* Flags */
u8 single_escalation;
+
+ struct kvmppc_xive_ops *ops;
+ struct address_space *mapping;
+ struct mutex mapping_lock;
};
#define KVMPPC_XIVE_Q_COUNT 8
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index c1b7aa7dbc28..480a3fc6b9fd 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -937,6 +937,13 @@ int kvmppc_xive_set_mapped(struct kvm *kvm, unsigned long guest_irq,
/* Turn the IPI hard off */
xive_vm_esb_load(&state->ipi_data, XIVE_ESB_SET_PQ_01);
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped)
+ xive->ops->reset_mapped(kvm, guest_irq);
+
/* Grab info about irq */
state->pt_number = hw_irq;
state->pt_data = irq_data_get_irq_handler_data(host_data);
@@ -1022,6 +1029,14 @@ int kvmppc_xive_clr_mapped(struct kvm *kvm, unsigned long guest_irq,
state->pt_number = 0;
state->pt_data = NULL;
+ /*
+ * Reset ESB guest mapping. Needed when ESB pages are exposed
+ * to the guest in XIVE native mode
+ */
+ if (xive->ops && xive->ops->reset_mapped) {
+ xive->ops->reset_mapped(kvm, guest_irq);
+ }
+
/* Reconfigure the IPI */
xive_native_configure_irq(state->ipi_number,
kvmppc_xive_vp(xive, state->act_server),
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 465eb90ff23e..62648f833adf 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -11,6 +11,7 @@
#include <linux/gfp.h>
#include <linux/spinlock.h>
#include <linux/delay.h>
+#include <linux/file.h>
#include <asm/uaccess.h>
#include <asm/kvm_book3s.h>
#include <asm/kvm_ppc.h>
@@ -165,6 +166,35 @@ int kvmppc_xive_native_connect_vcpu(struct kvm_device *dev,
return rc;
}
+/*
+ * Device passthrough support
+ */
+static int kvmppc_xive_native_reset_mapped(struct kvm *kvm, unsigned long irq)
+{
+ struct kvmppc_xive *xive = kvm->arch.xive;
+
+ if (irq >= KVMPPC_XIVE_NR_IRQS)
+ return -EINVAL;
+
+ /*
+ * Clear the ESB pages of the IRQ number being mapped (or
+ * unmapped) into the guest and let the the VM fault handler
+ * repopulate with the appropriate ESB pages (device or IC)
+ */
+ pr_debug("clearing esb pages for girq 0x%lx\n", irq);
+ mutex_lock(&xive->mapping_lock);
+ if (xive->mapping)
+ unmap_mapping_range(xive->mapping,
+ irq * (2ull << PAGE_SHIFT),
+ 2ull << PAGE_SHIFT, 1);
+ mutex_unlock(&xive->mapping_lock);
+ return 0;
+}
+
+static struct kvmppc_xive_ops kvmppc_xive_native_ops = {
+ .reset_mapped = kvmppc_xive_native_reset_mapped,
+};
+
static vm_fault_t xive_native_esb_fault(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
@@ -242,6 +272,8 @@ static const struct vm_operations_struct xive_native_tima_vmops = {
static int kvmppc_xive_native_mmap(struct kvm_device *dev,
struct vm_area_struct *vma)
{
+ struct kvmppc_xive *xive = dev->private;
+
/* We only allow mappings at fixed offset for now */
if (vma->vm_pgoff = KVM_XIVE_TIMA_PAGE_OFFSET) {
if (vma_pages(vma) > 4)
@@ -257,6 +289,13 @@ static int kvmppc_xive_native_mmap(struct kvm_device *dev,
vma->vm_flags |= VM_IO | VM_PFNMAP;
vma->vm_page_prot = pgprot_noncached_wc(vma->vm_page_prot);
+
+ /*
+ * Grab the KVM device file address_space to be able to clear
+ * the ESB pages mapping when a device is passed-through into
+ * the guest.
+ */
+ xive->mapping = vma->vm_file->f_mapping;
return 0;
}
@@ -971,6 +1010,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
xive->dev = dev;
xive->kvm = kvm;
kvm->arch.xive = xive;
+ mutex_init(&xive->mapping_lock);
/*
* Allocate a bunch of VPs. KVM_MAX_VCPUS is a large value for
@@ -984,6 +1024,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
ret = -ENXIO;
xive->single_escalation = xive_native_has_single_escalation();
+ xive->ops = &kvmppc_xive_native_ops;
if (ret)
kfree(xive);
diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
index 69ee62d3d4dc..9a24a4525253 100644
--- a/Documentation/virtual/kvm/devices/xive.txt
+++ b/Documentation/virtual/kvm/devices/xive.txt
@@ -43,6 +43,25 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
manage the source: to trigger, to EOI, to turn off the source for
instance.
+ 3. Device pass-through
+
+ When a device is passed-through into the guest, the source
+ interrupts are from a different HW controller (PHB4) and the ESB
+ pages exposed to the guest should accommadate this change.
+
+ The passthru_irq helpers, kvmppc_xive_set_mapped() and
+ kvmppc_xive_clr_mapped() are called when the device HW irqs are
+ mapped into or unmapped from the guest IRQ number space. The KVM
+ device extends these helpers to clear the ESB pages of the guest IRQ
+ number being mapped and then lets the VM fault handler repopulate.
+ The handler will insert the ESB page corresponding to the HW
+ interrupt of the device being passed-through or the initial IPI ESB
+ page if the device has being removed.
+
+ The ESB remapping is fully transparent to the guest and the OS
+ device driver. All handling is done within VFIO and the above
+ helpers in KVM-PPC.
+
* Groups:
1. KVM_DEV_XIVE_GRP_CTRL
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 15/16] KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Full support for the XIVE native exploitation mode is now available,
advertise the capability KVM_CAP_PPC_IRQ_XIVE for guests running on
PowerNV KVM Hypervisors only. Support for nested guests (pseries KVM
Hypervisor) is not yet available. XIVE should also have been activated
which is default setting on POWER9 systems running a recent Linux
kernel.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/kvm/powerpc.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index b0858ee61460..f54926c78320 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -573,10 +573,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
#ifdef CONFIG_KVM_XIVE
case KVM_CAP_PPC_IRQ_XIVE:
/*
- * Return false until all the XIVE infrastructure is
- * in place including support for migration.
+ * We need XIVE to be enabled on the platform (implies
+ * a POWER9 processor) and the PowerNV platform, as
+ * nested is not yet supported.
*/
- r = 0;
+ r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE);
break;
#endif
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 15/16] KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
Full support for the XIVE native exploitation mode is now available,
advertise the capability KVM_CAP_PPC_IRQ_XIVE for guests running on
PowerNV KVM Hypervisors only. Support for nested guests (pseries KVM
Hypervisor) is not yet available. XIVE should also have been activated
which is default setting on POWER9 systems running a recent Linux
kernel.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/kvm/powerpc.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index b0858ee61460..f54926c78320 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -573,10 +573,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
#ifdef CONFIG_KVM_XIVE
case KVM_CAP_PPC_IRQ_XIVE:
/*
- * Return false until all the XIVE infrastructure is
- * in place including support for migration.
+ * We need XIVE to be enabled on the platform (implies
+ * a POWER9 processor) and the PowerNV platform, as
+ * nested is not yet supported.
*/
- r = 0;
+ r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE);
break;
#endif
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 15/16] KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater
Full support for the XIVE native exploitation mode is now available,
advertise the capability KVM_CAP_PPC_IRQ_XIVE for guests running on
PowerNV KVM Hypervisors only. Support for nested guests (pseries KVM
Hypervisor) is not yet available. XIVE should also have been activated
which is default setting on POWER9 systems running a recent Linux
kernel.
Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
---
arch/powerpc/kvm/powerpc.c | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/arch/powerpc/kvm/powerpc.c b/arch/powerpc/kvm/powerpc.c
index b0858ee61460..f54926c78320 100644
--- a/arch/powerpc/kvm/powerpc.c
+++ b/arch/powerpc/kvm/powerpc.c
@@ -573,10 +573,11 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
#ifdef CONFIG_KVM_XIVE
case KVM_CAP_PPC_IRQ_XIVE:
/*
- * Return false until all the XIVE infrastructure is
- * in place including support for migration.
+ * We need XIVE to be enabled on the platform (implies
+ * a POWER9 processor) and the PowerNV platform, as
+ * nested is not yet supported.
*/
- r = 0;
+ r = xive_enabled() && !!cpu_has_feature(CPU_FTR_HVMODE);
break;
#endif
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-10 17:04 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater, Paolo Bonzini
When a P9 sPAPR VM boots, the CAS negotiation process determines which
interrupt mode to use (XICS legacy or XIVE native) and invokes a
machine reset to activate the chosen mode.
To be able to switch from one mode to another, we introduce the
capability to release a KVM device without destroying the VM. The KVM
device interface is extended with a new 'release' operation which is
called when the file descriptor of the device is closed.
Such operations are defined for the XICS-on-XIVE and the XIVE native
KVM devices. They clear the vCPU interrupt presenters that could be
attached and then destroy the device.
This is not considered as a safe operation as the vCPUs are still
running and could be referencing the KVM device through their
presenters. To protect the system from any breakage, the kvmppc_xive
objects representing both KVM devices are now stored in an array under
the VM. Allocation is performed on first usage and memory is freed
only when the VM exits.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_xive.h | 1 +
include/linux/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 99 ++++++++++++++++++++++++++-
arch/powerpc/kvm/book3s_xive_native.c | 45 +++++++++++-
virt/kvm/kvm_main.c | 13 ++++
6 files changed, 156 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 9cc6abdce1b9..ed059c95e56a 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -314,6 +314,7 @@ struct kvm_arch {
#ifdef CONFIG_KVM_XICS
struct kvmppc_xics *xics;
struct kvmppc_xive *xive;
+ struct kvmppc_xive *xive_devices[2];
struct kvmppc_passthru_irqmap *pimap;
#endif
struct kvmppc_ops *kvm_ops;
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index e011622dc038..426146332984 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -283,6 +283,7 @@ void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
bool single_escalation);
+struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 831d963451d8..3b444620d8fc 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1246,6 +1246,7 @@ struct kvm_device_ops {
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
+ void (*release)(struct kvm_device *dev);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 480a3fc6b9fd..064a9f2ae678 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- struct kvmppc_xive *xive = xc->xive;
+ struct kvmppc_xive *xive;
int i;
+ if (!kvmppc_xics_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
+ xive = xc->xive;
+
/* Ensure no interrupt is still routed to that VP */
xc->valid = false;
kvmppc_xive_disable_vcpu_interrupts(vcpu);
@@ -1141,6 +1149,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
}
/* Free the VP */
kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
}
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
@@ -1158,7 +1170,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
if (xive->kvm != vcpu->kvm)
return -EPERM;
- if (vcpu->arch.irq_type)
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
return -EBUSY;
if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
pr_devel("Duplicate !\n");
@@ -1824,6 +1836,9 @@ void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
}
}
+/*
+ * Called when VM is destroyed
+ */
static void kvmppc_xive_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
@@ -1851,6 +1866,83 @@ static void kvmppc_xive_free(struct kvm_device *dev)
kfree(dev);
}
+/*
+ * Called when device fd is closed
+ */
+static void kvmppc_xive_release(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ pr_devel("Releasing xive device\n");
+
+ /*
+ * When releasing the KVM device fd, the vCPUs can still be
+ * running and we should clean up the vCPU interrupt
+ * presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ }
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ /* Mask and free interrupts */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ /* Do no free device private data. Destroy handler will */
+ kfree(dev);
+}
+
+/*
+ * When the guest chooses the interrupt mode (XICS legacy or XIVE
+ * native), the VM will switch of KVM device. The previous device will
+ * be "released" before the new one is created.
+ *
+ * Until we are sure all execution paths are well protected, provide a
+ * fail safe (transitional) method for device destruction, in which
+ * the XIVE device pointer is recycled and not directly freed.
+ */
+struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type)
+{
+ struct kvmppc_xive *xive;
+ bool xive_native_index = type == KVM_DEV_TYPE_XIVE;
+
+ xive = kvm->arch.xive_devices[xive_native_index];
+
+ if (!xive) {
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ kvm->arch.xive_devices[xive_native_index] = xive;
+ } else {
+ memset(xive, 0, sizeof(*xive));
+ }
+
+ return xive;
+}
+
static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
{
struct kvmppc_xive *xive;
@@ -1859,7 +1951,7 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
pr_devel("Creating xive for partition\n");
- xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ xive = kvmppc_xive_get_device(kvm, type);
if (!xive)
return -ENOMEM;
@@ -2024,6 +2116,7 @@ struct kvm_device_ops kvm_xive_ops = {
.name = "kvm-xive",
.create = kvmppc_xive_create,
.init = kvmppc_xive_init,
+ .release = kvmppc_xive_release,
.destroy = kvmppc_xive_free,
.set_attr = xive_set_attr,
.get_attr = xive_get_attr,
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 62648f833adf..ba1c55a93673 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -964,6 +964,9 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
return -ENXIO;
}
+/*
+ * Called when VM is destroyed
+ */
static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
@@ -991,6 +994,45 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
kfree(dev);
}
+/*
+ * Called when device fd is closed
+ */
+static void kvmppc_xive_native_release(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ pr_devel("Releasing xive native device\n");
+
+ /*
+ * When releasing the KVM device fd, the vCPUs can still be
+ * running and we should clean up the vCPU interrupt
+ * presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ }
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ /* Do no free device private data. Destroy handler will */
+ kfree(dev);
+}
+
static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
{
struct kvmppc_xive *xive;
@@ -1002,7 +1044,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
if (kvm->arch.xive)
return -EEXIST;
- xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ xive = kvmppc_xive_get_device(kvm, type);
if (!xive)
return -ENOMEM;
@@ -1182,6 +1224,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.name = "kvm-xive-native",
.create = kvmppc_xive_native_create,
.init = kvmppc_xive_native_init,
+ .release = kvmppc_xive_native_release,
.destroy = kvmppc_xive_native_free,
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ea2018ae1cd7..ea2619d5ca98 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
struct kvm_device *dev = filp->private_data;
struct kvm *kvm = dev->kvm;
+ if (!dev)
+ return -ENODEV;
+
+ if (dev->kvm != kvm)
+ return -EPERM;
+
+ if (dev->ops->release) {
+ mutex_lock(&kvm->lock);
+ list_del(&dev->vm_node);
+ dev->ops->release(dev);
+ mutex_unlock(&kvm->lock);
+ }
+
kvm_put_kvm(kvm);
return 0;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, Paolo Bonzini,
linuxppc-dev, David Gibson
When a P9 sPAPR VM boots, the CAS negotiation process determines which
interrupt mode to use (XICS legacy or XIVE native) and invokes a
machine reset to activate the chosen mode.
To be able to switch from one mode to another, we introduce the
capability to release a KVM device without destroying the VM. The KVM
device interface is extended with a new 'release' operation which is
called when the file descriptor of the device is closed.
Such operations are defined for the XICS-on-XIVE and the XIVE native
KVM devices. They clear the vCPU interrupt presenters that could be
attached and then destroy the device.
This is not considered as a safe operation as the vCPUs are still
running and could be referencing the KVM device through their
presenters. To protect the system from any breakage, the kvmppc_xive
objects representing both KVM devices are now stored in an array under
the VM. Allocation is performed on first usage and memory is freed
only when the VM exits.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_xive.h | 1 +
include/linux/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 99 ++++++++++++++++++++++++++-
arch/powerpc/kvm/book3s_xive_native.c | 45 +++++++++++-
virt/kvm/kvm_main.c | 13 ++++
6 files changed, 156 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 9cc6abdce1b9..ed059c95e56a 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -314,6 +314,7 @@ struct kvm_arch {
#ifdef CONFIG_KVM_XICS
struct kvmppc_xics *xics;
struct kvmppc_xive *xive;
+ struct kvmppc_xive *xive_devices[2];
struct kvmppc_passthru_irqmap *pimap;
#endif
struct kvmppc_ops *kvm_ops;
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index e011622dc038..426146332984 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -283,6 +283,7 @@ void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
bool single_escalation);
+struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 831d963451d8..3b444620d8fc 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1246,6 +1246,7 @@ struct kvm_device_ops {
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
+ void (*release)(struct kvm_device *dev);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 480a3fc6b9fd..064a9f2ae678 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- struct kvmppc_xive *xive = xc->xive;
+ struct kvmppc_xive *xive;
int i;
+ if (!kvmppc_xics_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
+ xive = xc->xive;
+
/* Ensure no interrupt is still routed to that VP */
xc->valid = false;
kvmppc_xive_disable_vcpu_interrupts(vcpu);
@@ -1141,6 +1149,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
}
/* Free the VP */
kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
}
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
@@ -1158,7 +1170,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
if (xive->kvm != vcpu->kvm)
return -EPERM;
- if (vcpu->arch.irq_type)
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
return -EBUSY;
if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
pr_devel("Duplicate !\n");
@@ -1824,6 +1836,9 @@ void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
}
}
+/*
+ * Called when VM is destroyed
+ */
static void kvmppc_xive_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
@@ -1851,6 +1866,83 @@ static void kvmppc_xive_free(struct kvm_device *dev)
kfree(dev);
}
+/*
+ * Called when device fd is closed
+ */
+static void kvmppc_xive_release(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ pr_devel("Releasing xive device\n");
+
+ /*
+ * When releasing the KVM device fd, the vCPUs can still be
+ * running and we should clean up the vCPU interrupt
+ * presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ }
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ /* Mask and free interrupts */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ /* Do no free device private data. Destroy handler will */
+ kfree(dev);
+}
+
+/*
+ * When the guest chooses the interrupt mode (XICS legacy or XIVE
+ * native), the VM will switch of KVM device. The previous device will
+ * be "released" before the new one is created.
+ *
+ * Until we are sure all execution paths are well protected, provide a
+ * fail safe (transitional) method for device destruction, in which
+ * the XIVE device pointer is recycled and not directly freed.
+ */
+struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type)
+{
+ struct kvmppc_xive *xive;
+ bool xive_native_index = type == KVM_DEV_TYPE_XIVE;
+
+ xive = kvm->arch.xive_devices[xive_native_index];
+
+ if (!xive) {
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ kvm->arch.xive_devices[xive_native_index] = xive;
+ } else {
+ memset(xive, 0, sizeof(*xive));
+ }
+
+ return xive;
+}
+
static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
{
struct kvmppc_xive *xive;
@@ -1859,7 +1951,7 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
pr_devel("Creating xive for partition\n");
- xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ xive = kvmppc_xive_get_device(kvm, type);
if (!xive)
return -ENOMEM;
@@ -2024,6 +2116,7 @@ struct kvm_device_ops kvm_xive_ops = {
.name = "kvm-xive",
.create = kvmppc_xive_create,
.init = kvmppc_xive_init,
+ .release = kvmppc_xive_release,
.destroy = kvmppc_xive_free,
.set_attr = xive_set_attr,
.get_attr = xive_get_attr,
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 62648f833adf..ba1c55a93673 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -964,6 +964,9 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
return -ENXIO;
}
+/*
+ * Called when VM is destroyed
+ */
static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
@@ -991,6 +994,45 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
kfree(dev);
}
+/*
+ * Called when device fd is closed
+ */
+static void kvmppc_xive_native_release(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ pr_devel("Releasing xive native device\n");
+
+ /*
+ * When releasing the KVM device fd, the vCPUs can still be
+ * running and we should clean up the vCPU interrupt
+ * presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ }
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ /* Do no free device private data. Destroy handler will */
+ kfree(dev);
+}
+
static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
{
struct kvmppc_xive *xive;
@@ -1002,7 +1044,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
if (kvm->arch.xive)
return -EEXIST;
- xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ xive = kvmppc_xive_get_device(kvm, type);
if (!xive)
return -ENOMEM;
@@ -1182,6 +1224,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.name = "kvm-xive-native",
.create = kvmppc_xive_native_create,
.init = kvmppc_xive_native_init,
+ .release = kvmppc_xive_native_release,
.destroy = kvmppc_xive_native_free,
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ea2018ae1cd7..ea2619d5ca98 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
struct kvm_device *dev = filp->private_data;
struct kvm *kvm = dev->kvm;
+ if (!dev)
+ return -ENODEV;
+
+ if (dev->kvm != kvm)
+ return -EPERM;
+
+ if (dev->ops->release) {
+ mutex_lock(&kvm->lock);
+ list_del(&dev->vm_node);
+ dev->ops->release(dev);
+ mutex_unlock(&kvm->lock);
+ }
+
kvm_put_kvm(kvm);
return 0;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-10 17:04 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-10 17:04 UTC (permalink / raw)
To: kvm-ppc
Cc: Paul Mackerras, David Gibson, kvm, Michael Ellerman,
linuxppc-dev, Cédric Le Goater, Paolo Bonzini
When a P9 sPAPR VM boots, the CAS negotiation process determines which
interrupt mode to use (XICS legacy or XIVE native) and invokes a
machine reset to activate the chosen mode.
To be able to switch from one mode to another, we introduce the
capability to release a KVM device without destroying the VM. The KVM
device interface is extended with a new 'release' operation which is
called when the file descriptor of the device is closed.
Such operations are defined for the XICS-on-XIVE and the XIVE native
KVM devices. They clear the vCPU interrupt presenters that could be
attached and then destroy the device.
This is not considered as a safe operation as the vCPUs are still
running and could be referencing the KVM device through their
presenters. To protect the system from any breakage, the kvmppc_xive
objects representing both KVM devices are now stored in an array under
the VM. Allocation is performed on first usage and memory is freed
only when the VM exits.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Cédric Le Goater <clg@kaod.org>
---
arch/powerpc/include/asm/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_xive.h | 1 +
include/linux/kvm_host.h | 1 +
arch/powerpc/kvm/book3s_xive.c | 99 ++++++++++++++++++++++++++-
arch/powerpc/kvm/book3s_xive_native.c | 45 +++++++++++-
virt/kvm/kvm_main.c | 13 ++++
6 files changed, 156 insertions(+), 4 deletions(-)
diff --git a/arch/powerpc/include/asm/kvm_host.h b/arch/powerpc/include/asm/kvm_host.h
index 9cc6abdce1b9..ed059c95e56a 100644
--- a/arch/powerpc/include/asm/kvm_host.h
+++ b/arch/powerpc/include/asm/kvm_host.h
@@ -314,6 +314,7 @@ struct kvm_arch {
#ifdef CONFIG_KVM_XICS
struct kvmppc_xics *xics;
struct kvmppc_xive *xive;
+ struct kvmppc_xive *xive_devices[2];
struct kvmppc_passthru_irqmap *pimap;
#endif
struct kvmppc_ops *kvm_ops;
diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
index e011622dc038..426146332984 100644
--- a/arch/powerpc/kvm/book3s_xive.h
+++ b/arch/powerpc/kvm/book3s_xive.h
@@ -283,6 +283,7 @@ void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
bool single_escalation);
+struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type);
#endif /* CONFIG_KVM_XICS */
#endif /* _KVM_PPC_BOOK3S_XICS_H */
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 831d963451d8..3b444620d8fc 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1246,6 +1246,7 @@ struct kvm_device_ops {
long (*ioctl)(struct kvm_device *dev, unsigned int ioctl,
unsigned long arg);
int (*mmap)(struct kvm_device *dev, struct vm_area_struct *vma);
+ void (*release)(struct kvm_device *dev);
};
void kvm_device_get(struct kvm_device *dev);
diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
index 480a3fc6b9fd..064a9f2ae678 100644
--- a/arch/powerpc/kvm/book3s_xive.c
+++ b/arch/powerpc/kvm/book3s_xive.c
@@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
{
struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
- struct kvmppc_xive *xive = xc->xive;
+ struct kvmppc_xive *xive;
int i;
+ if (!kvmppc_xics_enabled(vcpu))
+ return;
+
+ if (!xc)
+ return;
+
pr_devel("cleanup_vcpu(cpu=%d)\n", xc->server_num);
+ xive = xc->xive;
+
/* Ensure no interrupt is still routed to that VP */
xc->valid = false;
kvmppc_xive_disable_vcpu_interrupts(vcpu);
@@ -1141,6 +1149,10 @@ void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
}
/* Free the VP */
kfree(xc);
+
+ /* Cleanup the vcpu */
+ vcpu->arch.irq_type = KVMPPC_IRQ_DEFAULT;
+ vcpu->arch.xive_vcpu = NULL;
}
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
@@ -1158,7 +1170,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
}
if (xive->kvm != vcpu->kvm)
return -EPERM;
- if (vcpu->arch.irq_type)
+ if (vcpu->arch.irq_type != KVMPPC_IRQ_DEFAULT)
return -EBUSY;
if (kvmppc_xive_find_server(vcpu->kvm, cpu)) {
pr_devel("Duplicate !\n");
@@ -1824,6 +1836,9 @@ void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb)
}
}
+/*
+ * Called when VM is destroyed
+ */
static void kvmppc_xive_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
@@ -1851,6 +1866,83 @@ static void kvmppc_xive_free(struct kvm_device *dev)
kfree(dev);
}
+/*
+ * Called when device fd is closed
+ */
+static void kvmppc_xive_release(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ pr_devel("Releasing xive device\n");
+
+ /*
+ * When releasing the KVM device fd, the vCPUs can still be
+ * running and we should clean up the vCPU interrupt
+ * presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ /*
+ * call kick_all_cpus_sync() to ensure that all CPUs
+ * have executed any pending interrupts
+ */
+ if (is_kvmppc_hv_enabled(kvm))
+ kick_all_cpus_sync();
+
+ /*
+ * TODO: There is still a race window with the early
+ * checks in kvmppc_native_connect_vcpu()
+ */
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_cleanup_vcpu(vcpu);
+ }
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ /* Mask and free interrupts */
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ /* Do no free device private data. Destroy handler will */
+ kfree(dev);
+}
+
+/*
+ * When the guest chooses the interrupt mode (XICS legacy or XIVE
+ * native), the VM will switch of KVM device. The previous device will
+ * be "released" before the new one is created.
+ *
+ * Until we are sure all execution paths are well protected, provide a
+ * fail safe (transitional) method for device destruction, in which
+ * the XIVE device pointer is recycled and not directly freed.
+ */
+struct kvmppc_xive *kvmppc_xive_get_device(struct kvm *kvm, u32 type)
+{
+ struct kvmppc_xive *xive;
+ bool xive_native_index = type = KVM_DEV_TYPE_XIVE;
+
+ xive = kvm->arch.xive_devices[xive_native_index];
+
+ if (!xive) {
+ xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ kvm->arch.xive_devices[xive_native_index] = xive;
+ } else {
+ memset(xive, 0, sizeof(*xive));
+ }
+
+ return xive;
+}
+
static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
{
struct kvmppc_xive *xive;
@@ -1859,7 +1951,7 @@ static int kvmppc_xive_create(struct kvm_device *dev, u32 type)
pr_devel("Creating xive for partition\n");
- xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ xive = kvmppc_xive_get_device(kvm, type);
if (!xive)
return -ENOMEM;
@@ -2024,6 +2116,7 @@ struct kvm_device_ops kvm_xive_ops = {
.name = "kvm-xive",
.create = kvmppc_xive_create,
.init = kvmppc_xive_init,
+ .release = kvmppc_xive_release,
.destroy = kvmppc_xive_free,
.set_attr = xive_set_attr,
.get_attr = xive_get_attr,
diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
index 62648f833adf..ba1c55a93673 100644
--- a/arch/powerpc/kvm/book3s_xive_native.c
+++ b/arch/powerpc/kvm/book3s_xive_native.c
@@ -964,6 +964,9 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
return -ENXIO;
}
+/*
+ * Called when VM is destroyed
+ */
static void kvmppc_xive_native_free(struct kvm_device *dev)
{
struct kvmppc_xive *xive = dev->private;
@@ -991,6 +994,45 @@ static void kvmppc_xive_native_free(struct kvm_device *dev)
kfree(dev);
}
+/*
+ * Called when device fd is closed
+ */
+static void kvmppc_xive_native_release(struct kvm_device *dev)
+{
+ struct kvmppc_xive *xive = dev->private;
+ struct kvm *kvm = xive->kvm;
+ struct kvm_vcpu *vcpu;
+ int i;
+
+ pr_devel("Releasing xive native device\n");
+
+ /*
+ * When releasing the KVM device fd, the vCPUs can still be
+ * running and we should clean up the vCPU interrupt
+ * presenters first.
+ */
+ if (atomic_read(&kvm->online_vcpus) != 0) {
+ kvm_for_each_vcpu(i, vcpu, kvm)
+ kvmppc_xive_native_cleanup_vcpu(vcpu);
+ }
+
+ if (kvm)
+ kvm->arch.xive = NULL;
+
+ for (i = 0; i <= xive->max_sbid; i++) {
+ if (xive->src_blocks[i])
+ kvmppc_xive_free_sources(xive->src_blocks[i]);
+ kfree(xive->src_blocks[i]);
+ xive->src_blocks[i] = NULL;
+ }
+
+ if (xive->vp_base != XIVE_INVALID_VP)
+ xive_native_free_vp_block(xive->vp_base);
+
+ /* Do no free device private data. Destroy handler will */
+ kfree(dev);
+}
+
static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
{
struct kvmppc_xive *xive;
@@ -1002,7 +1044,7 @@ static int kvmppc_xive_native_create(struct kvm_device *dev, u32 type)
if (kvm->arch.xive)
return -EEXIST;
- xive = kzalloc(sizeof(*xive), GFP_KERNEL);
+ xive = kvmppc_xive_get_device(kvm, type);
if (!xive)
return -ENOMEM;
@@ -1182,6 +1224,7 @@ struct kvm_device_ops kvm_xive_native_ops = {
.name = "kvm-xive-native",
.create = kvmppc_xive_native_create,
.init = kvmppc_xive_native_init,
+ .release = kvmppc_xive_native_release,
.destroy = kvmppc_xive_native_free,
.set_attr = kvmppc_xive_native_set_attr,
.get_attr = kvmppc_xive_native_get_attr,
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index ea2018ae1cd7..ea2619d5ca98 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
struct kvm_device *dev = filp->private_data;
struct kvm *kvm = dev->kvm;
+ if (!dev)
+ return -ENODEV;
+
+ if (dev->kvm != kvm)
+ return -EPERM;
+
+ if (dev->ops->release) {
+ mutex_lock(&kvm->lock);
+ list_del(&dev->vm_node);
+ dev->ops->release(dev);
+ mutex_unlock(&kvm->lock);
+ }
+
kvm_put_kvm(kvm);
return 0;
}
--
2.20.1
^ permalink raw reply related [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-11 3:16 ` Paul Mackerras
-1 siblings, 0 replies; 81+ messages in thread
From: Paul Mackerras @ 2019-04-11 3:16 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, David Gibson, kvm, Michael Ellerman, linuxppc-dev,
Paolo Bonzini
On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> When a P9 sPAPR VM boots, the CAS negotiation process determines which
> interrupt mode to use (XICS legacy or XIVE native) and invokes a
> machine reset to activate the chosen mode.
>
> To be able to switch from one mode to another, we introduce the
> capability to release a KVM device without destroying the VM. The KVM
> device interface is extended with a new 'release' operation which is
> called when the file descriptor of the device is closed.
I believe the release operation is not called until all of the mmaps
using the fd are unmapped - which is a good thing for us, since it
means the guest can't possibly be accessing the XIVE directly.
You might want to reword that last paragraph to mention that.
> Such operations are defined for the XICS-on-XIVE and the XIVE native
> KVM devices. They clear the vCPU interrupt presenters that could be
> attached and then destroy the device.
>
> This is not considered as a safe operation as the vCPUs are still
> running and could be referencing the KVM device through their
> presenters. To protect the system from any breakage, the kvmppc_xive
> objects representing both KVM devices are now stored in an array under
> the VM. Allocation is performed on first usage and memory is freed
> only when the VM exits.
One quick comment below:
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 480a3fc6b9fd..064a9f2ae678 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - struct kvmppc_xive *xive = xc->xive;
> + struct kvmppc_xive *xive;
> int i;
>
> + if (!kvmppc_xics_enabled(vcpu))
> + return;
Should that be kvmppc_xive_enabled() rather than xics?
Paul.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 3:16 ` Paul Mackerras
0 siblings, 0 replies; 81+ messages in thread
From: Paul Mackerras @ 2019-04-11 3:16 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm, kvm-ppc, Paolo Bonzini, linuxppc-dev, David Gibson
On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> When a P9 sPAPR VM boots, the CAS negotiation process determines which
> interrupt mode to use (XICS legacy or XIVE native) and invokes a
> machine reset to activate the chosen mode.
>
> To be able to switch from one mode to another, we introduce the
> capability to release a KVM device without destroying the VM. The KVM
> device interface is extended with a new 'release' operation which is
> called when the file descriptor of the device is closed.
I believe the release operation is not called until all of the mmaps
using the fd are unmapped - which is a good thing for us, since it
means the guest can't possibly be accessing the XIVE directly.
You might want to reword that last paragraph to mention that.
> Such operations are defined for the XICS-on-XIVE and the XIVE native
> KVM devices. They clear the vCPU interrupt presenters that could be
> attached and then destroy the device.
>
> This is not considered as a safe operation as the vCPUs are still
> running and could be referencing the KVM device through their
> presenters. To protect the system from any breakage, the kvmppc_xive
> objects representing both KVM devices are now stored in an array under
> the VM. Allocation is performed on first usage and memory is freed
> only when the VM exits.
One quick comment below:
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 480a3fc6b9fd..064a9f2ae678 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - struct kvmppc_xive *xive = xc->xive;
> + struct kvmppc_xive *xive;
> int i;
>
> + if (!kvmppc_xics_enabled(vcpu))
> + return;
Should that be kvmppc_xive_enabled() rather than xics?
Paul.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 3:16 ` Paul Mackerras
0 siblings, 0 replies; 81+ messages in thread
From: Paul Mackerras @ 2019-04-11 3:16 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, David Gibson, kvm, Michael Ellerman, linuxppc-dev,
Paolo Bonzini
On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> When a P9 sPAPR VM boots, the CAS negotiation process determines which
> interrupt mode to use (XICS legacy or XIVE native) and invokes a
> machine reset to activate the chosen mode.
>
> To be able to switch from one mode to another, we introduce the
> capability to release a KVM device without destroying the VM. The KVM
> device interface is extended with a new 'release' operation which is
> called when the file descriptor of the device is closed.
I believe the release operation is not called until all of the mmaps
using the fd are unmapped - which is a good thing for us, since it
means the guest can't possibly be accessing the XIVE directly.
You might want to reword that last paragraph to mention that.
> Such operations are defined for the XICS-on-XIVE and the XIVE native
> KVM devices. They clear the vCPU interrupt presenters that could be
> attached and then destroy the device.
>
> This is not considered as a safe operation as the vCPUs are still
> running and could be referencing the KVM device through their
> presenters. To protect the system from any breakage, the kvmppc_xive
> objects representing both KVM devices are now stored in an array under
> the VM. Allocation is performed on first usage and memory is freed
> only when the VM exits.
One quick comment below:
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index 480a3fc6b9fd..064a9f2ae678 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> - struct kvmppc_xive *xive = xc->xive;
> + struct kvmppc_xive *xive;
> int i;
>
> + if (!kvmppc_xics_enabled(vcpu))
> + return;
Should that be kvmppc_xive_enabled() rather than xics?
Paul.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
2019-04-11 3:16 ` Paul Mackerras
(?)
@ 2019-04-11 4:38 ` David Gibson
-1 siblings, 0 replies; 81+ messages in thread
From: David Gibson @ 2019-04-11 4:38 UTC (permalink / raw)
To: Paul Mackerras
Cc: Cédric Le Goater, kvm-ppc, kvm, Michael Ellerman,
linuxppc-dev, Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 2689 bytes --]
On Thu, Apr 11, 2019 at 01:16:25PM +1000, Paul Mackerras wrote:
> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> > When a P9 sPAPR VM boots, the CAS negotiation process determines which
> > interrupt mode to use (XICS legacy or XIVE native) and invokes a
> > machine reset to activate the chosen mode.
> >
> > To be able to switch from one mode to another, we introduce the
> > capability to release a KVM device without destroying the VM. The KVM
> > device interface is extended with a new 'release' operation which is
> > called when the file descriptor of the device is closed.
>
> I believe the release operation is not called until all of the mmaps
> using the fd are unmapped - which is a good thing for us, since it
> means the guest can't possibly be accessing the XIVE directly.
> You might want to reword that last paragraph to mention that.
>
> > Such operations are defined for the XICS-on-XIVE and the XIVE native
> > KVM devices. They clear the vCPU interrupt presenters that could be
> > attached and then destroy the device.
> >
> > This is not considered as a safe operation as the vCPUs are still
> > running and could be referencing the KVM device through their
> > presenters. To protect the system from any breakage, the kvmppc_xive
> > objects representing both KVM devices are now stored in an array under
> > the VM. Allocation is performed on first usage and memory is freed
> > only when the VM exits.
>
> One quick comment below:
>
> > diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> > index 480a3fc6b9fd..064a9f2ae678 100644
> > --- a/arch/powerpc/kvm/book3s_xive.c
> > +++ b/arch/powerpc/kvm/book3s_xive.c
> > @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> > void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> > {
> > struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> > - struct kvmppc_xive *xive = xc->xive;
> > + struct kvmppc_xive *xive;
> > int i;
> >
> > + if (!kvmppc_xics_enabled(vcpu))
> > + return;
>
> Should that be kvmppc_xive_enabled() rather than xics?
I think I asked that on an earlier iteration, and the answer is no.
The names are confusing, but this file is all about xics-on-xive
rather than xive native. So here we're checking what's available from
the guest's point of view, so "xics", but most of the surrounding
functions are named "xive" because that's the backend.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 4:38 ` David Gibson
0 siblings, 0 replies; 81+ messages in thread
From: David Gibson @ 2019-04-11 4:38 UTC (permalink / raw)
To: Paul Mackerras
Cc: kvm, kvm-ppc, Cédric Le Goater, Paolo Bonzini, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 2689 bytes --]
On Thu, Apr 11, 2019 at 01:16:25PM +1000, Paul Mackerras wrote:
> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> > When a P9 sPAPR VM boots, the CAS negotiation process determines which
> > interrupt mode to use (XICS legacy or XIVE native) and invokes a
> > machine reset to activate the chosen mode.
> >
> > To be able to switch from one mode to another, we introduce the
> > capability to release a KVM device without destroying the VM. The KVM
> > device interface is extended with a new 'release' operation which is
> > called when the file descriptor of the device is closed.
>
> I believe the release operation is not called until all of the mmaps
> using the fd are unmapped - which is a good thing for us, since it
> means the guest can't possibly be accessing the XIVE directly.
> You might want to reword that last paragraph to mention that.
>
> > Such operations are defined for the XICS-on-XIVE and the XIVE native
> > KVM devices. They clear the vCPU interrupt presenters that could be
> > attached and then destroy the device.
> >
> > This is not considered as a safe operation as the vCPUs are still
> > running and could be referencing the KVM device through their
> > presenters. To protect the system from any breakage, the kvmppc_xive
> > objects representing both KVM devices are now stored in an array under
> > the VM. Allocation is performed on first usage and memory is freed
> > only when the VM exits.
>
> One quick comment below:
>
> > diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> > index 480a3fc6b9fd..064a9f2ae678 100644
> > --- a/arch/powerpc/kvm/book3s_xive.c
> > +++ b/arch/powerpc/kvm/book3s_xive.c
> > @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> > void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> > {
> > struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> > - struct kvmppc_xive *xive = xc->xive;
> > + struct kvmppc_xive *xive;
> > int i;
> >
> > + if (!kvmppc_xics_enabled(vcpu))
> > + return;
>
> Should that be kvmppc_xive_enabled() rather than xics?
I think I asked that on an earlier iteration, and the answer is no.
The names are confusing, but this file is all about xics-on-xive
rather than xive native. So here we're checking what's available from
the guest's point of view, so "xics", but most of the surrounding
functions are named "xive" because that's the backend.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 4:38 ` David Gibson
0 siblings, 0 replies; 81+ messages in thread
From: David Gibson @ 2019-04-11 4:38 UTC (permalink / raw)
To: Paul Mackerras
Cc: Cédric Le Goater, kvm-ppc, kvm, Michael Ellerman,
linuxppc-dev, Paolo Bonzini
[-- Attachment #1: Type: text/plain, Size: 2689 bytes --]
On Thu, Apr 11, 2019 at 01:16:25PM +1000, Paul Mackerras wrote:
> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> > When a P9 sPAPR VM boots, the CAS negotiation process determines which
> > interrupt mode to use (XICS legacy or XIVE native) and invokes a
> > machine reset to activate the chosen mode.
> >
> > To be able to switch from one mode to another, we introduce the
> > capability to release a KVM device without destroying the VM. The KVM
> > device interface is extended with a new 'release' operation which is
> > called when the file descriptor of the device is closed.
>
> I believe the release operation is not called until all of the mmaps
> using the fd are unmapped - which is a good thing for us, since it
> means the guest can't possibly be accessing the XIVE directly.
> You might want to reword that last paragraph to mention that.
>
> > Such operations are defined for the XICS-on-XIVE and the XIVE native
> > KVM devices. They clear the vCPU interrupt presenters that could be
> > attached and then destroy the device.
> >
> > This is not considered as a safe operation as the vCPUs are still
> > running and could be referencing the KVM device through their
> > presenters. To protect the system from any breakage, the kvmppc_xive
> > objects representing both KVM devices are now stored in an array under
> > the VM. Allocation is performed on first usage and memory is freed
> > only when the VM exits.
>
> One quick comment below:
>
> > diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> > index 480a3fc6b9fd..064a9f2ae678 100644
> > --- a/arch/powerpc/kvm/book3s_xive.c
> > +++ b/arch/powerpc/kvm/book3s_xive.c
> > @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
> > void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
> > {
> > struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> > - struct kvmppc_xive *xive = xc->xive;
> > + struct kvmppc_xive *xive;
> > int i;
> >
> > + if (!kvmppc_xics_enabled(vcpu))
> > + return;
>
> Should that be kvmppc_xive_enabled() rather than xics?
I think I asked that on an earlier iteration, and the answer is no.
The names are confusing, but this file is all about xics-on-xive
rather than xive native. So here we're checking what's available from
the guest's point of view, so "xics", but most of the surrounding
functions are named "xive" because that's the backend.
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
2019-04-11 4:38 ` David Gibson
(?)
@ 2019-04-11 6:12 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-11 6:12 UTC (permalink / raw)
To: David Gibson, Paul Mackerras
Cc: kvm-ppc, kvm, Michael Ellerman, linuxppc-dev, Paolo Bonzini
On 4/11/19 6:38 AM, David Gibson wrote:
> On Thu, Apr 11, 2019 at 01:16:25PM +1000, Paul Mackerras wrote:
>> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
>>> When a P9 sPAPR VM boots, the CAS negotiation process determines which
>>> interrupt mode to use (XICS legacy or XIVE native) and invokes a
>>> machine reset to activate the chosen mode.
>>>
>>> To be able to switch from one mode to another, we introduce the
>>> capability to release a KVM device without destroying the VM. The KVM
>>> device interface is extended with a new 'release' operation which is
>>> called when the file descriptor of the device is closed.
>>
>> I believe the release operation is not called until all of the mmaps
>> using the fd are unmapped - which is a good thing for us, since it
>> means the guest can't possibly be accessing the XIVE directly.
yes.
>> You might want to reword that last paragraph to mention that.
ok.
>>> Such operations are defined for the XICS-on-XIVE and the XIVE native
>>> KVM devices. They clear the vCPU interrupt presenters that could be
>>> attached and then destroy the device.
>>>
>>> This is not considered as a safe operation as the vCPUs are still
>>> running and could be referencing the KVM device through their
>>> presenters. To protect the system from any breakage, the kvmppc_xive
>>> objects representing both KVM devices are now stored in an array under
>>> the VM. Allocation is performed on first usage and memory is freed
>>> only when the VM exits.
>>
>> One quick comment below:
>>
>>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>>> index 480a3fc6b9fd..064a9f2ae678 100644
>>> --- a/arch/powerpc/kvm/book3s_xive.c
>>> +++ b/arch/powerpc/kvm/book3s_xive.c
>>> @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>>> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>>> {
>>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>> - struct kvmppc_xive *xive = xc->xive;
>>> + struct kvmppc_xive *xive;
>>> int i;
>>>
>>> + if (!kvmppc_xics_enabled(vcpu))
>>> + return;
>>
>> Should that be kvmppc_xive_enabled() rather than xics?
>
> I think I asked that on an earlier iteration, and the answer is no.
> The names are confusing, but this file is all about xics-on-xive
> rather than xive native. So here we're checking what's available from
> the guest's point of view, so "xics", but most of the surrounding
> functions are named "xive" because that's the backend.
>
yes.
The relevant part is at the end of the kvmppc_xive_connect_vcpu() routine :
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
struct kvm_vcpu *vcpu, u32 cpu)
{
...
vcpu->arch.irq_type = KVMPPC_IRQ_XICS;
return 0;
}
David suggested a few cleanups that we could do in the xics-on-xive
device. We might want to introduce a KVMPPC_IRQ_XICS_ON_XIVE flag also.
First, I would like to get rid of references to the kvmppc_xive struct
and remove some useless attributes to improve locking.
Once the XIVE native mode is merged, all kernels above 4.14 running on
a P9 sPAPR guest will switch to XIVE and the xics-on-xive device will
only be useful for nested.
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 6:12 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-11 6:12 UTC (permalink / raw)
To: David Gibson, Paul Mackerras; +Cc: linuxppc-dev, kvm, kvm-ppc, Paolo Bonzini
On 4/11/19 6:38 AM, David Gibson wrote:
> On Thu, Apr 11, 2019 at 01:16:25PM +1000, Paul Mackerras wrote:
>> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
>>> When a P9 sPAPR VM boots, the CAS negotiation process determines which
>>> interrupt mode to use (XICS legacy or XIVE native) and invokes a
>>> machine reset to activate the chosen mode.
>>>
>>> To be able to switch from one mode to another, we introduce the
>>> capability to release a KVM device without destroying the VM. The KVM
>>> device interface is extended with a new 'release' operation which is
>>> called when the file descriptor of the device is closed.
>>
>> I believe the release operation is not called until all of the mmaps
>> using the fd are unmapped - which is a good thing for us, since it
>> means the guest can't possibly be accessing the XIVE directly.
yes.
>> You might want to reword that last paragraph to mention that.
ok.
>>> Such operations are defined for the XICS-on-XIVE and the XIVE native
>>> KVM devices. They clear the vCPU interrupt presenters that could be
>>> attached and then destroy the device.
>>>
>>> This is not considered as a safe operation as the vCPUs are still
>>> running and could be referencing the KVM device through their
>>> presenters. To protect the system from any breakage, the kvmppc_xive
>>> objects representing both KVM devices are now stored in an array under
>>> the VM. Allocation is performed on first usage and memory is freed
>>> only when the VM exits.
>>
>> One quick comment below:
>>
>>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>>> index 480a3fc6b9fd..064a9f2ae678 100644
>>> --- a/arch/powerpc/kvm/book3s_xive.c
>>> +++ b/arch/powerpc/kvm/book3s_xive.c
>>> @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>>> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>>> {
>>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>> - struct kvmppc_xive *xive = xc->xive;
>>> + struct kvmppc_xive *xive;
>>> int i;
>>>
>>> + if (!kvmppc_xics_enabled(vcpu))
>>> + return;
>>
>> Should that be kvmppc_xive_enabled() rather than xics?
>
> I think I asked that on an earlier iteration, and the answer is no.
> The names are confusing, but this file is all about xics-on-xive
> rather than xive native. So here we're checking what's available from
> the guest's point of view, so "xics", but most of the surrounding
> functions are named "xive" because that's the backend.
>
yes.
The relevant part is at the end of the kvmppc_xive_connect_vcpu() routine :
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
struct kvm_vcpu *vcpu, u32 cpu)
{
...
vcpu->arch.irq_type = KVMPPC_IRQ_XICS;
return 0;
}
David suggested a few cleanups that we could do in the xics-on-xive
device. We might want to introduce a KVMPPC_IRQ_XICS_ON_XIVE flag also.
First, I would like to get rid of references to the kvmppc_xive struct
and remove some useless attributes to improve locking.
Once the XIVE native mode is merged, all kernels above 4.14 running on
a P9 sPAPR guest will switch to XIVE and the xics-on-xive device will
only be useful for nested.
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 6:12 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-11 6:12 UTC (permalink / raw)
To: David Gibson, Paul Mackerras
Cc: kvm-ppc, kvm, Michael Ellerman, linuxppc-dev, Paolo Bonzini
On 4/11/19 6:38 AM, David Gibson wrote:
> On Thu, Apr 11, 2019 at 01:16:25PM +1000, Paul Mackerras wrote:
>> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
>>> When a P9 sPAPR VM boots, the CAS negotiation process determines which
>>> interrupt mode to use (XICS legacy or XIVE native) and invokes a
>>> machine reset to activate the chosen mode.
>>>
>>> To be able to switch from one mode to another, we introduce the
>>> capability to release a KVM device without destroying the VM. The KVM
>>> device interface is extended with a new 'release' operation which is
>>> called when the file descriptor of the device is closed.
>>
>> I believe the release operation is not called until all of the mmaps
>> using the fd are unmapped - which is a good thing for us, since it
>> means the guest can't possibly be accessing the XIVE directly.
yes.
>> You might want to reword that last paragraph to mention that.
ok.
>>> Such operations are defined for the XICS-on-XIVE and the XIVE native
>>> KVM devices. They clear the vCPU interrupt presenters that could be
>>> attached and then destroy the device.
>>>
>>> This is not considered as a safe operation as the vCPUs are still
>>> running and could be referencing the KVM device through their
>>> presenters. To protect the system from any breakage, the kvmppc_xive
>>> objects representing both KVM devices are now stored in an array under
>>> the VM. Allocation is performed on first usage and memory is freed
>>> only when the VM exits.
>>
>> One quick comment below:
>>
>>> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
>>> index 480a3fc6b9fd..064a9f2ae678 100644
>>> --- a/arch/powerpc/kvm/book3s_xive.c
>>> +++ b/arch/powerpc/kvm/book3s_xive.c
>>> @@ -1100,11 +1100,19 @@ void kvmppc_xive_disable_vcpu_interrupts(struct kvm_vcpu *vcpu)
>>> void kvmppc_xive_cleanup_vcpu(struct kvm_vcpu *vcpu)
>>> {
>>> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
>>> - struct kvmppc_xive *xive = xc->xive;
>>> + struct kvmppc_xive *xive;
>>> int i;
>>>
>>> + if (!kvmppc_xics_enabled(vcpu))
>>> + return;
>>
>> Should that be kvmppc_xive_enabled() rather than xics?
>
> I think I asked that on an earlier iteration, and the answer is no.
> The names are confusing, but this file is all about xics-on-xive
> rather than xive native. So here we're checking what's available from
> the guest's point of view, so "xics", but most of the surrounding
> functions are named "xive" because that's the backend.
>
yes.
The relevant part is at the end of the kvmppc_xive_connect_vcpu() routine :
int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
struct kvm_vcpu *vcpu, u32 cpu)
{
...
vcpu->arch.irq_type = KVMPPC_IRQ_XICS;
return 0;
}
David suggested a few cleanups that we could do in the xics-on-xive
device. We might want to introduce a KVMPPC_IRQ_XICS_ON_XIVE flag also.
First, I would like to get rid of references to the kvmppc_xive struct
and remove some useless attributes to improve locking.
Once the XIVE native mode is merged, all kernels above 4.14 running on
a P9 sPAPR guest will switch to XIVE and the xics-on-xive device will
only be useful for nested.
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-11 10:27 ` Paul Mackerras
-1 siblings, 0 replies; 81+ messages in thread
From: Paul Mackerras @ 2019-04-11 10:27 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, David Gibson, kvm, Michael Ellerman, linuxppc-dev,
Paolo Bonzini
On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> When a P9 sPAPR VM boots, the CAS negotiation process determines which
> interrupt mode to use (XICS legacy or XIVE native) and invokes a
> machine reset to activate the chosen mode.
>
> To be able to switch from one mode to another, we introduce the
> capability to release a KVM device without destroying the VM. The KVM
> device interface is extended with a new 'release' operation which is
> called when the file descriptor of the device is closed.
Unfortunately, I think there is now a memory leak:
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index ea2018ae1cd7..ea2619d5ca98 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
> struct kvm_device *dev = filp->private_data;
> struct kvm *kvm = dev->kvm;
>
> + if (!dev)
> + return -ENODEV;
> +
> + if (dev->kvm != kvm)
> + return -EPERM;
> +
> + if (dev->ops->release) {
> + mutex_lock(&kvm->lock);
> + list_del(&dev->vm_node);
Because the device is now no longer in the kvm->devices list,
kvm_destroy_devices() won't find it there and therefore won't call the
device's destroy method. In fact now the device's destroy method will
never get called; I can't see how kvmppc_xive_free() or
kvmppc_xive_native_free() will ever get called. Thus the memory for
the kvmppc_xive structs will never get freed as far as I can see.
We could fix that by freeing both of the kvm->arch.xive_devices
entries at VM destruction time.
If it is true that any device that has a release method will never see
its destroy method being called, then that needs to be documented
clearly somewhere.
Paul.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 10:27 ` Paul Mackerras
0 siblings, 0 replies; 81+ messages in thread
From: Paul Mackerras @ 2019-04-11 10:27 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm, kvm-ppc, Paolo Bonzini, linuxppc-dev, David Gibson
On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> When a P9 sPAPR VM boots, the CAS negotiation process determines which
> interrupt mode to use (XICS legacy or XIVE native) and invokes a
> machine reset to activate the chosen mode.
>
> To be able to switch from one mode to another, we introduce the
> capability to release a KVM device without destroying the VM. The KVM
> device interface is extended with a new 'release' operation which is
> called when the file descriptor of the device is closed.
Unfortunately, I think there is now a memory leak:
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index ea2018ae1cd7..ea2619d5ca98 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
> struct kvm_device *dev = filp->private_data;
> struct kvm *kvm = dev->kvm;
>
> + if (!dev)
> + return -ENODEV;
> +
> + if (dev->kvm != kvm)
> + return -EPERM;
> +
> + if (dev->ops->release) {
> + mutex_lock(&kvm->lock);
> + list_del(&dev->vm_node);
Because the device is now no longer in the kvm->devices list,
kvm_destroy_devices() won't find it there and therefore won't call the
device's destroy method. In fact now the device's destroy method will
never get called; I can't see how kvmppc_xive_free() or
kvmppc_xive_native_free() will ever get called. Thus the memory for
the kvmppc_xive structs will never get freed as far as I can see.
We could fix that by freeing both of the kvm->arch.xive_devices
entries at VM destruction time.
If it is true that any device that has a release method will never see
its destroy method being called, then that needs to be documented
clearly somewhere.
Paul.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 10:27 ` Paul Mackerras
0 siblings, 0 replies; 81+ messages in thread
From: Paul Mackerras @ 2019-04-11 10:27 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, David Gibson, kvm, Michael Ellerman, linuxppc-dev,
Paolo Bonzini
On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
> When a P9 sPAPR VM boots, the CAS negotiation process determines which
> interrupt mode to use (XICS legacy or XIVE native) and invokes a
> machine reset to activate the chosen mode.
>
> To be able to switch from one mode to another, we introduce the
> capability to release a KVM device without destroying the VM. The KVM
> device interface is extended with a new 'release' operation which is
> called when the file descriptor of the device is closed.
Unfortunately, I think there is now a memory leak:
> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
> index ea2018ae1cd7..ea2619d5ca98 100644
> --- a/virt/kvm/kvm_main.c
> +++ b/virt/kvm/kvm_main.c
> @@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
> struct kvm_device *dev = filp->private_data;
> struct kvm *kvm = dev->kvm;
>
> + if (!dev)
> + return -ENODEV;
> +
> + if (dev->kvm != kvm)
> + return -EPERM;
> +
> + if (dev->ops->release) {
> + mutex_lock(&kvm->lock);
> + list_del(&dev->vm_node);
Because the device is now no longer in the kvm->devices list,
kvm_destroy_devices() won't find it there and therefore won't call the
device's destroy method. In fact now the device's destroy method will
never get called; I can't see how kvmppc_xive_free() or
kvmppc_xive_native_free() will ever get called. Thus the memory for
the kvmppc_xive structs will never get freed as far as I can see.
We could fix that by freeing both of the kvm->arch.xive_devices
entries at VM destruction time.
If it is true that any device that has a release method will never see
its destroy method being called, then that needs to be documented
clearly somewhere.
Paul.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
2019-04-11 10:27 ` Paul Mackerras
(?)
@ 2019-04-11 11:48 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-11 11:48 UTC (permalink / raw)
To: Paul Mackerras
Cc: kvm-ppc, David Gibson, kvm, Michael Ellerman, linuxppc-dev,
Paolo Bonzini
On 4/11/19 12:27 PM, Paul Mackerras wrote:
> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
>> When a P9 sPAPR VM boots, the CAS negotiation process determines which
>> interrupt mode to use (XICS legacy or XIVE native) and invokes a
>> machine reset to activate the chosen mode.
>>
>> To be able to switch from one mode to another, we introduce the
>> capability to release a KVM device without destroying the VM. The KVM
>> device interface is extended with a new 'release' operation which is
>> called when the file descriptor of the device is closed.
>
> Unfortunately, I think there is now a memory leak:
>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index ea2018ae1cd7..ea2619d5ca98 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
>> struct kvm_device *dev = filp->private_data;
>> struct kvm *kvm = dev->kvm;
>>
>> + if (!dev)
>> + return -ENODEV;
>> +
>> + if (dev->kvm != kvm)
>> + return -EPERM;
>> +
>> + if (dev->ops->release) {
>> + mutex_lock(&kvm->lock);
>> + list_del(&dev->vm_node);
>
> Because the device is now no longer in the kvm->devices list,
> kvm_destroy_devices() won't find it there and therefore won't call the
> device's destroy method. In fact now the device's destroy method will
> never get called; I can't see how kvmppc_xive_free() or
> kvmppc_xive_native_free() will ever get called. Thus the memory for
> the kvmppc_xive structs will never get freed as far as I can see.
ah yes. indeed ...
> We could fix that by freeing both of the kvm->arch.xive_devices
> entries at VM destruction time.
That is what I was doing in the first patch I sent :
http://patchwork.ozlabs.org/patch/1082303/
It worked fine and then, I had this better (worse) idea which I included
in v5.
> If it is true that any device that has a release method will never see
> its destroy method being called, then that needs to be documented
> clearly somewhere.
Yes. Closing the fd should take care of it. I have to rework that patch.
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 11:48 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-11 11:48 UTC (permalink / raw)
To: Paul Mackerras; +Cc: kvm, kvm-ppc, Paolo Bonzini, linuxppc-dev, David Gibson
On 4/11/19 12:27 PM, Paul Mackerras wrote:
> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
>> When a P9 sPAPR VM boots, the CAS negotiation process determines which
>> interrupt mode to use (XICS legacy or XIVE native) and invokes a
>> machine reset to activate the chosen mode.
>>
>> To be able to switch from one mode to another, we introduce the
>> capability to release a KVM device without destroying the VM. The KVM
>> device interface is extended with a new 'release' operation which is
>> called when the file descriptor of the device is closed.
>
> Unfortunately, I think there is now a memory leak:
>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index ea2018ae1cd7..ea2619d5ca98 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
>> struct kvm_device *dev = filp->private_data;
>> struct kvm *kvm = dev->kvm;
>>
>> + if (!dev)
>> + return -ENODEV;
>> +
>> + if (dev->kvm != kvm)
>> + return -EPERM;
>> +
>> + if (dev->ops->release) {
>> + mutex_lock(&kvm->lock);
>> + list_del(&dev->vm_node);
>
> Because the device is now no longer in the kvm->devices list,
> kvm_destroy_devices() won't find it there and therefore won't call the
> device's destroy method. In fact now the device's destroy method will
> never get called; I can't see how kvmppc_xive_free() or
> kvmppc_xive_native_free() will ever get called. Thus the memory for
> the kvmppc_xive structs will never get freed as far as I can see.
ah yes. indeed ...
> We could fix that by freeing both of the kvm->arch.xive_devices
> entries at VM destruction time.
That is what I was doing in the first patch I sent :
http://patchwork.ozlabs.org/patch/1082303/
It worked fine and then, I had this better (worse) idea which I included
in v5.
> If it is true that any device that has a release method will never see
> its destroy method being called, then that needs to be documented
> clearly somewhere.
Yes. Closing the fd should take care of it. I have to rework that patch.
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
@ 2019-04-11 11:48 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-04-11 11:48 UTC (permalink / raw)
To: Paul Mackerras
Cc: kvm-ppc, David Gibson, kvm, Michael Ellerman, linuxppc-dev,
Paolo Bonzini
On 4/11/19 12:27 PM, Paul Mackerras wrote:
> On Wed, Apr 10, 2019 at 07:04:48PM +0200, Cédric Le Goater wrote:
>> When a P9 sPAPR VM boots, the CAS negotiation process determines which
>> interrupt mode to use (XICS legacy or XIVE native) and invokes a
>> machine reset to activate the chosen mode.
>>
>> To be able to switch from one mode to another, we introduce the
>> capability to release a KVM device without destroying the VM. The KVM
>> device interface is extended with a new 'release' operation which is
>> called when the file descriptor of the device is closed.
>
> Unfortunately, I think there is now a memory leak:
>
>> diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
>> index ea2018ae1cd7..ea2619d5ca98 100644
>> --- a/virt/kvm/kvm_main.c
>> +++ b/virt/kvm/kvm_main.c
>> @@ -2938,6 +2938,19 @@ static int kvm_device_release(struct inode *inode, struct file *filp)
>> struct kvm_device *dev = filp->private_data;
>> struct kvm *kvm = dev->kvm;
>>
>> + if (!dev)
>> + return -ENODEV;
>> +
>> + if (dev->kvm != kvm)
>> + return -EPERM;
>> +
>> + if (dev->ops->release) {
>> + mutex_lock(&kvm->lock);
>> + list_del(&dev->vm_node);
>
> Because the device is now no longer in the kvm->devices list,
> kvm_destroy_devices() won't find it there and therefore won't call the
> device's destroy method. In fact now the device's destroy method will
> never get called; I can't see how kvmppc_xive_free() or
> kvmppc_xive_native_free() will ever get called. Thus the memory for
> the kvmppc_xive structs will never get freed as far as I can see.
ah yes. indeed ...
> We could fix that by freeing both of the kvm->arch.xive_devices
> entries at VM destruction time.
That is what I was doing in the first patch I sent :
http://patchwork.ozlabs.org/patch/1082303/
It worked fine and then, I had this better (worse) idea which I included
in v5.
> If it is true that any device that has a release method will never see
> its destroy method being called, then that needs to be documented
> clearly somewhere.
Yes. Closing the fd should take care of it. I have to rework that patch.
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-12 5:19 ` David Gibson
-1 siblings, 0 replies; 81+ messages in thread
From: David Gibson @ 2019-04-12 5:19 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, Paul Mackerras, kvm, Michael Ellerman, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 16576 bytes --]
On Wed, Apr 10, 2019 at 07:04:38PM +0200, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU to configure the underlying
> Event Queue in the XIVE IC. They will also be used to restore the
> configuration of the XIVE EQs and to capture the internal run-time
> state of the EQs. Both 'get' and 'set' rely on an OPAL call to access
> the EQ toggle bit and EQ index which are updated by the XIVE IC when
> event notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes since v4 :
>
> - add check on EQ page alignment
> - add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
>
> arch/powerpc/include/asm/xive.h | 2 +
> arch/powerpc/include/uapi/asm/kvm.h | 19 ++
> arch/powerpc/kvm/book3s_xive.h | 2 +
> arch/powerpc/kvm/book3s_xive.c | 15 +-
> arch/powerpc/kvm/book3s_xive_native.c | 249 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 34 +++
> 6 files changed, 315 insertions(+), 6 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index b579a943407b..c4e88abd3b67 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -73,6 +73,8 @@ struct xive_q {
> u32 esc_irq;
> atomic_t count;
> atomic_t pending_count;
> + u64 guest_qaddr;
> + u32 guest_qshift;
> };
>
> /* Global enable flags for the XIVE support */
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index e8161e21629b..85005400fd86 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> @@ -696,4 +697,22 @@ struct kvm_ppc_cpu_char {
> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>
> +/* Layout of 64-bit EQ identifier */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of EQ configuration values (64 bytes) */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qshift;
> + __u64 qaddr;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
> +
> +#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index ae26fe653d98..622f594d93e1 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -272,6 +272,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index e09f3addffe5..c1b7aa7dbc28 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
> return IRQ_HANDLED;
> }
>
> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct xive_q *q = &xc->queues[prio];
> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> return -EIO;
> }
>
> - if (xc->xive->single_escalation)
> + if (single_escalation)
> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
> vcpu->kvm->arch.lpid, xc->server_num);
> else
> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> * interrupt, thus leaving it effectively masked after
> * it fires once.
> */
> - if (xc->xive->single_escalation) {
> + if (single_escalation) {
> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>
> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
> continue;
> rc = xive_provision_queue(vcpu, prio);
> if (rc == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, prio);
> + kvmppc_xive_attach_escalation(vcpu, prio,
> + xive->single_escalation);
> if (rc)
> return rc;
> }
> @@ -1214,7 +1216,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> if (xive->qmap & (1 << i)) {
> r = xive_provision_queue(vcpu, i);
> if (r == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, i);
> + kvmppc_xive_attach_escalation(
> + vcpu, i, xive->single_escalation);
> if (r)
> goto bail;
> } else {
> @@ -1229,7 +1232,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
>
> /* If not done above, attach priority 0 escalation */
> - r = xive_attach_escalation(vcpu, 0);
> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
> if (r)
> goto bail;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 492825a35958..3e7cdcacc932 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -335,6 +335,243 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, masked, eisn);
> }
>
> +static int xive_native_validate_queue_size(u32 qshift)
> +{
> + /*
> + * We only support 64K pages for the moment. This is also
> + * advertised in the DT property "ibm,xive-eq-sizes"
> + */
> + switch (qshift) {
> + case 0: /* EQ reset */
> + case 16:
> + return 0;
> + case 12:
> + case 21:
> + case 24:
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + void __user *ubufp = (void __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + int rc;
> + __be32 *qaddr = 0;
> + struct page *page;
> + struct xive_q *q;
> + gfn_t gfn;
> + unsigned long page_size;
> +
> + /*
> + * Demangle priority/server tuple from the EQ identifier
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + /*
> + * sPAPR specifies a "Unconditional Notify (n) flag" for the
> + * H_INT_SET_QUEUE_CONFIG hcall which forces notification
> + * without using the coalescing mechanisms provided by the
> + * XIVE END ESBs. This is required on KVM as notification
> + * using the END ESBs is not supported.
> + */
> + if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
> + pr_err("invalid flags %d\n", kvm_eq.flags);
> + return -EINVAL;
> + }
> +
> + rc = xive_native_validate_queue_size(kvm_eq.qshift);
> + if (rc) {
> + pr_err("invalid queue size %d\n", kvm_eq.qshift);
> + return rc;
> + }
> +
> + /* reset queue and disable queueing */
> + if (!kvm_eq.qshift) {
> + q->guest_qaddr = 0;
> + q->guest_qshift = 0;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + NULL, 0, true);
> + if (rc) {
> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + return rc;
> + }
> +
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +
> + return 0;
> + }
> +
> + if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
> + pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
> + 1ull << kvm_eq.qshift);
> + return -EINVAL;
> + }
> +
> + gfn = gpa_to_gfn(kvm_eq.qaddr);
> + page = gfn_to_page(kvm, gfn);
> + if (is_error_page(page)) {
> + pr_err("Couldn't get queue page %llx!\n", kvm_eq.qaddr);
> + return -EINVAL;
> + }
> + page_size = kvm_host_page_size(kvm, gfn);
> + if (1ull << kvm_eq.qshift > page_size) {
> + pr_warn("Incompatible host page size %lx!\n", page_size);
> + return -EINVAL;
> + }
> +
> + qaddr = page_to_virt(page) + (kvm_eq.qaddr & ~PAGE_MASK);
> +
> + /*
> + * Backup the queue page guest address to the mark EQ page
> + * dirty for migration.
> + */
> + q->guest_qaddr = kvm_eq.qaddr;
> + q->guest_qshift = kvm_eq.qshift;
> +
> + /*
> + * Unconditional Notification is forced by default at the
> + * OPAL level because the use of END ESBs is not supported by
> + * Linux.
> + */
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + (__be32 *) qaddr, kvm_eq.qshift, true);
> + if (rc) {
> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + put_page(page);
> + return rc;
> + }
> +
> + /*
> + * Only restore the queue state when needed. When doing the
> + * H_INT_SET_SOURCE_CONFIG hcall, it should not.
> + */
> + if (kvm_eq.qtoggle != 1 || kvm_eq.qindex != 0) {
> + rc = xive_native_set_queue_state(xc->vp_id, priority,
> + kvm_eq.qtoggle,
> + kvm_eq.qindex);
> + if (rc)
> + goto error;
> + }
> +
> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
> + xive->single_escalation);
> +error:
> + if (rc)
> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + struct xive_q *q;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + u64 qaddr;
> + u64 qshift;
> + u64 qeoi_page;
> + u32 escalate_irq;
> + u64 qflags;
> + int rc;
> +
> + /*
> + * Demangle priority/server tuple from the EQ identifier
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + memset(&kvm_eq, 0, sizeof(kvm_eq));
> +
> + if (!q->qpage)
> + return 0;
> +
> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qaddr, &qshift,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc)
> + return rc;
> +
> + kvm_eq.flags = 0;
> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
> + kvm_eq.flags |= KVM_XIVE_EQ_ALWAYS_NOTIFY;
> +
> + kvm_eq.qshift = q->guest_qshift;
> + kvm_eq.qaddr = q->guest_qaddr;
> +
> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
> + &kvm_eq.qindex);
> + if (rc)
> + return rc;
> +
> + pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -349,6 +586,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> return kvmppc_xive_native_set_source_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -356,6 +596,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
> + attr->addr);
> + }
> return -ENXIO;
> }
>
> @@ -371,6 +618,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> break;
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return 0;
> }
> return -ENXIO;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 33c64b2cdbe8..cc13bfd5cf53 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -53,3 +53,37 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENXIO: CPU event queues not configured or configuration of the
> underlying HW interrupt failed
> -EBUSY: No CPU available to serve interrupt
> +
> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
> + Configures an event queue of a CPU
> + Attributes:
> + EQ descriptor identifier (64-bit)
> + The EQ descriptor identifier is a tuple (server, priority) :
> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
> + values: | unused | server | priority
> + The kvm_device_attr.addr points to :
> + struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qshift;
> + __u64 qaddr;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> + };
> + - flags: queue flags
> + KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
> + forces notification without using the coalescing mechanism
> + provided by the XIVE END ESBs.
> + - qshift: queue size (power of 2)
> + - qaddr: real address of queue
> + - qtoggle: current queue toggle bit
> + - qindex: current queue index
> + - pad: reserved for future use
> + Errors:
> + -ENOENT: Invalid CPU number
> + -EINVAL: Invalid priority
> + -EINVAL: Invalid flags
> + -EINVAL: Invalid queue size
> + -EINVAL: Invalid queue address
> + -EFAULT: Invalid user pointer for attr->addr.
> + -EIO: Configuration of the underlying HW failed
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-04-12 5:19 ` David Gibson
0 siblings, 0 replies; 81+ messages in thread
From: David Gibson @ 2019-04-12 5:19 UTC (permalink / raw)
To: Cédric Le Goater; +Cc: linuxppc-dev, Paul Mackerras, kvm, kvm-ppc
[-- Attachment #1: Type: text/plain, Size: 16576 bytes --]
On Wed, Apr 10, 2019 at 07:04:38PM +0200, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU to configure the underlying
> Event Queue in the XIVE IC. They will also be used to restore the
> configuration of the XIVE EQs and to capture the internal run-time
> state of the EQs. Both 'get' and 'set' rely on an OPAL call to access
> the EQ toggle bit and EQ index which are updated by the XIVE IC when
> event notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes since v4 :
>
> - add check on EQ page alignment
> - add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
>
> arch/powerpc/include/asm/xive.h | 2 +
> arch/powerpc/include/uapi/asm/kvm.h | 19 ++
> arch/powerpc/kvm/book3s_xive.h | 2 +
> arch/powerpc/kvm/book3s_xive.c | 15 +-
> arch/powerpc/kvm/book3s_xive_native.c | 249 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 34 +++
> 6 files changed, 315 insertions(+), 6 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index b579a943407b..c4e88abd3b67 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -73,6 +73,8 @@ struct xive_q {
> u32 esc_irq;
> atomic_t count;
> atomic_t pending_count;
> + u64 guest_qaddr;
> + u32 guest_qshift;
> };
>
> /* Global enable flags for the XIVE support */
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index e8161e21629b..85005400fd86 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> @@ -696,4 +697,22 @@ struct kvm_ppc_cpu_char {
> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>
> +/* Layout of 64-bit EQ identifier */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of EQ configuration values (64 bytes) */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qshift;
> + __u64 qaddr;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
> +
> +#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index ae26fe653d98..622f594d93e1 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -272,6 +272,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index e09f3addffe5..c1b7aa7dbc28 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
> return IRQ_HANDLED;
> }
>
> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct xive_q *q = &xc->queues[prio];
> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> return -EIO;
> }
>
> - if (xc->xive->single_escalation)
> + if (single_escalation)
> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
> vcpu->kvm->arch.lpid, xc->server_num);
> else
> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> * interrupt, thus leaving it effectively masked after
> * it fires once.
> */
> - if (xc->xive->single_escalation) {
> + if (single_escalation) {
> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>
> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
> continue;
> rc = xive_provision_queue(vcpu, prio);
> if (rc == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, prio);
> + kvmppc_xive_attach_escalation(vcpu, prio,
> + xive->single_escalation);
> if (rc)
> return rc;
> }
> @@ -1214,7 +1216,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> if (xive->qmap & (1 << i)) {
> r = xive_provision_queue(vcpu, i);
> if (r == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, i);
> + kvmppc_xive_attach_escalation(
> + vcpu, i, xive->single_escalation);
> if (r)
> goto bail;
> } else {
> @@ -1229,7 +1232,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
>
> /* If not done above, attach priority 0 escalation */
> - r = xive_attach_escalation(vcpu, 0);
> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
> if (r)
> goto bail;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 492825a35958..3e7cdcacc932 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -335,6 +335,243 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, masked, eisn);
> }
>
> +static int xive_native_validate_queue_size(u32 qshift)
> +{
> + /*
> + * We only support 64K pages for the moment. This is also
> + * advertised in the DT property "ibm,xive-eq-sizes"
> + */
> + switch (qshift) {
> + case 0: /* EQ reset */
> + case 16:
> + return 0;
> + case 12:
> + case 21:
> + case 24:
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + void __user *ubufp = (void __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + int rc;
> + __be32 *qaddr = 0;
> + struct page *page;
> + struct xive_q *q;
> + gfn_t gfn;
> + unsigned long page_size;
> +
> + /*
> + * Demangle priority/server tuple from the EQ identifier
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + /*
> + * sPAPR specifies a "Unconditional Notify (n) flag" for the
> + * H_INT_SET_QUEUE_CONFIG hcall which forces notification
> + * without using the coalescing mechanisms provided by the
> + * XIVE END ESBs. This is required on KVM as notification
> + * using the END ESBs is not supported.
> + */
> + if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
> + pr_err("invalid flags %d\n", kvm_eq.flags);
> + return -EINVAL;
> + }
> +
> + rc = xive_native_validate_queue_size(kvm_eq.qshift);
> + if (rc) {
> + pr_err("invalid queue size %d\n", kvm_eq.qshift);
> + return rc;
> + }
> +
> + /* reset queue and disable queueing */
> + if (!kvm_eq.qshift) {
> + q->guest_qaddr = 0;
> + q->guest_qshift = 0;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + NULL, 0, true);
> + if (rc) {
> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + return rc;
> + }
> +
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +
> + return 0;
> + }
> +
> + if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
> + pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
> + 1ull << kvm_eq.qshift);
> + return -EINVAL;
> + }
> +
> + gfn = gpa_to_gfn(kvm_eq.qaddr);
> + page = gfn_to_page(kvm, gfn);
> + if (is_error_page(page)) {
> + pr_err("Couldn't get queue page %llx!\n", kvm_eq.qaddr);
> + return -EINVAL;
> + }
> + page_size = kvm_host_page_size(kvm, gfn);
> + if (1ull << kvm_eq.qshift > page_size) {
> + pr_warn("Incompatible host page size %lx!\n", page_size);
> + return -EINVAL;
> + }
> +
> + qaddr = page_to_virt(page) + (kvm_eq.qaddr & ~PAGE_MASK);
> +
> + /*
> + * Backup the queue page guest address to the mark EQ page
> + * dirty for migration.
> + */
> + q->guest_qaddr = kvm_eq.qaddr;
> + q->guest_qshift = kvm_eq.qshift;
> +
> + /*
> + * Unconditional Notification is forced by default at the
> + * OPAL level because the use of END ESBs is not supported by
> + * Linux.
> + */
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + (__be32 *) qaddr, kvm_eq.qshift, true);
> + if (rc) {
> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + put_page(page);
> + return rc;
> + }
> +
> + /*
> + * Only restore the queue state when needed. When doing the
> + * H_INT_SET_SOURCE_CONFIG hcall, it should not.
> + */
> + if (kvm_eq.qtoggle != 1 || kvm_eq.qindex != 0) {
> + rc = xive_native_set_queue_state(xc->vp_id, priority,
> + kvm_eq.qtoggle,
> + kvm_eq.qindex);
> + if (rc)
> + goto error;
> + }
> +
> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
> + xive->single_escalation);
> +error:
> + if (rc)
> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + struct xive_q *q;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + u64 qaddr;
> + u64 qshift;
> + u64 qeoi_page;
> + u32 escalate_irq;
> + u64 qflags;
> + int rc;
> +
> + /*
> + * Demangle priority/server tuple from the EQ identifier
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + memset(&kvm_eq, 0, sizeof(kvm_eq));
> +
> + if (!q->qpage)
> + return 0;
> +
> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qaddr, &qshift,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc)
> + return rc;
> +
> + kvm_eq.flags = 0;
> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
> + kvm_eq.flags |= KVM_XIVE_EQ_ALWAYS_NOTIFY;
> +
> + kvm_eq.qshift = q->guest_qshift;
> + kvm_eq.qaddr = q->guest_qaddr;
> +
> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
> + &kvm_eq.qindex);
> + if (rc)
> + return rc;
> +
> + pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -349,6 +586,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> return kvmppc_xive_native_set_source_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -356,6 +596,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
> + attr->addr);
> + }
> return -ENXIO;
> }
>
> @@ -371,6 +618,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> break;
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return 0;
> }
> return -ENXIO;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 33c64b2cdbe8..cc13bfd5cf53 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -53,3 +53,37 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENXIO: CPU event queues not configured or configuration of the
> underlying HW interrupt failed
> -EBUSY: No CPU available to serve interrupt
> +
> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
> + Configures an event queue of a CPU
> + Attributes:
> + EQ descriptor identifier (64-bit)
> + The EQ descriptor identifier is a tuple (server, priority) :
> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
> + values: | unused | server | priority
> + The kvm_device_attr.addr points to :
> + struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qshift;
> + __u64 qaddr;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> + };
> + - flags: queue flags
> + KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
> + forces notification without using the coalescing mechanism
> + provided by the XIVE END ESBs.
> + - qshift: queue size (power of 2)
> + - qaddr: real address of queue
> + - qtoggle: current queue toggle bit
> + - qindex: current queue index
> + - pad: reserved for future use
> + Errors:
> + -ENOENT: Invalid CPU number
> + -EINVAL: Invalid priority
> + -EINVAL: Invalid flags
> + -EINVAL: Invalid queue size
> + -EINVAL: Invalid queue address
> + -EFAULT: Invalid user pointer for attr->addr.
> + -EIO: Configuration of the underlying HW failed
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
@ 2019-04-12 5:19 ` David Gibson
0 siblings, 0 replies; 81+ messages in thread
From: David Gibson @ 2019-04-12 5:19 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, Paul Mackerras, kvm, Michael Ellerman, linuxppc-dev
[-- Attachment #1: Type: text/plain, Size: 16576 bytes --]
On Wed, Apr 10, 2019 at 07:04:38PM +0200, Cédric Le Goater wrote:
> These controls will be used by the H_INT_SET_QUEUE_CONFIG and
> H_INT_GET_QUEUE_CONFIG hcalls from QEMU to configure the underlying
> Event Queue in the XIVE IC. They will also be used to restore the
> configuration of the XIVE EQs and to capture the internal run-time
> state of the EQs. Both 'get' and 'set' rely on an OPAL call to access
> the EQ toggle bit and EQ index which are updated by the XIVE IC when
> event notifications are enqueued in the EQ.
>
> The value of the guest physical address of the event queue is saved in
> the XIVE internal xive_q structure for later use. That is when
> migration needs to mark the EQ pages dirty to capture a consistent
> memory state of the VM.
>
> To be noted that H_INT_SET_QUEUE_CONFIG does not require the extra
> OPAL call setting the EQ toggle bit and EQ index to configure the EQ,
> but restoring the EQ state will.
>
> Signed-off-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>
> Changes since v4 :
>
> - add check on EQ page alignment
> - add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
>
> arch/powerpc/include/asm/xive.h | 2 +
> arch/powerpc/include/uapi/asm/kvm.h | 19 ++
> arch/powerpc/kvm/book3s_xive.h | 2 +
> arch/powerpc/kvm/book3s_xive.c | 15 +-
> arch/powerpc/kvm/book3s_xive_native.c | 249 +++++++++++++++++++++
> Documentation/virtual/kvm/devices/xive.txt | 34 +++
> 6 files changed, 315 insertions(+), 6 deletions(-)
>
> diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
> index b579a943407b..c4e88abd3b67 100644
> --- a/arch/powerpc/include/asm/xive.h
> +++ b/arch/powerpc/include/asm/xive.h
> @@ -73,6 +73,8 @@ struct xive_q {
> u32 esc_irq;
> atomic_t count;
> atomic_t pending_count;
> + u64 guest_qaddr;
> + u32 guest_qshift;
> };
>
> /* Global enable flags for the XIVE support */
> diff --git a/arch/powerpc/include/uapi/asm/kvm.h b/arch/powerpc/include/uapi/asm/kvm.h
> index e8161e21629b..85005400fd86 100644
> --- a/arch/powerpc/include/uapi/asm/kvm.h
> +++ b/arch/powerpc/include/uapi/asm/kvm.h
> @@ -681,6 +681,7 @@ struct kvm_ppc_cpu_char {
> #define KVM_DEV_XIVE_GRP_CTRL 1
> #define KVM_DEV_XIVE_GRP_SOURCE 2 /* 64-bit source identifier */
> #define KVM_DEV_XIVE_GRP_SOURCE_CONFIG 3 /* 64-bit source identifier */
> +#define KVM_DEV_XIVE_GRP_EQ_CONFIG 4 /* 64-bit EQ identifier */
>
> /* Layout of 64-bit XIVE source attribute values */
> #define KVM_XIVE_LEVEL_SENSITIVE (1ULL << 0)
> @@ -696,4 +697,22 @@ struct kvm_ppc_cpu_char {
> #define KVM_XIVE_SOURCE_EISN_SHIFT 33
> #define KVM_XIVE_SOURCE_EISN_MASK 0xfffffffe00000000ULL
>
> +/* Layout of 64-bit EQ identifier */
> +#define KVM_XIVE_EQ_PRIORITY_SHIFT 0
> +#define KVM_XIVE_EQ_PRIORITY_MASK 0x7
> +#define KVM_XIVE_EQ_SERVER_SHIFT 3
> +#define KVM_XIVE_EQ_SERVER_MASK 0xfffffff8ULL
> +
> +/* Layout of EQ configuration values (64 bytes) */
> +struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qshift;
> + __u64 qaddr;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> +};
> +
> +#define KVM_XIVE_EQ_ALWAYS_NOTIFY 0x00000001
> +
> #endif /* __LINUX_KVM_POWERPC_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.h b/arch/powerpc/kvm/book3s_xive.h
> index ae26fe653d98..622f594d93e1 100644
> --- a/arch/powerpc/kvm/book3s_xive.h
> +++ b/arch/powerpc/kvm/book3s_xive.h
> @@ -272,6 +272,8 @@ struct kvmppc_xive_src_block *kvmppc_xive_create_src_block(
> struct kvmppc_xive *xive, int irq);
> void kvmppc_xive_free_sources(struct kvmppc_xive_src_block *sb);
> int kvmppc_xive_select_target(struct kvm *kvm, u32 *server, u8 prio);
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation);
>
> #endif /* CONFIG_KVM_XICS */
> #endif /* _KVM_PPC_BOOK3S_XICS_H */
> diff --git a/arch/powerpc/kvm/book3s_xive.c b/arch/powerpc/kvm/book3s_xive.c
> index e09f3addffe5..c1b7aa7dbc28 100644
> --- a/arch/powerpc/kvm/book3s_xive.c
> +++ b/arch/powerpc/kvm/book3s_xive.c
> @@ -166,7 +166,8 @@ static irqreturn_t xive_esc_irq(int irq, void *data)
> return IRQ_HANDLED;
> }
>
> -static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> +int kvmppc_xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio,
> + bool single_escalation)
> {
> struct kvmppc_xive_vcpu *xc = vcpu->arch.xive_vcpu;
> struct xive_q *q = &xc->queues[prio];
> @@ -185,7 +186,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> return -EIO;
> }
>
> - if (xc->xive->single_escalation)
> + if (single_escalation)
> name = kasprintf(GFP_KERNEL, "kvm-%d-%d",
> vcpu->kvm->arch.lpid, xc->server_num);
> else
> @@ -217,7 +218,7 @@ static int xive_attach_escalation(struct kvm_vcpu *vcpu, u8 prio)
> * interrupt, thus leaving it effectively masked after
> * it fires once.
> */
> - if (xc->xive->single_escalation) {
> + if (single_escalation) {
> struct irq_data *d = irq_get_irq_data(xc->esc_virq[prio]);
> struct xive_irq_data *xd = irq_data_get_irq_handler_data(d);
>
> @@ -291,7 +292,8 @@ static int xive_check_provisioning(struct kvm *kvm, u8 prio)
> continue;
> rc = xive_provision_queue(vcpu, prio);
> if (rc == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, prio);
> + kvmppc_xive_attach_escalation(vcpu, prio,
> + xive->single_escalation);
> if (rc)
> return rc;
> }
> @@ -1214,7 +1216,8 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> if (xive->qmap & (1 << i)) {
> r = xive_provision_queue(vcpu, i);
> if (r == 0 && !xive->single_escalation)
> - xive_attach_escalation(vcpu, i);
> + kvmppc_xive_attach_escalation(
> + vcpu, i, xive->single_escalation);
> if (r)
> goto bail;
> } else {
> @@ -1229,7 +1232,7 @@ int kvmppc_xive_connect_vcpu(struct kvm_device *dev,
> }
>
> /* If not done above, attach priority 0 escalation */
> - r = xive_attach_escalation(vcpu, 0);
> + r = kvmppc_xive_attach_escalation(vcpu, 0, xive->single_escalation);
> if (r)
> goto bail;
>
> diff --git a/arch/powerpc/kvm/book3s_xive_native.c b/arch/powerpc/kvm/book3s_xive_native.c
> index 492825a35958..3e7cdcacc932 100644
> --- a/arch/powerpc/kvm/book3s_xive_native.c
> +++ b/arch/powerpc/kvm/book3s_xive_native.c
> @@ -335,6 +335,243 @@ static int kvmppc_xive_native_set_source_config(struct kvmppc_xive *xive,
> priority, masked, eisn);
> }
>
> +static int xive_native_validate_queue_size(u32 qshift)
> +{
> + /*
> + * We only support 64K pages for the moment. This is also
> + * advertised in the DT property "ibm,xive-eq-sizes"
> + */
> + switch (qshift) {
> + case 0: /* EQ reset */
> + case 16:
> + return 0;
> + case 12:
> + case 21:
> + case 24:
> + default:
> + return -EINVAL;
> + }
> +}
> +
> +static int kvmppc_xive_native_set_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + void __user *ubufp = (void __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + int rc;
> + __be32 *qaddr = 0;
> + struct page *page;
> + struct xive_q *q;
> + gfn_t gfn;
> + unsigned long page_size;
> +
> + /*
> + * Demangle priority/server tuple from the EQ identifier
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + if (copy_from_user(&kvm_eq, ubufp, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("Trying to restore invalid queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + /*
> + * sPAPR specifies a "Unconditional Notify (n) flag" for the
> + * H_INT_SET_QUEUE_CONFIG hcall which forces notification
> + * without using the coalescing mechanisms provided by the
> + * XIVE END ESBs. This is required on KVM as notification
> + * using the END ESBs is not supported.
> + */
> + if (kvm_eq.flags != KVM_XIVE_EQ_ALWAYS_NOTIFY) {
> + pr_err("invalid flags %d\n", kvm_eq.flags);
> + return -EINVAL;
> + }
> +
> + rc = xive_native_validate_queue_size(kvm_eq.qshift);
> + if (rc) {
> + pr_err("invalid queue size %d\n", kvm_eq.qshift);
> + return rc;
> + }
> +
> + /* reset queue and disable queueing */
> + if (!kvm_eq.qshift) {
> + q->guest_qaddr = 0;
> + q->guest_qshift = 0;
> +
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + NULL, 0, true);
> + if (rc) {
> + pr_err("Failed to reset queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + return rc;
> + }
> +
> + if (q->qpage) {
> + put_page(virt_to_page(q->qpage));
> + q->qpage = NULL;
> + }
> +
> + return 0;
> + }
> +
> + if (kvm_eq.qaddr & ((1ull << kvm_eq.qshift) - 1)) {
> + pr_err("queue page is not aligned %llx/%llx\n", kvm_eq.qaddr,
> + 1ull << kvm_eq.qshift);
> + return -EINVAL;
> + }
> +
> + gfn = gpa_to_gfn(kvm_eq.qaddr);
> + page = gfn_to_page(kvm, gfn);
> + if (is_error_page(page)) {
> + pr_err("Couldn't get queue page %llx!\n", kvm_eq.qaddr);
> + return -EINVAL;
> + }
> + page_size = kvm_host_page_size(kvm, gfn);
> + if (1ull << kvm_eq.qshift > page_size) {
> + pr_warn("Incompatible host page size %lx!\n", page_size);
> + return -EINVAL;
> + }
> +
> + qaddr = page_to_virt(page) + (kvm_eq.qaddr & ~PAGE_MASK);
> +
> + /*
> + * Backup the queue page guest address to the mark EQ page
> + * dirty for migration.
> + */
> + q->guest_qaddr = kvm_eq.qaddr;
> + q->guest_qshift = kvm_eq.qshift;
> +
> + /*
> + * Unconditional Notification is forced by default at the
> + * OPAL level because the use of END ESBs is not supported by
> + * Linux.
> + */
> + rc = xive_native_configure_queue(xc->vp_id, q, priority,
> + (__be32 *) qaddr, kvm_eq.qshift, true);
> + if (rc) {
> + pr_err("Failed to configure queue %d for VCPU %d: %d\n",
> + priority, xc->server_num, rc);
> + put_page(page);
> + return rc;
> + }
> +
> + /*
> + * Only restore the queue state when needed. When doing the
> + * H_INT_SET_SOURCE_CONFIG hcall, it should not.
> + */
> + if (kvm_eq.qtoggle != 1 || kvm_eq.qindex != 0) {
> + rc = xive_native_set_queue_state(xc->vp_id, priority,
> + kvm_eq.qtoggle,
> + kvm_eq.qindex);
> + if (rc)
> + goto error;
> + }
> +
> + rc = kvmppc_xive_attach_escalation(vcpu, priority,
> + xive->single_escalation);
> +error:
> + if (rc)
> + kvmppc_xive_native_cleanup_queue(vcpu, priority);
> + return rc;
> +}
> +
> +static int kvmppc_xive_native_get_queue_config(struct kvmppc_xive *xive,
> + long eq_idx, u64 addr)
> +{
> + struct kvm *kvm = xive->kvm;
> + struct kvm_vcpu *vcpu;
> + struct kvmppc_xive_vcpu *xc;
> + struct xive_q *q;
> + void __user *ubufp = (u64 __user *) addr;
> + u32 server;
> + u8 priority;
> + struct kvm_ppc_xive_eq kvm_eq;
> + u64 qaddr;
> + u64 qshift;
> + u64 qeoi_page;
> + u32 escalate_irq;
> + u64 qflags;
> + int rc;
> +
> + /*
> + * Demangle priority/server tuple from the EQ identifier
> + */
> + priority = (eq_idx & KVM_XIVE_EQ_PRIORITY_MASK) >>
> + KVM_XIVE_EQ_PRIORITY_SHIFT;
> + server = (eq_idx & KVM_XIVE_EQ_SERVER_MASK) >>
> + KVM_XIVE_EQ_SERVER_SHIFT;
> +
> + vcpu = kvmppc_xive_find_server(kvm, server);
> + if (!vcpu) {
> + pr_err("Can't find server %d\n", server);
> + return -ENOENT;
> + }
> + xc = vcpu->arch.xive_vcpu;
> +
> + if (priority != xive_prio_from_guest(priority)) {
> + pr_err("invalid priority for queue %d for VCPU %d\n",
> + priority, server);
> + return -EINVAL;
> + }
> + q = &xc->queues[priority];
> +
> + memset(&kvm_eq, 0, sizeof(kvm_eq));
> +
> + if (!q->qpage)
> + return 0;
> +
> + rc = xive_native_get_queue_info(xc->vp_id, priority, &qaddr, &qshift,
> + &qeoi_page, &escalate_irq, &qflags);
> + if (rc)
> + return rc;
> +
> + kvm_eq.flags = 0;
> + if (qflags & OPAL_XIVE_EQ_ALWAYS_NOTIFY)
> + kvm_eq.flags |= KVM_XIVE_EQ_ALWAYS_NOTIFY;
> +
> + kvm_eq.qshift = q->guest_qshift;
> + kvm_eq.qaddr = q->guest_qaddr;
> +
> + rc = xive_native_get_queue_state(xc->vp_id, priority, &kvm_eq.qtoggle,
> + &kvm_eq.qindex);
> + if (rc)
> + return rc;
> +
> + pr_devel("%s VCPU %d priority %d fl:%x shift:%d addr:%llx g:%d idx:%d\n",
> + __func__, server, priority, kvm_eq.flags,
> + kvm_eq.qshift, kvm_eq.qaddr, kvm_eq.qtoggle, kvm_eq.qindex);
> +
> + if (copy_to_user(ubufp, &kvm_eq, sizeof(kvm_eq)))
> + return -EFAULT;
> +
> + return 0;
> +}
> +
> static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> @@ -349,6 +586,9 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> case KVM_DEV_XIVE_GRP_SOURCE_CONFIG:
> return kvmppc_xive_native_set_source_config(xive, attr->attr,
> attr->addr);
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_set_queue_config(xive, attr->attr,
> + attr->addr);
> }
> return -ENXIO;
> }
> @@ -356,6 +596,13 @@ static int kvmppc_xive_native_set_attr(struct kvm_device *dev,
> static int kvmppc_xive_native_get_attr(struct kvm_device *dev,
> struct kvm_device_attr *attr)
> {
> + struct kvmppc_xive *xive = dev->private;
> +
> + switch (attr->group) {
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return kvmppc_xive_native_get_queue_config(xive, attr->attr,
> + attr->addr);
> + }
> return -ENXIO;
> }
>
> @@ -371,6 +618,8 @@ static int kvmppc_xive_native_has_attr(struct kvm_device *dev,
> attr->attr < KVMPPC_XIVE_NR_IRQS)
> return 0;
> break;
> + case KVM_DEV_XIVE_GRP_EQ_CONFIG:
> + return 0;
> }
> return -ENXIO;
> }
> diff --git a/Documentation/virtual/kvm/devices/xive.txt b/Documentation/virtual/kvm/devices/xive.txt
> index 33c64b2cdbe8..cc13bfd5cf53 100644
> --- a/Documentation/virtual/kvm/devices/xive.txt
> +++ b/Documentation/virtual/kvm/devices/xive.txt
> @@ -53,3 +53,37 @@ the legacy interrupt mode, referred as XICS (POWER7/8).
> -ENXIO: CPU event queues not configured or configuration of the
> underlying HW interrupt failed
> -EBUSY: No CPU available to serve interrupt
> +
> + 4. KVM_DEV_XIVE_GRP_EQ_CONFIG (read-write)
> + Configures an event queue of a CPU
> + Attributes:
> + EQ descriptor identifier (64-bit)
> + The EQ descriptor identifier is a tuple (server, priority) :
> + bits: | 63 .... 32 | 31 .. 3 | 2 .. 0
> + values: | unused | server | priority
> + The kvm_device_attr.addr points to :
> + struct kvm_ppc_xive_eq {
> + __u32 flags;
> + __u32 qshift;
> + __u64 qaddr;
> + __u32 qtoggle;
> + __u32 qindex;
> + __u8 pad[40];
> + };
> + - flags: queue flags
> + KVM_XIVE_EQ_ALWAYS_NOTIFY (required)
> + forces notification without using the coalescing mechanism
> + provided by the XIVE END ESBs.
> + - qshift: queue size (power of 2)
> + - qaddr: real address of queue
> + - qtoggle: current queue toggle bit
> + - qindex: current queue index
> + - pad: reserved for future use
> + Errors:
> + -ENOENT: Invalid CPU number
> + -EINVAL: Invalid priority
> + -EINVAL: Invalid flags
> + -EINVAL: Invalid queue size
> + -EINVAL: Invalid queue address
> + -EFAULT: Invalid user pointer for attr->addr.
> + -EIO: Configuration of the underlying HW failed
--
David Gibson | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-04-29 8:05 ` Satheesh Rajendran
-1 siblings, 0 replies; 81+ messages in thread
From: Satheesh Rajendran @ 2019-04-29 8:05 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, kvm, Paul Mackerras, linuxppc-dev, David Gibson
On Wed, Apr 10, 2019 at 07:04:32PM +0200, Cédric Le Goater wrote:
> Hello,
>
> GitHub trees available here :
>
> QEMU sPAPR:
>
> https://github.com/legoater/qemu/commits/xive-next
>
> Linux/KVM:
>
> https://github.com/legoater/linux/commits/xive-5.1
Hi,
Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with guest memory > 64G, till 64G it boots fine.
Note: xics(ic-mode=xics) guest with the same configuration boots fine
Tested with below current latest code(v6).
HW: Power9 DD 2.2
Qemu:
# git log -1
commit 34cc68411a5ada92df6ef968c32bad424911474c (HEAD -> xive-next, origin/xive-next)
Author: Cédric Le Goater <clg@kaod.org>
Date: Thu Apr 18 18:31:37 2019 +0200
spapr/irq: add KVM support to the 'dual' machine
Kernel Guest/Host: (Host kernel built with `ppc64le_defconfig`, Guest kernel built with `ppc64le_guest_defconfig`)
# git log -1
commit fac6994841aa8cfa5af02552f2eb9858fee9a25d (HEAD -> xive-5.1, origin/xive-5.1, origin/HEAD)
Author: Cédric Le Goater <clg@kaod.org>
Date: Thu Apr 18 08:46:33 2019 +0200
KVM: PPC: Book3S HV: XIVE: replace the 'destroy' method by a 'release' method
Qemu Commandline:
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 -name guest=vm2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-vm2/master-key.aes -machine pseries-4.0,accel=kvm,usb=off,dump-guest-core=off -m 66560 -realtime mlock=off -smp 56,sockets=1,cores=28,threads=2 -uuid 5510791f-f156-4f5a-8c3d-30cfa7a4c7a2 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-vm2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /home/sath/linux/vmlinux -append 'root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on' -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -drive file=/home/sath/tests/data/avocado-vt/images/jeos-27-ppc64le_vm2.qcow2,fo
rmat=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:57:58:59,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -M pseries,ic-mode=dual -msg timestamp=on
Guest Console:
Escape character is ^]
Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
00 0800 (D) : 1af4 1000 virtio [ net ]
00 1000 (D) : 1af4 1004 virtio [ scsi ]
Populating /pci@800000020000000/scsi@2
SCSI: Looking for devices
100000000000000 DISK : "QEMU QEMU HARDDISK 2.5+"
00 1800 (D) : 1b36 000d serial bus [ usb-xhci ]
00 2000 (D) : 1af4 1002 unknown-legacy-device*
No NVRAM common partition, re-initializing...
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@30000000
Detected RAM kernel at 400000 (17fe068 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
Detected machine type: 0000000000000101
command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
Max number of cores passed to firmware: 1024 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...
SLOF **********************************************************************
QEMU Starting
Build Date = Jan 14 2019 18:00:39
FW Version = git-a5b428e1c1eae703
Press "s" to enter Open Firmware.
Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
00 0800 (D) : 1af4 1000 virtio [ net ]
00 1000 (D) : 1af4 1004 virtio [ scsi ]
Populating /pci@800000020000000/scsi@2
SCSI: Looking for devices
100000000000000 DISK : "QEMU QEMU HARDDISK 2.5+"
00 1800 (D) : 1b36 000d serial bus [ usb-xhci ]
00 2000 (D) : 1af4 1002 unknown-legacy-device*
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@30000000
Detected RAM kernel at 400000 (17fe068 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
Detected machine type: 0000000000000101
command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
Max number of cores passed to firmware: 1024 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000001c10000
alloc_top : 0000000010000000
alloc_top_hi : 0000001040000000
rmo_top : 0000000010000000
ram_top : 0000001040000000
instantiating rtas at 0x000000000daf0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000001c20000 -> 0x0000000001c20af8
Device tree struct 0x0000000001c30000 -> 0x0000000001c40000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000000400000 ...
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000001040000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 56 cpus.
[ 0.000000] CPU maps initialized for 2 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1040000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000c06f8f5f91a7
[ 0.000000] possible = 0x0000fbffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x00000005455a445f
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x103fe17000-0x103fe1bfff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] count-cache-flush: full software flush sequence enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] random: get_random_u64 called from start_kernel+0xbc/0x648 with crng_init=0
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s169368 r0 d92776 u262144
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1063920
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
[ 0.000000] printk: log_buf_len individual max cpu contribution: 8192 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 450560 bytes
[ 0.000000] printk: log_buf_len min size: 262144 bytes
[ 0.000000] printk: log_buf_len: 1048576 bytes
[ 0.000000] printk: early log buf free: 257356(98%)
[ 0.000000] Memory: 68034688K/68157440K available (12800K kernel code, 1728K rwdata, 3264K rodata, 4224K init, 2488K bss, 122752K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=56, Nodes=1
[ 0.000000] ftrace: allocating 32078 entries in 12 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=56.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=56
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-df]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000536] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001486] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002374] Console: colour dummy device 80x25
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000001040000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 56 cpus.
[ 0.000000] CPU maps initialized for 2 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1040000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000c06f8f5f91a7
[ 0.000000] possible = 0x0000fbffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x00000005455a445f
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x103fe17000-0x103fe1bfff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] count-cache-flush: full software flush sequence enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] random: get_random_u64 called from start_kernel+0xbc/0x648 with crng_init=0
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s169368 r0 d92776 u262144
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1063920
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
[ 0.000000] printk: log_buf_len individual max cpu contribution: 8192 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 450560 bytes
[ 0.000000] printk: log_buf_len min size: 262144 bytes
[ 0.000000] printk: log_buf_len: 1048576 bytes
[ 0.000000] printk: early log buf free: 257356(98%)
[ 0.000000] Memory: 68034688K/68157440K available (12800K kernel code, 1728K rwdata, 3264K rodata, 4224K init, 2488K bss, 122752K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=56, Nodes=1
[ 0.000000] ftrace: allocating 32078 entries in 12 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=56.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=56
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-df]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000536] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001486] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002374] Console: colour dummy device 80x25
[ 0.041204] printk: console [tty0] enabled
[ 0.041652] pid_max: default: 57344 minimum: 448
[ 0.047640] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
[ 0.051167] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
[ 0.052040] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[ 0.052748] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[ 0.053855] *** VALIDATE proc ***
[ 0.054280] *** VALIDATE cgroup1 ***
[ 0.054612] *** VALIDATE cgroup2 ***
[ 0.055735] EEH: pSeries platform initialized
[ 0.056187] POWER9 performance monitor hardware support registered
[ 0.056912] rcu: Hierarchical SRCU implementation.
[ 0.058350] smp: Bringing up secondary CPUs ...
[ 0.594474] smp: Brought up 1 node, 56 CPUs
[ 0.595265] numa: Node 0 CPUs: 0-55
[ 0.595639] Using standard scheduler topology
[ 0.624525] devtmpfs: initialized
[ 0.644891] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.646019] futex hash table entries: 16384 (order: 5, 2097152 bytes)
[ 0.647574] NET: Registered protocol family 16
[ 0.665143] kworker/u112:0 (294) used greatest stack depth: 12608 bytes left
[ 0.694670] cpuidle: using governor menu
Linux ppc64le
#2 SMP Wed Apr 2[ 4.103966] kworker/u112:1 (324) used greatest stack depth: 12464 bytes left
[ 197.780442] PCI: Probing PCI hardware
[ 197.781468] PCI host bridge to bus 0000:00
[ 197.782323] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
[ 197.784122] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[ 197.786303] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[ 197.787822] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 197.789299] pci 0000:00:01.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.790809] pci 0000:00:01.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.792106] pci 0000:00:01.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.793659] pci 0000:00:01.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.795269] pci 0000:00:01.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.796674] pci 0000:00:01.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.798102] pci 0000:00:01.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.799461] pci 0000:00:01.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.801612] pci 0000:00:02.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.802395] pci 0000:00:02.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.803195] pci 0000:00:02.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.804167] pci 0000:00:02.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.805140] pci 0000:00:02.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.806018] pci 0000:00:02.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.806902] pci 0000:00:02.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.807746] pci 0000:00:02.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.809277] pci 0000:00:03.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.810068] pci 0000:00:03.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.810956] pci 0000:00:03.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.811944] pci 0000:00:03.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.812930] pci 0000:00:03.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.813827] pci 0000:00:03.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.814739] pci 0000:00:03.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.815599] pci 0000:00:03.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.816935] pci 0000:00:04.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.817725] pci 0000:00:04.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.818487] pci 0000:00:04.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.819407] pci 0000:00:04.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.820414] pci 0000:00:04.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.821322] pci 0000:00:04.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.822219] pci 0000:00:04.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.823088] pci 0000:00:04.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.826289] IOMMU table initialized, virtual merging enabled
[ 197.827014] pci 0000:00:01.0: Adding to iommu group 0
[ 197.827749] pci 0000:00:02.0: Adding to iommu group 0
[ 197.828451] pci 0000:00:03.0: Adding to iommu group 0
[ 197.829158] pci 0000:00:04.0: Adding to iommu group 0
[ 197.831587] EEH: No capable adapters found
[ 200.724067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 200.724986] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 202.781050] vgaarb: loaded
[ 202.782052] SCSI subsystem initialized
[ 204.738109] usbcore: registered new interface driver usbfs
[ 204.738794] usbcore: registered new interface driver hub
[ 204.739502] usbcore: registered new device driver usb
[ 204.740234] pps_core: LinuxPPS API ver. 1 registered
[ 204.741014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 204.742571] PTP clock support registered
[ 208.752064] EDAC MC: Ver: 3.0.0
---gets stuck here
>
> Thanks,
>
> C.
>
> Caveats :
>
> - We should introduce a set of definitions common to XIVE and XICS
> - The XICS-over-XIVE device file book3s_xive.c could be renamed to
> book3s_xics_on_xive.c or book3s_xics_p9.c
> - The XICS-over-XIVE device still has locking issues in the setup.
> - xc->valid is not useful
> - xc->xive is not very useful either
>
> Changes since v4:
>
> - add check on EQ page alignment
> - add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
> - add documentation in Documentation/virtual/kvm/api.txt
> - remove 'destroy' method
> - introduce a 'release' device operation called when device fd is
> closed.
> - introduce a 'xive_devices' array under the VM to store kvmppc_xive
> objects until VM is destroyed.
>
> Changes since v3:
>
> - removed a couple of useless includes
> - fix the test ont the initial setting of the EQ toggle bit : 0 -> 1
> - renamed qsize to qshift
> - renamed qpage to qaddr
> - checked host page size
> - limited flags to KVM_XIVE_EQ_ALWAYS_NOTIFY to fit sPAPR specs
> - Fixed xive_timaval description in documentation
>
> Changes since v2:
>
> - removed extra OPAL call definitions
> - removed ->q_order setting. Only useful in the XICS-on-XIVE KVM
> device which allocates the EQs on behalf of the guest.
> - returned -ENXIO when VP base is invalid
> - made use of the xive_vp() macro to compute VP identifiers
> - reworked locking in kvmppc_xive_native_connect_vcpu() to fix races
> - stop advertising KVM_CAP_PPC_IRQ_XIVE as support is not fully
> available yet
> - fixed comment on XIVE IRQ number space
> - removed usage of the __x_* macros
> - fixed locking on source block
> - fixed comments on the KVM device attribute definitions
> - handled MASKED EAS configuration
> - fixed check on supported EQ size to restrict to 64K pages
> - checked kvm_eq.flags that need to be zero
> - removed the OPAL call when EQ qtoggle bit and index are zero.
> - reduced the size of kvmppc_one_reg timaval attribute to two u64s
> - stopped returning of the OS CAM line value
>
> Changes since v1:
>
> - Better documentation (was missing)
> - Nested support. XIVE not advertised on non PowerNV platforms. This
> is a good way to test the fallback on QEMU emulated devices.
> - ESB and TIMA special mapping done using the KVM device fd
> - All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
> - Reworked of the KVM device ioctl controls to support hcalls and
> migration needs to capture/save states
> - Merged the control syncing XIVE and marking the EQ page dirty
> - Fixed passthrough support using the KVM device file address_space
> to clear the ESB pages from the mapping
> - Misc enhancements and fixes
>
> Cédric Le Goater (16):
> powerpc/xive: add OPAL extensions for the XIVE native exploitation
> support
> KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
> exploitation mode
> KVM: PPC: Book3S HV: XIVE: introduce a new capability
> KVM_CAP_PPC_IRQ_XIVE
> KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
> KVM: PPC: Book3S HV: XIVE: add a control to configure a source
> KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
> KVM: PPC: Book3S HV: XIVE: add a global reset control
> KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
> KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
> KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
> KVM: introduce a 'mmap' method for KVM devices
> KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
> KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
> KVM: PPC: Book3S HV: XIVE: add passthrough support
> KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
> KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
>
> arch/powerpc/include/asm/kvm_host.h | 3 +
> arch/powerpc/include/asm/kvm_ppc.h | 32 +
> arch/powerpc/include/asm/opal-api.h | 7 +-
> arch/powerpc/include/asm/opal.h | 7 +
> arch/powerpc/include/asm/xive.h | 17 +
> arch/powerpc/include/uapi/asm/kvm.h | 46 +
> arch/powerpc/kvm/book3s_xive.h | 37 +
> include/linux/kvm_host.h | 2 +
> include/uapi/linux/kvm.h | 3 +
> arch/powerpc/kvm/book3s.c | 31 +-
> arch/powerpc/kvm/book3s_xive.c | 230 +++-
> arch/powerpc/kvm/book3s_xive_native.c | 1243 ++++++++++++++++++++
> arch/powerpc/kvm/powerpc.c | 37 +
> arch/powerpc/platforms/powernv/opal-call.c | 3 +
> arch/powerpc/sysdev/xive/native.c | 110 ++
> virt/kvm/kvm_main.c | 24 +
> Documentation/virtual/kvm/api.txt | 10 +
> Documentation/virtual/kvm/devices/xive.txt | 197 ++++
> arch/powerpc/kvm/Makefile | 2 +-
> 19 files changed, 1980 insertions(+), 61 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-04-29 8:05 ` Satheesh Rajendran
0 siblings, 0 replies; 81+ messages in thread
From: Satheesh Rajendran @ 2019-04-29 8:05 UTC (permalink / raw)
To: Cédric Le Goater
Cc: linuxppc-dev, Paul Mackerras, kvm, kvm-ppc, David Gibson
On Wed, Apr 10, 2019 at 07:04:32PM +0200, Cédric Le Goater wrote:
> Hello,
>
> GitHub trees available here :
>
> QEMU sPAPR:
>
> https://github.com/legoater/qemu/commits/xive-next
>
> Linux/KVM:
>
> https://github.com/legoater/linux/commits/xive-5.1
Hi,
Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with guest memory > 64G, till 64G it boots fine.
Note: xics(ic-mode=xics) guest with the same configuration boots fine
Tested with below current latest code(v6).
HW: Power9 DD 2.2
Qemu:
# git log -1
commit 34cc68411a5ada92df6ef968c32bad424911474c (HEAD -> xive-next, origin/xive-next)
Author: Cédric Le Goater <clg@kaod.org>
Date: Thu Apr 18 18:31:37 2019 +0200
spapr/irq: add KVM support to the 'dual' machine
Kernel Guest/Host: (Host kernel built with `ppc64le_defconfig`, Guest kernel built with `ppc64le_guest_defconfig`)
# git log -1
commit fac6994841aa8cfa5af02552f2eb9858fee9a25d (HEAD -> xive-5.1, origin/xive-5.1, origin/HEAD)
Author: Cédric Le Goater <clg@kaod.org>
Date: Thu Apr 18 08:46:33 2019 +0200
KVM: PPC: Book3S HV: XIVE: replace the 'destroy' method by a 'release' method
Qemu Commandline:
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 -name guest=vm2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-vm2/master-key.aes -machine pseries-4.0,accel=kvm,usb=off,dump-guest-core=off -m 66560 -realtime mlock=off -smp 56,sockets=1,cores=28,threads=2 -uuid 5510791f-f156-4f5a-8c3d-30cfa7a4c7a2 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-vm2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /home/sath/linux/vmlinux -append 'root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on' -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -drive file=/home/sath/tests/data/avocado-vt/images/jeos-27-ppc64le_vm2.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:57:58:59,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -M pseries,ic-mode=dual -msg timestamp=on
Guest Console:
Escape character is ^]
Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
00 0800 (D) : 1af4 1000 virtio [ net ]
00 1000 (D) : 1af4 1004 virtio [ scsi ]
Populating /pci@800000020000000/scsi@2
SCSI: Looking for devices
100000000000000 DISK : "QEMU QEMU HARDDISK 2.5+"
00 1800 (D) : 1b36 000d serial bus [ usb-xhci ]
00 2000 (D) : 1af4 1002 unknown-legacy-device*
No NVRAM common partition, re-initializing...
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@30000000
Detected RAM kernel at 400000 (17fe068 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
Detected machine type: 0000000000000101
command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
Max number of cores passed to firmware: 1024 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...
SLOF **********************************************************************
QEMU Starting
Build Date = Jan 14 2019 18:00:39
FW Version = git-a5b428e1c1eae703
Press "s" to enter Open Firmware.
Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
00 0800 (D) : 1af4 1000 virtio [ net ]
00 1000 (D) : 1af4 1004 virtio [ scsi ]
Populating /pci@800000020000000/scsi@2
SCSI: Looking for devices
100000000000000 DISK : "QEMU QEMU HARDDISK 2.5+"
00 1800 (D) : 1b36 000d serial bus [ usb-xhci ]
00 2000 (D) : 1af4 1002 unknown-legacy-device*
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@30000000
Detected RAM kernel at 400000 (17fe068 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
Detected machine type: 0000000000000101
command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
Max number of cores passed to firmware: 1024 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000001c10000
alloc_top : 0000000010000000
alloc_top_hi : 0000001040000000
rmo_top : 0000000010000000
ram_top : 0000001040000000
instantiating rtas at 0x000000000daf0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000001c20000 -> 0x0000000001c20af8
Device tree struct 0x0000000001c30000 -> 0x0000000001c40000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000000400000 ...
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000001040000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 56 cpus.
[ 0.000000] CPU maps initialized for 2 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1040000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000c06f8f5f91a7
[ 0.000000] possible = 0x0000fbffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x00000005455a445f
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x103fe17000-0x103fe1bfff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] count-cache-flush: full software flush sequence enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] random: get_random_u64 called from start_kernel+0xbc/0x648 with crng_init=0
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s169368 r0 d92776 u262144
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1063920
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
[ 0.000000] printk: log_buf_len individual max cpu contribution: 8192 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 450560 bytes
[ 0.000000] printk: log_buf_len min size: 262144 bytes
[ 0.000000] printk: log_buf_len: 1048576 bytes
[ 0.000000] printk: early log buf free: 257356(98%)
[ 0.000000] Memory: 68034688K/68157440K available (12800K kernel code, 1728K rwdata, 3264K rodata, 4224K init, 2488K bss, 122752K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=56, Nodes=1
[ 0.000000] ftrace: allocating 32078 entries in 12 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=56.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=56
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-df]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000536] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001486] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002374] Console: colour dummy device 80x25
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000001040000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 56 cpus.
[ 0.000000] CPU maps initialized for 2 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1040000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000c06f8f5f91a7
[ 0.000000] possible = 0x0000fbffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x00000005455a445f
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x103fe17000-0x103fe1bfff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] count-cache-flush: full software flush sequence enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] random: get_random_u64 called from start_kernel+0xbc/0x648 with crng_init=0
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s169368 r0 d92776 u262144
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1063920
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
[ 0.000000] printk: log_buf_len individual max cpu contribution: 8192 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 450560 bytes
[ 0.000000] printk: log_buf_len min size: 262144 bytes
[ 0.000000] printk: log_buf_len: 1048576 bytes
[ 0.000000] printk: early log buf free: 257356(98%)
[ 0.000000] Memory: 68034688K/68157440K available (12800K kernel code, 1728K rwdata, 3264K rodata, 4224K init, 2488K bss, 122752K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign=128, Order=0-3, MinObjects=0, CPUs=56, Nodes=1
[ 0.000000] ftrace: allocating 32078 entries in 12 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=2048 to nr_cpu_ids=56.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=56
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-df]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000536] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001486] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002374] Console: colour dummy device 80x25
[ 0.041204] printk: console [tty0] enabled
[ 0.041652] pid_max: default: 57344 minimum: 448
[ 0.047640] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
[ 0.051167] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
[ 0.052040] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[ 0.052748] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[ 0.053855] *** VALIDATE proc ***
[ 0.054280] *** VALIDATE cgroup1 ***
[ 0.054612] *** VALIDATE cgroup2 ***
[ 0.055735] EEH: pSeries platform initialized
[ 0.056187] POWER9 performance monitor hardware support registered
[ 0.056912] rcu: Hierarchical SRCU implementation.
[ 0.058350] smp: Bringing up secondary CPUs ...
[ 0.594474] smp: Brought up 1 node, 56 CPUs
[ 0.595265] numa: Node 0 CPUs: 0-55
[ 0.595639] Using standard scheduler topology
[ 0.624525] devtmpfs: initialized
[ 0.644891] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.646019] futex hash table entries: 16384 (order: 5, 2097152 bytes)
[ 0.647574] NET: Registered protocol family 16
[ 0.665143] kworker/u112:0 (294) used greatest stack depth: 12608 bytes left
[ 0.694670] cpuidle: using governor menu
Linux ppc64le
#2 SMP Wed Apr 2[ 4.103966] kworker/u112:1 (324) used greatest stack depth: 12464 bytes left
[ 197.780442] PCI: Probing PCI hardware
[ 197.781468] PCI host bridge to bus 0000:00
[ 197.782323] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
[ 197.784122] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[ 197.786303] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[ 197.787822] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 197.789299] pci 0000:00:01.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.790809] pci 0000:00:01.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.792106] pci 0000:00:01.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.793659] pci 0000:00:01.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.795269] pci 0000:00:01.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.796674] pci 0000:00:01.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.798102] pci 0000:00:01.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.799461] pci 0000:00:01.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.801612] pci 0000:00:02.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.802395] pci 0000:00:02.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.803195] pci 0000:00:02.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.804167] pci 0000:00:02.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.805140] pci 0000:00:02.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.806018] pci 0000:00:02.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.806902] pci 0000:00:02.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.807746] pci 0000:00:02.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.809277] pci 0000:00:03.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.810068] pci 0000:00:03.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.810956] pci 0000:00:03.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.811944] pci 0000:00:03.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.812930] pci 0000:00:03.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.813827] pci 0000:00:03.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.814739] pci 0000:00:03.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.815599] pci 0000:00:03.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.816935] pci 0000:00:04.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.817725] pci 0000:00:04.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.818487] pci 0000:00:04.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.819407] pci 0000:00:04.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.820414] pci 0000:00:04.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.821322] pci 0000:00:04.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.822219] pci 0000:00:04.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.823088] pci 0000:00:04.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.826289] IOMMU table initialized, virtual merging enabled
[ 197.827014] pci 0000:00:01.0: Adding to iommu group 0
[ 197.827749] pci 0000:00:02.0: Adding to iommu group 0
[ 197.828451] pci 0000:00:03.0: Adding to iommu group 0
[ 197.829158] pci 0000:00:04.0: Adding to iommu group 0
[ 197.831587] EEH: No capable adapters found
[ 200.724067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 200.724986] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 202.781050] vgaarb: loaded
[ 202.782052] SCSI subsystem initialized
[ 204.738109] usbcore: registered new interface driver usbfs
[ 204.738794] usbcore: registered new interface driver hub
[ 204.739502] usbcore: registered new device driver usb
[ 204.740234] pps_core: LinuxPPS API ver. 1 registered
[ 204.741014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 204.742571] PTP clock support registered
[ 208.752064] EDAC MC: Ver: 3.0.0
---gets stuck here
>
> Thanks,
>
> C.
>
> Caveats :
>
> - We should introduce a set of definitions common to XIVE and XICS
> - The XICS-over-XIVE device file book3s_xive.c could be renamed to
> book3s_xics_on_xive.c or book3s_xics_p9.c
> - The XICS-over-XIVE device still has locking issues in the setup.
> - xc->valid is not useful
> - xc->xive is not very useful either
>
> Changes since v4:
>
> - add check on EQ page alignment
> - add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
> - add documentation in Documentation/virtual/kvm/api.txt
> - remove 'destroy' method
> - introduce a 'release' device operation called when device fd is
> closed.
> - introduce a 'xive_devices' array under the VM to store kvmppc_xive
> objects until VM is destroyed.
>
> Changes since v3:
>
> - removed a couple of useless includes
> - fix the test ont the initial setting of the EQ toggle bit : 0 -> 1
> - renamed qsize to qshift
> - renamed qpage to qaddr
> - checked host page size
> - limited flags to KVM_XIVE_EQ_ALWAYS_NOTIFY to fit sPAPR specs
> - Fixed xive_timaval description in documentation
>
> Changes since v2:
>
> - removed extra OPAL call definitions
> - removed ->q_order setting. Only useful in the XICS-on-XIVE KVM
> device which allocates the EQs on behalf of the guest.
> - returned -ENXIO when VP base is invalid
> - made use of the xive_vp() macro to compute VP identifiers
> - reworked locking in kvmppc_xive_native_connect_vcpu() to fix races
> - stop advertising KVM_CAP_PPC_IRQ_XIVE as support is not fully
> available yet
> - fixed comment on XIVE IRQ number space
> - removed usage of the __x_* macros
> - fixed locking on source block
> - fixed comments on the KVM device attribute definitions
> - handled MASKED EAS configuration
> - fixed check on supported EQ size to restrict to 64K pages
> - checked kvm_eq.flags that need to be zero
> - removed the OPAL call when EQ qtoggle bit and index are zero.
> - reduced the size of kvmppc_one_reg timaval attribute to two u64s
> - stopped returning of the OS CAM line value
>
> Changes since v1:
>
> - Better documentation (was missing)
> - Nested support. XIVE not advertised on non PowerNV platforms. This
> is a good way to test the fallback on QEMU emulated devices.
> - ESB and TIMA special mapping done using the KVM device fd
> - All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
> - Reworked of the KVM device ioctl controls to support hcalls and
> migration needs to capture/save states
> - Merged the control syncing XIVE and marking the EQ page dirty
> - Fixed passthrough support using the KVM device file address_space
> to clear the ESB pages from the mapping
> - Misc enhancements and fixes
>
> Cédric Le Goater (16):
> powerpc/xive: add OPAL extensions for the XIVE native exploitation
> support
> KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
> exploitation mode
> KVM: PPC: Book3S HV: XIVE: introduce a new capability
> KVM_CAP_PPC_IRQ_XIVE
> KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
> KVM: PPC: Book3S HV: XIVE: add a control to configure a source
> KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
> KVM: PPC: Book3S HV: XIVE: add a global reset control
> KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
> KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
> KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
> KVM: introduce a 'mmap' method for KVM devices
> KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
> KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
> KVM: PPC: Book3S HV: XIVE: add passthrough support
> KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
> KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
>
> arch/powerpc/include/asm/kvm_host.h | 3 +
> arch/powerpc/include/asm/kvm_ppc.h | 32 +
> arch/powerpc/include/asm/opal-api.h | 7 +-
> arch/powerpc/include/asm/opal.h | 7 +
> arch/powerpc/include/asm/xive.h | 17 +
> arch/powerpc/include/uapi/asm/kvm.h | 46 +
> arch/powerpc/kvm/book3s_xive.h | 37 +
> include/linux/kvm_host.h | 2 +
> include/uapi/linux/kvm.h | 3 +
> arch/powerpc/kvm/book3s.c | 31 +-
> arch/powerpc/kvm/book3s_xive.c | 230 +++-
> arch/powerpc/kvm/book3s_xive_native.c | 1243 ++++++++++++++++++++
> arch/powerpc/kvm/powerpc.c | 37 +
> arch/powerpc/platforms/powernv/opal-call.c | 3 +
> arch/powerpc/sysdev/xive/native.c | 110 ++
> virt/kvm/kvm_main.c | 24 +
> Documentation/virtual/kvm/api.txt | 10 +
> Documentation/virtual/kvm/devices/xive.txt | 197 ++++
> arch/powerpc/kvm/Makefile | 2 +-
> 19 files changed, 1980 insertions(+), 61 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-04-29 8:05 ` Satheesh Rajendran
0 siblings, 0 replies; 81+ messages in thread
From: Satheesh Rajendran @ 2019-04-29 8:17 UTC (permalink / raw)
To: Cédric Le Goater
Cc: kvm-ppc, kvm, Paul Mackerras, linuxppc-dev, David Gibson
On Wed, Apr 10, 2019 at 07:04:32PM +0200, Cédric Le Goater wrote:
> Hello,
>
> GitHub trees available here :
>
> QEMU sPAPR:
>
> https://github.com/legoater/qemu/commits/xive-next
>
> Linux/KVM:
>
> https://github.com/legoater/linux/commits/xive-5.1
Hi,
Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with guest memory > 64G, till 64G it boots fine.
Note: xics(ic-mode=xics) guest with the same configuration boots fine
Tested with below current latest code(v6).
HW: Power9 DD 2.2
Qemu:
# git log -1
commit 34cc68411a5ada92df6ef968c32bad424911474c (HEAD -> xive-next, origin/xive-next)
Author: Cédric Le Goater <clg@kaod.org>
Date: Thu Apr 18 18:31:37 2019 +0200
spapr/irq: add KVM support to the 'dual' machine
Kernel Guest/Host: (Host kernel built with `ppc64le_defconfig`, Guest kernel built with `ppc64le_guest_defconfig`)
# git log -1
commit fac6994841aa8cfa5af02552f2eb9858fee9a25d (HEAD -> xive-5.1, origin/xive-5.1, origin/HEAD)
Author: Cédric Le Goater <clg@kaod.org>
Date: Thu Apr 18 08:46:33 2019 +0200
KVM: PPC: Book3S HV: XIVE: replace the 'destroy' method by a 'release' method
Qemu Commandline:
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=none /home/sath/qemu/ppc64-softmmu/qemu-system-ppc64 -name guest=vm2,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-13-vm2/master-key.aes -machine pseries-4.0,accel=kvm,usb=off,dump-guest-core=off -m 66560 -realtime mlock=off -smp 56,sockets=1,cores(,threads=2 -uuid 5510791f-f156-4f5a-8c3d-30cfa7a4c7a2 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-13-vm2/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /home/sath/linux/vmlinux -append 'root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on' -device qemu-xhci,id=usb,bus=pci.0,addr=0x3 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x2 -drive file=/home/sath/tests/data/avocado-vt/images/jeos-27-ppc64le_vm2.qcow2,format=qcow2,if=none,id=drive-scsi0-0-0-0 -device scsi-hd,bus=scsi0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0-0-0-0,id=scsi0-0-0-0,bootindex=1 -netdev tap,fd%,id=hostnet0,vhost=on,vhostfd' -device virtio-net-pci,netdev=hostnet0,id=net0,macR:54:00:57:58:59,bus=pci.0,addr=0x1 -chardev pty,id=charserial0 -device spapr-vty,chardev=charserial0,id=serial0,reg=0x30000000 -device virtio-balloon-pci,idºlloon0,bus=pci.0,addr=0x4 -M pseries,ic-mode=dual -msg timestamp=on
Guest Console:
Escape character is ^]
Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
00 0800 (D) : 1af4 1000 virtio [ net ]
00 1000 (D) : 1af4 1004 virtio [ scsi ]
Populating /pci@800000020000000/scsi@2
SCSI: Looking for devices
100000000000000 DISK : "QEMU QEMU HARDDISK 2.5+"
00 1800 (D) : 1b36 000d serial bus [ usb-xhci ]
00 2000 (D) : 1af4 1002 unknown-legacy-device*
No NVRAM common partition, re-initializing...
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@30000000
Detected RAM kernel at 400000 (17fe068 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
Detected machine type: 0000000000000101
command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
Max number of cores passed to firmware: 1024 (NR_CPUS = 2048)
Calling ibm,client-architecture-support...
SLOF **********************************************************************
QEMU Starting
Build Date = Jan 14 2019 18:00:39
FW Version = git-a5b428e1c1eae703
Press "s" to enter Open Firmware.
Populating /vdevice methods
Populating /vdevice/vty@30000000
Populating /vdevice/nvram@71000000
Populating /pci@800000020000000
00 0800 (D) : 1af4 1000 virtio [ net ]
00 1000 (D) : 1af4 1004 virtio [ scsi ]
Populating /pci@800000020000000/scsi@2
SCSI: Looking for devices
100000000000000 DISK : "QEMU QEMU HARDDISK 2.5+"
00 1800 (D) : 1b36 000d serial bus [ usb-xhci ]
00 2000 (D) : 1af4 1002 unknown-legacy-device*
Scanning USB
XHCI: Initializing
Using default console: /vdevice/vty@30000000
Detected RAM kernel at 400000 (17fe068 bytes)
Welcome to Open Firmware
Copyright (c) 2004, 2017 IBM Corporation All rights reserved.
This program and the accompanying materials are made available
under the terms of the BSD License available at
http://www.opensource.org/licenses/bsd-license.php
Booting from memory...
OF stdout device is: /vdevice/vty@30000000
Preparing to boot Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
Detected machine type: 0000000000000101
command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
Max number of cores passed to firmware: 1024 (NR_CPUS = 2048)
Calling ibm,client-architecture-support... done
memory layout at init:
memory_limit : 0000000000000000 (16 MB aligned)
alloc_bottom : 0000000001c10000
alloc_top : 0000000010000000
alloc_top_hi : 0000001040000000
rmo_top : 0000000010000000
ram_top : 0000001040000000
instantiating rtas at 0x000000000daf0000... done
prom_hold_cpus: skipped
copying OF device tree...
Building dt strings...
Building dt structure...
Device tree strings 0x0000000001c20000 -> 0x0000000001c20af8
Device tree struct 0x0000000001c30000 -> 0x0000000001c40000
Quiescing Open Firmware ...
Booting Linux via __start() @ 0x0000000000400000 ...
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000001040000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 56 cpus.
[ 0.000000] CPU maps initialized for 2 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1040000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000c06f8f5f91a7
[ 0.000000] possible = 0x0000fbffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x00000005455a445f
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x103fe17000-0x103fe1bfff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] count-cache-flush: full software flush sequence enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] random: get_random_u64 called from start_kernel+0xbc/0x648 with crng_init=0
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s169368 r0 d92776 u262144
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1063920
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
[ 0.000000] printk: log_buf_len individual max cpu contribution: 8192 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 450560 bytes
[ 0.000000] printk: log_buf_len min size: 262144 bytes
[ 0.000000] printk: log_buf_len: 1048576 bytes
[ 0.000000] printk: early log buf free: 257356(98%)
[ 0.000000] Memory: 68034688K/68157440K available (12800K kernel code, 1728K rwdata, 3264K rodata, 4224K init, 2488K bss, 122752K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign\x128, Order=0-3, MinObjects=0, CPUsV, Nodes=1
[ 0.000000] ftrace: allocating 32078 entries in 12 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS 48 to nr_cpu_idsV.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf\x16, nr_cpu_idsV
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-df]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000536] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001486] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002374] Console: colour dummy device 80x25
[ 0.000000] radix-mmu: Page sizes from device-tree:
[ 0.000000] radix-mmu: Page size shift = 12 AP=0x0
[ 0.000000] radix-mmu: Page size shift = 16 AP=0x5
[ 0.000000] radix-mmu: Page size shift = 21 AP=0x1
[ 0.000000] radix-mmu: Page size shift = 30 AP=0x2
[ 0.000000] lpar: Using radix MMU under hypervisor
[ 0.000000] radix-mmu: Mapped 0x0000000000000000-0x0000000040000000 with 1.00 GiB pages (exec)
[ 0.000000] radix-mmu: Mapped 0x0000000040000000-0x0000001040000000 with 1.00 GiB pages
[ 0.000000] radix-mmu: Process table (____ptrval____) and radix root for kernel: (____ptrval____)
[ 0.000000] Linux version 5.1.0-rc5-176614-gfac6994841aa (root@kvmupstream) (gcc version 7.4.0 (Ubuntu 7.4.0-1ubuntu1~18.04)) #2 SMP Wed Apr 24 07:58:04 EDT 2019
[ 0.000000] Using pSeries machine description
[ 0.000000] printk: bootconsole [udbg0] enabled
[ 0.000000] Partition configured for 56 cpus.
[ 0.000000] CPU maps initialized for 2 threads per core
[ 0.000000] -----------------------------------------------------
[ 0.000000] ppc64_pft_size = 0x0
[ 0.000000] phys_mem_size = 0x1040000000
[ 0.000000] dcache_bsize = 0x80
[ 0.000000] icache_bsize = 0x80
[ 0.000000] cpu_features = 0x0000c06f8f5f91a7
[ 0.000000] possible = 0x0000fbffcf5fb1a7
[ 0.000000] always = 0x00000003800081a1
[ 0.000000] cpu_user_features = 0xdc0065c2 0xaee00000
[ 0.000000] mmu_features = 0x3c006041
[ 0.000000] firmware_features = 0x00000005455a445f
[ 0.000000] -----------------------------------------------------
[ 0.000000] numa: NODE_DATA [mem 0x103fe17000-0x103fe1bfff]
[ 0.000000] rfi-flush: fallback displacement flush available
[ 0.000000] rfi-flush: ori type flush available
[ 0.000000] rfi-flush: mttrig type flush available
[ 0.000000] count-cache-flush: full software flush sequence enabled.
[ 0.000000] stf-barrier: eieio barrier available
[ 0.000000] PCI host bridge /pci@800000020000000 ranges:
[ 0.000000] IO 0x0000200000000000..0x000020000000ffff -> 0x0000000000000000
[ 0.000000] MEM 0x0000200080000000..0x00002000ffffffff -> 0x0000000080000000
[ 0.000000] MEM 0x0000210000000000..0x000021ffffffffff -> 0x0000210000000000
[ 0.000000] PPC64 nvram contains 65536 bytes
[ 0.000000] barrier-nospec: using ORI speculation barrier
[ 0.000000] Zone ranges:
[ 0.000000] Normal [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Movable zone start for each node
[ 0.000000] Early memory node ranges
[ 0.000000] node 0: [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x000000103fffffff]
[ 0.000000] random: get_random_u64 called from start_kernel+0xbc/0x648 with crng_init=0
[ 0.000000] percpu: Embedded 4 pages/cpu @(____ptrval____) s169368 r0 d92776 u262144
[ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1063920
[ 0.000000] Policy zone: Normal
[ 0.000000] Kernel command line: root=/dev/sda2 rw console=tty0 console=ttyS0,115200 init=/sbin/init initcall_debug selinux=0 secure=on
[ 0.000000] printk: log_buf_len individual max cpu contribution: 8192 bytes
[ 0.000000] printk: log_buf_len total cpu_extra contributions: 450560 bytes
[ 0.000000] printk: log_buf_len min size: 262144 bytes
[ 0.000000] printk: log_buf_len: 1048576 bytes
[ 0.000000] printk: early log buf free: 257356(98%)
[ 0.000000] Memory: 68034688K/68157440K available (12800K kernel code, 1728K rwdata, 3264K rodata, 4224K init, 2488K bss, 122752K reserved, 0K cma-reserved)
[ 0.000000] SLUB: HWalign\x128, Order=0-3, MinObjects=0, CPUsV, Nodes=1
[ 0.000000] ftrace: allocating 32078 entries in 12 pages
[ 0.000000] rcu: Hierarchical RCU implementation.
[ 0.000000] rcu: RCU event tracing is enabled.
[ 0.000000] rcu: RCU restricting CPUs from NR_CPUS 48 to nr_cpu_idsV.
[ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf\x16, nr_cpu_idsV
[ 0.000000] NR_IRQS: 512, nr_irqs: 512, preallocated irqs: 16
[ 0.000000] xive: Using IRQ range [0-df]
[ 0.000000] xive: Interrupt handling initialized with spapr backend
[ 0.000000] xive: Using priority 6 for all interrupts
[ 0.000000] xive: Using 64kB queues
[ 0.000002] time_init: 56 bit decrementer (max: 7fffffffffffff)
[ 0.000536] clocksource: timebase: mask: 0xffffffffffffffff max_cycles: 0x761537d007, max_idle_ns: 440795202126 ns
[ 0.001486] clocksource: timebase mult[1f40000] shift[24] registered
[ 0.002374] Console: colour dummy device 80x25
[ 0.041204] printk: console [tty0] enabled
[ 0.041652] pid_max: default: 57344 minimum: 448
[ 0.047640] Dentry cache hash table entries: 8388608 (order: 10, 67108864 bytes)
[ 0.051167] Inode-cache hash table entries: 4194304 (order: 9, 33554432 bytes)
[ 0.052040] Mount-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[ 0.052748] Mountpoint-cache hash table entries: 131072 (order: 4, 1048576 bytes)
[ 0.053855] *** VALIDATE proc ***
[ 0.054280] *** VALIDATE cgroup1 ***
[ 0.054612] *** VALIDATE cgroup2 ***
[ 0.055735] EEH: pSeries platform initialized
[ 0.056187] POWER9 performance monitor hardware support registered
[ 0.056912] rcu: Hierarchical SRCU implementation.
[ 0.058350] smp: Bringing up secondary CPUs ...
[ 0.594474] smp: Brought up 1 node, 56 CPUs
[ 0.595265] numa: Node 0 CPUs: 0-55
[ 0.595639] Using standard scheduler topology
[ 0.624525] devtmpfs: initialized
[ 0.644891] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[ 0.646019] futex hash table entries: 16384 (order: 5, 2097152 bytes)
[ 0.647574] NET: Registered protocol family 16
[ 0.665143] kworker/u112:0 (294) used greatest stack depth: 12608 bytes left
[ 0.694670] cpuidle: using governor menu
Linux ppc64le
#2 SMP Wed Apr 2[ 4.103966] kworker/u112:1 (324) used greatest stack depth: 12464 bytes left
[ 197.780442] PCI: Probing PCI hardware
[ 197.781468] PCI host bridge to bus 0000:00
[ 197.782323] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff])
[ 197.784122] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[ 197.786303] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[ 197.787822] pci_bus 0000:00: root bus resource [bus 00-ff]
[ 197.789299] pci 0000:00:01.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.790809] pci 0000:00:01.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.792106] pci 0000:00:01.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.793659] pci 0000:00:01.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.795269] pci 0000:00:01.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.796674] pci 0000:00:01.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.798102] pci 0000:00:01.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.799461] pci 0000:00:01.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.801612] pci 0000:00:02.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.802395] pci 0000:00:02.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.803195] pci 0000:00:02.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.804167] pci 0000:00:02.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.805140] pci 0000:00:02.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.806018] pci 0000:00:02.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.806902] pci 0000:00:02.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.807746] pci 0000:00:02.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.809277] pci 0000:00:03.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.810068] pci 0000:00:03.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.810956] pci 0000:00:03.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.811944] pci 0000:00:03.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.812930] pci 0000:00:03.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.813827] pci 0000:00:03.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.814739] pci 0000:00:03.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.815599] pci 0000:00:03.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.816935] pci 0000:00:04.0: calling pci_dev_pdn_setup+0x0/0x50 @ 1
[ 197.817725] pci 0000:00:04.0: pci_dev_pdn_setup+0x0/0x50 took 0 usecs
[ 197.818487] pci 0000:00:04.0: calling pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 @ 1
[ 197.819407] pci 0000:00:04.0: pnv_npu2_opencapi_cfg_size_fixup+0x0/0x50 took 0 usecs
[ 197.820414] pci 0000:00:04.0: calling pcibios_fixup_resources+0x0/0x1b0 @ 1
[ 197.821322] pci 0000:00:04.0: pcibios_fixup_resources+0x0/0x1b0 took 0 usecs
[ 197.822219] pci 0000:00:04.0: calling pnv_ocxl_fixup_actag+0x0/0x2a0 @ 1
[ 197.823088] pci 0000:00:04.0: pnv_ocxl_fixup_actag+0x0/0x2a0 took 0 usecs
[ 197.826289] IOMMU table initialized, virtual merging enabled
[ 197.827014] pci 0000:00:01.0: Adding to iommu group 0
[ 197.827749] pci 0000:00:02.0: Adding to iommu group 0
[ 197.828451] pci 0000:00:03.0: Adding to iommu group 0
[ 197.829158] pci 0000:00:04.0: Adding to iommu group 0
[ 197.831587] EEH: No capable adapters found
[ 200.724067] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[ 200.724986] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[ 202.781050] vgaarb: loaded
[ 202.782052] SCSI subsystem initialized
[ 204.738109] usbcore: registered new interface driver usbfs
[ 204.738794] usbcore: registered new interface driver hub
[ 204.739502] usbcore: registered new device driver usb
[ 204.740234] pps_core: LinuxPPS API ver. 1 registered
[ 204.741014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[ 204.742571] PTP clock support registered
[ 208.752064] EDAC MC: Ver: 3.0.0
---gets stuck here
>
> Thanks,
>
> C.
>
> Caveats :
>
> - We should introduce a set of definitions common to XIVE and XICS
> - The XICS-over-XIVE device file book3s_xive.c could be renamed to
> book3s_xics_on_xive.c or book3s_xics_p9.c
> - The XICS-over-XIVE device still has locking issues in the setup.
> - xc->valid is not useful
> - xc->xive is not very useful either
>
> Changes since v4:
>
> - add check on EQ page alignment
> - add requirement on KVM_XIVE_EQ_ALWAYS_NOTIFY
> - add documentation in Documentation/virtual/kvm/api.txt
> - remove 'destroy' method
> - introduce a 'release' device operation called when device fd is
> closed.
> - introduce a 'xive_devices' array under the VM to store kvmppc_xive
> objects until VM is destroyed.
>
> Changes since v3:
>
> - removed a couple of useless includes
> - fix the test ont the initial setting of the EQ toggle bit : 0 -> 1
> - renamed qsize to qshift
> - renamed qpage to qaddr
> - checked host page size
> - limited flags to KVM_XIVE_EQ_ALWAYS_NOTIFY to fit sPAPR specs
> - Fixed xive_timaval description in documentation
>
> Changes since v2:
>
> - removed extra OPAL call definitions
> - removed ->q_order setting. Only useful in the XICS-on-XIVE KVM
> device which allocates the EQs on behalf of the guest.
> - returned -ENXIO when VP base is invalid
> - made use of the xive_vp() macro to compute VP identifiers
> - reworked locking in kvmppc_xive_native_connect_vcpu() to fix races
> - stop advertising KVM_CAP_PPC_IRQ_XIVE as support is not fully
> available yet
> - fixed comment on XIVE IRQ number space
> - removed usage of the __x_* macros
> - fixed locking on source block
> - fixed comments on the KVM device attribute definitions
> - handled MASKED EAS configuration
> - fixed check on supported EQ size to restrict to 64K pages
> - checked kvm_eq.flags that need to be zero
> - removed the OPAL call when EQ qtoggle bit and index are zero.
> - reduced the size of kvmppc_one_reg timaval attribute to two u64s
> - stopped returning of the OS CAM line value
>
> Changes since v1:
>
> - Better documentation (was missing)
> - Nested support. XIVE not advertised on non PowerNV platforms. This
> is a good way to test the fallback on QEMU emulated devices.
> - ESB and TIMA special mapping done using the KVM device fd
> - All hcalls moved to QEMU. Dropped the patch moving the hcall flags.
> - Reworked of the KVM device ioctl controls to support hcalls and
> migration needs to capture/save states
> - Merged the control syncing XIVE and marking the EQ page dirty
> - Fixed passthrough support using the KVM device file address_space
> to clear the ESB pages from the mapping
> - Misc enhancements and fixes
>
> Cédric Le Goater (16):
> powerpc/xive: add OPAL extensions for the XIVE native exploitation
> support
> KVM: PPC: Book3S HV: add a new KVM device for the XIVE native
> exploitation mode
> KVM: PPC: Book3S HV: XIVE: introduce a new capability
> KVM_CAP_PPC_IRQ_XIVE
> KVM: PPC: Book3S HV: XIVE: add a control to initialize a source
> KVM: PPC: Book3S HV: XIVE: add a control to configure a source
> KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration
> KVM: PPC: Book3S HV: XIVE: add a global reset control
> KVM: PPC: Book3S HV: XIVE: add a control to sync the sources
> KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages
> KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state
> KVM: introduce a 'mmap' method for KVM devices
> KVM: PPC: Book3S HV: XIVE: add a TIMA mapping
> KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages
> KVM: PPC: Book3S HV: XIVE: add passthrough support
> KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode
> KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation
>
> arch/powerpc/include/asm/kvm_host.h | 3 +
> arch/powerpc/include/asm/kvm_ppc.h | 32 +
> arch/powerpc/include/asm/opal-api.h | 7 +-
> arch/powerpc/include/asm/opal.h | 7 +
> arch/powerpc/include/asm/xive.h | 17 +
> arch/powerpc/include/uapi/asm/kvm.h | 46 +
> arch/powerpc/kvm/book3s_xive.h | 37 +
> include/linux/kvm_host.h | 2 +
> include/uapi/linux/kvm.h | 3 +
> arch/powerpc/kvm/book3s.c | 31 +-
> arch/powerpc/kvm/book3s_xive.c | 230 +++-
> arch/powerpc/kvm/book3s_xive_native.c | 1243 ++++++++++++++++++++
> arch/powerpc/kvm/powerpc.c | 37 +
> arch/powerpc/platforms/powernv/opal-call.c | 3 +
> arch/powerpc/sysdev/xive/native.c | 110 ++
> virt/kvm/kvm_main.c | 24 +
> Documentation/virtual/kvm/api.txt | 10 +
> Documentation/virtual/kvm/devices/xive.txt | 197 ++++
> arch/powerpc/kvm/Makefile | 2 +-
> 19 files changed, 1980 insertions(+), 61 deletions(-)
> create mode 100644 arch/powerpc/kvm/book3s_xive_native.c
> create mode 100644 Documentation/virtual/kvm/devices/xive.txt
>
> --
> 2.20.1
>
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
2019-04-10 17:04 ` Cédric Le Goater
(?)
@ 2019-05-03 6:50 ` Michael Ellerman
-1 siblings, 0 replies; 81+ messages in thread
From: Michael Ellerman @ 2019-05-03 6:50 UTC (permalink / raw)
To: Cédric Le Goater, kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 489 bytes --]
On Wed, 2019-04-10 at 17:04:33 UTC, =?utf-8?q?C=C3=A9dric_Le_Goater?= wrote:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to get and set the state of the XIVE internal
> structures being used by a sPAPR guest.
>
> Signed-off-by: Céddric Le Goater <clg@kaod.org>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Applied to powerpc topic/ppc-kvm, thanks.
https://git.kernel.org/powerpc/c/88ec6b93c8e7d6d4ffaf6ad6395ceb3b
cheers
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-05-03 6:50 ` Michael Ellerman
0 siblings, 0 replies; 81+ messages in thread
From: Michael Ellerman @ 2019-05-03 6:50 UTC (permalink / raw)
To: Cédric Le Goater, kvm-ppc
Cc: linuxppc-dev, Paul Mackerras, Cédric Le Goater, kvm, David Gibson
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 489 bytes --]
On Wed, 2019-04-10 at 17:04:33 UTC, =?utf-8?q?C=C3=A9dric_Le_Goater?= wrote:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to get and set the state of the XIVE internal
> structures being used by a sPAPR guest.
>
> Signed-off-by: Céddric Le Goater <clg@kaod.org>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Applied to powerpc topic/ppc-kvm, thanks.
https://git.kernel.org/powerpc/c/88ec6b93c8e7d6d4ffaf6ad6395ceb3b
cheers
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support
@ 2019-05-03 6:50 ` Michael Ellerman
0 siblings, 0 replies; 81+ messages in thread
From: Michael Ellerman @ 2019-05-03 6:50 UTC (permalink / raw)
To: Cédric Le Goater, kvm-ppc
Cc: kvm, Paul Mackerras, Cédric Le Goater, linuxppc-dev, David Gibson
On Wed, 2019-04-10 at 17:04:33 UTC, =?utf-8?q?Cédric_Le_Goater?= wrote:
> The support for XIVE native exploitation mode in Linux/KVM needs a
> couple more OPAL calls to get and set the state of the XIVE internal
> structures being used by a sPAPR guest.
>
> Signed-off-by: Céddric Le Goater <clg@kaod.org>
> Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Applied to powerpc topic/ppc-kvm, thanks.
https://git.kernel.org/powerpc/c/88ec6b93c8e7d6d4ffaf6ad6395ceb3b
cheers
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
2019-04-29 8:05 ` Satheesh Rajendran
(?)
@ 2019-05-06 16:21 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-05-06 16:21 UTC (permalink / raw)
To: Satheesh Rajendran
Cc: kvm-ppc, kvm, Paul Mackerras, linuxppc-dev, David Gibson
Hello Satheesh,
On 4/29/19 10:05 AM, Satheesh Rajendran wrote:
> On Wed, Apr 10, 2019 at 07:04:32PM +0200, Cédric Le Goater wrote:
>> Hello,
>>
>> GitHub trees available here :
>>
>> QEMU sPAPR:
>>
>> https://github.com/legoater/qemu/commits/xive-next
>>
>> Linux/KVM:
>>
>> https://github.com/legoater/linux/commits/xive-5.1
>
> Hi,
>
> Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with
> guest memory > 64G, till 64G it boots fine.
>
> Note: xics(ic-mode=xics) guest with the same configuration boots fine
Indeed. The guest hangs because IPIs are not correctly received. The guest
sees the EQ page as being filled with zeroes and discards the interrupt
whereas the host, KVM and QEMU, sees the correct entries.
I haven't spotted anything bizarre from guest side. Do we have a 64GB
frontier somewhere in KVM ?
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-05-06 16:21 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-05-06 16:21 UTC (permalink / raw)
To: Satheesh Rajendran
Cc: linuxppc-dev, Paul Mackerras, kvm, kvm-ppc, David Gibson
Hello Satheesh,
On 4/29/19 10:05 AM, Satheesh Rajendran wrote:
> On Wed, Apr 10, 2019 at 07:04:32PM +0200, Cédric Le Goater wrote:
>> Hello,
>>
>> GitHub trees available here :
>>
>> QEMU sPAPR:
>>
>> https://github.com/legoater/qemu/commits/xive-next
>>
>> Linux/KVM:
>>
>> https://github.com/legoater/linux/commits/xive-5.1
>
> Hi,
>
> Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with
> guest memory > 64G, till 64G it boots fine.
>
> Note: xics(ic-mode=xics) guest with the same configuration boots fine
Indeed. The guest hangs because IPIs are not correctly received. The guest
sees the EQ page as being filled with zeroes and discards the interrupt
whereas the host, KVM and QEMU, sees the correct entries.
I haven't spotted anything bizarre from guest side. Do we have a 64GB
frontier somewhere in KVM ?
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-05-06 16:21 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-05-06 16:21 UTC (permalink / raw)
To: Satheesh Rajendran
Cc: kvm-ppc, kvm, Paul Mackerras, linuxppc-dev, David Gibson
Hello Satheesh,
On 4/29/19 10:05 AM, Satheesh Rajendran wrote:
> On Wed, Apr 10, 2019 at 07:04:32PM +0200, Cédric Le Goater wrote:
>> Hello,
>>
>> GitHub trees available here :
>>
>> QEMU sPAPR:
>>
>> https://github.com/legoater/qemu/commits/xive-next
>>
>> Linux/KVM:
>>
>> https://github.com/legoater/linux/commits/xive-5.1
>
> Hi,
>
> Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with
> guest memory > 64G, till 64G it boots fine.
>
> Note: xics(ic-mode=xics) guest with the same configuration boots fine
Indeed. The guest hangs because IPIs are not correctly received. The guest
sees the EQ page as being filled with zeroes and discards the interrupt
whereas the host, KVM and QEMU, sees the correct entries.
I haven't spotted anything bizarre from guest side. Do we have a 64GB
frontier somewhere in KVM ?
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
2019-05-06 16:21 ` Cédric Le Goater
(?)
@ 2019-05-09 6:34 ` Cédric Le Goater
-1 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-05-09 6:34 UTC (permalink / raw)
To: Satheesh Rajendran
Cc: kvm-ppc, kvm, Paul Mackerras, linuxppc-dev, David Gibson
Satheesh,
>> Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with
>> guest memory > 64G, till 64G it boots fine.
>>
>> Note: xics(ic-mode=xics) guest with the same configuration boots fine
>
> Indeed. The guest hangs because IPIs are not correctly received. The guest
> sees the EQ page as being filled with zeroes and discards the interrupt
> whereas the host, KVM and QEMU, sees the correct entries.
>
> I haven't spotted anything bizarre from guest side. Do we have a 64GB
> frontier somewhere in KVM ?
The issue was an erroneous assignment of the EQ page address in QEMU.
I pushed the fix in my QEMU branch :
https://github.com/legoater/qemu/commits/xive-next
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-05-09 6:34 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-05-09 6:34 UTC (permalink / raw)
To: Satheesh Rajendran
Cc: linuxppc-dev, Paul Mackerras, kvm, kvm-ppc, David Gibson
Satheesh,
>> Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with
>> guest memory > 64G, till 64G it boots fine.
>>
>> Note: xics(ic-mode=xics) guest with the same configuration boots fine
>
> Indeed. The guest hangs because IPIs are not correctly received. The guest
> sees the EQ page as being filled with zeroes and discards the interrupt
> whereas the host, KVM and QEMU, sees the correct entries.
>
> I haven't spotted anything bizarre from guest side. Do we have a 64GB
> frontier somewhere in KVM ?
The issue was an erroneous assignment of the EQ page address in QEMU.
I pushed the fix in my QEMU branch :
https://github.com/legoater/qemu/commits/xive-next
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
* Re: [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode
@ 2019-05-09 6:34 ` Cédric Le Goater
0 siblings, 0 replies; 81+ messages in thread
From: Cédric Le Goater @ 2019-05-09 6:34 UTC (permalink / raw)
To: Satheesh Rajendran
Cc: kvm-ppc, kvm, Paul Mackerras, linuxppc-dev, David Gibson
Satheesh,
>> Xive(both ic-mode=dual and ic-mode=xive) guest fails to boot with
>> guest memory > 64G, till 64G it boots fine.
>>
>> Note: xics(ic-mode=xics) guest with the same configuration boots fine
>
> Indeed. The guest hangs because IPIs are not correctly received. The guest
> sees the EQ page as being filled with zeroes and discards the interrupt
> whereas the host, KVM and QEMU, sees the correct entries.
>
> I haven't spotted anything bizarre from guest side. Do we have a 64GB
> frontier somewhere in KVM ?
The issue was an erroneous assignment of the EQ page address in QEMU.
I pushed the fix in my QEMU branch :
https://github.com/legoater/qemu/commits/xive-next
Thanks,
C.
^ permalink raw reply [flat|nested] 81+ messages in thread
end of thread, other threads:[~2019-05-09 9:02 UTC | newest]
Thread overview: 81+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-10 17:04 [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-05-03 6:50 ` Michael Ellerman
2019-05-03 6:50 ` Michael Ellerman
2019-05-03 6:50 ` Michael Ellerman
2019-04-10 17:04 ` [PATCH v5 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure " Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-12 5:19 ` David Gibson
2019-04-12 5:19 ` David Gibson
2019-04-12 5:19 ` David Gibson
2019-04-10 17:04 ` [PATCH v5 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 11/16] KVM: introduce a 'mmap' method for KVM devices Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 15/16] KVM: PPC: Book3S HV: XIVE: activate XIVE exploitation mode Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` [PATCH v5 16/16] KVM: PPC: Book3S HV: XIVE: introduce a 'release' device operation Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-10 17:04 ` Cédric Le Goater
2019-04-11 3:16 ` Paul Mackerras
2019-04-11 3:16 ` Paul Mackerras
2019-04-11 3:16 ` Paul Mackerras
2019-04-11 4:38 ` David Gibson
2019-04-11 4:38 ` David Gibson
2019-04-11 4:38 ` David Gibson
2019-04-11 6:12 ` Cédric Le Goater
2019-04-11 6:12 ` Cédric Le Goater
2019-04-11 6:12 ` Cédric Le Goater
2019-04-11 10:27 ` Paul Mackerras
2019-04-11 10:27 ` Paul Mackerras
2019-04-11 10:27 ` Paul Mackerras
2019-04-11 11:48 ` Cédric Le Goater
2019-04-11 11:48 ` Cédric Le Goater
2019-04-11 11:48 ` Cédric Le Goater
2019-04-29 8:05 ` [PATCH v5 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode Satheesh Rajendran
2019-04-29 8:17 ` Satheesh Rajendran
2019-04-29 8:05 ` Satheesh Rajendran
2019-05-06 16:21 ` Cédric Le Goater
2019-05-06 16:21 ` Cédric Le Goater
2019-05-06 16:21 ` Cédric Le Goater
2019-05-09 6:34 ` Cédric Le Goater
2019-05-09 6:34 ` Cédric Le Goater
2019-05-09 6:34 ` Cédric Le Goater
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.