From: "Cédric Le Goater" <clg@kaod.org> To: kvm-ppc@vger.kernel.org Cc: kvm@vger.kernel.org, "Paul Mackerras" <paulus@samba.org>, "Cédric Le Goater" <clg@kaod.org>, linuxppc-dev@lists.ozlabs.org, "David Gibson" <david@gibson.dropbear.id.au> Subject: [PATCH v2 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode Date: Fri, 22 Feb 2019 12:28:24 +0100 [thread overview] Message-ID: <20190222112840.25000-1-clg@kaod.org> (raw) Hello, On the POWER9 processor, the XIVE interrupt controller can control interrupt sources using MMIOs to trigger events, to EOI or to turn off the sources. Priority management and interrupt acknowledgment is also controlled by MMIO in the CPU presenter subengine. PowerNV/baremetal Linux runs natively under XIVE but sPAPR guests need special support from the hypervisor to do the same. This is called the XIVE native exploitation mode and today, it can be activated under the PowerPC Hypervisor, pHyp. However, Linux/KVM lacks XIVE native support and still offers the old interrupt mode interface using a KVM device implementing the XICS hcalls over XIVE. The following series is proposal to add the same support under KVM. A new KVM device is introduced for the XIVE native exploitation mode. It reuses most of the XICS-over-XIVE glue implementation structures which are internal to KVM but has a completely different interface. A set of KVM device ioctls provide support for the hypervisor calls, all handled in QEMU, to configure the sources and the event queues. From there, all interrupt control is transferred to the guest which can use MMIOs. These MMIO regions (ESB and TIMA) are exposed to guests in QEMU, similarly to VFIO, and the associated VMAs are populated dynamically with the appropriate pages using a fault handler. These are now implemented using mmap()s of the KVM device fd. Migration has its own specific needs regarding memory. The patchset provides a specific control to quiesce XIVE before capturing the memory. The save and restore of the internal state is based on the same ioctls used for the hcalls. On a POWER9 sPAPR machine, the Client Architecture Support (CAS) negotiation process determines whether the guest operates with a interrupt controller using the XICS legacy model, as found on POWER8, or in XIVE exploitation mode. Which means that the KVM interrupt device should be created at runtime, after the machine has started. This requires extra support from KVM to destroy KVM devices. It is introduced at the end of the patcshet as it still requires some attention and a XIVE-only VM would not need. This is 5.2 material. I expect a couple of respin for fixes, and the OPAL patches have not yet been merged. GitHub trees available here : QEMU sPAPR: https://github.com/legoater/qemu/commits/xive-next Linux/KVM: https://github.com/legoater/linux/commits/xive-5.0 OPAL: https://github.com/legoater/skiboot/commits/xive Thanks, C. Changes since v1: - Better documentation (was missing) - Nested support. XIVE not advertised on non PowerNV platforms. This is a good way to test the fallback on QEMU emulated devices. - ESB and TIMA special mapping done using the KVM device fd - All hcalls moved to QEMU. Dropped the patch moving the hcall flags. - Reworked of the KVM device ioctl controls to support hcalls and migration needs to capture/save states - Merged the control syncing XIVE and marking the EQ page dirty - Fixed passthrough support using the KVM device file address_space to clear the ESB pages from the mapping - Misc enhancements and fixes Cédric Le Goater (16): powerpc/xive: add OPAL extensions for the XIVE native exploitation support KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE KVM: PPC: Book3S HV: XIVE: add a control to initialize a source KVM: PPC: Book3S HV: XIVE: add a control to configure a source KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration KVM: PPC: Book3S HV: XIVE: add a global reset control KVM: PPC: Book3S HV: XIVE: add a control to sync the sources KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state KVM: introduce a 'mmap' method for KVM devices KVM: PPC: Book3S HV: XIVE: add a TIMA mapping KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages KVM: PPC: Book3S HV: XIVE: add passthrough support KVM: introduce a KVM_DESTROY_DEVICE ioctl KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters arch/powerpc/include/asm/kvm_host.h | 2 + arch/powerpc/include/asm/kvm_ppc.h | 32 + arch/powerpc/include/asm/opal-api.h | 11 +- arch/powerpc/include/asm/opal.h | 7 + arch/powerpc/include/asm/xive.h | 17 + arch/powerpc/include/uapi/asm/kvm.h | 48 + arch/powerpc/kvm/book3s_xive.h | 31 + include/linux/kvm_host.h | 1 + include/uapi/linux/kvm.h | 10 + arch/powerpc/kvm/book3s.c | 31 +- arch/powerpc/kvm/book3s_xics.c | 19 + arch/powerpc/kvm/book3s_xive.c | 149 ++- arch/powerpc/kvm/book3s_xive_native.c | 1171 +++++++++++++++++ arch/powerpc/kvm/powerpc.c | 33 + arch/powerpc/sysdev/xive/native.c | 110 ++ virt/kvm/kvm_main.c | 49 + Documentation/virtual/kvm/api.txt | 28 + Documentation/virtual/kvm/devices/xive.txt | 190 +++ arch/powerpc/kvm/Makefile | 2 +- .../powerpc/platforms/powernv/opal-wrappers.S | 3 + 20 files changed, 1896 insertions(+), 48 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_xive_native.c create mode 100644 Documentation/virtual/kvm/devices/xive.txt -- 2.20.1
WARNING: multiple messages have this Message-ID (diff)
From: "Cédric Le Goater" <clg@kaod.org> To: kvm-ppc@vger.kernel.org Cc: kvm@vger.kernel.org, "Paul Mackerras" <paulus@samba.org>, "Cédric Le Goater" <clg@kaod.org>, linuxppc-dev@lists.ozlabs.org, "David Gibson" <david@gibson.dropbear.id.au> Subject: [PATCH v2 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode Date: Fri, 22 Feb 2019 11:28:24 +0000 [thread overview] Message-ID: <20190222112840.25000-1-clg@kaod.org> (raw) Hello, On the POWER9 processor, the XIVE interrupt controller can control interrupt sources using MMIOs to trigger events, to EOI or to turn off the sources. Priority management and interrupt acknowledgment is also controlled by MMIO in the CPU presenter subengine. PowerNV/baremetal Linux runs natively under XIVE but sPAPR guests need special support from the hypervisor to do the same. This is called the XIVE native exploitation mode and today, it can be activated under the PowerPC Hypervisor, pHyp. However, Linux/KVM lacks XIVE native support and still offers the old interrupt mode interface using a KVM device implementing the XICS hcalls over XIVE. The following series is proposal to add the same support under KVM. A new KVM device is introduced for the XIVE native exploitation mode. It reuses most of the XICS-over-XIVE glue implementation structures which are internal to KVM but has a completely different interface. A set of KVM device ioctls provide support for the hypervisor calls, all handled in QEMU, to configure the sources and the event queues. From there, all interrupt control is transferred to the guest which can use MMIOs. These MMIO regions (ESB and TIMA) are exposed to guests in QEMU, similarly to VFIO, and the associated VMAs are populated dynamically with the appropriate pages using a fault handler. These are now implemented using mmap()s of the KVM device fd. Migration has its own specific needs regarding memory. The patchset provides a specific control to quiesce XIVE before capturing the memory. The save and restore of the internal state is based on the same ioctls used for the hcalls. On a POWER9 sPAPR machine, the Client Architecture Support (CAS) negotiation process determines whether the guest operates with a interrupt controller using the XICS legacy model, as found on POWER8, or in XIVE exploitation mode. Which means that the KVM interrupt device should be created at runtime, after the machine has started. This requires extra support from KVM to destroy KVM devices. It is introduced at the end of the patcshet as it still requires some attention and a XIVE-only VM would not need. This is 5.2 material. I expect a couple of respin for fixes, and the OPAL patches have not yet been merged. GitHub trees available here : QEMU sPAPR: https://github.com/legoater/qemu/commits/xive-next Linux/KVM: https://github.com/legoater/linux/commits/xive-5.0 OPAL: https://github.com/legoater/skiboot/commits/xive Thanks, C. Changes since v1: - Better documentation (was missing) - Nested support. XIVE not advertised on non PowerNV platforms. This is a good way to test the fallback on QEMU emulated devices. - ESB and TIMA special mapping done using the KVM device fd - All hcalls moved to QEMU. Dropped the patch moving the hcall flags. - Reworked of the KVM device ioctl controls to support hcalls and migration needs to capture/save states - Merged the control syncing XIVE and marking the EQ page dirty - Fixed passthrough support using the KVM device file address_space to clear the ESB pages from the mapping - Misc enhancements and fixes Cédric Le Goater (16): powerpc/xive: add OPAL extensions for the XIVE native exploitation support KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE KVM: PPC: Book3S HV: XIVE: add a control to initialize a source KVM: PPC: Book3S HV: XIVE: add a control to configure a source KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration KVM: PPC: Book3S HV: XIVE: add a global reset control KVM: PPC: Book3S HV: XIVE: add a control to sync the sources KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state KVM: introduce a 'mmap' method for KVM devices KVM: PPC: Book3S HV: XIVE: add a TIMA mapping KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages KVM: PPC: Book3S HV: XIVE: add passthrough support KVM: introduce a KVM_DESTROY_DEVICE ioctl KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters arch/powerpc/include/asm/kvm_host.h | 2 + arch/powerpc/include/asm/kvm_ppc.h | 32 + arch/powerpc/include/asm/opal-api.h | 11 +- arch/powerpc/include/asm/opal.h | 7 + arch/powerpc/include/asm/xive.h | 17 + arch/powerpc/include/uapi/asm/kvm.h | 48 + arch/powerpc/kvm/book3s_xive.h | 31 + include/linux/kvm_host.h | 1 + include/uapi/linux/kvm.h | 10 + arch/powerpc/kvm/book3s.c | 31 +- arch/powerpc/kvm/book3s_xics.c | 19 + arch/powerpc/kvm/book3s_xive.c | 149 ++- arch/powerpc/kvm/book3s_xive_native.c | 1171 +++++++++++++++++ arch/powerpc/kvm/powerpc.c | 33 + arch/powerpc/sysdev/xive/native.c | 110 ++ virt/kvm/kvm_main.c | 49 + Documentation/virtual/kvm/api.txt | 28 + Documentation/virtual/kvm/devices/xive.txt | 190 +++ arch/powerpc/kvm/Makefile | 2 +- .../powerpc/platforms/powernv/opal-wrappers.S | 3 + 20 files changed, 1896 insertions(+), 48 deletions(-) create mode 100644 arch/powerpc/kvm/book3s_xive_native.c create mode 100644 Documentation/virtual/kvm/devices/xive.txt -- 2.20.1
next reply other threads:[~2019-02-22 11:28 UTC|newest] Thread overview: 142+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-02-22 11:28 Cédric Le Goater [this message] 2019-02-22 11:28 ` [PATCH v2 00/16] KVM: PPC: Book3S HV: add XIVE native exploitation mode Cédric Le Goater 2019-02-22 11:28 ` [PATCH v2 01/16] powerpc/xive: add OPAL extensions for the XIVE native exploitation support Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-24 23:42 ` David Gibson 2019-02-24 23:42 ` David Gibson 2019-02-25 3:50 ` Michael Ellerman 2019-02-25 3:50 ` Michael Ellerman 2019-02-25 10:11 ` Cédric Le Goater 2019-02-25 10:11 ` Cédric Le Goater 2019-02-26 4:21 ` David Gibson 2019-02-26 4:21 ` David Gibson 2019-03-12 18:25 ` Cédric Le Goater 2019-03-12 18:25 ` Cédric Le Goater 2019-02-22 11:28 ` [PATCH v2 02/16] KVM: PPC: Book3S HV: add a new KVM device for the XIVE native exploitation mode Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 0:08 ` David Gibson 2019-02-25 0:08 ` David Gibson 2019-03-12 11:14 ` Cédric Le Goater 2019-03-12 11:14 ` Cédric Le Goater 2019-02-22 11:28 ` [PATCH v2 03/16] KVM: PPC: Book3S HV: XIVE: introduce a new capability KVM_CAP_PPC_IRQ_XIVE Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 0:35 ` David Gibson 2019-02-25 0:35 ` David Gibson 2019-02-25 4:59 ` Paul Mackerras 2019-02-25 4:59 ` Paul Mackerras 2019-03-12 14:10 ` Cédric Le Goater 2019-03-12 14:10 ` Cédric Le Goater 2019-03-12 14:03 ` Cédric Le Goater 2019-03-12 14:03 ` Cédric Le Goater 2019-03-13 4:05 ` David Gibson 2019-03-13 4:05 ` David Gibson 2019-02-25 4:35 ` Paul Mackerras 2019-02-25 4:35 ` Paul Mackerras 2019-03-13 8:34 ` Cédric Le Goater 2019-03-13 8:34 ` Cédric Le Goater 2019-03-14 2:29 ` David Gibson 2019-03-14 2:29 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 04/16] KVM: PPC: Book3S HV: XIVE: add a control to initialize a source Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 2:10 ` David Gibson 2019-02-25 2:10 ` David Gibson 2019-02-26 4:25 ` Paul Mackerras 2019-02-26 4:25 ` Paul Mackerras 2019-02-26 23:20 ` David Gibson 2019-02-26 23:20 ` David Gibson 2019-03-12 15:19 ` Cédric Le Goater 2019-03-12 15:19 ` Cédric Le Goater 2019-03-14 2:15 ` David Gibson 2019-03-14 2:15 ` David Gibson 2019-02-25 5:30 ` Paul Mackerras 2019-02-25 5:30 ` Paul Mackerras 2019-02-22 11:28 ` [PATCH v2 05/16] KVM: PPC: Book3S HV: XIVE: add a control to configure " Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 2:21 ` David Gibson 2019-02-25 2:21 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 06/16] KVM: PPC: Book3S HV: XIVE: add controls for the EQ configuration Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 2:39 ` David Gibson 2019-02-25 2:39 ` David Gibson 2019-03-12 17:00 ` Cédric Le Goater 2019-03-12 17:00 ` Cédric Le Goater 2019-03-13 4:03 ` David Gibson 2019-03-13 4:03 ` David Gibson 2019-03-13 8:46 ` Cédric Le Goater 2019-03-13 8:46 ` Cédric Le Goater 2019-03-14 3:29 ` David Gibson 2019-03-14 3:29 ` David Gibson 2019-02-26 5:24 ` Paul Mackerras 2019-02-26 5:24 ` Paul Mackerras 2019-03-13 9:40 ` Cédric Le Goater 2019-03-13 9:40 ` Cédric Le Goater 2019-03-14 2:32 ` David Gibson 2019-03-14 2:32 ` David Gibson 2019-03-14 7:11 ` Cédric Le Goater 2019-03-14 7:11 ` Cédric Le Goater 2019-03-15 0:29 ` David Gibson 2019-03-15 0:29 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 07/16] KVM: PPC: Book3S HV: XIVE: add a global reset control Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 2:43 ` David Gibson 2019-02-25 2:43 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 08/16] KVM: PPC: Book3S HV: XIVE: add a control to sync the sources Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 2:45 ` David Gibson 2019-02-25 2:45 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 09/16] KVM: PPC: Book3S HV: XIVE: add a control to dirty the XIVE EQ pages Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 2:53 ` David Gibson 2019-02-25 2:53 ` David Gibson 2019-03-13 11:48 ` Cédric Le Goater 2019-03-13 11:48 ` Cédric Le Goater 2019-03-14 2:33 ` David Gibson 2019-03-14 2:33 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 10/16] KVM: PPC: Book3S HV: XIVE: add get/set accessors for the VP XIVE state Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 3:31 ` David Gibson 2019-02-25 3:31 ` David Gibson 2019-03-13 13:19 ` Cédric Le Goater 2019-03-13 13:19 ` Cédric Le Goater 2019-03-14 3:09 ` David Gibson 2019-03-14 3:09 ` David Gibson 2019-03-14 7:08 ` Cédric Le Goater 2019-03-14 7:08 ` Cédric Le Goater 2019-02-22 11:28 ` [PATCH v2 11/16] KVM: introduce a 'mmap' method for KVM devices Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 3:33 ` David Gibson 2019-02-25 3:33 ` David Gibson 2019-02-25 10:57 ` Cédric Le Goater 2019-02-25 10:57 ` Cédric Le Goater 2019-02-26 12:52 ` Paolo Bonzini 2019-02-26 12:52 ` Paolo Bonzini 2019-02-26 23:22 ` David Gibson 2019-02-26 23:22 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 12/16] KVM: PPC: Book3S HV: XIVE: add a TIMA mapping Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 3:42 ` David Gibson 2019-02-25 3:42 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 13/16] KVM: PPC: Book3S HV: XIVE: add a mapping for the source ESB pages Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 3:47 ` David Gibson 2019-02-25 3:47 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 14/16] KVM: PPC: Book3S HV: XIVE: add passthrough support Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 4:13 ` David Gibson 2019-02-25 4:13 ` David Gibson 2019-02-22 11:28 ` [PATCH v2 15/16] KVM: introduce a KVM_DESTROY_DEVICE ioctl Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 4:15 ` David Gibson 2019-02-25 4:15 ` David Gibson 2019-03-13 8:02 ` Cédric Le Goater 2019-03-13 8:02 ` Cédric Le Goater 2019-03-15 17:57 ` Paolo Bonzini 2019-03-15 17:57 ` Paolo Bonzini 2019-02-22 11:28 ` [PATCH v2 16/16] KVM: PPC: Book3S HV: XIVE: clear the vCPU interrupt presenters Cédric Le Goater 2019-02-22 11:28 ` Cédric Le Goater 2019-02-25 4:18 ` David Gibson 2019-02-25 4:18 ` David Gibson 2019-03-13 8:17 ` Cédric Le Goater 2019-03-13 8:17 ` Cédric Le Goater 2019-03-14 2:26 ` David Gibson 2019-03-14 2:26 ` David Gibson
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190222112840.25000-1-clg@kaod.org \ --to=clg@kaod.org \ --cc=david@gibson.dropbear.id.au \ --cc=kvm-ppc@vger.kernel.org \ --cc=kvm@vger.kernel.org \ --cc=linuxppc-dev@lists.ozlabs.org \ --cc=paulus@samba.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.