All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v5 00/22] Support SDEI Virtualization
@ 2022-03-22  8:06 ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This series intends to virtualize Software Delegated Exception Interface
(SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to
deliver NMI-alike SDEI event to guest and it's needed by Async PF to
deliver page-not-present notification from hypervisor to guest. The code
and the required qemu changes can be found from:

   https://developer.arm.com/documentation/den0054/c
   https://github.com/gwshan/linux    ("kvm/arm64_sdei")
   https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

For the design and migration needs, please refer to the document in
PATCH[21/22] in this series. The series is organized as below:

  PATCH[01]    Introduces template for smccc_get_argx()
  PATCH[02]    Adds SDEI virtualization infrastructure
  PATCH[03-17] Supports various SDEI hypercalls and event handling
  PATCH[18-20] Adds ioctl commands to support migration and configuration
               and exports SDEI capability
  PATCH[21]    Adds SDEI document 
  PATCH[22]    Adds SDEI selftest case

Testing
=======

[1] The selftest case included in this series works fine. The default SDEI
    event, whose number is zero, can be registered, enabled, raised. The
    SDEI event handler can be invoked.

    [host]# pwd
    /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
    [root@virtlab-arm01 kvm]# ./aarch64/sdei 

        NR_VCPUS: 2    SDEI Event: 0x00000000

    --- VERSION
        Version:              1.1 (vendor: 0x4b564d)
    --- FEATURES
        Shared event slots:   0
        Private event slots:  0
        Relative mode:        No
    --- PRIVATE_RESET
    --- SHARED_RESET
    --- PE_UNMASK
    --- EVENT_GET_INFO
        Type:                 Private
        Priority:             Normal
        Signaled:             Yes
    --- EVENT_REGISTER
    --- EVENT_ENABLE
    --- EVENT_SIGNAL
        Handled:              Yes
        IRQ:                  No
        Status:               Registered-Enabled-Running
        PC/PSTATE:            000000000040232c 00000000600003c5
        Regs:                 0000000000000000 0000000000000000
                              0000000000000000 0000000000000000
    --- PE_MASK
    --- EVENT_DISABLE
    --- EVENT_UNREGISTER

        Result: OK

[2] There are additional patches in the following repositories to create
    procfs entries, allowing to inject SDEI event from host side. The
    SDEI client in the guest side registers the SDEI default event, whose
    number is zero. Also, the QEMU exports SDEI ACPI table and supports
    migration for SDEI.

    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

    [2.1] Start the guests and migrate the source VM to the destination
          VM.

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
               :

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
            -incoming tcp:0:4444                                          \
               :

    [2.2] Check kernel log on the source VM. The SDEI service is enabled
          and the default SDEI event (0x0) is enabled.

     [guest-src]# dmesg | grep -i sdei
     ACPI: SDEI 0x000000005BC80000 000024 \
                (v00 BOCHS  BXPC     00000001 BXPC 00000001)
     sdei: SDEIv1.1 (0x4b564d) detected in firmware.
     SDEI TEST: Version 1.1, Vendor 0x4b564d
     sdei_init: SDEI event (0x0) registered
     sdei_init: SDEI event (0x0) enabled

 
     (qemu) migrate -d tcp:localhost:4444

    [2.3] Migrate the source VM to the destination VM. Inject SDEI event
          to the destination VM. The event is raised and handled.

    (qemu) migrate -d tcp:localhost:4444

    [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1

    [guest-dst]#
    =========== SDEI Event (CPU#1) ===========
    Event: 0000000000000000  Parameter: 00000000dabfdabf
    PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
    Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 ffff800009399008 
             ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 
             0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 
             0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 
    Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 
             ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 

Changelog
========= 
v5:
   * Rebased to v5.17.rc7                                     (Gavin)
   * Unified names for the objects, data structures, variables
     and functions. The events have been named as exposed,
     registered and vcpu event. The staes needs to be migrated
     is put into kvm_sdei_state.h                             (Eric)
   * More inline functions to visit SDEI event's properties   (Eric)
   * Support unregistration pending state                     (Eric)
   * Support v1.1 SDEI specification                          (Eric)
   * Fold the code to inject, deliver and handle SDEI event
     from PATCH[v4 13/18/19] into PATCH[v5 13]                (Eric)
   * Simplified ioctl interface to visit all events at once   (Eric/Gavin)
   * Improved reference count and avoid its migration. Also,
     the limit to memory allocation is added based on it.     (Eric)
   * Change the return values from hypercall functions        (Eric) 
   * Validate @ksdei and @vsdi in kvm_sdei_hypercall()        (Shannon)
   * Add document to explain how SDEI virutalization and the
     migration are supported                                  (Eric)
   * Improved selftest case to inject and handle SDEI event   (Gavin)
   * Improved comments and commit logs                        (Eric)
   * Address misc comments from Eric. Hopefully, all of them
     are covered in v5 because Eric provided lots of comments
     in the last round of review                              (Eric)
v4:
   * Rebased to v5.14.rc5                                         (Gavin)
v3:
   * Rebased to v5.13.rc1                                         (Gavin)
   * Use linux data types in kvm_sdei.h                           (Gavin)
v2:
   * Rebased to v5.11.rc6                                         (Gavin)
   * Dropped changes related to SDEI client driver                (Gavin)
   * Removed support for passthrou SDEI events                    (Gavin)
   * Redesigned data structures                                   (Gavin)
   * Implementation is almost rewritten as the data structures
     are totally changed                                          (Gavin)
   * Added ioctl commands to support migration                    (Gavin)

Gavin Shan (22):
  KVM: arm64: Introduce template for inline functions
  KVM: arm64: Add SDEI virtualization infrastructure
  KVM: arm64: Support SDEI_VERSION hypercall
  KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
  KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
  KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
  KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
  KVM: arm64: Support SDEI_EVENT_STATUS hypercall
  KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
  KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
  KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
  KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
  KVM: arm64: Support SDEI_FEATURES hypercall
  KVM: arm64: Support SDEI event injection, delivery and cancellation
  KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
  KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME}
    hypercall
  KVM: arm64: Support SDEI event notifier
  KVM: arm64: Support SDEI ioctl commands on VM
  KVM: arm64: Support SDEI ioctl commands on vCPU
  KVM: arm64: Export SDEI capability
  KVM: arm64: Add SDEI document
  KVM: selftests: Add SDEI test case

 Documentation/virt/kvm/api.rst               |   10 +
 Documentation/virt/kvm/arm/sdei.rst          |  325 +++
 arch/arm64/include/asm/kvm_emulate.h         |    1 +
 arch/arm64/include/asm/kvm_host.h            |    5 +
 arch/arm64/include/asm/kvm_sdei.h            |  187 ++
 arch/arm64/include/uapi/asm/kvm.h            |    1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  101 +
 arch/arm64/kvm/Makefile                      |    2 +-
 arch/arm64/kvm/arm.c                         |   20 +
 arch/arm64/kvm/hypercalls.c                  |   21 +
 arch/arm64/kvm/inject_fault.c                |   29 +
 arch/arm64/kvm/sdei.c                        | 1900 ++++++++++++++++++
 include/kvm/arm_hypercalls.h                 |   24 +-
 include/uapi/linux/arm_sdei.h                |    2 +
 include/uapi/linux/kvm.h                     |    4 +
 tools/testing/selftests/kvm/Makefile         |    1 +
 tools/testing/selftests/kvm/aarch64/sdei.c   |  525 +++++
 17 files changed, 3145 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/virt/kvm/arm/sdei.rst
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c
 create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c

-- 
2.23.0


^ permalink raw reply	[flat|nested] 98+ messages in thread

* [PATCH v5 00/22] Support SDEI Virtualization
@ 2022-03-22  8:06 ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This series intends to virtualize Software Delegated Exception Interface
(SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to
deliver NMI-alike SDEI event to guest and it's needed by Async PF to
deliver page-not-present notification from hypervisor to guest. The code
and the required qemu changes can be found from:

   https://developer.arm.com/documentation/den0054/c
   https://github.com/gwshan/linux    ("kvm/arm64_sdei")
   https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

For the design and migration needs, please refer to the document in
PATCH[21/22] in this series. The series is organized as below:

  PATCH[01]    Introduces template for smccc_get_argx()
  PATCH[02]    Adds SDEI virtualization infrastructure
  PATCH[03-17] Supports various SDEI hypercalls and event handling
  PATCH[18-20] Adds ioctl commands to support migration and configuration
               and exports SDEI capability
  PATCH[21]    Adds SDEI document 
  PATCH[22]    Adds SDEI selftest case

Testing
=======

[1] The selftest case included in this series works fine. The default SDEI
    event, whose number is zero, can be registered, enabled, raised. The
    SDEI event handler can be invoked.

    [host]# pwd
    /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
    [root@virtlab-arm01 kvm]# ./aarch64/sdei 

        NR_VCPUS: 2    SDEI Event: 0x00000000

    --- VERSION
        Version:              1.1 (vendor: 0x4b564d)
    --- FEATURES
        Shared event slots:   0
        Private event slots:  0
        Relative mode:        No
    --- PRIVATE_RESET
    --- SHARED_RESET
    --- PE_UNMASK
    --- EVENT_GET_INFO
        Type:                 Private
        Priority:             Normal
        Signaled:             Yes
    --- EVENT_REGISTER
    --- EVENT_ENABLE
    --- EVENT_SIGNAL
        Handled:              Yes
        IRQ:                  No
        Status:               Registered-Enabled-Running
        PC/PSTATE:            000000000040232c 00000000600003c5
        Regs:                 0000000000000000 0000000000000000
                              0000000000000000 0000000000000000
    --- PE_MASK
    --- EVENT_DISABLE
    --- EVENT_UNREGISTER

        Result: OK

[2] There are additional patches in the following repositories to create
    procfs entries, allowing to inject SDEI event from host side. The
    SDEI client in the guest side registers the SDEI default event, whose
    number is zero. Also, the QEMU exports SDEI ACPI table and supports
    migration for SDEI.

    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")

    [2.1] Start the guests and migrate the source VM to the destination
          VM.

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
               :

    [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
            -accel kvm -machine virt,gic-version=host                     \
            -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
            -m 1024M,slots=16,maxmem=64G                                  \
               :                                                          \
            -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
            -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
            -append earlycon=pl011,mmio,0x9000000                         \
            -incoming tcp:0:4444                                          \
               :

    [2.2] Check kernel log on the source VM. The SDEI service is enabled
          and the default SDEI event (0x0) is enabled.

     [guest-src]# dmesg | grep -i sdei
     ACPI: SDEI 0x000000005BC80000 000024 \
                (v00 BOCHS  BXPC     00000001 BXPC 00000001)
     sdei: SDEIv1.1 (0x4b564d) detected in firmware.
     SDEI TEST: Version 1.1, Vendor 0x4b564d
     sdei_init: SDEI event (0x0) registered
     sdei_init: SDEI event (0x0) enabled

 
     (qemu) migrate -d tcp:localhost:4444

    [2.3] Migrate the source VM to the destination VM. Inject SDEI event
          to the destination VM. The event is raised and handled.

    (qemu) migrate -d tcp:localhost:4444

    [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1

    [guest-dst]#
    =========== SDEI Event (CPU#1) ===========
    Event: 0000000000000000  Parameter: 00000000dabfdabf
    PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
    Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 0000000000000000 
             0000000000000000 0000000000000000 0000000000000000 ffff800009399008 
             ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 
             0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 
             0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 
    Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
             ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 
             ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 

Changelog
========= 
v5:
   * Rebased to v5.17.rc7                                     (Gavin)
   * Unified names for the objects, data structures, variables
     and functions. The events have been named as exposed,
     registered and vcpu event. The staes needs to be migrated
     is put into kvm_sdei_state.h                             (Eric)
   * More inline functions to visit SDEI event's properties   (Eric)
   * Support unregistration pending state                     (Eric)
   * Support v1.1 SDEI specification                          (Eric)
   * Fold the code to inject, deliver and handle SDEI event
     from PATCH[v4 13/18/19] into PATCH[v5 13]                (Eric)
   * Simplified ioctl interface to visit all events at once   (Eric/Gavin)
   * Improved reference count and avoid its migration. Also,
     the limit to memory allocation is added based on it.     (Eric)
   * Change the return values from hypercall functions        (Eric) 
   * Validate @ksdei and @vsdi in kvm_sdei_hypercall()        (Shannon)
   * Add document to explain how SDEI virutalization and the
     migration are supported                                  (Eric)
   * Improved selftest case to inject and handle SDEI event   (Gavin)
   * Improved comments and commit logs                        (Eric)
   * Address misc comments from Eric. Hopefully, all of them
     are covered in v5 because Eric provided lots of comments
     in the last round of review                              (Eric)
v4:
   * Rebased to v5.14.rc5                                         (Gavin)
v3:
   * Rebased to v5.13.rc1                                         (Gavin)
   * Use linux data types in kvm_sdei.h                           (Gavin)
v2:
   * Rebased to v5.11.rc6                                         (Gavin)
   * Dropped changes related to SDEI client driver                (Gavin)
   * Removed support for passthrou SDEI events                    (Gavin)
   * Redesigned data structures                                   (Gavin)
   * Implementation is almost rewritten as the data structures
     are totally changed                                          (Gavin)
   * Added ioctl commands to support migration                    (Gavin)

Gavin Shan (22):
  KVM: arm64: Introduce template for inline functions
  KVM: arm64: Add SDEI virtualization infrastructure
  KVM: arm64: Support SDEI_VERSION hypercall
  KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
  KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
  KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
  KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
  KVM: arm64: Support SDEI_EVENT_STATUS hypercall
  KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
  KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
  KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
  KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
  KVM: arm64: Support SDEI_FEATURES hypercall
  KVM: arm64: Support SDEI event injection, delivery and cancellation
  KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
  KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME}
    hypercall
  KVM: arm64: Support SDEI event notifier
  KVM: arm64: Support SDEI ioctl commands on VM
  KVM: arm64: Support SDEI ioctl commands on vCPU
  KVM: arm64: Export SDEI capability
  KVM: arm64: Add SDEI document
  KVM: selftests: Add SDEI test case

 Documentation/virt/kvm/api.rst               |   10 +
 Documentation/virt/kvm/arm/sdei.rst          |  325 +++
 arch/arm64/include/asm/kvm_emulate.h         |    1 +
 arch/arm64/include/asm/kvm_host.h            |    5 +
 arch/arm64/include/asm/kvm_sdei.h            |  187 ++
 arch/arm64/include/uapi/asm/kvm.h            |    1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  101 +
 arch/arm64/kvm/Makefile                      |    2 +-
 arch/arm64/kvm/arm.c                         |   20 +
 arch/arm64/kvm/hypercalls.c                  |   21 +
 arch/arm64/kvm/inject_fault.c                |   29 +
 arch/arm64/kvm/sdei.c                        | 1900 ++++++++++++++++++
 include/kvm/arm_hypercalls.h                 |   24 +-
 include/uapi/linux/arm_sdei.h                |    2 +
 include/uapi/linux/kvm.h                     |    4 +
 tools/testing/selftests/kvm/Makefile         |    1 +
 tools/testing/selftests/kvm/aarch64/sdei.c   |  525 +++++
 17 files changed, 3145 insertions(+), 13 deletions(-)
 create mode 100644 Documentation/virt/kvm/arm/sdei.rst
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c
 create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c

-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

The inline functions used to get the SMCCC parameters have same
layout. It means these functions can be presented by an unified
template, to make the code simplified. Besides, this adds more
similar inline functions like smccc_get_arg{4,5,6,7,8}() to get
more SMCCC arguments, which are needed by SDEI virtualization
support.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/include/kvm/arm_hypercalls.h b/include/kvm/arm_hypercalls.h
index 0e2509d27910..d5144c852fe4 100644
--- a/include/kvm/arm_hypercalls.h
+++ b/include/kvm/arm_hypercalls.h
@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
 	return vcpu_get_reg(vcpu, 0);
 }
 
-static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 1);
-}
-
-static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 2);
+#define SMCCC_DECLARE_GET_ARG(reg)					\
+static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
+{									\
+	return vcpu_get_reg(vcpu, reg);					\
 }
 
-static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 3);
-}
+SMCCC_DECLARE_GET_ARG(1)
+SMCCC_DECLARE_GET_ARG(2)
+SMCCC_DECLARE_GET_ARG(3)
+SMCCC_DECLARE_GET_ARG(4)
+SMCCC_DECLARE_GET_ARG(5)
+SMCCC_DECLARE_GET_ARG(6)
+SMCCC_DECLARE_GET_ARG(7)
+SMCCC_DECLARE_GET_ARG(8)
 
 static inline void smccc_set_retval(struct kvm_vcpu *vcpu,
 				    unsigned long a0,
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

The inline functions used to get the SMCCC parameters have same
layout. It means these functions can be presented by an unified
template, to make the code simplified. Besides, this adds more
similar inline functions like smccc_get_arg{4,5,6,7,8}() to get
more SMCCC arguments, which are needed by SDEI virtualization
support.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
 1 file changed, 12 insertions(+), 12 deletions(-)

diff --git a/include/kvm/arm_hypercalls.h b/include/kvm/arm_hypercalls.h
index 0e2509d27910..d5144c852fe4 100644
--- a/include/kvm/arm_hypercalls.h
+++ b/include/kvm/arm_hypercalls.h
@@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
 	return vcpu_get_reg(vcpu, 0);
 }
 
-static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 1);
-}
-
-static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 2);
+#define SMCCC_DECLARE_GET_ARG(reg)					\
+static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
+{									\
+	return vcpu_get_reg(vcpu, reg);					\
 }
 
-static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
-{
-	return vcpu_get_reg(vcpu, 3);
-}
+SMCCC_DECLARE_GET_ARG(1)
+SMCCC_DECLARE_GET_ARG(2)
+SMCCC_DECLARE_GET_ARG(3)
+SMCCC_DECLARE_GET_ARG(4)
+SMCCC_DECLARE_GET_ARG(5)
+SMCCC_DECLARE_GET_ARG(6)
+SMCCC_DECLARE_GET_ARG(7)
+SMCCC_DECLARE_GET_ARG(8)
 
 static inline void smccc_set_retval(struct kvm_vcpu *vcpu,
 				    unsigned long a0,
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

Software Delegated Exception Interface (SDEI) provides a mechanism for
registering and servicing system events. Those system events are high
priority events, which must be serviced immediately. It's going to be
used by Asynchronous Page Fault (APF) to deliver notification from KVM
to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

This introduces SDEI virtualization infrastructure where the SDEI events
are registered and manipulated by the guest through hypercall. The SDEI
event is delivered to one specific vCPU by KVM once it's raised. This
introduces data structures to represent the needed objects to support
the feature, which is highlighted as below. As those objects could be
migrated between VMs, these data structures are partially exposed to
user space.

   * kvm_sdei_exposed_event
     The exposed events are determined and added by VMM through ioctl
     interface. Only the exposed events can be registered from the
     guest.

   * kvm_sdei_registered_event
     The events that have been registered from the guest through the
     SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.

   * kvm_sdei_vcpu_event
     The events that have been delivered to the target vCPU.

   * kvm_sdei_vcpu
     Used to save the preempted context when the SDEI event is serviced
     and delivered. After the SDEI event handling is completed, the
     execution is resumed from the preempted context.

   * kvm_sdei_kvm
     Place holder for the exposed and registered events.

The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for
now. They will be supported in the subsequent patches.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_host.h            |   3 +
 arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
 arch/arm64/include/uapi/asm/kvm.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
 arch/arm64/kvm/Makefile                      |   2 +-
 arch/arm64/kvm/arm.c                         |   8 +
 arch/arm64/kvm/hypercalls.c                  |  21 ++
 arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
 include/uapi/linux/arm_sdei.h                |   2 +
 9 files changed, 523 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 031e3a2537fc..5d37e046a458 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -113,6 +113,8 @@ struct kvm_arch {
 	/* Interrupt controller */
 	struct vgic_dist	vgic;
 
+	struct kvm_sdei_kvm *sdei;
+
 	/* Mandated version of PSCI */
 	u32 psci_version;
 
@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
 	 * Anything that is not used directly from assembly code goes
 	 * here.
 	 */
+	struct kvm_sdei_vcpu *sdei;
 
 	/*
 	 * Guest registers we preserve during guest debugging.
diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
new file mode 100644
index 000000000000..6f58a846d05c
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -0,0 +1,171 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Definitions of various KVM SDEI events.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#ifndef __ARM64_KVM_SDEI_H__
+#define __ARM64_KVM_SDEI_H__
+
+#include <uapi/linux/arm_sdei.h>
+#include <uapi/asm/kvm_sdei_state.h>
+#include <linux/bitmap.h>
+#include <linux/list.h>
+#include <linux/spinlock.h>
+
+struct kvm_vcpu;
+
+struct kvm_sdei_exposed_event {
+	struct kvm_sdei_exposed_event_state	state;
+	struct kvm				*kvm;
+	unsigned int				registered_event_count;
+	struct list_head			link;
+};
+
+struct kvm_sdei_registered_event {
+	struct kvm_sdei_registered_event_state	state;
+	struct kvm				*kvm;
+	struct kvm_sdei_exposed_event		*exposed_event;
+	unsigned int				vcpu_event_count;
+	struct list_head			link;
+};
+
+struct kvm_sdei_vcpu_event {
+	struct kvm_sdei_vcpu_event_state	state;
+	struct kvm_vcpu				*vcpu;
+	struct kvm_sdei_registered_event	*registered_event;
+	struct list_head			link;
+};
+
+struct kvm_sdei_kvm {
+	spinlock_t		lock;
+	unsigned int		exposed_event_count;
+	unsigned int		registered_event_count;
+	struct list_head	exposed_events;
+	struct list_head	registered_events;
+};
+
+struct kvm_sdei_vcpu {
+	spinlock_t                      lock;
+	struct kvm_sdei_vcpu_state      state;
+	struct kvm_sdei_vcpu_event      *critical_event;
+	struct kvm_sdei_vcpu_event      *normal_event;
+	unsigned int			critical_event_count;
+	unsigned int			normal_event_count;
+	struct list_head                critical_events;
+	struct list_head                normal_events;
+};
+
+/*
+ * According to SDEI specification (v1.1), the event number spans 32-bits
+ * and the lower 24-bits are used as the (real) event number. I don't
+ * think we can use that much SDEI numbers in one system. So we reserve
+ * two bits from the 24-bits real event number, to indicate its types:
+ * physical or virtual event. One reserved bit is enough for now, but
+ * two bits are reserved for possible extension in future.
+ *
+ * The physical events are owned by firmware while the virtual events
+ * are used by VMM and KVM.
+ */
+#define KVM_SDEI_EVENT_NUM_TYPE_SHIFT	22
+#define KVM_SDEI_EVENT_NUM_TYPE_MASK	(3UL << KVM_SDEI_EVENT_NUM_TYPE_SHIFT)
+#define KVM_SDEI_EVENT_NUM_TYPE_PHYS	0
+#define KVM_SDEI_EVENT_NUM_TYPE_VIRT	1
+
+static inline bool kvm_sdei_is_virtual(unsigned long num)
+{
+	unsigned long type;
+
+	if (num >> 32)
+		return false;
+
+	type = (num & KVM_SDEI_EVENT_NUM_TYPE_MASK) >>
+		KVM_SDEI_EVENT_NUM_TYPE_SHIFT;
+	if (type != KVM_SDEI_EVENT_NUM_TYPE_VIRT)
+		return false;
+
+	return true;
+}
+
+static inline bool kvm_sdei_is_default(unsigned long num)
+{
+	return num == KVM_SDEI_DEFAULT_EVENT;
+}
+
+static inline bool kvm_sdei_is_supported(unsigned long num)
+{
+	return kvm_sdei_is_default(num) || kvm_sdei_is_virtual(num);
+}
+
+static inline bool kvm_sdei_is_shared(unsigned char type)
+{
+	return type == SDEI_EVENT_TYPE_SHARED;
+}
+
+static inline bool kvm_sdei_is_private(unsigned char type)
+{
+	return type == SDEI_EVENT_TYPE_PRIVATE;
+}
+
+static inline bool kvm_sdei_is_critical(unsigned char priority)
+{
+	return priority == SDEI_EVENT_PRIORITY_CRITICAL;
+}
+
+static inline bool kvm_sdei_is_normal(unsigned char priority)
+{
+	return priority == SDEI_EVENT_PRIORITY_NORMAL;
+}
+
+static inline unsigned int
+kvm_sdei_vcpu_index(struct kvm_vcpu *vcpu,
+		    struct kvm_sdei_exposed_event *event)
+{
+	return kvm_sdei_is_private(event->state.type) ? vcpu->vcpu_idx : 0;
+}
+
+/* Accessors for the registered event */
+#define KVM_SDEI_REGISTERED_EVENT_FUNC(field)				\
+static inline bool							\
+kvm_sdei_is_##field(struct kvm_sdei_registered_event *event,		\
+		    unsigned int index)					\
+{									\
+	return !!test_bit(index, (void *)(event->state.field));		\
+}									\
+									\
+static inline bool							\
+kvm_sdei_none_##field(struct kvm_sdei_registered_event *event)		\
+{									\
+	return bitmap_empty((void *)(event->state.field),		\
+			    KVM_SDEI_MAX_VCPUS);			\
+}									\
+									\
+static inline void							\
+kvm_sdei_set_##field(struct kvm_sdei_registered_event *event,		\
+		     unsigned int index)				\
+{									\
+	set_bit(index, (void *)(event->state.field));			\
+}									\
+									\
+static inline void							\
+kvm_sdei_clear_##field(struct kvm_sdei_registered_event *event,		\
+		       unsigned int index)				\
+{									\
+	clear_bit(index, (void *)(event->state.field));			\
+}
+
+KVM_SDEI_REGISTERED_EVENT_FUNC(registered)
+KVM_SDEI_REGISTERED_EVENT_FUNC(enabled)
+KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
+
+/* APIs */
+void kvm_sdei_init_vm(struct kvm *kvm);
+void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
+int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
+void kvm_sdei_destroy_vm(struct kvm *kvm);
+
+#endif /* __ARM64_KVM_SDEI_H__ */
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 323e251ed37b..33ee95bc088e 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -36,6 +36,7 @@
 #include <linux/types.h>
 #include <asm/ptrace.h>
 #include <asm/sve_context.h>
+#include <asm/kvm_sdei_state.h>
 
 #define __KVM_HAVE_GUEST_DEBUG
 #define __KVM_HAVE_IRQ_LINE
diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
new file mode 100644
index 000000000000..b14844230117
--- /dev/null
+++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Definitions of various KVM SDEI event states.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
+#define _UAPI__ASM_KVM_SDEI_STATE_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+/*
+ * The software signaled event is the default one, which is
+ * defined in v1.1 specification.
+ */
+#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF
+#define KVM_SDEI_DEFAULT_EVENT	0
+
+#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
+#define KVM_SDEI_MAX_EVENTS	128
+
+struct kvm_sdei_exposed_event_state {
+	__u64	num;
+
+	__u8	type;
+	__u8	signaled;
+	__u8	priority;
+	__u8	padding[5];
+	__u64	notifier;
+};
+
+struct kvm_sdei_registered_event_state {
+	__u64	num;
+
+	__u8	route_mode;
+	__u8	padding[3];
+	__u64	route_affinity;
+	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
+	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
+	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
+	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
+	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+};
+
+struct kvm_sdei_vcpu_event_state {
+	__u64	num;
+
+	__u32	event_count;
+	__u32	padding;
+};
+
+struct kvm_sdei_vcpu_regs_state {
+	__u64	regs[18];
+	__u64	pc;
+	__u64	pstate;
+};
+
+struct kvm_sdei_vcpu_state {
+	__u8				masked;
+	__u8				padding[7];
+	__u64				critical_num;
+	__u64				normal_num;
+	struct kvm_sdei_vcpu_regs_state	critical_regs;
+	struct kvm_sdei_vcpu_regs_state	normal_regs;
+};
+
+#endif /* !__ASSEMBLY__ */
+#endif /* _UAPI__ASM_KVM_SDEI_STATE_H */
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 91861fd8b897..a12903a21d1f 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o pkvm.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o sdei.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 4dca6ffd03d4..96fcae5beee4 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -38,6 +38,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 #include <asm/sections.h>
 
 #include <kvm/arm_hypercalls.h>
@@ -152,6 +153,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	kvm_vgic_early_init(kvm);
 
+	kvm_sdei_init_vm(kvm);
+
 	/* The maximum number of VCPUs is limited by the host's GIC model */
 	kvm->arch.max_vcpus = kvm_arm_default_max_vcpus();
 
@@ -179,6 +182,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 
 	kvm_vgic_destroy(kvm);
 
+	kvm_sdei_destroy_vm(kvm);
+
 	kvm_destroy_vcpus(kvm);
 
 	kvm_unshare_hyp(kvm, kvm + 1);
@@ -330,6 +335,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	kvm_arm_pvtime_vcpu_init(&vcpu->arch);
 
+	kvm_sdei_create_vcpu(vcpu);
+
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
 	err = kvm_vgic_vcpu_init(vcpu);
@@ -351,6 +358,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
 	kvm_timer_vcpu_terminate(vcpu);
 	kvm_pmu_vcpu_destroy(vcpu);
+	kvm_sdei_destroy_vcpu(vcpu);
 
 	kvm_arm_vcpu_destroy(vcpu);
 }
diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
index 202b8c455724..3c20fee72bb4 100644
--- a/arch/arm64/kvm/hypercalls.c
+++ b/arch/arm64/kvm/hypercalls.c
@@ -5,6 +5,7 @@
 #include <linux/kvm_host.h>
 
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 
 #include <kvm/arm_hypercalls.h>
 #include <kvm/arm_psci.h>
@@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 	case ARM_SMCCC_TRNG_RND32:
 	case ARM_SMCCC_TRNG_RND64:
 		return kvm_trng_call(vcpu);
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:
+		return kvm_sdei_hypercall(vcpu);
 	default:
 		return kvm_psci_call(vcpu);
 	}
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
new file mode 100644
index 000000000000..8a9b477b8977
--- /dev/null
+++ b/arch/arm64/kvm/sdei.c
@@ -0,0 +1,244 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDEI virtualization support.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/spinlock.h>
+#include <linux/slab.h>
+#include <kvm/arm_hypercalls.h>
+#include <asm/kvm_sdei.h>
+
+static void remove_all_exposed_events(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event, *tmp;
+
+	list_for_each_entry_safe(exposed_event, tmp,
+				 &ksdei->exposed_events, link) {
+		ksdei->exposed_event_count--;
+		list_del(&exposed_event->link);
+
+		kfree(exposed_event);
+	}
+}
+
+static void remove_one_registered_event(struct kvm *kvm,
+		struct kvm_sdei_registered_event *registered_event)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+
+	exposed_event = registered_event->exposed_event;
+	ksdei->registered_event_count--;
+	exposed_event->registered_event_count--;
+
+	list_del(&registered_event->link);
+	kfree(registered_event);
+}
+
+static void remove_all_registered_events(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event, *tmp;
+
+	list_for_each_entry_safe(registered_event, tmp,
+				 &ksdei->registered_events, link) {
+		remove_one_registered_event(kvm, registered_event);
+	}
+}
+
+static void remove_one_vcpu_event(struct kvm_vcpu *vcpu,
+				  struct kvm_sdei_vcpu_event *vcpu_event)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	registered_event->vcpu_event_count--;
+	if (kvm_sdei_is_critical(exposed_event->state.priority))
+		vsdei->critical_event_count--;
+	else
+		vsdei->normal_event_count--;
+
+	list_del(&vcpu_event->link);
+	kfree(vcpu_event);
+}
+
+static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
+				   unsigned long num)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *vcpu_event, *tmp;
+	bool pending = false;
+
+	list_for_each_entry_safe(vcpu_event, tmp,
+				 &vsdei->critical_events, link) {
+		if (!kvm_sdei_is_supported(num)) {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+			continue;
+		}
+
+		if (vcpu_event->state.num != num)
+			continue;
+
+		if (vsdei->critical_event == vcpu_event) {
+			vcpu_event->state.event_count = 1;
+			pending = true;
+		} else {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+		}
+	}
+
+	list_for_each_entry_safe(vcpu_event, tmp,
+				 &vsdei->normal_events, link) {
+		if (!kvm_sdei_is_supported(num)) {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+			continue;
+		}
+
+		if (vcpu_event->state.num != num)
+			continue;
+
+		if (vsdei->normal_event == vcpu_event) {
+			vcpu_event->state.event_count = 1;
+			pending = true;
+		} else {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+		}
+	}
+
+	return pending;
+}
+
+int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	u32 func = smccc_get_function(vcpu);
+	bool has_result = true;
+	unsigned long ret;
+
+	/*
+	 * We don't have return value for COMPLETE or COMPLETE_AND_RESUME
+	 * hypercalls. Otherwise, the restored context will be corrupted.
+	 */
+	if (func == SDEI_1_0_FN_SDEI_EVENT_COMPLETE ||
+	    func == SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME)
+		has_result = false;
+
+	if (!(ksdei && vsdei)) {
+		ret = SDEI_NOT_SUPPORTED;
+		goto out;
+	}
+
+	switch (func) {
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:
+	default:
+		ret = SDEI_NOT_SUPPORTED;
+	}
+
+out:
+	if (has_result)
+		smccc_set_retval(vcpu, ret, 0, 0, 0);
+
+	return 1;
+}
+
+void kvm_sdei_init_vm(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei;
+
+	ksdei = kzalloc(sizeof(*ksdei), GFP_KERNEL_ACCOUNT);
+	if (!ksdei)
+		return;
+
+	spin_lock_init(&ksdei->lock);
+	INIT_LIST_HEAD(&ksdei->exposed_events);
+	INIT_LIST_HEAD(&ksdei->registered_events);
+
+	kvm->arch.sdei = ksdei;
+}
+
+void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_vcpu *vsdei;
+
+	if (!kvm->arch.sdei)
+		return;
+
+	vsdei = kzalloc(sizeof(*vsdei), GFP_KERNEL_ACCOUNT);
+	if (!vsdei)
+		return;
+
+	spin_lock_init(&vsdei->lock);
+	vsdei->state.masked       = 1;
+	vsdei->state.critical_num = KVM_SDEI_INVALID_EVENT;
+	vsdei->state.normal_num   = KVM_SDEI_INVALID_EVENT;
+	vsdei->critical_event     = NULL;
+	vsdei->normal_event       = NULL;
+	INIT_LIST_HEAD(&vsdei->critical_events);
+	INIT_LIST_HEAD(&vsdei->normal_events);
+
+	vcpu->arch.sdei = vsdei;
+}
+
+void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+
+	if (ksdei && vsdei) {
+		spin_lock(&ksdei->lock);
+		spin_lock(&vsdei->lock);
+		remove_all_vcpu_events(vcpu, KVM_SDEI_INVALID_EVENT);
+		spin_unlock(&vsdei->lock);
+		spin_unlock(&ksdei->lock);
+
+		kfree(vsdei);
+		vcpu->arch.sdei = NULL;
+	}
+}
+
+void kvm_sdei_destroy_vm(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+
+	if (ksdei) {
+		spin_lock(&ksdei->lock);
+		remove_all_registered_events(kvm);
+		remove_all_exposed_events(kvm);
+		spin_unlock(&ksdei->lock);
+
+		kfree(ksdei);
+		kvm->arch.sdei = NULL;
+	}
+}
diff --git a/include/uapi/linux/arm_sdei.h b/include/uapi/linux/arm_sdei.h
index af0630ba5437..39bcf6dbbea0 100644
--- a/include/uapi/linux/arm_sdei.h
+++ b/include/uapi/linux/arm_sdei.h
@@ -22,8 +22,10 @@
 #define SDEI_1_0_FN_SDEI_PE_UNMASK			SDEI_1_0_FN(0x0C)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_BIND			SDEI_1_0_FN(0x0D)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE		SDEI_1_0_FN(0x0E)
+#define SDEI_1_1_FN_SDEI_EVENT_SIGNAL			SDEI_1_0_FN(0x0F)
 #define SDEI_1_0_FN_SDEI_PRIVATE_RESET			SDEI_1_0_FN(0x11)
 #define SDEI_1_0_FN_SDEI_SHARED_RESET			SDEI_1_0_FN(0x12)
+#define SDEI_1_1_FN_SDEI_FEATURES			SDEI_1_0_FN(0x30)
 
 #define SDEI_VERSION_MAJOR_SHIFT			48
 #define SDEI_VERSION_MAJOR_MASK				0x7fff
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Software Delegated Exception Interface (SDEI) provides a mechanism for
registering and servicing system events. Those system events are high
priority events, which must be serviced immediately. It's going to be
used by Asynchronous Page Fault (APF) to deliver notification from KVM
to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

This introduces SDEI virtualization infrastructure where the SDEI events
are registered and manipulated by the guest through hypercall. The SDEI
event is delivered to one specific vCPU by KVM once it's raised. This
introduces data structures to represent the needed objects to support
the feature, which is highlighted as below. As those objects could be
migrated between VMs, these data structures are partially exposed to
user space.

   * kvm_sdei_exposed_event
     The exposed events are determined and added by VMM through ioctl
     interface. Only the exposed events can be registered from the
     guest.

   * kvm_sdei_registered_event
     The events that have been registered from the guest through the
     SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.

   * kvm_sdei_vcpu_event
     The events that have been delivered to the target vCPU.

   * kvm_sdei_vcpu
     Used to save the preempted context when the SDEI event is serviced
     and delivered. After the SDEI event handling is completed, the
     execution is resumed from the preempted context.

   * kvm_sdei_kvm
     Place holder for the exposed and registered events.

The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for
now. They will be supported in the subsequent patches.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_host.h            |   3 +
 arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
 arch/arm64/include/uapi/asm/kvm.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
 arch/arm64/kvm/Makefile                      |   2 +-
 arch/arm64/kvm/arm.c                         |   8 +
 arch/arm64/kvm/hypercalls.c                  |  21 ++
 arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
 include/uapi/linux/arm_sdei.h                |   2 +
 9 files changed, 523 insertions(+), 1 deletion(-)
 create mode 100644 arch/arm64/include/asm/kvm_sdei.h
 create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
 create mode 100644 arch/arm64/kvm/sdei.c

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 031e3a2537fc..5d37e046a458 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -113,6 +113,8 @@ struct kvm_arch {
 	/* Interrupt controller */
 	struct vgic_dist	vgic;
 
+	struct kvm_sdei_kvm *sdei;
+
 	/* Mandated version of PSCI */
 	u32 psci_version;
 
@@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
 	 * Anything that is not used directly from assembly code goes
 	 * here.
 	 */
+	struct kvm_sdei_vcpu *sdei;
 
 	/*
 	 * Guest registers we preserve during guest debugging.
diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
new file mode 100644
index 000000000000..6f58a846d05c
--- /dev/null
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -0,0 +1,171 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * Definitions of various KVM SDEI events.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#ifndef __ARM64_KVM_SDEI_H__
+#define __ARM64_KVM_SDEI_H__
+
+#include <uapi/linux/arm_sdei.h>
+#include <uapi/asm/kvm_sdei_state.h>
+#include <linux/bitmap.h>
+#include <linux/list.h>
+#include <linux/spinlock.h>
+
+struct kvm_vcpu;
+
+struct kvm_sdei_exposed_event {
+	struct kvm_sdei_exposed_event_state	state;
+	struct kvm				*kvm;
+	unsigned int				registered_event_count;
+	struct list_head			link;
+};
+
+struct kvm_sdei_registered_event {
+	struct kvm_sdei_registered_event_state	state;
+	struct kvm				*kvm;
+	struct kvm_sdei_exposed_event		*exposed_event;
+	unsigned int				vcpu_event_count;
+	struct list_head			link;
+};
+
+struct kvm_sdei_vcpu_event {
+	struct kvm_sdei_vcpu_event_state	state;
+	struct kvm_vcpu				*vcpu;
+	struct kvm_sdei_registered_event	*registered_event;
+	struct list_head			link;
+};
+
+struct kvm_sdei_kvm {
+	spinlock_t		lock;
+	unsigned int		exposed_event_count;
+	unsigned int		registered_event_count;
+	struct list_head	exposed_events;
+	struct list_head	registered_events;
+};
+
+struct kvm_sdei_vcpu {
+	spinlock_t                      lock;
+	struct kvm_sdei_vcpu_state      state;
+	struct kvm_sdei_vcpu_event      *critical_event;
+	struct kvm_sdei_vcpu_event      *normal_event;
+	unsigned int			critical_event_count;
+	unsigned int			normal_event_count;
+	struct list_head                critical_events;
+	struct list_head                normal_events;
+};
+
+/*
+ * According to SDEI specification (v1.1), the event number spans 32-bits
+ * and the lower 24-bits are used as the (real) event number. I don't
+ * think we can use that much SDEI numbers in one system. So we reserve
+ * two bits from the 24-bits real event number, to indicate its types:
+ * physical or virtual event. One reserved bit is enough for now, but
+ * two bits are reserved for possible extension in future.
+ *
+ * The physical events are owned by firmware while the virtual events
+ * are used by VMM and KVM.
+ */
+#define KVM_SDEI_EVENT_NUM_TYPE_SHIFT	22
+#define KVM_SDEI_EVENT_NUM_TYPE_MASK	(3UL << KVM_SDEI_EVENT_NUM_TYPE_SHIFT)
+#define KVM_SDEI_EVENT_NUM_TYPE_PHYS	0
+#define KVM_SDEI_EVENT_NUM_TYPE_VIRT	1
+
+static inline bool kvm_sdei_is_virtual(unsigned long num)
+{
+	unsigned long type;
+
+	if (num >> 32)
+		return false;
+
+	type = (num & KVM_SDEI_EVENT_NUM_TYPE_MASK) >>
+		KVM_SDEI_EVENT_NUM_TYPE_SHIFT;
+	if (type != KVM_SDEI_EVENT_NUM_TYPE_VIRT)
+		return false;
+
+	return true;
+}
+
+static inline bool kvm_sdei_is_default(unsigned long num)
+{
+	return num == KVM_SDEI_DEFAULT_EVENT;
+}
+
+static inline bool kvm_sdei_is_supported(unsigned long num)
+{
+	return kvm_sdei_is_default(num) || kvm_sdei_is_virtual(num);
+}
+
+static inline bool kvm_sdei_is_shared(unsigned char type)
+{
+	return type == SDEI_EVENT_TYPE_SHARED;
+}
+
+static inline bool kvm_sdei_is_private(unsigned char type)
+{
+	return type == SDEI_EVENT_TYPE_PRIVATE;
+}
+
+static inline bool kvm_sdei_is_critical(unsigned char priority)
+{
+	return priority == SDEI_EVENT_PRIORITY_CRITICAL;
+}
+
+static inline bool kvm_sdei_is_normal(unsigned char priority)
+{
+	return priority == SDEI_EVENT_PRIORITY_NORMAL;
+}
+
+static inline unsigned int
+kvm_sdei_vcpu_index(struct kvm_vcpu *vcpu,
+		    struct kvm_sdei_exposed_event *event)
+{
+	return kvm_sdei_is_private(event->state.type) ? vcpu->vcpu_idx : 0;
+}
+
+/* Accessors for the registered event */
+#define KVM_SDEI_REGISTERED_EVENT_FUNC(field)				\
+static inline bool							\
+kvm_sdei_is_##field(struct kvm_sdei_registered_event *event,		\
+		    unsigned int index)					\
+{									\
+	return !!test_bit(index, (void *)(event->state.field));		\
+}									\
+									\
+static inline bool							\
+kvm_sdei_none_##field(struct kvm_sdei_registered_event *event)		\
+{									\
+	return bitmap_empty((void *)(event->state.field),		\
+			    KVM_SDEI_MAX_VCPUS);			\
+}									\
+									\
+static inline void							\
+kvm_sdei_set_##field(struct kvm_sdei_registered_event *event,		\
+		     unsigned int index)				\
+{									\
+	set_bit(index, (void *)(event->state.field));			\
+}									\
+									\
+static inline void							\
+kvm_sdei_clear_##field(struct kvm_sdei_registered_event *event,		\
+		       unsigned int index)				\
+{									\
+	clear_bit(index, (void *)(event->state.field));			\
+}
+
+KVM_SDEI_REGISTERED_EVENT_FUNC(registered)
+KVM_SDEI_REGISTERED_EVENT_FUNC(enabled)
+KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
+
+/* APIs */
+void kvm_sdei_init_vm(struct kvm *kvm);
+void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
+int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
+void kvm_sdei_destroy_vm(struct kvm *kvm);
+
+#endif /* __ARM64_KVM_SDEI_H__ */
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index 323e251ed37b..33ee95bc088e 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -36,6 +36,7 @@
 #include <linux/types.h>
 #include <asm/ptrace.h>
 #include <asm/sve_context.h>
+#include <asm/kvm_sdei_state.h>
 
 #define __KVM_HAVE_GUEST_DEBUG
 #define __KVM_HAVE_IRQ_LINE
diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
new file mode 100644
index 000000000000..b14844230117
--- /dev/null
+++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
@@ -0,0 +1,72 @@
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Definitions of various KVM SDEI event states.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
+#define _UAPI__ASM_KVM_SDEI_STATE_H
+
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
+/*
+ * The software signaled event is the default one, which is
+ * defined in v1.1 specification.
+ */
+#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF
+#define KVM_SDEI_DEFAULT_EVENT	0
+
+#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
+#define KVM_SDEI_MAX_EVENTS	128
+
+struct kvm_sdei_exposed_event_state {
+	__u64	num;
+
+	__u8	type;
+	__u8	signaled;
+	__u8	priority;
+	__u8	padding[5];
+	__u64	notifier;
+};
+
+struct kvm_sdei_registered_event_state {
+	__u64	num;
+
+	__u8	route_mode;
+	__u8	padding[3];
+	__u64	route_affinity;
+	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
+	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
+	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
+	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
+	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+};
+
+struct kvm_sdei_vcpu_event_state {
+	__u64	num;
+
+	__u32	event_count;
+	__u32	padding;
+};
+
+struct kvm_sdei_vcpu_regs_state {
+	__u64	regs[18];
+	__u64	pc;
+	__u64	pstate;
+};
+
+struct kvm_sdei_vcpu_state {
+	__u8				masked;
+	__u8				padding[7];
+	__u64				critical_num;
+	__u64				normal_num;
+	struct kvm_sdei_vcpu_regs_state	critical_regs;
+	struct kvm_sdei_vcpu_regs_state	normal_regs;
+};
+
+#endif /* !__ASSEMBLY__ */
+#endif /* _UAPI__ASM_KVM_SDEI_STATE_H */
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 91861fd8b897..a12903a21d1f 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -14,7 +14,7 @@ kvm-y += arm.o mmu.o mmio.o psci.o hypercalls.o pvtime.o \
 	 inject_fault.o va_layout.o handle_exit.o \
 	 guest.o debug.o reset.o sys_regs.o \
 	 vgic-sys-reg-v3.o fpsimd.o pmu.o pkvm.o \
-	 arch_timer.o trng.o\
+	 arch_timer.o trng.o sdei.o \
 	 vgic/vgic.o vgic/vgic-init.o \
 	 vgic/vgic-irqfd.o vgic/vgic-v2.o \
 	 vgic/vgic-v3.o vgic/vgic-v4.o \
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 4dca6ffd03d4..96fcae5beee4 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -38,6 +38,7 @@
 #include <asm/kvm_asm.h>
 #include <asm/kvm_mmu.h>
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 #include <asm/sections.h>
 
 #include <kvm/arm_hypercalls.h>
@@ -152,6 +153,8 @@ int kvm_arch_init_vm(struct kvm *kvm, unsigned long type)
 
 	kvm_vgic_early_init(kvm);
 
+	kvm_sdei_init_vm(kvm);
+
 	/* The maximum number of VCPUs is limited by the host's GIC model */
 	kvm->arch.max_vcpus = kvm_arm_default_max_vcpus();
 
@@ -179,6 +182,8 @@ void kvm_arch_destroy_vm(struct kvm *kvm)
 
 	kvm_vgic_destroy(kvm);
 
+	kvm_sdei_destroy_vm(kvm);
+
 	kvm_destroy_vcpus(kvm);
 
 	kvm_unshare_hyp(kvm, kvm + 1);
@@ -330,6 +335,8 @@ int kvm_arch_vcpu_create(struct kvm_vcpu *vcpu)
 
 	kvm_arm_pvtime_vcpu_init(&vcpu->arch);
 
+	kvm_sdei_create_vcpu(vcpu);
+
 	vcpu->arch.hw_mmu = &vcpu->kvm->arch.mmu;
 
 	err = kvm_vgic_vcpu_init(vcpu);
@@ -351,6 +358,7 @@ void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
 	kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
 	kvm_timer_vcpu_terminate(vcpu);
 	kvm_pmu_vcpu_destroy(vcpu);
+	kvm_sdei_destroy_vcpu(vcpu);
 
 	kvm_arm_vcpu_destroy(vcpu);
 }
diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
index 202b8c455724..3c20fee72bb4 100644
--- a/arch/arm64/kvm/hypercalls.c
+++ b/arch/arm64/kvm/hypercalls.c
@@ -5,6 +5,7 @@
 #include <linux/kvm_host.h>
 
 #include <asm/kvm_emulate.h>
+#include <asm/kvm_sdei.h>
 
 #include <kvm/arm_hypercalls.h>
 #include <kvm/arm_psci.h>
@@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
 	case ARM_SMCCC_TRNG_RND32:
 	case ARM_SMCCC_TRNG_RND64:
 		return kvm_trng_call(vcpu);
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:
+		return kvm_sdei_hypercall(vcpu);
 	default:
 		return kvm_psci_call(vcpu);
 	}
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
new file mode 100644
index 000000000000..8a9b477b8977
--- /dev/null
+++ b/arch/arm64/kvm/sdei.c
@@ -0,0 +1,244 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SDEI virtualization support.
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/kvm_host.h>
+#include <linux/spinlock.h>
+#include <linux/slab.h>
+#include <kvm/arm_hypercalls.h>
+#include <asm/kvm_sdei.h>
+
+static void remove_all_exposed_events(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event, *tmp;
+
+	list_for_each_entry_safe(exposed_event, tmp,
+				 &ksdei->exposed_events, link) {
+		ksdei->exposed_event_count--;
+		list_del(&exposed_event->link);
+
+		kfree(exposed_event);
+	}
+}
+
+static void remove_one_registered_event(struct kvm *kvm,
+		struct kvm_sdei_registered_event *registered_event)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+
+	exposed_event = registered_event->exposed_event;
+	ksdei->registered_event_count--;
+	exposed_event->registered_event_count--;
+
+	list_del(&registered_event->link);
+	kfree(registered_event);
+}
+
+static void remove_all_registered_events(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event, *tmp;
+
+	list_for_each_entry_safe(registered_event, tmp,
+				 &ksdei->registered_events, link) {
+		remove_one_registered_event(kvm, registered_event);
+	}
+}
+
+static void remove_one_vcpu_event(struct kvm_vcpu *vcpu,
+				  struct kvm_sdei_vcpu_event *vcpu_event)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	registered_event->vcpu_event_count--;
+	if (kvm_sdei_is_critical(exposed_event->state.priority))
+		vsdei->critical_event_count--;
+	else
+		vsdei->normal_event_count--;
+
+	list_del(&vcpu_event->link);
+	kfree(vcpu_event);
+}
+
+static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
+				   unsigned long num)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *vcpu_event, *tmp;
+	bool pending = false;
+
+	list_for_each_entry_safe(vcpu_event, tmp,
+				 &vsdei->critical_events, link) {
+		if (!kvm_sdei_is_supported(num)) {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+			continue;
+		}
+
+		if (vcpu_event->state.num != num)
+			continue;
+
+		if (vsdei->critical_event == vcpu_event) {
+			vcpu_event->state.event_count = 1;
+			pending = true;
+		} else {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+		}
+	}
+
+	list_for_each_entry_safe(vcpu_event, tmp,
+				 &vsdei->normal_events, link) {
+		if (!kvm_sdei_is_supported(num)) {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+			continue;
+		}
+
+		if (vcpu_event->state.num != num)
+			continue;
+
+		if (vsdei->normal_event == vcpu_event) {
+			vcpu_event->state.event_count = 1;
+			pending = true;
+		} else {
+			remove_one_vcpu_event(vcpu, vcpu_event);
+		}
+	}
+
+	return pending;
+}
+
+int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	u32 func = smccc_get_function(vcpu);
+	bool has_result = true;
+	unsigned long ret;
+
+	/*
+	 * We don't have return value for COMPLETE or COMPLETE_AND_RESUME
+	 * hypercalls. Otherwise, the restored context will be corrupted.
+	 */
+	if (func == SDEI_1_0_FN_SDEI_EVENT_COMPLETE ||
+	    func == SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME)
+		has_result = false;
+
+	if (!(ksdei && vsdei)) {
+		ret = SDEI_NOT_SUPPORTED;
+		goto out;
+	}
+
+	switch (func) {
+	case SDEI_1_0_FN_SDEI_VERSION:
+	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+	case SDEI_1_0_FN_SDEI_PE_MASK:
+	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
+	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+	case SDEI_1_1_FN_SDEI_FEATURES:
+	default:
+		ret = SDEI_NOT_SUPPORTED;
+	}
+
+out:
+	if (has_result)
+		smccc_set_retval(vcpu, ret, 0, 0, 0);
+
+	return 1;
+}
+
+void kvm_sdei_init_vm(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei;
+
+	ksdei = kzalloc(sizeof(*ksdei), GFP_KERNEL_ACCOUNT);
+	if (!ksdei)
+		return;
+
+	spin_lock_init(&ksdei->lock);
+	INIT_LIST_HEAD(&ksdei->exposed_events);
+	INIT_LIST_HEAD(&ksdei->registered_events);
+
+	kvm->arch.sdei = ksdei;
+}
+
+void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_vcpu *vsdei;
+
+	if (!kvm->arch.sdei)
+		return;
+
+	vsdei = kzalloc(sizeof(*vsdei), GFP_KERNEL_ACCOUNT);
+	if (!vsdei)
+		return;
+
+	spin_lock_init(&vsdei->lock);
+	vsdei->state.masked       = 1;
+	vsdei->state.critical_num = KVM_SDEI_INVALID_EVENT;
+	vsdei->state.normal_num   = KVM_SDEI_INVALID_EVENT;
+	vsdei->critical_event     = NULL;
+	vsdei->normal_event       = NULL;
+	INIT_LIST_HEAD(&vsdei->critical_events);
+	INIT_LIST_HEAD(&vsdei->normal_events);
+
+	vcpu->arch.sdei = vsdei;
+}
+
+void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+
+	if (ksdei && vsdei) {
+		spin_lock(&ksdei->lock);
+		spin_lock(&vsdei->lock);
+		remove_all_vcpu_events(vcpu, KVM_SDEI_INVALID_EVENT);
+		spin_unlock(&vsdei->lock);
+		spin_unlock(&ksdei->lock);
+
+		kfree(vsdei);
+		vcpu->arch.sdei = NULL;
+	}
+}
+
+void kvm_sdei_destroy_vm(struct kvm *kvm)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+
+	if (ksdei) {
+		spin_lock(&ksdei->lock);
+		remove_all_registered_events(kvm);
+		remove_all_exposed_events(kvm);
+		spin_unlock(&ksdei->lock);
+
+		kfree(ksdei);
+		kvm->arch.sdei = NULL;
+	}
+}
diff --git a/include/uapi/linux/arm_sdei.h b/include/uapi/linux/arm_sdei.h
index af0630ba5437..39bcf6dbbea0 100644
--- a/include/uapi/linux/arm_sdei.h
+++ b/include/uapi/linux/arm_sdei.h
@@ -22,8 +22,10 @@
 #define SDEI_1_0_FN_SDEI_PE_UNMASK			SDEI_1_0_FN(0x0C)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_BIND			SDEI_1_0_FN(0x0D)
 #define SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE		SDEI_1_0_FN(0x0E)
+#define SDEI_1_1_FN_SDEI_EVENT_SIGNAL			SDEI_1_0_FN(0x0F)
 #define SDEI_1_0_FN_SDEI_PRIVATE_RESET			SDEI_1_0_FN(0x11)
 #define SDEI_1_0_FN_SDEI_SHARED_RESET			SDEI_1_0_FN(0x12)
+#define SDEI_1_1_FN_SDEI_FEATURES			SDEI_1_0_FN(0x30)
 
 #define SDEI_VERSION_MAJOR_SHIFT			48
 #define SDEI_VERSION_MAJOR_MASK				0x7fff
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_VERSION hypercall by returning v1.1, which is
the specification version we're following. The vendor is set to
'KVM'.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 8a9b477b8977..5a3a64cd6e84 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
 	return pending;
 }
 
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -142,6 +150,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 
 	switch (func) {
 	case SDEI_1_0_FN_SDEI_VERSION:
+		ret = hypercall_version(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_VERSION hypercall by returning v1.1, which is
the specification version we're following. The vendor is set to
'KVM'.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 8a9b477b8977..5a3a64cd6e84 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
 	return pending;
 }
 
+static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
+{
+	/* v1.1 and the vendor is KVM */
+	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
+	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
+	       0x4b564d;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -142,6 +150,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 
 	switch (func) {
 	case SDEI_1_0_FN_SDEI_VERSION:
+		ret = hypercall_version(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 04/22] KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_REGISTER hypercall, which is used by guest
to register SDEI events. The SDEI event won't be raised until it's
registered and enabled explicitly.

Only the exposed events can be registered. For shared event, the
registered event instance is created. However, the instance may be
not created for the private events.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 128 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 5a3a64cd6e84..2458dc666445 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -14,6 +14,35 @@
 #include <kvm/arm_hypercalls.h>
 #include <asm/kvm_sdei.h>
 
+static struct kvm_sdei_exposed_event *
+find_exposed_event(struct kvm *kvm, unsigned long num)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+
+	list_for_each_entry(exposed_event, &ksdei->exposed_events, link) {
+		if (exposed_event->state.num == num)
+			return exposed_event;
+	}
+
+	return NULL;
+}
+
+static struct kvm_sdei_registered_event *
+find_registered_event(struct kvm *kvm, unsigned long num)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event;
+
+	list_for_each_entry(registered_event,
+			    &ksdei->registered_events, link) {
+		if (registered_event->state.num == num)
+			return registered_event;
+	}
+
+	return NULL;
+}
+
 static void remove_all_exposed_events(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
@@ -126,6 +155,103 @@ static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
 	       0x4b564d;
 }
 
+static unsigned long hypercall_register(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long event_ep_address = smccc_get_arg2(vcpu);
+	unsigned long event_ep_arg = smccc_get_arg3(vcpu);
+	unsigned long route_mode = smccc_get_arg4(vcpu);
+	unsigned long route_affinity = smccc_get_arg5(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	if (route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+	    route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * The registered event could have been existing if it's a private
+	 * one. We needn't to create another registered event instance
+	 * in this case.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	if (registered_event) {
+		exposed_event = registered_event->exposed_event;
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (kvm_sdei_is_registered(registered_event, index) ||
+		    kvm_sdei_is_unregister_pending(registered_event, index)) {
+			ret = SDEI_DENIED;
+			goto unlock;
+		}
+
+		registered_event->state.route_mode        = route_mode;
+		registered_event->state.route_affinity    = route_affinity;
+		registered_event->state.ep_address[index] = event_ep_address;
+		registered_event->state.ep_arg[index]     = event_ep_arg;
+		kvm_sdei_set_registered(registered_event, index);
+		goto unlock;
+	}
+
+	/* Check if the exposed event exists */
+	exposed_event = find_exposed_event(kvm, event_num);
+	if (!exposed_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/*
+	 * Check if the count of registered event instances exceeds
+	 * the limit.
+	 */
+	if (ksdei->registered_event_count >= KVM_SDEI_MAX_EVENTS) {
+		ret = SDEI_OUT_OF_RESOURCE;
+		goto unlock;
+	}
+
+	/* Allocate the registered event instance */
+	registered_event = kzalloc(sizeof(*registered_event),
+				   GFP_KERNEL_ACCOUNT);
+	if (!registered_event) {
+		ret = SDEI_OUT_OF_RESOURCE;
+		goto unlock;
+	}
+
+	/* Initialize the registered event state */
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	registered_event->state.num               = event_num;
+	registered_event->state.route_mode        = route_affinity;
+	registered_event->state.route_affinity    = route_affinity;
+	registered_event->state.ep_address[index] = event_ep_address;
+	registered_event->state.ep_arg[index]     = event_ep_arg;
+	registered_event->kvm                     = kvm;
+	registered_event->exposed_event           = exposed_event;
+	registered_event->vcpu_event_count        = 0;
+	kvm_sdei_set_registered(registered_event, index);
+
+	/* Add the registered event instance */
+	ksdei->registered_event_count++;
+	exposed_event->registered_event_count++;
+	list_add_tail(&registered_event->link, &ksdei->registered_events);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -153,6 +279,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_version(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+		ret = hypercall_register(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 04/22] KVM: arm64: Support SDEI_EVENT_REGISTER hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_REGISTER hypercall, which is used by guest
to register SDEI events. The SDEI event won't be raised until it's
registered and enabled explicitly.

Only the exposed events can be registered. For shared event, the
registered event instance is created. However, the instance may be
not created for the private events.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 128 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 128 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 5a3a64cd6e84..2458dc666445 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -14,6 +14,35 @@
 #include <kvm/arm_hypercalls.h>
 #include <asm/kvm_sdei.h>
 
+static struct kvm_sdei_exposed_event *
+find_exposed_event(struct kvm *kvm, unsigned long num)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+
+	list_for_each_entry(exposed_event, &ksdei->exposed_events, link) {
+		if (exposed_event->state.num == num)
+			return exposed_event;
+	}
+
+	return NULL;
+}
+
+static struct kvm_sdei_registered_event *
+find_registered_event(struct kvm *kvm, unsigned long num)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event;
+
+	list_for_each_entry(registered_event,
+			    &ksdei->registered_events, link) {
+		if (registered_event->state.num == num)
+			return registered_event;
+	}
+
+	return NULL;
+}
+
 static void remove_all_exposed_events(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
@@ -126,6 +155,103 @@ static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
 	       0x4b564d;
 }
 
+static unsigned long hypercall_register(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long event_ep_address = smccc_get_arg2(vcpu);
+	unsigned long event_ep_arg = smccc_get_arg3(vcpu);
+	unsigned long route_mode = smccc_get_arg4(vcpu);
+	unsigned long route_affinity = smccc_get_arg5(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	if (route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+	    route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * The registered event could have been existing if it's a private
+	 * one. We needn't to create another registered event instance
+	 * in this case.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	if (registered_event) {
+		exposed_event = registered_event->exposed_event;
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (kvm_sdei_is_registered(registered_event, index) ||
+		    kvm_sdei_is_unregister_pending(registered_event, index)) {
+			ret = SDEI_DENIED;
+			goto unlock;
+		}
+
+		registered_event->state.route_mode        = route_mode;
+		registered_event->state.route_affinity    = route_affinity;
+		registered_event->state.ep_address[index] = event_ep_address;
+		registered_event->state.ep_arg[index]     = event_ep_arg;
+		kvm_sdei_set_registered(registered_event, index);
+		goto unlock;
+	}
+
+	/* Check if the exposed event exists */
+	exposed_event = find_exposed_event(kvm, event_num);
+	if (!exposed_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/*
+	 * Check if the count of registered event instances exceeds
+	 * the limit.
+	 */
+	if (ksdei->registered_event_count >= KVM_SDEI_MAX_EVENTS) {
+		ret = SDEI_OUT_OF_RESOURCE;
+		goto unlock;
+	}
+
+	/* Allocate the registered event instance */
+	registered_event = kzalloc(sizeof(*registered_event),
+				   GFP_KERNEL_ACCOUNT);
+	if (!registered_event) {
+		ret = SDEI_OUT_OF_RESOURCE;
+		goto unlock;
+	}
+
+	/* Initialize the registered event state */
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	registered_event->state.num               = event_num;
+	registered_event->state.route_mode        = route_affinity;
+	registered_event->state.route_affinity    = route_affinity;
+	registered_event->state.ep_address[index] = event_ep_address;
+	registered_event->state.ep_arg[index]     = event_ep_arg;
+	registered_event->kvm                     = kvm;
+	registered_event->exposed_event           = exposed_event;
+	registered_event->vcpu_event_count        = 0;
+	kvm_sdei_set_registered(registered_event, index);
+
+	/* Add the registered event instance */
+	ksdei->registered_event_count++;
+	exposed_event->registered_event_count++;
+	list_add_tail(&registered_event->link, &ksdei->registered_events);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -153,6 +279,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_version(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+		ret = hypercall_register(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 05/22] KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_{ENABLE, DISABLE} hypercall. After SDEI
event is registered by guest, it won't be delivered to the guest
until it's enabled. For unregistration pending event, we can't
enable or disable it as the registered event is going to be
destroyed after current event is handled.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 2458dc666445..4ab58f264992 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -252,6 +252,51 @@ static unsigned long hypercall_register(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Check if the event is registered and pending for unregistration */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Update the enablement state */
+	if (enable)
+		kvm_sdei_set_enabled(registered_event, index);
+	else
+		kvm_sdei_clear_enabled(registered_event, index);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -282,7 +327,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_register(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+		ret = hypercall_enable(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+		ret = hypercall_enable(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 05/22] KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_{ENABLE, DISABLE} hypercall. After SDEI
event is registered by guest, it won't be delivered to the guest
until it's enabled. For unregistration pending event, we can't
enable or disable it as the registered event is going to be
destroyed after current event is handled.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 49 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 49 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 2458dc666445..4ab58f264992 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -252,6 +252,51 @@ static unsigned long hypercall_register(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Check if the event is registered and pending for unregistration */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Update the enablement state */
+	if (enable)
+		kvm_sdei_set_enabled(registered_event, index);
+	else
+		kvm_sdei_clear_enabled(registered_event, index);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -282,7 +327,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_register(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+		ret = hypercall_enable(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+		ret = hypercall_enable(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 06/22] KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_CONTEXT hypercall. It's used by the guest
to retrieve the registers (R0 - R17) in the preempted context
in the SDEI event handler. The preempted context is saved prior to
servicing or handling the SDEI event and restored after that.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 4ab58f264992..4488d3f044f2 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -297,6 +297,34 @@ static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)
 	return ret;
 }
 
+static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long param_id = smccc_get_arg1(vcpu);
+	unsigned long ret = SDEI_SUCCESS;
+
+	spin_lock(&vsdei->lock);
+
+	/* Check if the pending event exists */
+	if (!vsdei->critical_event && !vsdei->normal_event) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Fetch the requested register */
+	regs = vsdei->critical_event ? &vsdei->state.critical_regs :
+				       &vsdei->state.normal_regs;
+	if (param_id < ARRAY_SIZE(regs->regs))
+		ret = regs->regs[param_id];
+	else
+		ret = SDEI_INVALID_PARAMETERS;
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -333,6 +361,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_enable(vcpu, false);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+		ret = hypercall_context(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 06/22] KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_CONTEXT hypercall. It's used by the guest
to retrieve the registers (R0 - R17) in the preempted context
in the SDEI event handler. The preempted context is saved prior to
servicing or handling the SDEI event and restored after that.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 4ab58f264992..4488d3f044f2 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -297,6 +297,34 @@ static unsigned long hypercall_enable(struct kvm_vcpu *vcpu, bool enable)
 	return ret;
 }
 
+static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long param_id = smccc_get_arg1(vcpu);
+	unsigned long ret = SDEI_SUCCESS;
+
+	spin_lock(&vsdei->lock);
+
+	/* Check if the pending event exists */
+	if (!vsdei->critical_event && !vsdei->normal_event) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Fetch the requested register */
+	regs = vsdei->critical_event ? &vsdei->state.critical_regs :
+				       &vsdei->state.normal_regs;
+	if (param_id < ARRAY_SIZE(regs->regs))
+		ret = regs->regs[param_id];
+	else
+		ret = SDEI_INVALID_PARAMETERS;
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -333,6 +361,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_enable(vcpu, false);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
+		ret = hypercall_context(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 07/22] KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_UNREGISTER hypercall. It's used by the
guest to unregister SDEI event. The SDEI event won't be raised to
the guest after it's unregistered. The SDEI event is disabled
automatically on unregistration.

The currently handled events can't be unregistered. We set the
unregistration pending state for the event so that it can be
unregistered when the event handler is completed by receiving
SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 133 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 4488d3f044f2..36eda31e0392 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -325,6 +325,135 @@ static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long
+unregister_one_event(struct kvm *kvm, struct kvm_vcpu *vcpu,
+		     struct kvm_sdei_registered_event *registered_event)
+{
+	struct kvm_vcpu *vcpup;
+	struct kvm_sdei_vcpu *vsdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	int index;
+	bool pending;
+	unsigned long i, ret = SDEI_SUCCESS;
+
+	/*
+	 * Cancel all vcpu events that have been queued, but not dispatched
+	 * yet. If the vcpu event has been dispatched, we should mark it
+	 * as pending for unregistration. The unregistration will be executed
+	 * when the event handler is to be completed.
+	 */
+	exposed_event = registered_event->exposed_event;
+	kvm_for_each_vcpu(i, vcpup, kvm) {
+		vsdei = vcpup->arch.sdei;
+		if (!vsdei)
+			continue;
+
+		/*
+		 * The private vcpu events are requested to be unregistered
+		 * on the specific vcpu or all vcpus. @vcpu is used to
+		 * identify the cases. For the shared vcpu events, we need
+		 * to unregister them on all vcpus.
+		 */
+		if (kvm_sdei_is_private(exposed_event->state.type) &&
+		    vcpu && vcpu != vcpup)
+			continue;
+
+		if (registered_event->vcpu_event_count > 0) {
+			spin_lock(&vsdei->lock);
+			pending = remove_all_vcpu_events(vcpup,
+					registered_event->state.num);
+			spin_unlock(&vsdei->lock);
+		} else {
+			pending = false;
+		}
+
+		/*
+		 * For the private event, the pending state for unregistration
+		 * is scattered and we need to update them individually.
+		 * However, that same state for the shared event has to be
+		 * updated at once according to @ret after the iteration is
+		 * done.
+		 */
+		ret = pending ? SDEI_PENDING : ret;
+		if (!kvm_sdei_is_private(exposed_event->state.type))
+			continue;
+
+		index = kvm_sdei_vcpu_index(vcpup, exposed_event);
+		if (pending) {
+			kvm_sdei_set_unregister_pending(registered_event,
+							index);
+		} else {
+			kvm_sdei_clear_enabled(registered_event, index);
+			kvm_sdei_clear_registered(registered_event, index);
+		}
+	}
+
+	/*
+	 * Update the pending state for unregistration for the shared event
+	 * at once.
+	 */
+	if (kvm_sdei_is_shared(exposed_event->state.type)) {
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (ret == SDEI_PENDING) {
+			kvm_sdei_set_unregister_pending(registered_event,
+							index);
+		} else {
+			kvm_sdei_clear_enabled(registered_event, index);
+			kvm_sdei_clear_registered(registered_event, index);
+		}
+	}
+
+	/* Destroy the registered event instance if needed */
+	if (kvm_sdei_none_registered(registered_event))
+		remove_one_registered_event(kvm, registered_event);
+
+	return ret;
+}
+
+static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num))
+		return SDEI_INVALID_PARAMETERS;
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Check if the event has been registered */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Check if the event has been pending for unregistration */
+	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_PENDING;
+		goto unlock;
+	}
+
+	ret = unregister_one_event(kvm, vcpu, registered_event);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -365,7 +494,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+		ret = SDEI_NOT_SUPPORTED;
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+		ret = hypercall_unregister(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 07/22] KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_UNREGISTER hypercall. It's used by the
guest to unregister SDEI event. The SDEI event won't be raised to
the guest after it's unregistered. The SDEI event is disabled
automatically on unregistration.

The currently handled events can't be unregistered. We set the
unregistration pending state for the event so that it can be
unregistered when the event handler is completed by receiving
SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 133 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 133 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 4488d3f044f2..36eda31e0392 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -325,6 +325,135 @@ static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long
+unregister_one_event(struct kvm *kvm, struct kvm_vcpu *vcpu,
+		     struct kvm_sdei_registered_event *registered_event)
+{
+	struct kvm_vcpu *vcpup;
+	struct kvm_sdei_vcpu *vsdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	int index;
+	bool pending;
+	unsigned long i, ret = SDEI_SUCCESS;
+
+	/*
+	 * Cancel all vcpu events that have been queued, but not dispatched
+	 * yet. If the vcpu event has been dispatched, we should mark it
+	 * as pending for unregistration. The unregistration will be executed
+	 * when the event handler is to be completed.
+	 */
+	exposed_event = registered_event->exposed_event;
+	kvm_for_each_vcpu(i, vcpup, kvm) {
+		vsdei = vcpup->arch.sdei;
+		if (!vsdei)
+			continue;
+
+		/*
+		 * The private vcpu events are requested to be unregistered
+		 * on the specific vcpu or all vcpus. @vcpu is used to
+		 * identify the cases. For the shared vcpu events, we need
+		 * to unregister them on all vcpus.
+		 */
+		if (kvm_sdei_is_private(exposed_event->state.type) &&
+		    vcpu && vcpu != vcpup)
+			continue;
+
+		if (registered_event->vcpu_event_count > 0) {
+			spin_lock(&vsdei->lock);
+			pending = remove_all_vcpu_events(vcpup,
+					registered_event->state.num);
+			spin_unlock(&vsdei->lock);
+		} else {
+			pending = false;
+		}
+
+		/*
+		 * For the private event, the pending state for unregistration
+		 * is scattered and we need to update them individually.
+		 * However, that same state for the shared event has to be
+		 * updated at once according to @ret after the iteration is
+		 * done.
+		 */
+		ret = pending ? SDEI_PENDING : ret;
+		if (!kvm_sdei_is_private(exposed_event->state.type))
+			continue;
+
+		index = kvm_sdei_vcpu_index(vcpup, exposed_event);
+		if (pending) {
+			kvm_sdei_set_unregister_pending(registered_event,
+							index);
+		} else {
+			kvm_sdei_clear_enabled(registered_event, index);
+			kvm_sdei_clear_registered(registered_event, index);
+		}
+	}
+
+	/*
+	 * Update the pending state for unregistration for the shared event
+	 * at once.
+	 */
+	if (kvm_sdei_is_shared(exposed_event->state.type)) {
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (ret == SDEI_PENDING) {
+			kvm_sdei_set_unregister_pending(registered_event,
+							index);
+		} else {
+			kvm_sdei_clear_enabled(registered_event, index);
+			kvm_sdei_clear_registered(registered_event, index);
+		}
+	}
+
+	/* Destroy the registered event instance if needed */
+	if (kvm_sdei_none_registered(registered_event))
+		remove_one_registered_event(kvm, registered_event);
+
+	return ret;
+}
+
+static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num))
+		return SDEI_INVALID_PARAMETERS;
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Check if the event has been registered */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Check if the event has been pending for unregistration */
+	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_PENDING;
+		goto unlock;
+	}
+
+	ret = unregister_one_event(kvm, vcpu, registered_event);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -365,7 +494,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
+		ret = SDEI_NOT_SUPPORTED;
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+		ret = hypercall_unregister(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 08/22] KVM: arm64: Support SDEI_EVENT_STATUS hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_STATUS hypercall. It's used by the guest
to retrieve the status about the specified SDEI event. A bitmap
is returned to indicate the corresponding status, including
registration, enablement and delivery state.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 36eda31e0392..5c43c8912ea1 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -454,6 +454,46 @@ static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_status(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = 0;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * Check if the registered event exists. None of the flags
+	 * will be set if it doesn't exist.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event)
+		goto unlock;
+
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (kvm_sdei_is_registered(registered_event, index))
+		ret |= (1UL << SDEI_EVENT_STATUS_REGISTERED);
+	if (kvm_sdei_is_enabled(registered_event, index))
+		ret |= (1UL << SDEI_EVENT_STATUS_ENABLED);
+	if (registered_event->vcpu_event_count > 0)
+		ret |= (1UL << SDEI_EVENT_STATUS_RUNNING);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -500,6 +540,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_unregister(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+		ret = hypercall_status(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
 	case SDEI_1_0_FN_SDEI_PE_MASK:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 08/22] KVM: arm64: Support SDEI_EVENT_STATUS hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_STATUS hypercall. It's used by the guest
to retrieve the status about the specified SDEI event. A bitmap
is returned to indicate the corresponding status, including
registration, enablement and delivery state.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 36eda31e0392..5c43c8912ea1 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -454,6 +454,46 @@ static unsigned long hypercall_unregister(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_status(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = 0;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * Check if the registered event exists. None of the flags
+	 * will be set if it doesn't exist.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event)
+		goto unlock;
+
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (kvm_sdei_is_registered(registered_event, index))
+		ret |= (1UL << SDEI_EVENT_STATUS_REGISTERED);
+	if (kvm_sdei_is_enabled(registered_event, index))
+		ret |= (1UL << SDEI_EVENT_STATUS_ENABLED);
+	if (registered_event->vcpu_event_count > 0)
+		ret |= (1UL << SDEI_EVENT_STATUS_RUNNING);
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -500,6 +540,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_unregister(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
+		ret = hypercall_status(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
 	case SDEI_1_0_FN_SDEI_PE_MASK:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 09/22] KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_GET_INFO hypercall. It's used by the guest
to retrieve various information about the exposed or registered event,
including type, signaled, routing mode and affinity. The routing
mode and affinity information is only valid to the shared and
registered event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 73 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 5c43c8912ea1..4f26e5f70bff 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -494,6 +494,77 @@ static unsigned long hypercall_status(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_info(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long event_info = smccc_get_arg2(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * Retrieve the information from the registered event if it exists.
+	 * Otherwise, we turn into the exposed event if needed.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	exposed_event = registered_event ? registered_event->exposed_event :
+					   find_exposed_event(kvm, event_num);
+	if (!exposed_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Retrieve the requested information */
+	switch (event_info) {
+	case SDEI_EVENT_INFO_EV_TYPE:
+		ret = exposed_event->state.type;
+		break;
+	case SDEI_EVENT_INFO_EV_SIGNALED:
+		ret = exposed_event->state.signaled;
+		break;
+	case SDEI_EVENT_INFO_EV_PRIORITY:
+		ret = exposed_event->state.priority;
+		break;
+	case SDEI_EVENT_INFO_EV_ROUTING_MODE:
+	case SDEI_EVENT_INFO_EV_ROUTING_AFF:
+		if (!kvm_sdei_is_shared(exposed_event->state.type)) {
+			ret = SDEI_INVALID_PARAMETERS;
+			break;
+		}
+
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (!registered_event ||
+		    !kvm_sdei_is_registered(registered_event, index)) {
+			ret = SDEI_DENIED;
+			break;
+		}
+
+		if (event_info == SDEI_EVENT_INFO_EV_ROUTING_MODE)
+			ret = registered_event->state.route_mode;
+		else
+			ret = registered_event->state.route_affinity;
+
+		break;
+	default:
+		ret = SDEI_INVALID_PARAMETERS;
+	}
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -543,6 +614,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_status(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+		ret = hypercall_info(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
 	case SDEI_1_0_FN_SDEI_PE_MASK:
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 09/22] KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_GET_INFO hypercall. It's used by the guest
to retrieve various information about the exposed or registered event,
including type, signaled, routing mode and affinity. The routing
mode and affinity information is only valid to the shared and
registered event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 73 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 73 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 5c43c8912ea1..4f26e5f70bff 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -494,6 +494,77 @@ static unsigned long hypercall_status(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_info(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long event_info = smccc_get_arg2(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/*
+	 * Retrieve the information from the registered event if it exists.
+	 * Otherwise, we turn into the exposed event if needed.
+	 */
+	registered_event = find_registered_event(kvm, event_num);
+	exposed_event = registered_event ? registered_event->exposed_event :
+					   find_exposed_event(kvm, event_num);
+	if (!exposed_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Retrieve the requested information */
+	switch (event_info) {
+	case SDEI_EVENT_INFO_EV_TYPE:
+		ret = exposed_event->state.type;
+		break;
+	case SDEI_EVENT_INFO_EV_SIGNALED:
+		ret = exposed_event->state.signaled;
+		break;
+	case SDEI_EVENT_INFO_EV_PRIORITY:
+		ret = exposed_event->state.priority;
+		break;
+	case SDEI_EVENT_INFO_EV_ROUTING_MODE:
+	case SDEI_EVENT_INFO_EV_ROUTING_AFF:
+		if (!kvm_sdei_is_shared(exposed_event->state.type)) {
+			ret = SDEI_INVALID_PARAMETERS;
+			break;
+		}
+
+		index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+		if (!registered_event ||
+		    !kvm_sdei_is_registered(registered_event, index)) {
+			ret = SDEI_DENIED;
+			break;
+		}
+
+		if (event_info == SDEI_EVENT_INFO_EV_ROUTING_MODE)
+			ret = registered_event->state.route_mode;
+		else
+			ret = registered_event->state.route_affinity;
+
+		break;
+	default:
+		ret = SDEI_INVALID_PARAMETERS;
+	}
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -543,6 +614,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_status(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+		ret = hypercall_info(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
 	case SDEI_1_0_FN_SDEI_PE_MASK:
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 10/22] KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_ROUTING_SET hypercall. It's used by the
guest to set route mode and affinity for the shared and registered
events. The request to configure the routing mode and affinity for
the private events are disallowed. Besides, It's not allowed to do
when the corresponding vCPU events are existing.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 62 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 4f26e5f70bff..db82ea441eae 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -565,6 +565,66 @@ static unsigned long hypercall_info(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_route(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long route_mode = smccc_get_arg2(vcpu);
+	unsigned long route_affinity = smccc_get_arg3(vcpu);
+	int index = 0;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	/*
+	 * FIXME: The affinity should be verified when it's supported. We
+	 * accept anything for now.
+	 */
+	if (route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+	    route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Check the registered event is a shared one */
+	exposed_event = registered_event->exposed_event;
+	if (!kvm_sdei_is_shared(exposed_event->state.type)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    kvm_sdei_is_enabled(registered_event, index)     ||
+	    registered_event->vcpu_event_count > 0) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Update the registered event state */
+	registered_event->state.route_mode     = route_mode;
+	registered_event->state.route_affinity = route_affinity;
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -617,6 +677,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_info(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+		ret = hypercall_route(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_PE_MASK:
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 10/22] KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_ROUTING_SET hypercall. It's used by the
guest to set route mode and affinity for the shared and registered
events. The request to configure the routing mode and affinity for
the private events are disallowed. Besides, It's not allowed to do
when the corresponding vCPU events are existing.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 62 +++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 62 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 4f26e5f70bff..db82ea441eae 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -565,6 +565,66 @@ static unsigned long hypercall_info(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_route(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	unsigned long route_mode = smccc_get_arg2(vcpu);
+	unsigned long route_affinity = smccc_get_arg3(vcpu);
+	int index = 0;
+	unsigned long ret = SDEI_SUCCESS;
+
+	if (!kvm_sdei_is_supported(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	/*
+	 * FIXME: The affinity should be verified when it's supported. We
+	 * accept anything for now.
+	 */
+	if (route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+	    route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock;
+	}
+
+	/* Check the registered event is a shared one */
+	exposed_event = registered_event->exposed_event;
+	if (!kvm_sdei_is_shared(exposed_event->state.type)) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    kvm_sdei_is_enabled(registered_event, index)     ||
+	    registered_event->vcpu_event_count > 0) {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Update the registered event state */
+	registered_event->state.route_mode     = route_mode;
+	registered_event->state.route_affinity = route_affinity;
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -617,6 +677,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_info(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
+		ret = hypercall_route(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_PE_MASK:
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 11/22] KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:06   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_PE_{MASK, UNMASK} hypercall. They are used by
the guest to stop or start receiving SDEI event on the specified
vCPU.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index db82ea441eae..b2a916724cfa 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -625,6 +625,18 @@ static unsigned long hypercall_route(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	unsigned long ret = SDEI_SUCCESS;
+
+	spin_lock(&vsdei->lock);
+	vsdei->state.masked = mask ? 1 : 0;
+	spin_unlock(&vsdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -680,7 +692,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_route(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_PE_MASK:
+		ret = hypercall_mask(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+		ret = hypercall_mask(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
 	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 11/22] KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall
@ 2022-03-22  8:06   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:06 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_PE_{MASK, UNMASK} hypercall. They are used by
the guest to stop or start receiving SDEI event on the specified
vCPU.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index db82ea441eae..b2a916724cfa 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -625,6 +625,18 @@ static unsigned long hypercall_route(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	unsigned long ret = SDEI_SUCCESS;
+
+	spin_lock(&vsdei->lock);
+	vsdei->state.masked = mask ? 1 : 0;
+	spin_unlock(&vsdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -680,7 +692,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_route(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_PE_MASK:
+		ret = hypercall_mask(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_PE_UNMASK:
+		ret = hypercall_mask(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
 	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 12/22] KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_{PRIVATE, SHARED}_RESET. They are used by the
guest to reset the private events on the calling vCPU or the
shared events on all vCPUs.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index b2a916724cfa..0dec35a0eed1 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -637,6 +637,31 @@ static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
 	return ret;
 }
 
+static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event, *tmp;
+	unsigned long r, ret = SDEI_SUCCESS;
+
+	spin_lock(&ksdei->lock);
+
+	list_for_each_entry_safe(registered_event, tmp,
+				 &ksdei->registered_events, link) {
+		exposed_event = registered_event->exposed_event;
+		if (private ^ kvm_sdei_is_shared(exposed_event->state.type))
+			continue;
+
+		r = unregister_one_event(kvm, NULL, registered_event);
+		ret = (r == SDEI_SUCCESS) ? ret : r;
+	}
+
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -700,8 +725,14 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
 	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		ret = SDEI_NOT_SUPPORTED;
+		break;
 	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+		ret = hypercall_reset(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+		ret = hypercall_reset(vcpu, false);
+		break;
 	case SDEI_1_1_FN_SDEI_FEATURES:
 	default:
 		ret = SDEI_NOT_SUPPORTED;
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 12/22] KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_{PRIVATE, SHARED}_RESET. They are used by the
guest to reset the private events on the calling vCPU or the
shared events on all vCPUs.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index b2a916724cfa..0dec35a0eed1 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -637,6 +637,31 @@ static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
 	return ret;
 }
 
+static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event, *tmp;
+	unsigned long r, ret = SDEI_SUCCESS;
+
+	spin_lock(&ksdei->lock);
+
+	list_for_each_entry_safe(registered_event, tmp,
+				 &ksdei->registered_events, link) {
+		exposed_event = registered_event->exposed_event;
+		if (private ^ kvm_sdei_is_shared(exposed_event->state.type))
+			continue;
+
+		r = unregister_one_event(kvm, NULL, registered_event);
+		ret = (r == SDEI_SUCCESS) ? ret : r;
+	}
+
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -700,8 +725,14 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
 	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		ret = SDEI_NOT_SUPPORTED;
+		break;
 	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+		ret = hypercall_reset(vcpu, true);
+		break;
 	case SDEI_1_0_FN_SDEI_SHARED_RESET:
+		ret = hypercall_reset(vcpu, false);
+		break;
 	case SDEI_1_1_FN_SDEI_FEATURES:
 	default:
 		ret = SDEI_NOT_SUPPORTED;
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 13/22] KVM: arm64: Support SDEI_FEATURES hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_FEATURES hypercall. It's used by the guest to
retrieve the supported features, which are number of binding slots
and relative mode for the event handler. Currently, none of them
is supported.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 0dec35a0eed1..1e0ca9022eaa 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -662,6 +662,20 @@ static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 	return ret;
 }
 
+static unsigned long hypercall_features(struct kvm_vcpu *vcpu)
+{
+	unsigned long feature = smccc_get_arg1(vcpu);
+
+	switch (feature) {
+	case 0: /* BIND_SLOTS */
+		return 0;
+	case 1: /* RELATIVE_MODE */
+		return 0;
+	}
+
+	return SDEI_INVALID_PARAMETERS;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -734,6 +748,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_reset(vcpu, false);
 		break;
 	case SDEI_1_1_FN_SDEI_FEATURES:
+		ret = hypercall_features(vcpu);
+		break;
 	default:
 		ret = SDEI_NOT_SUPPORTED;
 	}
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 13/22] KVM: arm64: Support SDEI_FEATURES hypercall
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_FEATURES hypercall. It's used by the guest to
retrieve the supported features, which are number of binding slots
and relative mode for the event handler. Currently, none of them
is supported.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 0dec35a0eed1..1e0ca9022eaa 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -662,6 +662,20 @@ static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 	return ret;
 }
 
+static unsigned long hypercall_features(struct kvm_vcpu *vcpu)
+{
+	unsigned long feature = smccc_get_arg1(vcpu);
+
+	switch (feature) {
+	case 0: /* BIND_SLOTS */
+		return 0;
+	case 1: /* RELATIVE_MODE */
+		return 0;
+	}
+
+	return SDEI_INVALID_PARAMETERS;
+}
+
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -734,6 +748,8 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_reset(vcpu, false);
 		break;
 	case SDEI_1_1_FN_SDEI_FEATURES:
+		ret = hypercall_features(vcpu);
+		break;
 	default:
 		ret = SDEI_NOT_SUPPORTED;
 	}
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 14/22] KVM: arm64: Support SDEI event injection, delivery and cancellation
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI event injection, delivery and cancellation. The
SDEI event is injected by kvm_sdei_inject_event(). The injected
event can be cancelled by kvm_sdei_cancel_event() before it's
delivered and handled.

KVM_REQ_SDEI request becomes pending once the SDEI event is injected
and kvm_sdei_deliver_event() is called to accommodate the request.
The injected SDEI event is delivered and handled in this way. The
context for execution is switched like below:

   * x0 - x17 are saved. All of them are cleared except the following
     registers:

     x0: SDEI event number
     x1: user argument associated with the SDEI event
     x2: PC of the interrupted or preempted context
     x3: PSTATE of the interrupted or preempted context

   * PC is set to the handler of the SDEI event, which was provided
     during its registration. PSTATE is modified according to the
     SDEI specification.

   * The SDEI event with normal priority can be preempted by that
     with critical priority. However, no one can preempt the SDEI
     event with critical event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_host.h |   1 +
 arch/arm64/include/asm/kvm_sdei.h |   4 +
 arch/arm64/kvm/arm.c              |   3 +
 arch/arm64/kvm/sdei.c             | 284 ++++++++++++++++++++++++++++++
 4 files changed, 292 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 5d37e046a458..e2762d08ab1c 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -46,6 +46,7 @@
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
 #define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 #define KVM_REQ_RELOAD_PMU	KVM_ARCH_REQ(5)
+#define KVM_REQ_SDEI		KVM_ARCH_REQ(6)
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 6f58a846d05c..54c730acd298 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -165,6 +165,10 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
 void kvm_sdei_init_vm(struct kvm *kvm);
 void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
+			  unsigned long num, bool immediate);
+int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
+void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 96fcae5beee4..00c136a6e8df 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -734,6 +734,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 		if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
 			kvm_reset_vcpu(vcpu);
 
+		if (kvm_check_request(KVM_REQ_SDEI, vcpu))
+			kvm_sdei_deliver_event(vcpu);
+
 		/*
 		 * Clear IRQ_PENDING requests that were made to guarantee
 		 * that a VCPU sees new virtual interrupts.
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 1e0ca9022eaa..a24270378305 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -43,6 +43,25 @@ find_registered_event(struct kvm *kvm, unsigned long num)
 	return NULL;
 }
 
+static struct kvm_sdei_vcpu_event *
+find_vcpu_event(struct kvm_vcpu *vcpu, unsigned long num)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+
+	list_for_each_entry(vcpu_event, &vsdei->critical_events, link) {
+		if (vcpu_event->state.num == num)
+			return vcpu_event;
+	}
+
+	list_for_each_entry(vcpu_event, &vsdei->normal_events, link) {
+		if (vcpu_event->state.num == num)
+			return vcpu_event;
+	}
+
+	return NULL;
+}
+
 static void remove_all_exposed_events(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
@@ -637,6 +656,76 @@ static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
 	return ret;
 }
 
+static int do_inject_event(struct kvm_vcpu *vcpu,
+			   struct kvm_sdei_registered_event *registered_event,
+			   bool immediate)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	unsigned int vcpu_event_count;
+
+	/*
+	 * In some cases, the injected event is expected to be delivered
+	 * immediately. However, there are two cases the injected event
+	 * can't be delivered immediately: (a) the injected event is a
+	 * critical one, but we already have pending critical events for
+	 * delivery. (b) the injected event is a normal one, but we have
+	 * pending events for delivery, regardless of their priorities.
+	 */
+	exposed_event = registered_event->exposed_event;
+	if (immediate) {
+		vcpu_event_count = vsdei->critical_event_count;
+		if (kvm_sdei_is_normal(exposed_event->state.priority))
+			vcpu_event_count += vsdei->normal_event_count;
+
+		if (vcpu_event_count > 0)
+			return -ENOSPC;
+	}
+
+	/* Check if the vcpu event exists */
+	vcpu_event = find_vcpu_event(vcpu, registered_event->state.num);
+	if (vcpu_event) {
+		vcpu_event->state.event_count++;
+		kvm_make_request(KVM_REQ_SDEI, vcpu);
+		return 0;
+	}
+
+	/* Check if the count of vcpu event instances exceeds the limit */
+	vcpu_event_count = vsdei->critical_event_count +
+			   vsdei->normal_event_count;
+	if (vcpu_event_count >= KVM_SDEI_MAX_EVENTS)
+		return -ERANGE;
+
+	/* Allocate the vcpu event */
+	vcpu_event = kzalloc(sizeof(*vcpu_event), GFP_KERNEL_ACCOUNT);
+	if (!vcpu_event)
+		return -ENOMEM;
+
+	/*
+	 * We should take lock to update the registered event because its
+	 * reference count might be zero. In that case, the registered event
+	 * could be released.
+	 */
+	vcpu_event->state.num         = registered_event->state.num;
+	vcpu_event->state.event_count = 1;
+	vcpu_event->vcpu              = vcpu;
+	vcpu_event->registered_event  = registered_event;
+
+	registered_event->vcpu_event_count++;
+	if (kvm_sdei_is_critical(exposed_event->state.priority)) {
+		list_add_tail(&vcpu_event->link, &vsdei->critical_events);
+		vsdei->critical_event_count++;
+	} else {
+		list_add_tail(&vcpu_event->link, &vsdei->normal_events);
+		vsdei->normal_event_count++;
+	}
+
+	kvm_make_request(KVM_REQ_SDEI, vcpu);
+
+	return 0;
+}
+
 static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -761,6 +850,201 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
+			  unsigned long num,
+			  bool immediate)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	int index, ret = 0;
+
+	if (!(ksdei && vsdei)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	if (!kvm_sdei_is_supported(num)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, num);
+	if (!registered_event) {
+		ret = -ENOENT;
+		goto unlock_kvm;
+	}
+
+	/* Check if the event has been registered and enabled */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = -EPERM;
+		goto unlock_kvm;
+	}
+
+	/* Check if the vcpu has been masked off */
+	spin_lock(&vsdei->lock);
+	if (vsdei->state.masked) {
+		ret = -EPERM;
+		goto unlock_vcpu;
+	}
+
+	/* Inject the event */
+	ret = do_inject_event(vcpu, registered_event, immediate);
+
+unlock_vcpu:
+	spin_unlock(&vsdei->lock);
+unlock_kvm:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
+int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	struct kvm_sdei_vcpu_event *vcpu_event = NULL;
+	int ret = 0;
+
+	if (!(ksdei && vsdei)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+	spin_lock(&vsdei->lock);
+
+	/* Find the vcpu event */
+	vcpu_event = find_vcpu_event(vcpu, num);
+	if (!vcpu_event) {
+		ret = -EINVAL;
+		goto unlock;
+	}
+
+	/* We can't cancel the event if it has been delivered */
+	if (vcpu_event->state.event_count <= 1 &&
+	    (vsdei->critical_event == vcpu_event ||
+	     vsdei->normal_event == vcpu_event)) {
+		ret = -EINPROGRESS;
+		goto unlock;
+	}
+
+	/* Destroy the vcpu event instance if needed */
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	vcpu_event->state.event_count--;
+	if (!vcpu_event->state.event_count)
+		remove_one_vcpu_event(vcpu, vcpu_event);
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
+void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long pstate;
+	int index;
+
+	if (!(ksdei && vsdei))
+		return;
+
+	spin_lock(&vsdei->lock);
+
+	/* The critical event can't be preempted */
+	if (vsdei->critical_event)
+		goto unlock;
+
+	/*
+	 * The normal event can be preempted by the critical event.
+	 * However, the normal event can't be preempted by another
+	 * normal event.
+	 */
+	vcpu_event = list_first_entry_or_null(&vsdei->critical_events,
+				struct kvm_sdei_vcpu_event, link);
+	if (!vcpu_event && !vsdei->normal_event) {
+		vcpu_event = list_first_entry_or_null(&vsdei->normal_events,
+					struct kvm_sdei_vcpu_event, link);
+	}
+
+	if (!vcpu_event)
+		goto unlock;
+
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	if (kvm_sdei_is_critical(exposed_event->state.priority)) {
+		vsdei->critical_event = vcpu_event;
+		vsdei->state.critical_num = vcpu_event->state.num;
+		regs = &vsdei->state.critical_regs;
+	} else {
+		vsdei->normal_event = vcpu_event;
+		vsdei->state.normal_num = vcpu_event->state.num;
+		regs = &vsdei->state.normal_regs;
+	}
+
+	/*
+	 * Save registers: x0 -> x17, PC, PState. There might be pending
+	 * exception or PC increment request in the last run on this vCPU.
+	 * In this case, we need to save the site in advance. Otherwise,
+	 * the passed entry point could be floated by 4 bytes in the
+	 * subsequent call to __kvm_adjust_pc().
+	 */
+	__kvm_adjust_pc(vcpu);
+	for (index = 0; index < ARRAY_SIZE(regs->regs); index++)
+		regs->regs[index] = vcpu_get_reg(vcpu, index);
+
+	regs->pc = *vcpu_pc(vcpu);
+	regs->pstate = *vcpu_cpsr(vcpu);
+
+	/*
+	 * Inject SDEI event: x0 -> x3, PC, PState. We needn't take lock
+	 * for the registered event as it can't be released because of
+	 * its reference count.
+	 */
+	for (index = 0; index < ARRAY_SIZE(regs->regs); index++)
+		vcpu_set_reg(vcpu, index, 0);
+
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	vcpu_set_reg(vcpu, 0, registered_event->state.num);
+	vcpu_set_reg(vcpu, 1, registered_event->state.ep_arg[index]);
+	vcpu_set_reg(vcpu, 2, regs->pc);
+	vcpu_set_reg(vcpu, 3, regs->pstate);
+
+	pstate = regs->pstate;
+	pstate |= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT);
+	pstate &= ~PSR_MODE_MASK;
+	pstate |= PSR_MODE_EL1h;
+	pstate &= ~PSR_MODE32_BIT;
+
+	vcpu_write_sys_reg(vcpu, regs->pstate, SPSR_EL1);
+	*vcpu_cpsr(vcpu) = pstate;
+	*vcpu_pc(vcpu) = registered_event->state.ep_address[index];
+
+unlock:
+	spin_unlock(&vsdei->lock);
+}
+
 void kvm_sdei_init_vm(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei;
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 14/22] KVM: arm64: Support SDEI event injection, delivery and cancellation
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI event injection, delivery and cancellation. The
SDEI event is injected by kvm_sdei_inject_event(). The injected
event can be cancelled by kvm_sdei_cancel_event() before it's
delivered and handled.

KVM_REQ_SDEI request becomes pending once the SDEI event is injected
and kvm_sdei_deliver_event() is called to accommodate the request.
The injected SDEI event is delivered and handled in this way. The
context for execution is switched like below:

   * x0 - x17 are saved. All of them are cleared except the following
     registers:

     x0: SDEI event number
     x1: user argument associated with the SDEI event
     x2: PC of the interrupted or preempted context
     x3: PSTATE of the interrupted or preempted context

   * PC is set to the handler of the SDEI event, which was provided
     during its registration. PSTATE is modified according to the
     SDEI specification.

   * The SDEI event with normal priority can be preempted by that
     with critical priority. However, no one can preempt the SDEI
     event with critical event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_host.h |   1 +
 arch/arm64/include/asm/kvm_sdei.h |   4 +
 arch/arm64/kvm/arm.c              |   3 +
 arch/arm64/kvm/sdei.c             | 284 ++++++++++++++++++++++++++++++
 4 files changed, 292 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 5d37e046a458..e2762d08ab1c 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -46,6 +46,7 @@
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
 #define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 #define KVM_REQ_RELOAD_PMU	KVM_ARCH_REQ(5)
+#define KVM_REQ_SDEI		KVM_ARCH_REQ(6)
 
 #define KVM_DIRTY_LOG_MANUAL_CAPS   (KVM_DIRTY_LOG_MANUAL_PROTECT_ENABLE | \
 				     KVM_DIRTY_LOG_INITIALLY_SET)
diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 6f58a846d05c..54c730acd298 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -165,6 +165,10 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
 void kvm_sdei_init_vm(struct kvm *kvm);
 void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
+			  unsigned long num, bool immediate);
+int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
+void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 96fcae5beee4..00c136a6e8df 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -734,6 +734,9 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 		if (kvm_check_request(KVM_REQ_VCPU_RESET, vcpu))
 			kvm_reset_vcpu(vcpu);
 
+		if (kvm_check_request(KVM_REQ_SDEI, vcpu))
+			kvm_sdei_deliver_event(vcpu);
+
 		/*
 		 * Clear IRQ_PENDING requests that were made to guarantee
 		 * that a VCPU sees new virtual interrupts.
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 1e0ca9022eaa..a24270378305 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -43,6 +43,25 @@ find_registered_event(struct kvm *kvm, unsigned long num)
 	return NULL;
 }
 
+static struct kvm_sdei_vcpu_event *
+find_vcpu_event(struct kvm_vcpu *vcpu, unsigned long num)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+
+	list_for_each_entry(vcpu_event, &vsdei->critical_events, link) {
+		if (vcpu_event->state.num == num)
+			return vcpu_event;
+	}
+
+	list_for_each_entry(vcpu_event, &vsdei->normal_events, link) {
+		if (vcpu_event->state.num == num)
+			return vcpu_event;
+	}
+
+	return NULL;
+}
+
 static void remove_all_exposed_events(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
@@ -637,6 +656,76 @@ static unsigned long hypercall_mask(struct kvm_vcpu *vcpu, bool mask)
 	return ret;
 }
 
+static int do_inject_event(struct kvm_vcpu *vcpu,
+			   struct kvm_sdei_registered_event *registered_event,
+			   bool immediate)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	unsigned int vcpu_event_count;
+
+	/*
+	 * In some cases, the injected event is expected to be delivered
+	 * immediately. However, there are two cases the injected event
+	 * can't be delivered immediately: (a) the injected event is a
+	 * critical one, but we already have pending critical events for
+	 * delivery. (b) the injected event is a normal one, but we have
+	 * pending events for delivery, regardless of their priorities.
+	 */
+	exposed_event = registered_event->exposed_event;
+	if (immediate) {
+		vcpu_event_count = vsdei->critical_event_count;
+		if (kvm_sdei_is_normal(exposed_event->state.priority))
+			vcpu_event_count += vsdei->normal_event_count;
+
+		if (vcpu_event_count > 0)
+			return -ENOSPC;
+	}
+
+	/* Check if the vcpu event exists */
+	vcpu_event = find_vcpu_event(vcpu, registered_event->state.num);
+	if (vcpu_event) {
+		vcpu_event->state.event_count++;
+		kvm_make_request(KVM_REQ_SDEI, vcpu);
+		return 0;
+	}
+
+	/* Check if the count of vcpu event instances exceeds the limit */
+	vcpu_event_count = vsdei->critical_event_count +
+			   vsdei->normal_event_count;
+	if (vcpu_event_count >= KVM_SDEI_MAX_EVENTS)
+		return -ERANGE;
+
+	/* Allocate the vcpu event */
+	vcpu_event = kzalloc(sizeof(*vcpu_event), GFP_KERNEL_ACCOUNT);
+	if (!vcpu_event)
+		return -ENOMEM;
+
+	/*
+	 * We should take lock to update the registered event because its
+	 * reference count might be zero. In that case, the registered event
+	 * could be released.
+	 */
+	vcpu_event->state.num         = registered_event->state.num;
+	vcpu_event->state.event_count = 1;
+	vcpu_event->vcpu              = vcpu;
+	vcpu_event->registered_event  = registered_event;
+
+	registered_event->vcpu_event_count++;
+	if (kvm_sdei_is_critical(exposed_event->state.priority)) {
+		list_add_tail(&vcpu_event->link, &vsdei->critical_events);
+		vsdei->critical_event_count++;
+	} else {
+		list_add_tail(&vcpu_event->link, &vsdei->normal_events);
+		vsdei->normal_event_count++;
+	}
+
+	kvm_make_request(KVM_REQ_SDEI, vcpu);
+
+	return 0;
+}
+
 static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -761,6 +850,201 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
+			  unsigned long num,
+			  bool immediate)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	int index, ret = 0;
+
+	if (!(ksdei && vsdei)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	if (!kvm_sdei_is_supported(num)) {
+		ret = -EINVAL;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, num);
+	if (!registered_event) {
+		ret = -ENOENT;
+		goto unlock_kvm;
+	}
+
+	/* Check if the event has been registered and enabled */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = -EPERM;
+		goto unlock_kvm;
+	}
+
+	/* Check if the vcpu has been masked off */
+	spin_lock(&vsdei->lock);
+	if (vsdei->state.masked) {
+		ret = -EPERM;
+		goto unlock_vcpu;
+	}
+
+	/* Inject the event */
+	ret = do_inject_event(vcpu, registered_event, immediate);
+
+unlock_vcpu:
+	spin_unlock(&vsdei->lock);
+unlock_kvm:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
+int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	struct kvm_sdei_registered_event *registered_event = NULL;
+	struct kvm_sdei_vcpu_event *vcpu_event = NULL;
+	int ret = 0;
+
+	if (!(ksdei && vsdei)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+	spin_lock(&vsdei->lock);
+
+	/* Find the vcpu event */
+	vcpu_event = find_vcpu_event(vcpu, num);
+	if (!vcpu_event) {
+		ret = -EINVAL;
+		goto unlock;
+	}
+
+	/* We can't cancel the event if it has been delivered */
+	if (vcpu_event->state.event_count <= 1 &&
+	    (vsdei->critical_event == vcpu_event ||
+	     vsdei->normal_event == vcpu_event)) {
+		ret = -EINPROGRESS;
+		goto unlock;
+	}
+
+	/* Destroy the vcpu event instance if needed */
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	vcpu_event->state.event_count--;
+	if (!vcpu_event->state.event_count)
+		remove_one_vcpu_event(vcpu, vcpu_event);
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
+void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long pstate;
+	int index;
+
+	if (!(ksdei && vsdei))
+		return;
+
+	spin_lock(&vsdei->lock);
+
+	/* The critical event can't be preempted */
+	if (vsdei->critical_event)
+		goto unlock;
+
+	/*
+	 * The normal event can be preempted by the critical event.
+	 * However, the normal event can't be preempted by another
+	 * normal event.
+	 */
+	vcpu_event = list_first_entry_or_null(&vsdei->critical_events,
+				struct kvm_sdei_vcpu_event, link);
+	if (!vcpu_event && !vsdei->normal_event) {
+		vcpu_event = list_first_entry_or_null(&vsdei->normal_events,
+					struct kvm_sdei_vcpu_event, link);
+	}
+
+	if (!vcpu_event)
+		goto unlock;
+
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	if (kvm_sdei_is_critical(exposed_event->state.priority)) {
+		vsdei->critical_event = vcpu_event;
+		vsdei->state.critical_num = vcpu_event->state.num;
+		regs = &vsdei->state.critical_regs;
+	} else {
+		vsdei->normal_event = vcpu_event;
+		vsdei->state.normal_num = vcpu_event->state.num;
+		regs = &vsdei->state.normal_regs;
+	}
+
+	/*
+	 * Save registers: x0 -> x17, PC, PState. There might be pending
+	 * exception or PC increment request in the last run on this vCPU.
+	 * In this case, we need to save the site in advance. Otherwise,
+	 * the passed entry point could be floated by 4 bytes in the
+	 * subsequent call to __kvm_adjust_pc().
+	 */
+	__kvm_adjust_pc(vcpu);
+	for (index = 0; index < ARRAY_SIZE(regs->regs); index++)
+		regs->regs[index] = vcpu_get_reg(vcpu, index);
+
+	regs->pc = *vcpu_pc(vcpu);
+	regs->pstate = *vcpu_cpsr(vcpu);
+
+	/*
+	 * Inject SDEI event: x0 -> x3, PC, PState. We needn't take lock
+	 * for the registered event as it can't be released because of
+	 * its reference count.
+	 */
+	for (index = 0; index < ARRAY_SIZE(regs->regs); index++)
+		vcpu_set_reg(vcpu, index, 0);
+
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	vcpu_set_reg(vcpu, 0, registered_event->state.num);
+	vcpu_set_reg(vcpu, 1, registered_event->state.ep_arg[index]);
+	vcpu_set_reg(vcpu, 2, regs->pc);
+	vcpu_set_reg(vcpu, 3, regs->pstate);
+
+	pstate = regs->pstate;
+	pstate |= (PSR_D_BIT | PSR_A_BIT | PSR_I_BIT | PSR_F_BIT);
+	pstate &= ~PSR_MODE_MASK;
+	pstate |= PSR_MODE_EL1h;
+	pstate &= ~PSR_MODE32_BIT;
+
+	vcpu_write_sys_reg(vcpu, regs->pstate, SPSR_EL1);
+	*vcpu_cpsr(vcpu) = pstate;
+	*vcpu_pc(vcpu) = registered_event->state.ep_address[index];
+
+unlock:
+	spin_unlock(&vsdei->lock);
+}
+
 void kvm_sdei_init_vm(struct kvm *kvm)
 {
 	struct kvm_sdei_kvm *ksdei;
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest
to inject SDEI event, whose number must be zero to the specified
vCPU. As the routing mode and affinity isn't supported yet, the
calling vCPU is assumed to be the target.

The SDEI event 0x0 is a private one, with normal priority. It's
usually used for testing.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index a24270378305..ba2ca65c871b 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	/* @event_num must be zero */
+	if (!kvm_sdei_is_default(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_kvm;
+	}
+
+	/*
+	 * Check if the event has been registered and enabled. The
+	 * @target_pe parameter isn't checked for now and the event
+	 * is assumed to injected the current vcpu. We should fix it
+	 * when the routing mode and affinity are supported.
+	 */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_kvm;
+	}
+
+	/* Check if the vcpu has been masked off */
+	spin_lock(&vsdei->lock);
+	if (vsdei->state.masked) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_vcpu;
+	}
+
+	/* Inject the event */
+	if (do_inject_event(vcpu, registered_event, false))
+		ret = SDEI_INVALID_PARAMETERS;
+
+unlock_vcpu:
+	spin_unlock(&vsdei->lock);
+unlock_kvm:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -827,9 +887,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
-	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
 		ret = SDEI_NOT_SUPPORTED;
 		break;
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		ret = hypercall_signal(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
 		ret = hypercall_reset(vcpu, true);
 		break;
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest
to inject SDEI event, whose number must be zero to the specified
vCPU. As the routing mode and affinity isn't supported yet, the
calling vCPU is assumed to be the target.

The SDEI event 0x0 is a private one, with normal priority. It's
usually used for testing.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index a24270378305..ba2ca65c871b 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	unsigned long event_num = smccc_get_arg1(vcpu);
+	int index;
+	unsigned long ret = SDEI_SUCCESS;
+
+	/* @event_num must be zero */
+	if (!kvm_sdei_is_default(event_num)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	/* Check if the registered event exists */
+	registered_event = find_registered_event(kvm, event_num);
+	if (!registered_event) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_kvm;
+	}
+
+	/*
+	 * Check if the event has been registered and enabled. The
+	 * @target_pe parameter isn't checked for now and the event
+	 * is assumed to injected the current vcpu. We should fix it
+	 * when the routing mode and affinity are supported.
+	 */
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index)) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_kvm;
+	}
+
+	/* Check if the vcpu has been masked off */
+	spin_lock(&vsdei->lock);
+	if (vsdei->state.masked) {
+		ret = SDEI_INVALID_PARAMETERS;
+		goto unlock_vcpu;
+	}
+
+	/* Inject the event */
+	if (do_inject_event(vcpu, registered_event, false))
+		ret = SDEI_INVALID_PARAMETERS;
+
+unlock_vcpu:
+	spin_unlock(&vsdei->lock);
+unlock_kvm:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 static unsigned long hypercall_reset(struct kvm_vcpu *vcpu, bool private)
 {
 	struct kvm *kvm = vcpu->kvm;
@@ -827,9 +887,11 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		break;
 	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
 	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
-	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
 		ret = SDEI_NOT_SUPPORTED;
 		break;
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		ret = hypercall_signal(vcpu);
+		break;
 	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
 		ret = hypercall_reset(vcpu, true);
 		break;
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME} hypercall
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall.
They are used by the guest to notify the completion of the SDEI
event in the handler. The executing context or registers are modified
according to the SDEI specification like below:

   * x0 - x17, PC and PState are restored to what values we had in
     the interrupted or preempted context.

   * If it's SDEI_EVENT_COMPLETE_AND_RESUME hypercall, IRQ exception
     is injected.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  1 +
 arch/arm64/include/asm/kvm_host.h    |  1 +
 arch/arm64/kvm/inject_fault.c        | 29 +++++++++++
 arch/arm64/kvm/sdei.c                | 76 +++++++++++++++++++++++++++-
 4 files changed, 106 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index d62405ce3e6d..ca9de9f24923 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -37,6 +37,7 @@ bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
 void kvm_inject_undefined(struct kvm_vcpu *vcpu);
+void kvm_inject_irq(struct kvm_vcpu *vcpu);
 void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index e2762d08ab1c..282913e1afb0 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -428,6 +428,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_AA32_UND	(0 << 9)
 #define KVM_ARM64_EXCEPT_AA32_IABT	(1 << 9)
 #define KVM_ARM64_EXCEPT_AA32_DABT	(2 << 9)
+#define KVM_ARM64_EXCEPT_AA32_IRQ	(3 << 9)
 /* For AArch64: */
 #define KVM_ARM64_EXCEPT_AA64_ELx_SYNC	(0 << 9)
 #define KVM_ARM64_EXCEPT_AA64_ELx_IRQ	(1 << 9)
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index b47df73e98d7..c8a8791bdf28 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -66,6 +66,13 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
 	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
 }
 
+static void inject_irq64(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1     |
+			     KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+			     KVM_ARM64_PENDING_EXCEPTION);
+}
+
 #define DFSR_FSC_EXTABT_LPAE	0x10
 #define DFSR_FSC_EXTABT_nLPAE	0x08
 #define DFSR_LPAE		BIT(9)
@@ -77,6 +84,12 @@ static void inject_undef32(struct kvm_vcpu *vcpu)
 			     KVM_ARM64_PENDING_EXCEPTION);
 }
 
+static void inject_irq32(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA32_IRQ |
+			     KVM_ARM64_PENDING_EXCEPTION);
+}
+
 /*
  * Modelled after TakeDataAbortException() and TakePrefetchAbortException
  * pseudocode.
@@ -160,6 +173,22 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu)
 		inject_undef64(vcpu);
 }
 
+/**
+ * kvm_inject_irq - inject an IRQ into the guest
+ * @vcpu: The vCPU in which to inject IRQ
+ *
+ * Inject IRQs to the target vCPU. It is assumed that this code is
+ * called from the VCPU thread and that the VCPU therefore is not
+ * currently executing guest code.
+ */
+void kvm_inject_irq(struct kvm_vcpu *vcpu)
+{
+	if (vcpu_el1_is_32bit(vcpu))
+		inject_irq32(vcpu);
+	else
+		inject_irq64(vcpu);
+}
+
 void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 esr)
 {
 	vcpu_set_vsesr(vcpu, esr & ESR_ELx_ISS_MASK);
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index ba2ca65c871b..3019ac196e76 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -344,6 +344,78 @@ static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long ret = SDEI_SUCCESS;
+	int index;
+
+	spin_lock(&ksdei->lock);
+	spin_lock(&vsdei->lock);
+
+	if (vsdei->critical_event) {
+		vcpu_event = vsdei->critical_event;
+		regs = &vsdei->state.critical_regs;
+		vsdei->critical_event = NULL;
+		vsdei->state.critical_num = KVM_SDEI_INVALID_EVENT;
+	} else if (vsdei->normal_event) {
+		vcpu_event = vsdei->normal_event;
+		regs = &vsdei->state.normal_regs;
+		vsdei->normal_event = NULL;
+		vsdei->state.normal_num = KVM_SDEI_INVALID_EVENT;
+	} else {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Restore registers: x0 -> x17, PC, PState */
+	for (index = 0; index < ARRAY_SIZE(regs->regs); index++)
+		vcpu_set_reg(vcpu, index, regs->regs[index]);
+
+	*vcpu_cpsr(vcpu) = regs->pstate;
+	*vcpu_pc(vcpu) = regs->pc;
+
+	/* Inject interrupt if needed */
+	if (resume)
+		kvm_inject_irq(vcpu);
+
+	/* Dereference the vcpu event and destroy it if needed */
+	vcpu_event->state.event_count--;
+	if (!vcpu_event->state.event_count)
+		remove_one_vcpu_event(vcpu, vcpu_event);
+
+	/*
+	 * We need to check if the registered event is pending for
+	 * unregistration. In that case, the registered event should
+	 * be unregistered and destroyed if needed.
+	 */
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
+		kvm_sdei_clear_enabled(registered_event, index);
+		kvm_sdei_clear_registered(registered_event, index);
+		if (kvm_sdei_none_registered(registered_event))
+			remove_one_registered_event(kvm, registered_event);
+	}
+
+	/* Make another request if we have any pending events */
+	if ((vsdei->critical_event_count + vsdei->normal_event_count) > 0)
+		kvm_make_request(KVM_REQ_SDEI, vcpu);
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 static unsigned long
 unregister_one_event(struct kvm *kvm, struct kvm_vcpu *vcpu,
 		     struct kvm_sdei_registered_event *registered_event)
@@ -864,8 +936,10 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_context(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+		ret = hypercall_complete(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
-		ret = SDEI_NOT_SUPPORTED;
+		ret = hypercall_complete(vcpu, true);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
 		ret = hypercall_unregister(vcpu);
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall.
They are used by the guest to notify the completion of the SDEI
event in the handler. The executing context or registers are modified
according to the SDEI specification like below:

   * x0 - x17, PC and PState are restored to what values we had in
     the interrupted or preempted context.

   * If it's SDEI_EVENT_COMPLETE_AND_RESUME hypercall, IRQ exception
     is injected.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_emulate.h |  1 +
 arch/arm64/include/asm/kvm_host.h    |  1 +
 arch/arm64/kvm/inject_fault.c        | 29 +++++++++++
 arch/arm64/kvm/sdei.c                | 76 +++++++++++++++++++++++++++-
 4 files changed, 106 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index d62405ce3e6d..ca9de9f24923 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -37,6 +37,7 @@ bool kvm_condition_valid32(const struct kvm_vcpu *vcpu);
 void kvm_skip_instr32(struct kvm_vcpu *vcpu);
 
 void kvm_inject_undefined(struct kvm_vcpu *vcpu);
+void kvm_inject_irq(struct kvm_vcpu *vcpu);
 void kvm_inject_vabt(struct kvm_vcpu *vcpu);
 void kvm_inject_dabt(struct kvm_vcpu *vcpu, unsigned long addr);
 void kvm_inject_pabt(struct kvm_vcpu *vcpu, unsigned long addr);
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index e2762d08ab1c..282913e1afb0 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -428,6 +428,7 @@ struct kvm_vcpu_arch {
 #define KVM_ARM64_EXCEPT_AA32_UND	(0 << 9)
 #define KVM_ARM64_EXCEPT_AA32_IABT	(1 << 9)
 #define KVM_ARM64_EXCEPT_AA32_DABT	(2 << 9)
+#define KVM_ARM64_EXCEPT_AA32_IRQ	(3 << 9)
 /* For AArch64: */
 #define KVM_ARM64_EXCEPT_AA64_ELx_SYNC	(0 << 9)
 #define KVM_ARM64_EXCEPT_AA64_ELx_IRQ	(1 << 9)
diff --git a/arch/arm64/kvm/inject_fault.c b/arch/arm64/kvm/inject_fault.c
index b47df73e98d7..c8a8791bdf28 100644
--- a/arch/arm64/kvm/inject_fault.c
+++ b/arch/arm64/kvm/inject_fault.c
@@ -66,6 +66,13 @@ static void inject_undef64(struct kvm_vcpu *vcpu)
 	vcpu_write_sys_reg(vcpu, esr, ESR_EL1);
 }
 
+static void inject_irq64(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA64_EL1     |
+			     KVM_ARM64_EXCEPT_AA64_ELx_IRQ |
+			     KVM_ARM64_PENDING_EXCEPTION);
+}
+
 #define DFSR_FSC_EXTABT_LPAE	0x10
 #define DFSR_FSC_EXTABT_nLPAE	0x08
 #define DFSR_LPAE		BIT(9)
@@ -77,6 +84,12 @@ static void inject_undef32(struct kvm_vcpu *vcpu)
 			     KVM_ARM64_PENDING_EXCEPTION);
 }
 
+static void inject_irq32(struct kvm_vcpu *vcpu)
+{
+	vcpu->arch.flags |= (KVM_ARM64_EXCEPT_AA32_IRQ |
+			     KVM_ARM64_PENDING_EXCEPTION);
+}
+
 /*
  * Modelled after TakeDataAbortException() and TakePrefetchAbortException
  * pseudocode.
@@ -160,6 +173,22 @@ void kvm_inject_undefined(struct kvm_vcpu *vcpu)
 		inject_undef64(vcpu);
 }
 
+/**
+ * kvm_inject_irq - inject an IRQ into the guest
+ * @vcpu: The vCPU in which to inject IRQ
+ *
+ * Inject IRQs to the target vCPU. It is assumed that this code is
+ * called from the VCPU thread and that the VCPU therefore is not
+ * currently executing guest code.
+ */
+void kvm_inject_irq(struct kvm_vcpu *vcpu)
+{
+	if (vcpu_el1_is_32bit(vcpu))
+		inject_irq32(vcpu);
+	else
+		inject_irq64(vcpu);
+}
+
 void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 esr)
 {
 	vcpu_set_vsesr(vcpu, esr & ESR_ELx_ISS_MASK);
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index ba2ca65c871b..3019ac196e76 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -344,6 +344,78 @@ static unsigned long hypercall_context(struct kvm_vcpu *vcpu)
 	return ret;
 }
 
+static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_regs_state *regs;
+	unsigned long ret = SDEI_SUCCESS;
+	int index;
+
+	spin_lock(&ksdei->lock);
+	spin_lock(&vsdei->lock);
+
+	if (vsdei->critical_event) {
+		vcpu_event = vsdei->critical_event;
+		regs = &vsdei->state.critical_regs;
+		vsdei->critical_event = NULL;
+		vsdei->state.critical_num = KVM_SDEI_INVALID_EVENT;
+	} else if (vsdei->normal_event) {
+		vcpu_event = vsdei->normal_event;
+		regs = &vsdei->state.normal_regs;
+		vsdei->normal_event = NULL;
+		vsdei->state.normal_num = KVM_SDEI_INVALID_EVENT;
+	} else {
+		ret = SDEI_DENIED;
+		goto unlock;
+	}
+
+	/* Restore registers: x0 -> x17, PC, PState */
+	for (index = 0; index < ARRAY_SIZE(regs->regs); index++)
+		vcpu_set_reg(vcpu, index, regs->regs[index]);
+
+	*vcpu_cpsr(vcpu) = regs->pstate;
+	*vcpu_pc(vcpu) = regs->pc;
+
+	/* Inject interrupt if needed */
+	if (resume)
+		kvm_inject_irq(vcpu);
+
+	/* Dereference the vcpu event and destroy it if needed */
+	vcpu_event->state.event_count--;
+	if (!vcpu_event->state.event_count)
+		remove_one_vcpu_event(vcpu, vcpu_event);
+
+	/*
+	 * We need to check if the registered event is pending for
+	 * unregistration. In that case, the registered event should
+	 * be unregistered and destroyed if needed.
+	 */
+	registered_event = vcpu_event->registered_event;
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
+		kvm_sdei_clear_enabled(registered_event, index);
+		kvm_sdei_clear_registered(registered_event, index);
+		if (kvm_sdei_none_registered(registered_event))
+			remove_one_registered_event(kvm, registered_event);
+	}
+
+	/* Make another request if we have any pending events */
+	if ((vsdei->critical_event_count + vsdei->normal_event_count) > 0)
+		kvm_make_request(KVM_REQ_SDEI, vcpu);
+
+unlock:
+	spin_unlock(&vsdei->lock);
+	spin_unlock(&ksdei->lock);
+
+	return ret;
+}
+
 static unsigned long
 unregister_one_event(struct kvm *kvm, struct kvm_vcpu *vcpu,
 		     struct kvm_sdei_registered_event *registered_event)
@@ -864,8 +936,10 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 		ret = hypercall_context(vcpu);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
+		ret = hypercall_complete(vcpu, false);
+		break;
 	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
-		ret = SDEI_NOT_SUPPORTED;
+		ret = hypercall_complete(vcpu, true);
 		break;
 	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
 		ret = hypercall_unregister(vcpu);
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 17/22] KVM: arm64: Support SDEI event notifier
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

The owner of the SDEI event may want to know if the injected event
has been delivered and handled. For example, Async PF cancels the
injected SDEI event at early stage of the work, driven by the
event has been completed asynchronously.

This supports SDEI event state updating by introducing notifier
mechanism. It's notable the notifier (handler) should be capable of
migration.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_sdei.h | 10 +++++++
 arch/arm64/kvm/sdei.c             | 45 +++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 54c730acd298..2480ec0e9824 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -18,6 +18,14 @@
 
 struct kvm_vcpu;
 
+typedef void (*kvm_sdei_notifier)(struct kvm_vcpu *vcpu,
+				  unsigned long num,
+				  unsigned int state);
+enum {
+	KVM_SDEI_NOTIFY_DELIVERED,
+	KVM_SDEI_NOTIFY_COMPLETED,
+};
+
 struct kvm_sdei_exposed_event {
 	struct kvm_sdei_exposed_event_state	state;
 	struct kvm				*kvm;
@@ -165,6 +173,8 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
 void kvm_sdei_init_vm(struct kvm *kvm);
 void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+int kvm_sdei_register_event_notifier(struct kvm *kvm, unsigned long num,
+				     kvm_sdei_notifier notifier);
 int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num, bool immediate);
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 3019ac196e76..9f1959653318 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -353,6 +353,7 @@ static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
 	struct kvm_sdei_registered_event *registered_event;
 	struct kvm_sdei_vcpu_event *vcpu_event;
 	struct kvm_sdei_vcpu_regs_state *regs;
+	kvm_sdei_notifier notifier = NULL;
 	unsigned long ret = SDEI_SUCCESS;
 	int index;
 
@@ -397,6 +398,7 @@ static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
 	 */
 	registered_event = vcpu_event->registered_event;
 	exposed_event = registered_event->exposed_event;
+	notifier = (kvm_sdei_notifier)(exposed_event->state.notifier);
 	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
 	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
 		kvm_sdei_clear_enabled(registered_event, index);
@@ -413,6 +415,12 @@ static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
 	spin_unlock(&vsdei->lock);
 	spin_unlock(&ksdei->lock);
 
+	/* Notifier */
+	if (notifier) {
+		notifier(vcpu, exposed_event->state.num,
+			 KVM_SDEI_NOTIFY_COMPLETED);
+	}
+
 	return ret;
 }
 
@@ -986,6 +994,35 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+int kvm_sdei_register_event_notifier(struct kvm *kvm,
+				     unsigned long num,
+				     kvm_sdei_notifier notifier)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	int ret = 0;
+
+	if (!ksdei) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	exposed_event = find_exposed_event(kvm, num);
+	if (!exposed_event) {
+		ret = -EINVAL;
+		goto unlock;
+	}
+
+	exposed_event->state.notifier = (unsigned long)notifier;
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num,
 			  bool immediate)
@@ -1100,6 +1137,7 @@ void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
 	struct kvm_sdei_registered_event *registered_event;
 	struct kvm_sdei_vcpu_event *vcpu_event;
 	struct kvm_sdei_vcpu_regs_state *regs;
+	kvm_sdei_notifier notifier = NULL;
 	unsigned long pstate;
 	int index;
 
@@ -1129,6 +1167,7 @@ void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
 
 	registered_event = vcpu_event->registered_event;
 	exposed_event = registered_event->exposed_event;
+	notifier = (kvm_sdei_notifier)(exposed_event->state.notifier);
 	if (kvm_sdei_is_critical(exposed_event->state.priority)) {
 		vsdei->critical_event = vcpu_event;
 		vsdei->state.critical_num = vcpu_event->state.num;
@@ -1179,6 +1218,12 @@ void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
 
 unlock:
 	spin_unlock(&vsdei->lock);
+
+	/* Notifier */
+	if (notifier) {
+		notifier(vcpu, exposed_event->state.num,
+			 KVM_SDEI_NOTIFY_DELIVERED);
+	}
 }
 
 void kvm_sdei_init_vm(struct kvm *kvm)
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 17/22] KVM: arm64: Support SDEI event notifier
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

The owner of the SDEI event may want to know if the injected event
has been delivered and handled. For example, Async PF cancels the
injected SDEI event at early stage of the work, driven by the
event has been completed asynchronously.

This supports SDEI event state updating by introducing notifier
mechanism. It's notable the notifier (handler) should be capable of
migration.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_sdei.h | 10 +++++++
 arch/arm64/kvm/sdei.c             | 45 +++++++++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 54c730acd298..2480ec0e9824 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -18,6 +18,14 @@
 
 struct kvm_vcpu;
 
+typedef void (*kvm_sdei_notifier)(struct kvm_vcpu *vcpu,
+				  unsigned long num,
+				  unsigned int state);
+enum {
+	KVM_SDEI_NOTIFY_DELIVERED,
+	KVM_SDEI_NOTIFY_COMPLETED,
+};
+
 struct kvm_sdei_exposed_event {
 	struct kvm_sdei_exposed_event_state	state;
 	struct kvm				*kvm;
@@ -165,6 +173,8 @@ KVM_SDEI_REGISTERED_EVENT_FUNC(unregister_pending)
 void kvm_sdei_init_vm(struct kvm *kvm);
 void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu);
 int kvm_sdei_hypercall(struct kvm_vcpu *vcpu);
+int kvm_sdei_register_event_notifier(struct kvm *kvm, unsigned long num,
+				     kvm_sdei_notifier notifier);
 int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num, bool immediate);
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 3019ac196e76..9f1959653318 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -353,6 +353,7 @@ static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
 	struct kvm_sdei_registered_event *registered_event;
 	struct kvm_sdei_vcpu_event *vcpu_event;
 	struct kvm_sdei_vcpu_regs_state *regs;
+	kvm_sdei_notifier notifier = NULL;
 	unsigned long ret = SDEI_SUCCESS;
 	int index;
 
@@ -397,6 +398,7 @@ static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
 	 */
 	registered_event = vcpu_event->registered_event;
 	exposed_event = registered_event->exposed_event;
+	notifier = (kvm_sdei_notifier)(exposed_event->state.notifier);
 	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
 	if (kvm_sdei_is_unregister_pending(registered_event, index)) {
 		kvm_sdei_clear_enabled(registered_event, index);
@@ -413,6 +415,12 @@ static unsigned long hypercall_complete(struct kvm_vcpu *vcpu, bool resume)
 	spin_unlock(&vsdei->lock);
 	spin_unlock(&ksdei->lock);
 
+	/* Notifier */
+	if (notifier) {
+		notifier(vcpu, exposed_event->state.num,
+			 KVM_SDEI_NOTIFY_COMPLETED);
+	}
+
 	return ret;
 }
 
@@ -986,6 +994,35 @@ int kvm_sdei_hypercall(struct kvm_vcpu *vcpu)
 	return 1;
 }
 
+int kvm_sdei_register_event_notifier(struct kvm *kvm,
+				     unsigned long num,
+				     kvm_sdei_notifier notifier)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event = NULL;
+	int ret = 0;
+
+	if (!ksdei) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	exposed_event = find_exposed_event(kvm, num);
+	if (!exposed_event) {
+		ret = -EINVAL;
+		goto unlock;
+	}
+
+	exposed_event->state.notifier = (unsigned long)notifier;
+
+unlock:
+	spin_unlock(&ksdei->lock);
+out:
+	return ret;
+}
+
 int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num,
 			  bool immediate)
@@ -1100,6 +1137,7 @@ void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
 	struct kvm_sdei_registered_event *registered_event;
 	struct kvm_sdei_vcpu_event *vcpu_event;
 	struct kvm_sdei_vcpu_regs_state *regs;
+	kvm_sdei_notifier notifier = NULL;
 	unsigned long pstate;
 	int index;
 
@@ -1129,6 +1167,7 @@ void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
 
 	registered_event = vcpu_event->registered_event;
 	exposed_event = registered_event->exposed_event;
+	notifier = (kvm_sdei_notifier)(exposed_event->state.notifier);
 	if (kvm_sdei_is_critical(exposed_event->state.priority)) {
 		vsdei->critical_event = vcpu_event;
 		vsdei->state.critical_num = vcpu_event->state.num;
@@ -1179,6 +1218,12 @@ void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu)
 
 unlock:
 	spin_unlock(&vsdei->lock);
+
+	/* Notifier */
+	if (notifier) {
+		notifier(vcpu, exposed_event->state.num,
+			 KVM_SDEI_NOTIFY_DELIVERED);
+	}
 }
 
 void kvm_sdei_init_vm(struct kvm *kvm)
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports ioctl commands on VM to manage the various objects.
It's primarily used by VMM to accomplish migration. The ioctl
commands introduced by this are highlighted as below:

   * KVM_SDEI_CMD_GET_VERSION
     Retrieve the version of current implementation. It's different
     from the version of the followed SDEI specification. This version
     is used to indicates what functionalities documented in the SDEI
     specification have been supported or not supported.

   * KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
     Return the total count of exposed events.

   * KVM_SDEI_CMD_GET_EXPOSED_EVENT
   * KVM_SDEI_CMD_SET_EXPOSED_EVENT
     Get or set exposed event

   * KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
     Return the total count of registered events.

   * KVM_SDEI_CMD_GET_REGISTERED_EVENT
   * KVM_SDEI_CMD_SET_REGISTERED_EVENT
     Get or set registered event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_sdei.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  20 ++
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/sdei.c                        | 302 +++++++++++++++++++
 include/uapi/linux/kvm.h                     |   3 +
 5 files changed, 329 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 2480ec0e9824..64f00cc79162 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -179,6 +179,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num, bool immediate);
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
 void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
+long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
index b14844230117..2bd6d11627bc 100644
--- a/arch/arm64/include/uapi/asm/kvm_sdei_state.h
+++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
@@ -68,5 +68,25 @@ struct kvm_sdei_vcpu_state {
 	struct kvm_sdei_vcpu_regs_state	normal_regs;
 };
 
+#define KVM_SDEI_CMD_GET_VERSION		0
+#define KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT	1
+#define KVM_SDEI_CMD_GET_EXPOSED_EVENT		2
+#define KVM_SDEI_CMD_SET_EXPOSED_EVENT		3
+#define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT	4
+#define KVM_SDEI_CMD_GET_REGISTERED_EVENT	5
+#define KVM_SDEI_CMD_SET_REGISTERED_EVENT	6
+
+struct kvm_sdei_cmd {
+	__u32                                           cmd;
+	union {
+		__u32                                   version;
+		__u32                                   count;
+	};
+	union {
+		struct kvm_sdei_exposed_event_state     *exposed_event_state;
+		struct kvm_sdei_registered_event_state  *registered_event_state;
+	};
+};
+
 #endif /* !__ASSEMBLY__ */
 #endif /* _UAPI__ASM_KVM_SDEI_STATE_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 00c136a6e8df..ebfd504a1c08 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1465,6 +1465,9 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			return -EFAULT;
 		return kvm_vm_ioctl_mte_copy_tags(kvm, &copy_tags);
 	}
+	case KVM_ARM_SDEI_COMMAND: {
+		return kvm_sdei_vm_ioctl(kvm, arg);
+	}
 	default:
 		return -EINVAL;
 	}
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 9f1959653318..d9cf494990a9 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -1265,6 +1265,308 @@ void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu)
 	vcpu->arch.sdei = vsdei;
 }
 
+static long vm_ioctl_get_exposed_event(struct kvm *kvm,
+				       struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_exposed_event_state *state;
+	void __user *user_state = (void __user *)(cmd->exposed_event_state);
+	unsigned int count, i;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	i = 0;
+	count = cmd->count;
+	list_for_each_entry(exposed_event, &ksdei->exposed_events, link) {
+		state[i++] = exposed_event->state;
+		if (!--count)
+			break;
+	}
+
+	if (copy_to_user(user_state, state, sizeof(*state) * cmd->count))
+		ret = -EFAULT;
+
+	kfree(state);
+	return ret;
+}
+
+static long vm_ioctl_set_exposed_event(struct kvm *kvm,
+				       struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_exposed_event_state *state;
+	void __user *user_state = (void __user *)(cmd->exposed_event_state);
+	unsigned int i, j;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	if ((ksdei->exposed_event_count + cmd->count) > KVM_SDEI_MAX_EVENTS)
+		return -ERANGE;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state) * cmd->count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		if (!kvm_sdei_is_supported(state[i].num)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (!kvm_sdei_is_shared(state[i].type) &&
+		    !kvm_sdei_is_private(state[i].type)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (!kvm_sdei_is_critical(state[i].priority) &&
+		    !kvm_sdei_is_normal(state[i].priority)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		/*
+		 * Check if the event has been exposed. The notifier is
+		 * allowed to be changed.
+		 */
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (exposed_event &&
+		    (state[i].num != exposed_event->state.num           ||
+		     state[i].type != exposed_event->state.type         ||
+		     state[i].signaled != exposed_event->state.signaled ||
+		     state[i].priority != exposed_event->state.priority)) {
+			ret = -EEXIST;
+			goto out;
+		}
+
+		/* Avoid the duplicated event */
+		for (j = 0; j < cmd->count; j++) {
+			if (i != j && state[i].num == state[j].num) {
+				ret = -EINVAL;
+				goto out;
+			}
+		}
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (exposed_event) {
+			exposed_event->state = state[i];
+			continue;
+		}
+
+		exposed_event = kzalloc(sizeof(*exposed_event),
+					GFP_KERNEL_ACCOUNT);
+		if (!exposed_event) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		exposed_event->state = state[i];
+		exposed_event->kvm   = kvm;
+
+		ksdei->exposed_event_count++;
+		list_add_tail(&exposed_event->link, &ksdei->exposed_events);
+	}
+
+out:
+	kfree(state);
+	return ret;
+}
+
+static long vm_ioctl_get_registered_event(struct kvm *kvm,
+					  struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_registered_event_state *state;
+	void __user *user_state = (void __user *)(cmd->registered_event_state);
+	unsigned int count, i;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	i = 0;
+	count = cmd->count;
+	list_for_each_entry(registered_event,
+			    &ksdei->registered_events, link) {
+		state[i++] = registered_event->state;
+		if (!--count)
+			break;
+	}
+
+	if (copy_to_user(user_state, state, sizeof(*state) * cmd->count))
+		ret = -EFAULT;
+
+	kfree(state);
+	return ret;
+}
+
+static long vm_ioctl_set_registered_event(struct kvm *kvm,
+					  struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_registered_event_state *state;
+	void __user *user_state = (void __user *)(cmd->registered_event_state);
+	unsigned int i, j;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	if ((ksdei->registered_event_count + cmd->count) > KVM_SDEI_MAX_EVENTS)
+		return -ERANGE;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state) * cmd->count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		if (!kvm_sdei_is_supported(state[i].num)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (state[i].route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+		    state[i].route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		/* Check if the event has been exposed */
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (!exposed_event) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		/* Check if the event has been registered */
+		registered_event = find_registered_event(kvm, state[i].num);
+		if (registered_event) {
+			ret = -EEXIST;
+			goto out;
+		}
+
+		/* Avoid the duplicated event */
+		for (j = 0; j < cmd->count; j++) {
+			if (i != j && state[i].num == state[j].num) {
+				ret = -EINVAL;
+				goto out;
+			}
+		}
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		registered_event = kzalloc(sizeof(*registered_event),
+					   GFP_KERNEL_ACCOUNT);
+		if (!registered_event) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		registered_event->state         = state[i];
+		registered_event->kvm           = kvm;
+		registered_event->exposed_event = exposed_event;
+
+		ksdei->registered_event_count++;
+		exposed_event->registered_event_count++;
+		list_add_tail(&registered_event->link,
+			      &ksdei->registered_events);
+	}
+
+out:
+	kfree(state);
+	return ret;
+}
+
+long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_cmd *cmd;
+	void __user *argp = (void __user *)arg;
+	long ret = 0;
+
+	if (!ksdei)
+		return -EPERM;
+
+	cmd = kzalloc(sizeof(*cmd), GFP_KERNEL_ACCOUNT);
+	if (!cmd)
+		return -ENOMEM;
+
+	if (copy_from_user(cmd, argp, sizeof(*cmd))) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	switch (cmd->cmd) {
+	case KVM_SDEI_CMD_GET_VERSION:
+		cmd->version = (1 << 16);       /* v1.0.0 */
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT:
+		cmd->count = ksdei->exposed_event_count;
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_EXPOSED_EVENT:
+		ret = vm_ioctl_get_exposed_event(kvm, cmd);
+		break;
+	case KVM_SDEI_CMD_SET_EXPOSED_EVENT:
+		ret = vm_ioctl_set_exposed_event(kvm, cmd);
+		break;
+	case KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT:
+		cmd->count = ksdei->registered_event_count;
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_REGISTERED_EVENT:
+		ret = vm_ioctl_get_registered_event(kvm, cmd);
+		break;
+	case KVM_SDEI_CMD_SET_REGISTERED_EVENT:
+		ret = vm_ioctl_set_registered_event(kvm, cmd);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	spin_unlock(&ksdei->lock);
+out:
+
+	kfree(cmd);
+	return ret;
+}
+
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 507ee1f2aa96..2d11c909ec42 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -2049,4 +2049,7 @@ struct kvm_stats_desc {
 /* Available with KVM_CAP_XSAVE2 */
 #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
 
+/* Available with KVM_CAP_ARM_SDEI */
+#define KVM_ARM_SDEI_COMMAND	_IOWR(KVMIO, 0xd0, struct kvm_sdei_cmd)
+
 #endif /* __LINUX_KVM_H */
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports ioctl commands on VM to manage the various objects.
It's primarily used by VMM to accomplish migration. The ioctl
commands introduced by this are highlighted as below:

   * KVM_SDEI_CMD_GET_VERSION
     Retrieve the version of current implementation. It's different
     from the version of the followed SDEI specification. This version
     is used to indicates what functionalities documented in the SDEI
     specification have been supported or not supported.

   * KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
     Return the total count of exposed events.

   * KVM_SDEI_CMD_GET_EXPOSED_EVENT
   * KVM_SDEI_CMD_SET_EXPOSED_EVENT
     Get or set exposed event

   * KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
     Return the total count of registered events.

   * KVM_SDEI_CMD_GET_REGISTERED_EVENT
   * KVM_SDEI_CMD_SET_REGISTERED_EVENT
     Get or set registered event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_sdei.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |  20 ++
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/sdei.c                        | 302 +++++++++++++++++++
 include/uapi/linux/kvm.h                     |   3 +
 5 files changed, 329 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 2480ec0e9824..64f00cc79162 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -179,6 +179,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 			  unsigned long num, bool immediate);
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
 void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
+long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
index b14844230117..2bd6d11627bc 100644
--- a/arch/arm64/include/uapi/asm/kvm_sdei_state.h
+++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
@@ -68,5 +68,25 @@ struct kvm_sdei_vcpu_state {
 	struct kvm_sdei_vcpu_regs_state	normal_regs;
 };
 
+#define KVM_SDEI_CMD_GET_VERSION		0
+#define KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT	1
+#define KVM_SDEI_CMD_GET_EXPOSED_EVENT		2
+#define KVM_SDEI_CMD_SET_EXPOSED_EVENT		3
+#define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT	4
+#define KVM_SDEI_CMD_GET_REGISTERED_EVENT	5
+#define KVM_SDEI_CMD_SET_REGISTERED_EVENT	6
+
+struct kvm_sdei_cmd {
+	__u32                                           cmd;
+	union {
+		__u32                                   version;
+		__u32                                   count;
+	};
+	union {
+		struct kvm_sdei_exposed_event_state     *exposed_event_state;
+		struct kvm_sdei_registered_event_state  *registered_event_state;
+	};
+};
+
 #endif /* !__ASSEMBLY__ */
 #endif /* _UAPI__ASM_KVM_SDEI_STATE_H */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 00c136a6e8df..ebfd504a1c08 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1465,6 +1465,9 @@ long kvm_arch_vm_ioctl(struct file *filp,
 			return -EFAULT;
 		return kvm_vm_ioctl_mte_copy_tags(kvm, &copy_tags);
 	}
+	case KVM_ARM_SDEI_COMMAND: {
+		return kvm_sdei_vm_ioctl(kvm, arg);
+	}
 	default:
 		return -EINVAL;
 	}
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index 9f1959653318..d9cf494990a9 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -1265,6 +1265,308 @@ void kvm_sdei_create_vcpu(struct kvm_vcpu *vcpu)
 	vcpu->arch.sdei = vsdei;
 }
 
+static long vm_ioctl_get_exposed_event(struct kvm *kvm,
+				       struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_exposed_event_state *state;
+	void __user *user_state = (void __user *)(cmd->exposed_event_state);
+	unsigned int count, i;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	i = 0;
+	count = cmd->count;
+	list_for_each_entry(exposed_event, &ksdei->exposed_events, link) {
+		state[i++] = exposed_event->state;
+		if (!--count)
+			break;
+	}
+
+	if (copy_to_user(user_state, state, sizeof(*state) * cmd->count))
+		ret = -EFAULT;
+
+	kfree(state);
+	return ret;
+}
+
+static long vm_ioctl_set_exposed_event(struct kvm *kvm,
+				       struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_exposed_event_state *state;
+	void __user *user_state = (void __user *)(cmd->exposed_event_state);
+	unsigned int i, j;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	if ((ksdei->exposed_event_count + cmd->count) > KVM_SDEI_MAX_EVENTS)
+		return -ERANGE;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state) * cmd->count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		if (!kvm_sdei_is_supported(state[i].num)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (!kvm_sdei_is_shared(state[i].type) &&
+		    !kvm_sdei_is_private(state[i].type)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (!kvm_sdei_is_critical(state[i].priority) &&
+		    !kvm_sdei_is_normal(state[i].priority)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		/*
+		 * Check if the event has been exposed. The notifier is
+		 * allowed to be changed.
+		 */
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (exposed_event &&
+		    (state[i].num != exposed_event->state.num           ||
+		     state[i].type != exposed_event->state.type         ||
+		     state[i].signaled != exposed_event->state.signaled ||
+		     state[i].priority != exposed_event->state.priority)) {
+			ret = -EEXIST;
+			goto out;
+		}
+
+		/* Avoid the duplicated event */
+		for (j = 0; j < cmd->count; j++) {
+			if (i != j && state[i].num == state[j].num) {
+				ret = -EINVAL;
+				goto out;
+			}
+		}
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (exposed_event) {
+			exposed_event->state = state[i];
+			continue;
+		}
+
+		exposed_event = kzalloc(sizeof(*exposed_event),
+					GFP_KERNEL_ACCOUNT);
+		if (!exposed_event) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		exposed_event->state = state[i];
+		exposed_event->kvm   = kvm;
+
+		ksdei->exposed_event_count++;
+		list_add_tail(&exposed_event->link, &ksdei->exposed_events);
+	}
+
+out:
+	kfree(state);
+	return ret;
+}
+
+static long vm_ioctl_get_registered_event(struct kvm *kvm,
+					  struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_registered_event_state *state;
+	void __user *user_state = (void __user *)(cmd->registered_event_state);
+	unsigned int count, i;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	i = 0;
+	count = cmd->count;
+	list_for_each_entry(registered_event,
+			    &ksdei->registered_events, link) {
+		state[i++] = registered_event->state;
+		if (!--count)
+			break;
+	}
+
+	if (copy_to_user(user_state, state, sizeof(*state) * cmd->count))
+		ret = -EFAULT;
+
+	kfree(state);
+	return ret;
+}
+
+static long vm_ioctl_set_registered_event(struct kvm *kvm,
+					  struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_registered_event_state *state;
+	void __user *user_state = (void __user *)(cmd->registered_event_state);
+	unsigned int i, j;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	if ((ksdei->registered_event_count + cmd->count) > KVM_SDEI_MAX_EVENTS)
+		return -ERANGE;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state) * cmd->count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		if (!kvm_sdei_is_supported(state[i].num)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		if (state[i].route_mode != SDEI_EVENT_REGISTER_RM_ANY &&
+		    state[i].route_mode != SDEI_EVENT_REGISTER_RM_PE) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		/* Check if the event has been exposed */
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (!exposed_event) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		/* Check if the event has been registered */
+		registered_event = find_registered_event(kvm, state[i].num);
+		if (registered_event) {
+			ret = -EEXIST;
+			goto out;
+		}
+
+		/* Avoid the duplicated event */
+		for (j = 0; j < cmd->count; j++) {
+			if (i != j && state[i].num == state[j].num) {
+				ret = -EINVAL;
+				goto out;
+			}
+		}
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		registered_event = kzalloc(sizeof(*registered_event),
+					   GFP_KERNEL_ACCOUNT);
+		if (!registered_event) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		registered_event->state         = state[i];
+		registered_event->kvm           = kvm;
+		registered_event->exposed_event = exposed_event;
+
+		ksdei->registered_event_count++;
+		exposed_event->registered_event_count++;
+		list_add_tail(&registered_event->link,
+			      &ksdei->registered_events);
+	}
+
+out:
+	kfree(state);
+	return ret;
+}
+
+long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg)
+{
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_cmd *cmd;
+	void __user *argp = (void __user *)arg;
+	long ret = 0;
+
+	if (!ksdei)
+		return -EPERM;
+
+	cmd = kzalloc(sizeof(*cmd), GFP_KERNEL_ACCOUNT);
+	if (!cmd)
+		return -ENOMEM;
+
+	if (copy_from_user(cmd, argp, sizeof(*cmd))) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+
+	switch (cmd->cmd) {
+	case KVM_SDEI_CMD_GET_VERSION:
+		cmd->version = (1 << 16);       /* v1.0.0 */
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT:
+		cmd->count = ksdei->exposed_event_count;
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_EXPOSED_EVENT:
+		ret = vm_ioctl_get_exposed_event(kvm, cmd);
+		break;
+	case KVM_SDEI_CMD_SET_EXPOSED_EVENT:
+		ret = vm_ioctl_set_exposed_event(kvm, cmd);
+		break;
+	case KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT:
+		cmd->count = ksdei->registered_event_count;
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_REGISTERED_EVENT:
+		ret = vm_ioctl_get_registered_event(kvm, cmd);
+		break;
+	case KVM_SDEI_CMD_SET_REGISTERED_EVENT:
+		ret = vm_ioctl_set_registered_event(kvm, cmd);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	spin_unlock(&ksdei->lock);
+out:
+
+	kfree(cmd);
+	return ret;
+}
+
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 507ee1f2aa96..2d11c909ec42 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -2049,4 +2049,7 @@ struct kvm_stats_desc {
 /* Available with KVM_CAP_XSAVE2 */
 #define KVM_GET_XSAVE2		  _IOR(KVMIO,  0xcf, struct kvm_xsave)
 
+/* Available with KVM_CAP_ARM_SDEI */
+#define KVM_ARM_SDEI_COMMAND	_IOWR(KVMIO, 0xd0, struct kvm_sdei_cmd)
+
 #endif /* __LINUX_KVM_H */
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This supports ioctl commands on vCPU to manage the various object.
It's primarily used by VMM to accomplish migration. The ioctl
commands introduced by this are highlighted as below:

   * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
     Return the total count of vCPU events, which have been queued
     on the target vCPU.

   * KVM_SDEI_CMD_GET_VCPU_EVENT
   * KVM_SDEI_CMD_SET_VCPU_EVENT
     Get or set vCPU events.

   * KVM_SDEI_CMD_GET_VCPU_STATE
   * KVM_SDEI_CMD_SET_VCPU_STATE
     Get or set vCPU state.

   * KVM_SDEI_CMD_INJECT_EVENT
     Inject SDEI event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_sdei.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |   9 +
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/sdei.c                        | 299 +++++++++++++++++++
 4 files changed, 312 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 64f00cc79162..ea4f222cf73d 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -180,6 +180,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
 void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
 long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg);
+long kvm_sdei_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned long arg);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
index 2bd6d11627bc..149451c5584f 100644
--- a/arch/arm64/include/uapi/asm/kvm_sdei_state.h
+++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
@@ -75,6 +75,12 @@ struct kvm_sdei_vcpu_state {
 #define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT	4
 #define KVM_SDEI_CMD_GET_REGISTERED_EVENT	5
 #define KVM_SDEI_CMD_SET_REGISTERED_EVENT	6
+#define KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT	7
+#define KVM_SDEI_CMD_GET_VCPU_EVENT		8
+#define KVM_SDEI_CMD_SET_VCPU_EVENT		9
+#define KVM_SDEI_CMD_GET_VCPU_STATE		10
+#define KVM_SDEI_CMD_SET_VCPU_STATE		11
+#define KVM_SDEI_CMD_INJECT_EVENT		12
 
 struct kvm_sdei_cmd {
 	__u32                                           cmd;
@@ -85,6 +91,9 @@ struct kvm_sdei_cmd {
 	union {
 		struct kvm_sdei_exposed_event_state     *exposed_event_state;
 		struct kvm_sdei_registered_event_state  *registered_event_state;
+		struct kvm_sdei_vcpu_event_state	*vcpu_event_state;
+		struct kvm_sdei_vcpu_state		*vcpu_state;
+		__u64					num;
 	};
 };
 
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index ebfd504a1c08..3f532e1c4a95 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1387,6 +1387,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 
 		return kvm_arm_vcpu_finalize(vcpu, what);
 	}
+	case KVM_ARM_SDEI_COMMAND: {
+		return kvm_sdei_vcpu_ioctl(vcpu, arg);
+	}
 	default:
 		r = -EINVAL;
 	}
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index d9cf494990a9..06895ac73c24 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -1567,6 +1567,305 @@ long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg)
 	return ret;
 }
 
+static long vcpu_ioctl_get_vcpu_event(struct kvm_vcpu *vcpu,
+				      struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_event_state *state;
+	void __user *user_state = (void __user *)(cmd->vcpu_event_state);
+	unsigned int count, i;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	i = 0;
+	count = cmd->count;
+	list_for_each_entry(vcpu_event, &vsdei->critical_events, link) {
+		state[i++] = vcpu_event->state;
+		if (!--count)
+			break;
+	}
+
+	if (count) {
+		list_for_each_entry(vcpu_event, &vsdei->normal_events, link) {
+			state[i++] = vcpu_event->state;
+			if (!--count)
+				break;
+		}
+	}
+
+	if (copy_to_user(user_state, state, sizeof(*state) * cmd->count))
+		ret = -EFAULT;
+
+	kfree(state);
+	return ret;
+}
+
+static long vcpu_ioctl_set_vcpu_event(struct kvm_vcpu *vcpu,
+				      struct kvm_sdei_cmd *cmd)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_event_state *state;
+	void __user *user_state = (void __user *)(cmd->vcpu_event_state);
+	unsigned int vcpu_event_count, i, j;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state) * cmd->count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	vcpu_event_count = vsdei->critical_event_count +
+			   vsdei->normal_event_count;
+	for (i = 0; i < cmd->count; i++) {
+		if (!kvm_sdei_is_supported(state[i].num)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		/* Check if the event has been exposed */
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (!exposed_event) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		/* Check if the event has been registered */
+		registered_event = find_registered_event(kvm, state[i].num);
+		if (!registered_event) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		/*
+		 * Calculate the total count of the vcpu event instances.
+		 * We needn't a new vcpu event instance if it is existing
+		 * or a duplicated event.
+		 */
+		vcpu_event = find_vcpu_event(vcpu, state[i].num);
+		if (vcpu_event)
+			continue;
+
+		for (j = 0; j < cmd->count; j++) {
+			if (j != i && state[j].num == state[i].num)
+				break;
+		}
+
+		if (j >= cmd->count || i < j)
+			vcpu_event_count++;
+	}
+
+	/*
+	 * Check if the required count of vcpu event instances exceeds
+	 * the limit.
+	 */
+	if (vcpu_event_count > KVM_SDEI_MAX_EVENTS) {
+		ret = -ERANGE;
+		goto out;
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		/* The vcpu event might have been existing */
+		vcpu_event = find_vcpu_event(vcpu, state[i].num);
+		if (vcpu_event) {
+			vcpu_event->state.event_count += state[i].event_count;
+			continue;
+		}
+
+		vcpu_event = kzalloc(sizeof(*vcpu_event), GFP_KERNEL_ACCOUNT);
+		if (!vcpu_event) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		registered_event = find_registered_event(kvm, state[i].num);
+		exposed_event = registered_event->exposed_event;
+
+		vcpu_event->state            = state[i];
+		vcpu_event->registered_event = registered_event;
+		vcpu_event->vcpu             = vcpu;
+
+		registered_event->vcpu_event_count++;
+		if (kvm_sdei_is_critical(exposed_event->state.priority)) {
+			list_add_tail(&vcpu_event->link,
+				      &vsdei->critical_events);
+			vsdei->critical_event_count++;
+		} else {
+			list_add_tail(&vcpu_event->link,
+				      &vsdei->normal_events);
+			vsdei->normal_event_count++;
+		}
+	}
+
+out:
+	kfree(state);
+	return ret;
+}
+
+static long vcpu_ioctl_set_vcpu_state(struct kvm_vcpu *vcpu,
+				      struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *critical_vcpu_event = NULL;
+	struct kvm_sdei_vcpu_event *normal_vcpu_event = NULL;
+	struct kvm_sdei_vcpu_state *state;
+	void __user *user_state = (void __user *)(cmd->vcpu_state);
+	long ret = 0;
+
+	state = kzalloc(sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state))) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	if (kvm_sdei_is_supported(state->critical_num)) {
+		critical_vcpu_event = find_vcpu_event(vcpu,
+						      state->critical_num);
+		if (!critical_vcpu_event) {
+			ret = -EINVAL;
+			goto out;
+		}
+	}
+
+	if (kvm_sdei_is_supported(state->normal_num)) {
+		normal_vcpu_event = find_vcpu_event(vcpu, state->normal_num);
+		if (!normal_vcpu_event) {
+			ret = -EINVAL;
+			goto out;
+		}
+	}
+
+	vsdei->state          = *state;
+	vsdei->critical_event = critical_vcpu_event;
+	vsdei->normal_event   = normal_vcpu_event;
+
+	/*
+	 * To deliver the vCPU events if we don't have a valid handler
+	 * running. Otherwise, the vCPU events should be delivered when
+	 * the running handler is completed.
+	 */
+	if (!vsdei->critical_event && !vsdei->normal_event &&
+	    (vsdei->critical_event_count + vsdei->normal_event_count) > 0)
+		kvm_make_request(KVM_REQ_SDEI, vcpu);
+
+out:
+	kfree(state);
+	return ret;
+}
+
+static long vcpu_ioctl_inject_event(struct kvm_vcpu *vcpu,
+				    struct kvm_sdei_cmd *cmd)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	int index;
+
+	if (!kvm_sdei_is_supported(cmd->num))
+		return -EINVAL;
+
+	registered_event = find_registered_event(kvm, cmd->num);
+	if (!registered_event)
+		return -ENOENT;
+
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index))
+		return -EPERM;
+
+	if (vsdei->state.masked)
+		return -EPERM;
+
+	return do_inject_event(vcpu, registered_event, false);
+}
+
+long kvm_sdei_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned long arg)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_cmd *cmd = NULL;
+	void __user *argp = (void __user *)arg;
+	long ret = 0;
+
+	if (!(ksdei && vsdei)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	cmd = kzalloc(sizeof(*cmd), GFP_KERNEL_ACCOUNT);
+	if (!cmd) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (copy_from_user(cmd, argp, sizeof(*cmd))) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+	spin_lock(&vsdei->lock);
+
+	switch (cmd->cmd) {
+	case KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT:
+		cmd->count = vsdei->critical_event_count +
+			     vsdei->normal_event_count;
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_VCPU_EVENT:
+		ret = vcpu_ioctl_get_vcpu_event(vcpu, cmd);
+		break;
+	case KVM_SDEI_CMD_SET_VCPU_EVENT:
+		ret = vcpu_ioctl_set_vcpu_event(vcpu, cmd);
+		break;
+	case KVM_SDEI_CMD_GET_VCPU_STATE:
+		if (copy_to_user(cmd->vcpu_state, &vsdei->state,
+				 sizeof(vsdei->state)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_SET_VCPU_STATE:
+		ret = vcpu_ioctl_set_vcpu_state(vcpu, cmd);
+		break;
+	case KVM_SDEI_CMD_INJECT_EVENT:
+		ret = vcpu_ioctl_inject_event(vcpu, cmd);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	spin_unlock(&vsdei->lock);
+	spin_unlock(&ksdei->lock);
+
+out:
+	kfree(cmd);
+	return ret;
+}
+
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This supports ioctl commands on vCPU to manage the various object.
It's primarily used by VMM to accomplish migration. The ioctl
commands introduced by this are highlighted as below:

   * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
     Return the total count of vCPU events, which have been queued
     on the target vCPU.

   * KVM_SDEI_CMD_GET_VCPU_EVENT
   * KVM_SDEI_CMD_SET_VCPU_EVENT
     Get or set vCPU events.

   * KVM_SDEI_CMD_GET_VCPU_STATE
   * KVM_SDEI_CMD_SET_VCPU_STATE
     Get or set vCPU state.

   * KVM_SDEI_CMD_INJECT_EVENT
     Inject SDEI event.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 arch/arm64/include/asm/kvm_sdei.h            |   1 +
 arch/arm64/include/uapi/asm/kvm_sdei_state.h |   9 +
 arch/arm64/kvm/arm.c                         |   3 +
 arch/arm64/kvm/sdei.c                        | 299 +++++++++++++++++++
 4 files changed, 312 insertions(+)

diff --git a/arch/arm64/include/asm/kvm_sdei.h b/arch/arm64/include/asm/kvm_sdei.h
index 64f00cc79162..ea4f222cf73d 100644
--- a/arch/arm64/include/asm/kvm_sdei.h
+++ b/arch/arm64/include/asm/kvm_sdei.h
@@ -180,6 +180,7 @@ int kvm_sdei_inject_event(struct kvm_vcpu *vcpu,
 int kvm_sdei_cancel_event(struct kvm_vcpu *vcpu, unsigned long num);
 void kvm_sdei_deliver_event(struct kvm_vcpu *vcpu);
 long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg);
+long kvm_sdei_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned long arg);
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu);
 void kvm_sdei_destroy_vm(struct kvm *kvm);
 
diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
index 2bd6d11627bc..149451c5584f 100644
--- a/arch/arm64/include/uapi/asm/kvm_sdei_state.h
+++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
@@ -75,6 +75,12 @@ struct kvm_sdei_vcpu_state {
 #define KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT	4
 #define KVM_SDEI_CMD_GET_REGISTERED_EVENT	5
 #define KVM_SDEI_CMD_SET_REGISTERED_EVENT	6
+#define KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT	7
+#define KVM_SDEI_CMD_GET_VCPU_EVENT		8
+#define KVM_SDEI_CMD_SET_VCPU_EVENT		9
+#define KVM_SDEI_CMD_GET_VCPU_STATE		10
+#define KVM_SDEI_CMD_SET_VCPU_STATE		11
+#define KVM_SDEI_CMD_INJECT_EVENT		12
 
 struct kvm_sdei_cmd {
 	__u32                                           cmd;
@@ -85,6 +91,9 @@ struct kvm_sdei_cmd {
 	union {
 		struct kvm_sdei_exposed_event_state     *exposed_event_state;
 		struct kvm_sdei_registered_event_state  *registered_event_state;
+		struct kvm_sdei_vcpu_event_state	*vcpu_event_state;
+		struct kvm_sdei_vcpu_state		*vcpu_state;
+		__u64					num;
 	};
 };
 
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index ebfd504a1c08..3f532e1c4a95 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1387,6 +1387,9 @@ long kvm_arch_vcpu_ioctl(struct file *filp,
 
 		return kvm_arm_vcpu_finalize(vcpu, what);
 	}
+	case KVM_ARM_SDEI_COMMAND: {
+		return kvm_sdei_vcpu_ioctl(vcpu, arg);
+	}
 	default:
 		r = -EINVAL;
 	}
diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
index d9cf494990a9..06895ac73c24 100644
--- a/arch/arm64/kvm/sdei.c
+++ b/arch/arm64/kvm/sdei.c
@@ -1567,6 +1567,305 @@ long kvm_sdei_vm_ioctl(struct kvm *kvm, unsigned long arg)
 	return ret;
 }
 
+static long vcpu_ioctl_get_vcpu_event(struct kvm_vcpu *vcpu,
+				      struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_event_state *state;
+	void __user *user_state = (void __user *)(cmd->vcpu_event_state);
+	unsigned int count, i;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	i = 0;
+	count = cmd->count;
+	list_for_each_entry(vcpu_event, &vsdei->critical_events, link) {
+		state[i++] = vcpu_event->state;
+		if (!--count)
+			break;
+	}
+
+	if (count) {
+		list_for_each_entry(vcpu_event, &vsdei->normal_events, link) {
+			state[i++] = vcpu_event->state;
+			if (!--count)
+				break;
+		}
+	}
+
+	if (copy_to_user(user_state, state, sizeof(*state) * cmd->count))
+		ret = -EFAULT;
+
+	kfree(state);
+	return ret;
+}
+
+static long vcpu_ioctl_set_vcpu_event(struct kvm_vcpu *vcpu,
+				      struct kvm_sdei_cmd *cmd)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	struct kvm_sdei_vcpu_event *vcpu_event;
+	struct kvm_sdei_vcpu_event_state *state;
+	void __user *user_state = (void __user *)(cmd->vcpu_event_state);
+	unsigned int vcpu_event_count, i, j;
+	long ret = 0;
+
+	if (!cmd->count)
+		return 0;
+
+	state = kcalloc(cmd->count, sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state) * cmd->count)) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	vcpu_event_count = vsdei->critical_event_count +
+			   vsdei->normal_event_count;
+	for (i = 0; i < cmd->count; i++) {
+		if (!kvm_sdei_is_supported(state[i].num)) {
+			ret = -EINVAL;
+			goto out;
+		}
+
+		/* Check if the event has been exposed */
+		exposed_event = find_exposed_event(kvm, state[i].num);
+		if (!exposed_event) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		/* Check if the event has been registered */
+		registered_event = find_registered_event(kvm, state[i].num);
+		if (!registered_event) {
+			ret = -ENOENT;
+			goto out;
+		}
+
+		/*
+		 * Calculate the total count of the vcpu event instances.
+		 * We needn't a new vcpu event instance if it is existing
+		 * or a duplicated event.
+		 */
+		vcpu_event = find_vcpu_event(vcpu, state[i].num);
+		if (vcpu_event)
+			continue;
+
+		for (j = 0; j < cmd->count; j++) {
+			if (j != i && state[j].num == state[i].num)
+				break;
+		}
+
+		if (j >= cmd->count || i < j)
+			vcpu_event_count++;
+	}
+
+	/*
+	 * Check if the required count of vcpu event instances exceeds
+	 * the limit.
+	 */
+	if (vcpu_event_count > KVM_SDEI_MAX_EVENTS) {
+		ret = -ERANGE;
+		goto out;
+	}
+
+	for (i = 0; i < cmd->count; i++) {
+		/* The vcpu event might have been existing */
+		vcpu_event = find_vcpu_event(vcpu, state[i].num);
+		if (vcpu_event) {
+			vcpu_event->state.event_count += state[i].event_count;
+			continue;
+		}
+
+		vcpu_event = kzalloc(sizeof(*vcpu_event), GFP_KERNEL_ACCOUNT);
+		if (!vcpu_event) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		registered_event = find_registered_event(kvm, state[i].num);
+		exposed_event = registered_event->exposed_event;
+
+		vcpu_event->state            = state[i];
+		vcpu_event->registered_event = registered_event;
+		vcpu_event->vcpu             = vcpu;
+
+		registered_event->vcpu_event_count++;
+		if (kvm_sdei_is_critical(exposed_event->state.priority)) {
+			list_add_tail(&vcpu_event->link,
+				      &vsdei->critical_events);
+			vsdei->critical_event_count++;
+		} else {
+			list_add_tail(&vcpu_event->link,
+				      &vsdei->normal_events);
+			vsdei->normal_event_count++;
+		}
+	}
+
+out:
+	kfree(state);
+	return ret;
+}
+
+static long vcpu_ioctl_set_vcpu_state(struct kvm_vcpu *vcpu,
+				      struct kvm_sdei_cmd *cmd)
+{
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_vcpu_event *critical_vcpu_event = NULL;
+	struct kvm_sdei_vcpu_event *normal_vcpu_event = NULL;
+	struct kvm_sdei_vcpu_state *state;
+	void __user *user_state = (void __user *)(cmd->vcpu_state);
+	long ret = 0;
+
+	state = kzalloc(sizeof(*state), GFP_KERNEL_ACCOUNT);
+	if (!state)
+		return -ENOMEM;
+
+	if (copy_from_user(state, user_state, sizeof(*state))) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	if (kvm_sdei_is_supported(state->critical_num)) {
+		critical_vcpu_event = find_vcpu_event(vcpu,
+						      state->critical_num);
+		if (!critical_vcpu_event) {
+			ret = -EINVAL;
+			goto out;
+		}
+	}
+
+	if (kvm_sdei_is_supported(state->normal_num)) {
+		normal_vcpu_event = find_vcpu_event(vcpu, state->normal_num);
+		if (!normal_vcpu_event) {
+			ret = -EINVAL;
+			goto out;
+		}
+	}
+
+	vsdei->state          = *state;
+	vsdei->critical_event = critical_vcpu_event;
+	vsdei->normal_event   = normal_vcpu_event;
+
+	/*
+	 * To deliver the vCPU events if we don't have a valid handler
+	 * running. Otherwise, the vCPU events should be delivered when
+	 * the running handler is completed.
+	 */
+	if (!vsdei->critical_event && !vsdei->normal_event &&
+	    (vsdei->critical_event_count + vsdei->normal_event_count) > 0)
+		kvm_make_request(KVM_REQ_SDEI, vcpu);
+
+out:
+	kfree(state);
+	return ret;
+}
+
+static long vcpu_ioctl_inject_event(struct kvm_vcpu *vcpu,
+				    struct kvm_sdei_cmd *cmd)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_exposed_event *exposed_event;
+	struct kvm_sdei_registered_event *registered_event;
+	int index;
+
+	if (!kvm_sdei_is_supported(cmd->num))
+		return -EINVAL;
+
+	registered_event = find_registered_event(kvm, cmd->num);
+	if (!registered_event)
+		return -ENOENT;
+
+	exposed_event = registered_event->exposed_event;
+	index = kvm_sdei_vcpu_index(vcpu, exposed_event);
+	if (!kvm_sdei_is_registered(registered_event, index) ||
+	    !kvm_sdei_is_enabled(registered_event, index) ||
+	    kvm_sdei_is_unregister_pending(registered_event, index))
+		return -EPERM;
+
+	if (vsdei->state.masked)
+		return -EPERM;
+
+	return do_inject_event(vcpu, registered_event, false);
+}
+
+long kvm_sdei_vcpu_ioctl(struct kvm_vcpu *vcpu, unsigned long arg)
+{
+	struct kvm *kvm = vcpu->kvm;
+	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
+	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
+	struct kvm_sdei_cmd *cmd = NULL;
+	void __user *argp = (void __user *)arg;
+	long ret = 0;
+
+	if (!(ksdei && vsdei)) {
+		ret = -EPERM;
+		goto out;
+	}
+
+	cmd = kzalloc(sizeof(*cmd), GFP_KERNEL_ACCOUNT);
+	if (!cmd) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (copy_from_user(cmd, argp, sizeof(*cmd))) {
+		ret = -EFAULT;
+		goto out;
+	}
+
+	spin_lock(&ksdei->lock);
+	spin_lock(&vsdei->lock);
+
+	switch (cmd->cmd) {
+	case KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT:
+		cmd->count = vsdei->critical_event_count +
+			     vsdei->normal_event_count;
+		if (copy_to_user(argp, cmd, sizeof(*cmd)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_GET_VCPU_EVENT:
+		ret = vcpu_ioctl_get_vcpu_event(vcpu, cmd);
+		break;
+	case KVM_SDEI_CMD_SET_VCPU_EVENT:
+		ret = vcpu_ioctl_set_vcpu_event(vcpu, cmd);
+		break;
+	case KVM_SDEI_CMD_GET_VCPU_STATE:
+		if (copy_to_user(cmd->vcpu_state, &vsdei->state,
+				 sizeof(vsdei->state)))
+			ret = -EFAULT;
+		break;
+	case KVM_SDEI_CMD_SET_VCPU_STATE:
+		ret = vcpu_ioctl_set_vcpu_state(vcpu, cmd);
+		break;
+	case KVM_SDEI_CMD_INJECT_EVENT:
+		ret = vcpu_ioctl_inject_event(vcpu, cmd);
+		break;
+	default:
+		ret = -EINVAL;
+	}
+
+	spin_unlock(&vsdei->lock);
+	spin_unlock(&ksdei->lock);
+
+out:
+	kfree(cmd);
+	return ret;
+}
+
 void kvm_sdei_destroy_vcpu(struct kvm_vcpu *vcpu)
 {
 	struct kvm *kvm = vcpu->kvm;
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 20/22] KVM: arm64: Export SDEI capability
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

The SDEI functionality is ready to be exported so far. This adds
new capability (KVM_CAP_ARM_SDEI) and exports it.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 Documentation/virt/kvm/api.rst | 10 ++++++++++
 arch/arm64/kvm/arm.c           |  3 +++
 include/uapi/linux/kvm.h       |  1 +
 3 files changed, 14 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 9f3172376ec3..06cf27f37b4d 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7575,3 +7575,13 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset
 of the result of KVM_CHECK_EXTENSION.  KVM will forward to userspace
 the hypercalls whose corresponding bit is in the argument, and return
 ENOSYS for the others.
+
+8.35 KVM_CAP_ARM_SDEI
+---------------------
+
+:Capability: KVM_CAP_ARM_SDEI
+:Architectures: arm64
+
+This capability indicates that the SDEI virtual service is supported
+in the host. A VMM can check whether the service is available to enable
+it.
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 3f532e1c4a95..ae3b53dfe88f 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -282,6 +282,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_SDEI:
+		r = 1;
+		break;
 	default:
 		r = 0;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 2d11c909ec42..5772385639b8 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1135,6 +1135,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_XSAVE2 208
 #define KVM_CAP_SYS_ATTRIBUTES 209
 #define KVM_CAP_PPC_AIL_MODE_3 210
+#define KVM_CAP_ARM_SDEI 211
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 20/22] KVM: arm64: Export SDEI capability
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

The SDEI functionality is ready to be exported so far. This adds
new capability (KVM_CAP_ARM_SDEI) and exports it.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 Documentation/virt/kvm/api.rst | 10 ++++++++++
 arch/arm64/kvm/arm.c           |  3 +++
 include/uapi/linux/kvm.h       |  1 +
 3 files changed, 14 insertions(+)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 9f3172376ec3..06cf27f37b4d 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7575,3 +7575,13 @@ The argument to KVM_ENABLE_CAP is also a bitmask, and must be a subset
 of the result of KVM_CHECK_EXTENSION.  KVM will forward to userspace
 the hypercalls whose corresponding bit is in the argument, and return
 ENOSYS for the others.
+
+8.35 KVM_CAP_ARM_SDEI
+---------------------
+
+:Capability: KVM_CAP_ARM_SDEI
+:Architectures: arm64
+
+This capability indicates that the SDEI virtual service is supported
+in the host. A VMM can check whether the service is available to enable
+it.
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 3f532e1c4a95..ae3b53dfe88f 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -282,6 +282,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_ARM_PTRAUTH_GENERIC:
 		r = system_has_full_ptr_auth();
 		break;
+	case KVM_CAP_ARM_SDEI:
+		r = 1;
+		break;
 	default:
 		r = 0;
 	}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 2d11c909ec42..5772385639b8 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1135,6 +1135,7 @@ struct kvm_ppc_resize_hpt {
 #define KVM_CAP_XSAVE2 208
 #define KVM_CAP_SYS_ATTRIBUTES 209
 #define KVM_CAP_PPC_AIL_MODE_3 210
+#define KVM_CAP_ARM_SDEI 211
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 21/22] KVM: arm64: Add SDEI document
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This adds one document to explain how virtualized SDEI service is
implemented and supported.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 Documentation/virt/kvm/arm/sdei.rst | 325 ++++++++++++++++++++++++++++
 1 file changed, 325 insertions(+)
 create mode 100644 Documentation/virt/kvm/arm/sdei.rst

diff --git a/Documentation/virt/kvm/arm/sdei.rst b/Documentation/virt/kvm/arm/sdei.rst
new file mode 100644
index 000000000000..61213e4b9aea
--- /dev/null
+++ b/Documentation/virt/kvm/arm/sdei.rst
@@ -0,0 +1,325 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================
+SDEI Virtualization Support for ARM64
+=====================================
+
+Arm specification DEN0054/C defines Software Delegated Exception Interface
+(SDEI). It provides a mechanism for registering and servicing system events
+from system firmware. The interface is offered by a higher exception level
+to a lower exception level, in other words by a secure platform firmware
+to hypervisor or hypervisor to OS or both.
+
+https://developer.arm.com/documentation/den0054/c
+
+KVM/arm64 implements the defined hypercalls in the specification so that
+the system events can be registered and serviced from KVM hypervisor to
+the guest OS.
+
+SDEI Event Management
+=====================
+
+Each SDEI event is associated and identified with a 32-bit number. The
+lower 24-bits is the event number, but other bits are reserved or used
+for vendor defined events. In KVM/arm64 implementation, bit 22 and 23
+are further reserved to identify the events visible to the implementation.
+Value 0x1 should be seen in this field for those KVM/arm64 visible events.
+In the meanwhile, event number 0x0 is also supported since it is used by
+SDEI_EVENT_SIGNAL hypercall.
+
+The SDEI event needs to be exposed through ioctl interface by VMM to
+hypervisor before the guest is able to register it. The exposed event
+is represented by ``struct kvm_sdei_exposed_event``. The registered event
+is represented by ``struct kvm_sdei_registered_event``, whose instances
+are created on SDEI_EVENT_REGISTER hypercall. There is only one registered
+event instance in one particular VM no matter what event type is. The
+registered event can be injected and delivered to one specific vcpu in
+forms of vcpu event. ``struct kvm_sdei_vcpu_event`` describes vcpu events.
+On the particular vcpu, one vcpu event instance can be shared by multiple
+events.
+
+The execution runs into SDEI context when vcpu event is handled. The
+interrupted or preempted context should be saved to vcpu state, which
+is represented by ``struct kvm_sdei_vcpu``. After that, the SDEI event
+handler, which was provided by SDEI_EVENT_REGISTER hypercall is invoked.
+SDEI_EVENT_COMPLETE or SDEI_EVENT_COMPLETE_AND_RESUME hypercall must be
+issued before the SDEI event handler is to complete. When one of these
+two hypercalls is received, the interrupted or preempted context is
+restored from vcpu state for execution.
+
+When migration happens, the status of one particular SDEI event is not
+deterministic. So we need to support migration for the aforementioned
+structures or objects. The information capable of migration in these
+objects are put into separate data structures, which are the corresponding
+state variant in kvm_sdei_state.h. Besides, ioctl command are introduced
+to read and write them on the source and destination VM during migration.
+
+IOCTL Commands
+==============
+
+KVM_ARM_SDEI_COMMAND
+--------------------
+
+:Capability: KVM_CAP_ARM_SDEI
+:Type: vm ioctl, vcpu ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_cmd {
+        __u32        cmd;
+        union {
+            __u32    version;
+            __u32    count;
+        };
+        union {
+            struct kvm_sdei_exposed_event_state     *exposed_event_state;
+            struct kvm_sdei_registered_event_state  *registered_event_state;
+            struct kvm_sdei_vcpu_event_state        *vcpu_event_state;
+            struct kvm_sdei_vcpu_state              *vcpu_state;
+            __u64                                   num;
+        };
+    };
+
+The SDEI ioctl command is identified by KVM_ARM_SDEI_COMMAND and ``cmd``
+in the argument ``struct kvm_sdei_cmd`` provides further command to be
+executed.
+
+KVM_SDEI_CMD_GET_VERSION
+------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+On success, the implementation version is returned in ``version`` of
+``struct kvm_sdei_cmd``. This version is different from that of the
+followed SDEI specification. The implementation version is used to tell
+the coherence extent to the following specification. For example, the
+SDEI interrupt binding event is defined in SDEI v1.1 specification,
+but it is not supported in current implementation.
+
+KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
+------------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_EXPOSED_EVENT, to
+prepare ``exposed_event_state`` of ``struct kvm_sdei_cmd`` for that
+command during migration.
+
+On success, the number of exposed events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_EXPOSED_EVENT
+------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_exposed_event_state {
+        __u64   num;
+
+        __u8    type;
+        __u8    signaled;
+        __u8    priority;
+        __u8    padding[5];
+        __u64   notifier;
+    };
+
+This ioctl command is used to retrieve the exposed events on the source
+VM during migration.
+
+The number of exposed events to be retrieved is specified by ``count``
+of ``struct kvm_sdei_cmd``. On success, the retrieved exposed events are
+returned by ``exposed_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_EXPOSED_EVENT
+------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to expose SDEI events and migrate
+them. The ``notifier`` of ``struct kvm_sdei_exposed_event_state``
+will be modified if the specified exposed event has been existing.
+
+The number of events to be exposed is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the exposed events is
+passed by ``exposed_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
+---------------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_REGISTERED_EVENT,
+to prepare ``registered_event_state`` of ``struct kvm_sdei_cmd`` for
+that command during migration.
+
+On success, the number of registered events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_REGISTERED_EVENT
+---------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+    struct kvm_sdei_registered_event_state {
+        __u64   num;
+
+        __u8    route_mode;
+        __u8    padding[3];
+        __u64   route_affinity;
+        __u64   ep_address[KVM_SDEI_MAX_VCPUS];
+        __u64   ep_arg[KVM_SDEI_MAX_VCPUS];
+        __u64   registered[KVM_SDEI_MAX_VCPUS/64];
+        __u64   enabled[KVM_SDEI_MAX_VCPUS/64];
+        __u64   unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+    };
+
+This ioctl command is used to retrieve the registered events on the
+source VM during migration.
+
+The number of registered events to be retrieved is specified by ``count``
+of ``struct kvm_sdei_cmd``. On success, the retrieved registered events
+are returned by ``registered_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_REGISTERED_EVENT
+---------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the registered events
+on the destination VM.
+
+The number of events to be registered is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the registered events
+is passed by ``registered_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
+---------------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_VCPU_EVENT, to
+prepare ``vcpu_event_state`` of ``struct kvm_sdei_cmd`` for that
+command during migration.
+
+On success, the number of vcpu events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_EVENT
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_vcpu_event_state {
+        __u64   num;
+
+        __u32   event_count;
+        __u32   padding;
+    };
+
+This ioctl command is used to retrieve the vcpu events on the source
+VM during migration.
+
+The number of vcpu events to be retrieved is specified by ``count`` of
+``struct kvm_sdei_cmd``. On success, the retrieved exposed events are
+returned by ``vcpu_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_VCPU_EVENT
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the vcpu events on the
+destination VM.
+
+The number of vcpu events to be added is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the vcpu events is
+passed by ``vcpu_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_STATE
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_vcpu_regs_state {
+        __u64   regs[18];
+        __u64   pc;
+        __u64   pstate;
+    };
+
+    struct kvm_sdei_vcpu_state {
+        __u8                            masked;
+        __u8                            padding[7];
+        __u64                           critical_num;
+        __u64                           normal_num;
+        struct kvm_sdei_vcpu_regs_state critical_regs;
+        struct kvm_sdei_vcpu_regs_state normal_regs;
+    };
+
+This ioctl command is used to retrieve the vcpu state on the source VM
+during migration.
+
+On success, the current vcpu state is returned by ``vcpu_state`` of
+``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_SET_VCPU_STATE
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the vcpu state on the
+destination VM.
+
+The vcpu state to be configured is passed by ``vcpu_state`` of
+``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_INJECT_EVENT
+-------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to inject SDEI event to the specified
+vcpu.
+
+The number of the SDEI event to be injected is passed by ``num`` of
+``struct kvm_sdei_cmd``.
+
+Future Work
+===========
+
+1. Support the routing mode and affinity for the shared events.
+2. Support interrupt binding events.
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 21/22] KVM: arm64: Add SDEI document
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This adds one document to explain how virtualized SDEI service is
implemented and supported.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 Documentation/virt/kvm/arm/sdei.rst | 325 ++++++++++++++++++++++++++++
 1 file changed, 325 insertions(+)
 create mode 100644 Documentation/virt/kvm/arm/sdei.rst

diff --git a/Documentation/virt/kvm/arm/sdei.rst b/Documentation/virt/kvm/arm/sdei.rst
new file mode 100644
index 000000000000..61213e4b9aea
--- /dev/null
+++ b/Documentation/virt/kvm/arm/sdei.rst
@@ -0,0 +1,325 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+=====================================
+SDEI Virtualization Support for ARM64
+=====================================
+
+Arm specification DEN0054/C defines Software Delegated Exception Interface
+(SDEI). It provides a mechanism for registering and servicing system events
+from system firmware. The interface is offered by a higher exception level
+to a lower exception level, in other words by a secure platform firmware
+to hypervisor or hypervisor to OS or both.
+
+https://developer.arm.com/documentation/den0054/c
+
+KVM/arm64 implements the defined hypercalls in the specification so that
+the system events can be registered and serviced from KVM hypervisor to
+the guest OS.
+
+SDEI Event Management
+=====================
+
+Each SDEI event is associated and identified with a 32-bit number. The
+lower 24-bits is the event number, but other bits are reserved or used
+for vendor defined events. In KVM/arm64 implementation, bit 22 and 23
+are further reserved to identify the events visible to the implementation.
+Value 0x1 should be seen in this field for those KVM/arm64 visible events.
+In the meanwhile, event number 0x0 is also supported since it is used by
+SDEI_EVENT_SIGNAL hypercall.
+
+The SDEI event needs to be exposed through ioctl interface by VMM to
+hypervisor before the guest is able to register it. The exposed event
+is represented by ``struct kvm_sdei_exposed_event``. The registered event
+is represented by ``struct kvm_sdei_registered_event``, whose instances
+are created on SDEI_EVENT_REGISTER hypercall. There is only one registered
+event instance in one particular VM no matter what event type is. The
+registered event can be injected and delivered to one specific vcpu in
+forms of vcpu event. ``struct kvm_sdei_vcpu_event`` describes vcpu events.
+On the particular vcpu, one vcpu event instance can be shared by multiple
+events.
+
+The execution runs into SDEI context when vcpu event is handled. The
+interrupted or preempted context should be saved to vcpu state, which
+is represented by ``struct kvm_sdei_vcpu``. After that, the SDEI event
+handler, which was provided by SDEI_EVENT_REGISTER hypercall is invoked.
+SDEI_EVENT_COMPLETE or SDEI_EVENT_COMPLETE_AND_RESUME hypercall must be
+issued before the SDEI event handler is to complete. When one of these
+two hypercalls is received, the interrupted or preempted context is
+restored from vcpu state for execution.
+
+When migration happens, the status of one particular SDEI event is not
+deterministic. So we need to support migration for the aforementioned
+structures or objects. The information capable of migration in these
+objects are put into separate data structures, which are the corresponding
+state variant in kvm_sdei_state.h. Besides, ioctl command are introduced
+to read and write them on the source and destination VM during migration.
+
+IOCTL Commands
+==============
+
+KVM_ARM_SDEI_COMMAND
+--------------------
+
+:Capability: KVM_CAP_ARM_SDEI
+:Type: vm ioctl, vcpu ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_cmd {
+        __u32        cmd;
+        union {
+            __u32    version;
+            __u32    count;
+        };
+        union {
+            struct kvm_sdei_exposed_event_state     *exposed_event_state;
+            struct kvm_sdei_registered_event_state  *registered_event_state;
+            struct kvm_sdei_vcpu_event_state        *vcpu_event_state;
+            struct kvm_sdei_vcpu_state              *vcpu_state;
+            __u64                                   num;
+        };
+    };
+
+The SDEI ioctl command is identified by KVM_ARM_SDEI_COMMAND and ``cmd``
+in the argument ``struct kvm_sdei_cmd`` provides further command to be
+executed.
+
+KVM_SDEI_CMD_GET_VERSION
+------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+On success, the implementation version is returned in ``version`` of
+``struct kvm_sdei_cmd``. This version is different from that of the
+followed SDEI specification. The implementation version is used to tell
+the coherence extent to the following specification. For example, the
+SDEI interrupt binding event is defined in SDEI v1.1 specification,
+but it is not supported in current implementation.
+
+KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
+------------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_EXPOSED_EVENT, to
+prepare ``exposed_event_state`` of ``struct kvm_sdei_cmd`` for that
+command during migration.
+
+On success, the number of exposed events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_EXPOSED_EVENT
+------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_exposed_event_state {
+        __u64   num;
+
+        __u8    type;
+        __u8    signaled;
+        __u8    priority;
+        __u8    padding[5];
+        __u64   notifier;
+    };
+
+This ioctl command is used to retrieve the exposed events on the source
+VM during migration.
+
+The number of exposed events to be retrieved is specified by ``count``
+of ``struct kvm_sdei_cmd``. On success, the retrieved exposed events are
+returned by ``exposed_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_EXPOSED_EVENT
+------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_exposed_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to expose SDEI events and migrate
+them. The ``notifier`` of ``struct kvm_sdei_exposed_event_state``
+will be modified if the specified exposed event has been existing.
+
+The number of events to be exposed is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the exposed events is
+passed by ``exposed_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
+---------------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_REGISTERED_EVENT,
+to prepare ``registered_event_state`` of ``struct kvm_sdei_cmd`` for
+that command during migration.
+
+On success, the number of registered events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_REGISTERED_EVENT
+---------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+    struct kvm_sdei_registered_event_state {
+        __u64   num;
+
+        __u8    route_mode;
+        __u8    padding[3];
+        __u64   route_affinity;
+        __u64   ep_address[KVM_SDEI_MAX_VCPUS];
+        __u64   ep_arg[KVM_SDEI_MAX_VCPUS];
+        __u64   registered[KVM_SDEI_MAX_VCPUS/64];
+        __u64   enabled[KVM_SDEI_MAX_VCPUS/64];
+        __u64   unregister_pending[KVM_SDEI_MAX_VCPUS/64];
+    };
+
+This ioctl command is used to retrieve the registered events on the
+source VM during migration.
+
+The number of registered events to be retrieved is specified by ``count``
+of ``struct kvm_sdei_cmd``. On success, the retrieved registered events
+are returned by ``registered_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_REGISTERED_EVENT
+---------------------------------
+
+:Type: vm ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_registered_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the registered events
+on the destination VM.
+
+The number of events to be registered is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the registered events
+is passed by ``registered_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
+---------------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used prior to KVM_SDEI_CMD_GET_VCPU_EVENT, to
+prepare ``vcpu_event_state`` of ``struct kvm_sdei_cmd`` for that
+command during migration.
+
+On success, the number of vcpu events is returned by ``count``
+of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_EVENT
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_vcpu_event_state {
+        __u64   num;
+
+        __u32   event_count;
+        __u32   padding;
+    };
+
+This ioctl command is used to retrieve the vcpu events on the source
+VM during migration.
+
+The number of vcpu events to be retrieved is specified by ``count`` of
+``struct kvm_sdei_cmd``. On success, the retrieved exposed events are
+returned by ``vcpu_event_state`` of ``struct kvm_sdei_state``.
+
+KVM_SDEI_CMD_SET_VCPU_EVENT
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_event_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the vcpu events on the
+destination VM.
+
+The number of vcpu events to be added is specified by ``count`` of
+``struct kvm_sdei_cmd``. The information about the vcpu events is
+passed by ``vcpu_event_state`` of ``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_GET_VCPU_STATE
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+::
+
+    struct kvm_sdei_vcpu_regs_state {
+        __u64   regs[18];
+        __u64   pc;
+        __u64   pstate;
+    };
+
+    struct kvm_sdei_vcpu_state {
+        __u8                            masked;
+        __u8                            padding[7];
+        __u64                           critical_num;
+        __u64                           normal_num;
+        struct kvm_sdei_vcpu_regs_state critical_regs;
+        struct kvm_sdei_vcpu_regs_state normal_regs;
+    };
+
+This ioctl command is used to retrieve the vcpu state on the source VM
+during migration.
+
+On success, the current vcpu state is returned by ``vcpu_state`` of
+``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_SET_VCPU_STATE
+---------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to migrate the vcpu state on the
+destination VM.
+
+The vcpu state to be configured is passed by ``vcpu_state`` of
+``struct kvm_sdei_cmd``.
+
+KVM_SDEI_CMD_INJECT_EVENT
+-------------------------
+
+:Type: vcpu ioctl
+:Parameters: struct kvm_sdei_cmd, struct kvm_sdei_vcpu_state
+:Returns: 0 on success, < 0 on error
+
+This ioctl command is used by VMM to inject SDEI event to the specified
+vcpu.
+
+The number of the SDEI event to be injected is passed by ``num`` of
+``struct kvm_sdei_cmd``.
+
+Future Work
+===========
+
+1. Support the routing mode and affinity for the shared events.
+2. Support interrupt binding events.
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 22/22] KVM: selftests: Add SDEI test case
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22  8:07   ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: linux-kernel, eauger, shannon.zhaosl, maz, Jonathan.Cameron,
	will, pbonzini, james.morse, mark.rutland, drjones, vkuznets,
	shan.gavin

This adds SDEI self-test case where the various hypercalls are issued
to default event (0x0). The default event is private, signaled and in
normal priority.

By default, two vCPUs are started and the following ioctl commands
or hypercalls are sent to them in sequence, to simulate how they
are used in VMM and the linux guest:

   kvm_check_cap(KVM_CAP_ARM_SDEI)
   KVM_SDEI_CMD_GET_VERSION
   KVM_SDEI_CMD_SET_EXPOSED_EVENT      (expose event)

   SDEI_1_0_FN_SDEI_VERSION
   SDEI_1_1_FN_SDEI_FEATURES           (SDEI capability probing)
   SDEI_1_0_FN_SDEI_SHARED_RESET       (restart SDEI)
   SDEI_1_0_FN_SDEI_PE_UNMASK          (CPU online)

   SDEI_1_0_FN_SDEI_EVENT_GET_INFO
   SDEI_1_0_FN_SDEI_EVENT_REGISTER     (register event)
   SDEI_1_0_FN_SDEI_EVENT_ENABLE       (enable event)
   SDEI_1_1_FN_SDEI_EVENT_SIGNAL       (event injection)

   SDEI_1_0_FN_SDEI_EVENT_DISABLE      (disable event)
   SDEI_1_0_FN_SDEI_EVENT_UNREGISTER   (unregister event)
   SDEI_1_0_FN_SDEI_PE_MASK            (CPU offline)

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 tools/testing/selftests/kvm/Makefile       |   1 +
 tools/testing/selftests/kvm/aarch64/sdei.c | 525 +++++++++++++++++++++
 2 files changed, 526 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c

diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index 17c3f0749f05..4e4c03cc316b 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -105,6 +105,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list
 TEST_GEN_PROGS_aarch64 += aarch64/psci_cpu_on_test
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq
+TEST_GEN_PROGS_aarch64 += aarch64/sdei
 TEST_GEN_PROGS_aarch64 += demand_paging_test
 TEST_GEN_PROGS_aarch64 += dirty_log_test
 TEST_GEN_PROGS_aarch64 += dirty_log_perf_test
diff --git a/tools/testing/selftests/kvm/aarch64/sdei.c b/tools/testing/selftests/kvm/aarch64/sdei.c
new file mode 100644
index 000000000000..2a7d816ce438
--- /dev/null
+++ b/tools/testing/selftests/kvm/aarch64/sdei.c
@@ -0,0 +1,525 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM SDEI test
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <string.h>
+#include <pthread.h>
+#include <linux/bitmap.h>
+
+#include "kvm_util.h"
+#include "processor.h"
+#include "linux/arm_sdei.h"
+#include "asm/kvm_sdei_state.h"
+
+#define NR_VCPUS	2
+#define SDEI_EVENT_NUM	KVM_SDEI_DEFAULT_EVENT
+
+#define VCPU_COMMAND_IDLE		0
+#define VCPU_COMMAND_EXIT		1
+
+struct vcpu_command {
+	const char	*name;
+	uint64_t	command;
+};
+
+struct sdei_feature {
+	uint16_t	shared_slots;
+	uint16_t	private_slots;
+	uint8_t		relative_mode;
+};
+
+struct sdei_event_info {
+	uint8_t		type;
+	uint8_t		priority;
+	uint8_t		signaled;
+};
+
+struct sdei_event_signal {
+	uint8_t		handled;
+	uint8_t		irq;
+	uint64_t	status;
+	uint64_t	pc;
+	uint64_t	pstate;
+	uint64_t	regs[18];
+};
+
+struct sdei_state {
+	uint64_t	command;
+	uint64_t	num;
+	uint64_t	status;
+	union {
+		uint64_t			version;
+		struct sdei_feature		feature;
+		struct sdei_event_info		info;
+		struct sdei_event_signal	signal;
+	};
+
+	uint8_t		command_completed;
+};
+
+struct vcpu_state {
+	struct kvm_vm		*vm;
+	uint32_t		vcpu_id;
+	pthread_t		thread;
+	struct sdei_state	state;
+};
+
+static struct vcpu_state vcpu_states[NR_VCPUS];
+static struct vcpu_command vcpu_commands[] = {
+	{ "VERSION",          SDEI_1_0_FN_SDEI_VERSION          },
+	{ "FEATURES",         SDEI_1_1_FN_SDEI_FEATURES         },
+	{ "PRIVATE_RESET",    SDEI_1_0_FN_SDEI_PRIVATE_RESET    },
+	{ "SHARED_RESET",     SDEI_1_0_FN_SDEI_SHARED_RESET     },
+	{ "PE_UNMASK",        SDEI_1_0_FN_SDEI_PE_UNMASK        },
+	{ "EVENT_GET_INFO",   SDEI_1_0_FN_SDEI_EVENT_GET_INFO   },
+	{ "EVENT_REGISTER",   SDEI_1_0_FN_SDEI_EVENT_REGISTER   },
+	{ "EVENT_ENABLE",     SDEI_1_0_FN_SDEI_EVENT_ENABLE     },
+	{ "EVENT_SIGNAL",     SDEI_1_1_FN_SDEI_EVENT_SIGNAL     },
+	{ "PE_MASK",          SDEI_1_0_FN_SDEI_PE_MASK          },
+	{ "EVENT_DISABLE",    SDEI_1_0_FN_SDEI_EVENT_DISABLE    },
+	{ "EVENT_UNREGISTER", SDEI_1_0_FN_SDEI_EVENT_UNREGISTER },
+};
+
+static inline int64_t smccc(uint32_t func, uint64_t arg0, uint64_t arg1,
+			    uint64_t arg2, uint64_t arg3, uint64_t arg4)
+{
+	int64_t ret;
+
+	asm volatile (
+		"mov    x0, %1\n"
+		"mov    x1, %2\n"
+		"mov    x2, %3\n"
+		"mov    x3, %4\n"
+		"mov    x4, %5\n"
+		"mov    x5, %6\n"
+		"hvc    #0\n"
+		"mov    %0, x0\n"
+	: "=r" (ret) : "r" (func), "r" (arg0), "r" (arg1),
+	"r" (arg2), "r" (arg3), "r" (arg4) :
+	"x0", "x1", "x2", "x3", "x4", "x5");
+
+	return ret;
+}
+
+static inline bool is_error(int64_t status)
+{
+	if (status == SDEI_NOT_SUPPORTED      ||
+	    status == SDEI_INVALID_PARAMETERS ||
+	    status == SDEI_DENIED             ||
+	    status == SDEI_PENDING            ||
+	    status == SDEI_OUT_OF_RESOURCE)
+		return true;
+
+	return false;
+}
+
+static void guest_irq_handler(struct ex_regs *regs)
+{
+	int vcpu_id = guest_get_vcpuid();
+	struct sdei_state *state = &vcpu_states[vcpu_id].state;
+
+	WRITE_ONCE(state->signal.irq, true);
+}
+
+static void sdei_event_handler(uint64_t num, uint64_t arg,
+			       uint64_t pc, uint64_t pstate)
+{
+	struct sdei_state *state = (struct sdei_state *)arg;
+	uint64_t status;
+
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_STATUS, num, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.status, status);
+
+	WRITE_ONCE(state->signal.pc, pc);
+	WRITE_ONCE(state->signal.pstate, pstate);
+
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 0, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[0], status);
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 1, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[1], status);
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 2, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[2], status);
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 3, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[3], status);
+
+	WRITE_ONCE(state->signal.handled, true);
+	smccc(SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME,
+	      num, 0, 0, 0, 0);
+}
+
+static bool sdei_event_wait(struct sdei_state *state,
+			    uint64_t timeout_in_seconds)
+{
+	uint64_t limit, count = 0;
+
+	limit = (timeout_in_seconds * 1000000) / 10;
+
+	while (1) {
+		if (READ_ONCE(state->signal.handled))
+			return true;
+
+		if (++count >= limit)
+			return false;
+
+		/*
+		 * We issues HVC calls here to ensure the injected
+		 * event can be delivered in time.
+		 */
+		smccc(SDEI_1_0_FN_SDEI_EVENT_GET_INFO,
+		      READ_ONCE(state->num), SDEI_EVENT_INFO_EV_TYPE,
+		      0, 0, 0);
+
+		usleep(10);
+	}
+
+	return false;
+}
+
+static void guest_code(int vcpu_id)
+{
+	struct sdei_state *state;
+	uint64_t command, last_command = -1UL, num, status;
+
+	state = &vcpu_states[vcpu_id].state;
+
+	while (1) {
+		command = READ_ONCE(state->command);
+		if (command == last_command)
+			continue;
+
+		num = READ_ONCE(state->num);
+		switch (command) {
+		case VCPU_COMMAND_IDLE:
+			WRITE_ONCE(state->status, SDEI_SUCCESS);
+			break;
+		case SDEI_1_0_FN_SDEI_VERSION:
+			status = smccc(command, 0, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->version, status);
+			break;
+		case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+		case SDEI_1_0_FN_SDEI_SHARED_RESET:
+		case SDEI_1_0_FN_SDEI_PE_UNMASK:
+		case SDEI_1_0_FN_SDEI_PE_MASK:
+			status = smccc(command, 0, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			break;
+		case SDEI_1_1_FN_SDEI_FEATURES:
+			status = smccc(command, 0, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->feature.shared_slots,
+				   (status & 0xffff0000) >> 16);
+			WRITE_ONCE(state->feature.private_slots,
+				   (status & 0x0000ffff));
+			status = smccc(command, 1, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->feature.relative_mode, status);
+			break;
+		case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+			status = smccc(command, num,
+				       SDEI_EVENT_INFO_EV_TYPE, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->info.type, status);
+			status = smccc(command, num,
+				       SDEI_EVENT_INFO_EV_PRIORITY, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->info.priority, status);
+			status = smccc(command, num,
+				       SDEI_EVENT_INFO_EV_SIGNALED, 0, 0, 0);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->info.signaled, status);
+			break;
+		case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+			status = smccc(command, num,
+				       (uint64_t)sdei_event_handler,
+				       (uint64_t)state,
+				       SDEI_EVENT_REGISTER_RM_ANY, 0);
+			WRITE_ONCE(state->status, status);
+			break;
+		case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+		case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+		case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+			status = smccc(command, num, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			break;
+		case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+			status = smccc(command, num, (uint64_t)state, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			if (!sdei_event_wait(state, 5))
+				WRITE_ONCE(state->status, SDEI_DENIED);
+
+			break;
+		case VCPU_COMMAND_EXIT:
+			WRITE_ONCE(state->status, SDEI_SUCCESS);
+			GUEST_DONE();
+			break;
+		default:
+			WRITE_ONCE(state->status, SDEI_INVALID_PARAMETERS);
+		}
+
+		last_command = command;
+		WRITE_ONCE(state->command_completed, true);
+	}
+}
+
+static void *vcpu_thread(void *arg)
+{
+	struct vcpu_state *state = arg;
+
+	vcpu_run(state->vm, state->vcpu_id);
+
+	return NULL;
+}
+
+static bool vcpu_wait(struct kvm_vm *vm, int timeout_in_seconds)
+{
+	unsigned long count, limit;
+	int i;
+
+	count = 0;
+	limit = (timeout_in_seconds * 1000000) / 50;
+	while (1) {
+		for (i = 0; i < NR_VCPUS; i++) {
+			sync_global_from_guest(vm, vcpu_states[i].state);
+			if (!vcpu_states[i].state.command_completed)
+				break;
+		}
+
+		if (i >= NR_VCPUS)
+			return true;
+
+		if (++count > limit)
+			return false;
+
+		usleep(50);
+	}
+
+	return false;
+}
+
+static void vcpu_send_command(struct kvm_vm *vm, uint64_t command)
+{
+	int i;
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		memset(&vcpu_states[i].state, 0,
+		       sizeof(vcpu_states[0].state));
+		vcpu_states[i].state.num = SDEI_EVENT_NUM;
+		vcpu_states[i].state.status = SDEI_SUCCESS;
+		vcpu_states[i].state.command = command;
+		vcpu_states[i].state.command_completed = false;
+
+		sync_global_to_guest(vm, vcpu_states[i].state);
+	}
+}
+
+static bool vcpu_check_state(struct kvm_vm *vm)
+{
+	int i, j, ret;
+
+	for (i = 0; i < NR_VCPUS; i++)
+		sync_global_from_guest(vm, vcpu_states[i].state);
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		if (is_error(vcpu_states[i].state.status))
+			return false;
+
+		for (j = 0; j < NR_VCPUS; j++) {
+			ret = memcmp(&vcpu_states[i].state,
+				     &vcpu_states[j].state,
+				     sizeof(vcpu_states[0].state));
+			if (ret)
+				return false;
+		}
+	}
+
+	return true;
+}
+
+static void vcpu_dump_state(int index)
+{
+	struct sdei_state *state = &vcpu_states[0].state;
+
+	pr_info("--- %s\n", vcpu_commands[index].name);
+	switch (state->command) {
+	case SDEI_1_0_FN_SDEI_VERSION:
+		pr_info("    Version:              %ld.%ld (vendor: 0x%lx)\n",
+			SDEI_VERSION_MAJOR(state->version),
+			SDEI_VERSION_MINOR(state->version),
+			SDEI_VERSION_VENDOR(state->version));
+		break;
+	case SDEI_1_1_FN_SDEI_FEATURES:
+		pr_info("    Shared event slots:   %d\n",
+			state->feature.shared_slots);
+		pr_info("    Private event slots:  %d\n",
+			state->feature.private_slots);
+		pr_info("    Relative mode:        %s\n",
+			state->feature.relative_mode ? "Yes" : "No");
+			break;
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+		pr_info("    Type:                 %s\n",
+			state->info.type == SDEI_EVENT_TYPE_SHARED ?
+			"Shared" : "Private");
+		pr_info("    Priority:             %s\n",
+			state->info.priority == SDEI_EVENT_PRIORITY_NORMAL ?
+			"Normal" : "Critical");
+		pr_info("    Signaled:             %s\n",
+			state->info.signaled ? "Yes" : "No");
+		break;
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		pr_info("    Handled:              %s\n",
+			state->signal.handled ? "Yes" : "No");
+		pr_info("    IRQ:                  %s\n",
+			state->signal.irq ? "Yes" : "No");
+		pr_info("    Status:               %s-%s-%s\n",
+			state->signal.status & (1 << SDEI_EVENT_STATUS_REGISTERED) ?
+			"Registered" : "x",
+			state->signal.status & (1 << SDEI_EVENT_STATUS_ENABLED) ?
+			"Enabled" : "x",
+			state->signal.status & (1 << SDEI_EVENT_STATUS_RUNNING) ?
+			"Running" : "x");
+		pr_info("    PC/PSTATE:            %016lx %016lx\n",
+			state->signal.pc, state->signal.pstate);
+		pr_info("    Regs:                 %016lx %016lx %016lx %016lx\n",
+			state->signal.regs[0], state->signal.regs[1],
+			state->signal.regs[2], state->signal.regs[3]);
+		break;
+	}
+
+	if (index == ARRAY_SIZE(vcpu_commands))
+		pr_info("\n");
+}
+
+int main(int argc, char **argv)
+{
+	struct kvm_vm *vm;
+	struct kvm_sdei_cmd cmd;
+	struct kvm_sdei_exposed_event_state exposed_event;
+	uint32_t vcpu_ids[NR_VCPUS];
+	int i, ret;
+
+	if (!kvm_check_cap(KVM_CAP_ARM_SDEI)) {
+		pr_info("SDEI not supported\n");
+		return 0;
+	}
+
+	/* Create VM */
+	for (i = 0; i < NR_VCPUS; i++) {
+		vcpu_states[i].vcpu_id = i;
+		vcpu_ids[i] = i;
+	}
+
+	vm = vm_create_default_with_vcpus(NR_VCPUS, 0, 0,
+					  guest_code, vcpu_ids);
+	vm_init_descriptor_tables(vm);
+	vm_install_exception_handler(vm, VECTOR_IRQ_CURRENT,
+				     guest_irq_handler);
+
+	ucall_init(vm, NULL);
+
+	/* Ensure the version is v1.0.0 */
+	cmd.cmd = KVM_SDEI_CMD_GET_VERSION;
+	cmd.version = 0;
+	vm_ioctl(vm, KVM_ARM_SDEI_COMMAND, &cmd);
+	if (cmd.version != 0x10000) {
+		pr_info("v%d.%d.%d doesn't match with v1.0.0\n",
+			(cmd.version & 0xFF0000) >> 16,
+			(cmd.version & 0xFF00) >> 8,
+			(cmd.version & 0xFF));
+		return 0;
+	}
+
+	/* Expose the default SDEI event */
+	exposed_event.num = SDEI_EVENT_NUM;
+	exposed_event.type = SDEI_EVENT_TYPE_PRIVATE;
+	exposed_event.priority = SDEI_EVENT_PRIORITY_NORMAL;
+	exposed_event.signaled = 1;
+	exposed_event.notifier = 0;
+	cmd.cmd = KVM_SDEI_CMD_SET_EXPOSED_EVENT;
+	cmd.count = 1;
+	cmd.exposed_event_state = &exposed_event;
+	vm_ioctl(vm, KVM_ARM_SDEI_COMMAND, &cmd);
+
+	/* Start the vCPUs */
+	vcpu_send_command(vm, VCPU_COMMAND_IDLE);
+	for (i = 0; i < NR_VCPUS; i++) {
+		vcpu_states[i].vm = vm;
+		vcpu_args_set(vm, i, 1, i);
+		vcpu_init_descriptor_tables(vm, i);
+
+		ret = pthread_create(&vcpu_states[i].thread, NULL,
+				     vcpu_thread, &vcpu_states[i]);
+		TEST_ASSERT(!ret, "Failed to create vCPU-%d pthread\n", i);
+	}
+
+	/* Wait the idle command to complete */
+	ret = vcpu_wait(vm, 5);
+	TEST_ASSERT(ret, "Timeout to execute IDLE command\n");
+
+	/* Start the tests */
+	pr_info("\n");
+	pr_info("    NR_VCPUS: %d    SDEI Event: 0x%08x\n\n",
+		NR_VCPUS, SDEI_EVENT_NUM);
+	for (i = 0; i < ARRAY_SIZE(vcpu_commands); i++) {
+		/*
+		 * We depends on SDEI_1_1_FN_SDEI_EVENT_SIGNAL hypercall
+		 * to inject SDEI event. The number of the injected event
+		 * must be zero. So we have to skip the corresponding test
+		 * if the SDEI event number isn't zero.
+		 */
+		if (SDEI_EVENT_NUM != 0x0 &&
+		    vcpu_commands[i].command == SDEI_1_1_FN_SDEI_EVENT_SIGNAL)
+			continue;
+
+		vcpu_send_command(vm, vcpu_commands[i].command);
+		ret = vcpu_wait(vm, 5);
+		if (!ret) {
+			pr_info("%s: Timeout\n", vcpu_commands[i].name);
+			return -1;
+		}
+
+		ret = vcpu_check_state(vm);
+		if (!ret) {
+			pr_info("%s: Fail\n", vcpu_commands[i].name);
+			return -1;
+		}
+
+		vcpu_dump_state(i);
+	}
+
+	/* Terminate the guests */
+	pr_info("\n    Result: OK\n\n");
+	vcpu_send_command(vm, VCPU_COMMAND_EXIT);
+	sleep(1);
+
+	return 0;
+}
-- 
2.23.0


^ permalink raw reply related	[flat|nested] 98+ messages in thread

* [PATCH v5 22/22] KVM: selftests: Add SDEI test case
@ 2022-03-22  8:07   ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-22  8:07 UTC (permalink / raw)
  To: kvmarm
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

This adds SDEI self-test case where the various hypercalls are issued
to default event (0x0). The default event is private, signaled and in
normal priority.

By default, two vCPUs are started and the following ioctl commands
or hypercalls are sent to them in sequence, to simulate how they
are used in VMM and the linux guest:

   kvm_check_cap(KVM_CAP_ARM_SDEI)
   KVM_SDEI_CMD_GET_VERSION
   KVM_SDEI_CMD_SET_EXPOSED_EVENT      (expose event)

   SDEI_1_0_FN_SDEI_VERSION
   SDEI_1_1_FN_SDEI_FEATURES           (SDEI capability probing)
   SDEI_1_0_FN_SDEI_SHARED_RESET       (restart SDEI)
   SDEI_1_0_FN_SDEI_PE_UNMASK          (CPU online)

   SDEI_1_0_FN_SDEI_EVENT_GET_INFO
   SDEI_1_0_FN_SDEI_EVENT_REGISTER     (register event)
   SDEI_1_0_FN_SDEI_EVENT_ENABLE       (enable event)
   SDEI_1_1_FN_SDEI_EVENT_SIGNAL       (event injection)

   SDEI_1_0_FN_SDEI_EVENT_DISABLE      (disable event)
   SDEI_1_0_FN_SDEI_EVENT_UNREGISTER   (unregister event)
   SDEI_1_0_FN_SDEI_PE_MASK            (CPU offline)

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 tools/testing/selftests/kvm/Makefile       |   1 +
 tools/testing/selftests/kvm/aarch64/sdei.c | 525 +++++++++++++++++++++
 2 files changed, 526 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/sdei.c

diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index 17c3f0749f05..4e4c03cc316b 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -105,6 +105,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list
 TEST_GEN_PROGS_aarch64 += aarch64/psci_cpu_on_test
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_irq
+TEST_GEN_PROGS_aarch64 += aarch64/sdei
 TEST_GEN_PROGS_aarch64 += demand_paging_test
 TEST_GEN_PROGS_aarch64 += dirty_log_test
 TEST_GEN_PROGS_aarch64 += dirty_log_perf_test
diff --git a/tools/testing/selftests/kvm/aarch64/sdei.c b/tools/testing/selftests/kvm/aarch64/sdei.c
new file mode 100644
index 000000000000..2a7d816ce438
--- /dev/null
+++ b/tools/testing/selftests/kvm/aarch64/sdei.c
@@ -0,0 +1,525 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * ARM SDEI test
+ *
+ * Copyright (C) 2022 Red Hat, Inc.
+ *
+ * Author(s): Gavin Shan <gshan@redhat.com>
+ */
+
+#define _GNU_SOURCE
+#include <stdio.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <string.h>
+#include <pthread.h>
+#include <linux/bitmap.h>
+
+#include "kvm_util.h"
+#include "processor.h"
+#include "linux/arm_sdei.h"
+#include "asm/kvm_sdei_state.h"
+
+#define NR_VCPUS	2
+#define SDEI_EVENT_NUM	KVM_SDEI_DEFAULT_EVENT
+
+#define VCPU_COMMAND_IDLE		0
+#define VCPU_COMMAND_EXIT		1
+
+struct vcpu_command {
+	const char	*name;
+	uint64_t	command;
+};
+
+struct sdei_feature {
+	uint16_t	shared_slots;
+	uint16_t	private_slots;
+	uint8_t		relative_mode;
+};
+
+struct sdei_event_info {
+	uint8_t		type;
+	uint8_t		priority;
+	uint8_t		signaled;
+};
+
+struct sdei_event_signal {
+	uint8_t		handled;
+	uint8_t		irq;
+	uint64_t	status;
+	uint64_t	pc;
+	uint64_t	pstate;
+	uint64_t	regs[18];
+};
+
+struct sdei_state {
+	uint64_t	command;
+	uint64_t	num;
+	uint64_t	status;
+	union {
+		uint64_t			version;
+		struct sdei_feature		feature;
+		struct sdei_event_info		info;
+		struct sdei_event_signal	signal;
+	};
+
+	uint8_t		command_completed;
+};
+
+struct vcpu_state {
+	struct kvm_vm		*vm;
+	uint32_t		vcpu_id;
+	pthread_t		thread;
+	struct sdei_state	state;
+};
+
+static struct vcpu_state vcpu_states[NR_VCPUS];
+static struct vcpu_command vcpu_commands[] = {
+	{ "VERSION",          SDEI_1_0_FN_SDEI_VERSION          },
+	{ "FEATURES",         SDEI_1_1_FN_SDEI_FEATURES         },
+	{ "PRIVATE_RESET",    SDEI_1_0_FN_SDEI_PRIVATE_RESET    },
+	{ "SHARED_RESET",     SDEI_1_0_FN_SDEI_SHARED_RESET     },
+	{ "PE_UNMASK",        SDEI_1_0_FN_SDEI_PE_UNMASK        },
+	{ "EVENT_GET_INFO",   SDEI_1_0_FN_SDEI_EVENT_GET_INFO   },
+	{ "EVENT_REGISTER",   SDEI_1_0_FN_SDEI_EVENT_REGISTER   },
+	{ "EVENT_ENABLE",     SDEI_1_0_FN_SDEI_EVENT_ENABLE     },
+	{ "EVENT_SIGNAL",     SDEI_1_1_FN_SDEI_EVENT_SIGNAL     },
+	{ "PE_MASK",          SDEI_1_0_FN_SDEI_PE_MASK          },
+	{ "EVENT_DISABLE",    SDEI_1_0_FN_SDEI_EVENT_DISABLE    },
+	{ "EVENT_UNREGISTER", SDEI_1_0_FN_SDEI_EVENT_UNREGISTER },
+};
+
+static inline int64_t smccc(uint32_t func, uint64_t arg0, uint64_t arg1,
+			    uint64_t arg2, uint64_t arg3, uint64_t arg4)
+{
+	int64_t ret;
+
+	asm volatile (
+		"mov    x0, %1\n"
+		"mov    x1, %2\n"
+		"mov    x2, %3\n"
+		"mov    x3, %4\n"
+		"mov    x4, %5\n"
+		"mov    x5, %6\n"
+		"hvc    #0\n"
+		"mov    %0, x0\n"
+	: "=r" (ret) : "r" (func), "r" (arg0), "r" (arg1),
+	"r" (arg2), "r" (arg3), "r" (arg4) :
+	"x0", "x1", "x2", "x3", "x4", "x5");
+
+	return ret;
+}
+
+static inline bool is_error(int64_t status)
+{
+	if (status == SDEI_NOT_SUPPORTED      ||
+	    status == SDEI_INVALID_PARAMETERS ||
+	    status == SDEI_DENIED             ||
+	    status == SDEI_PENDING            ||
+	    status == SDEI_OUT_OF_RESOURCE)
+		return true;
+
+	return false;
+}
+
+static void guest_irq_handler(struct ex_regs *regs)
+{
+	int vcpu_id = guest_get_vcpuid();
+	struct sdei_state *state = &vcpu_states[vcpu_id].state;
+
+	WRITE_ONCE(state->signal.irq, true);
+}
+
+static void sdei_event_handler(uint64_t num, uint64_t arg,
+			       uint64_t pc, uint64_t pstate)
+{
+	struct sdei_state *state = (struct sdei_state *)arg;
+	uint64_t status;
+
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_STATUS, num, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.status, status);
+
+	WRITE_ONCE(state->signal.pc, pc);
+	WRITE_ONCE(state->signal.pstate, pstate);
+
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 0, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[0], status);
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 1, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[1], status);
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 2, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[2], status);
+	status = smccc(SDEI_1_0_FN_SDEI_EVENT_CONTEXT, 3, 0, 0, 0, 0);
+	WRITE_ONCE(state->signal.regs[3], status);
+
+	WRITE_ONCE(state->signal.handled, true);
+	smccc(SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME,
+	      num, 0, 0, 0, 0);
+}
+
+static bool sdei_event_wait(struct sdei_state *state,
+			    uint64_t timeout_in_seconds)
+{
+	uint64_t limit, count = 0;
+
+	limit = (timeout_in_seconds * 1000000) / 10;
+
+	while (1) {
+		if (READ_ONCE(state->signal.handled))
+			return true;
+
+		if (++count >= limit)
+			return false;
+
+		/*
+		 * We issues HVC calls here to ensure the injected
+		 * event can be delivered in time.
+		 */
+		smccc(SDEI_1_0_FN_SDEI_EVENT_GET_INFO,
+		      READ_ONCE(state->num), SDEI_EVENT_INFO_EV_TYPE,
+		      0, 0, 0);
+
+		usleep(10);
+	}
+
+	return false;
+}
+
+static void guest_code(int vcpu_id)
+{
+	struct sdei_state *state;
+	uint64_t command, last_command = -1UL, num, status;
+
+	state = &vcpu_states[vcpu_id].state;
+
+	while (1) {
+		command = READ_ONCE(state->command);
+		if (command == last_command)
+			continue;
+
+		num = READ_ONCE(state->num);
+		switch (command) {
+		case VCPU_COMMAND_IDLE:
+			WRITE_ONCE(state->status, SDEI_SUCCESS);
+			break;
+		case SDEI_1_0_FN_SDEI_VERSION:
+			status = smccc(command, 0, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->version, status);
+			break;
+		case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
+		case SDEI_1_0_FN_SDEI_SHARED_RESET:
+		case SDEI_1_0_FN_SDEI_PE_UNMASK:
+		case SDEI_1_0_FN_SDEI_PE_MASK:
+			status = smccc(command, 0, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			break;
+		case SDEI_1_1_FN_SDEI_FEATURES:
+			status = smccc(command, 0, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->feature.shared_slots,
+				   (status & 0xffff0000) >> 16);
+			WRITE_ONCE(state->feature.private_slots,
+				   (status & 0x0000ffff));
+			status = smccc(command, 1, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->feature.relative_mode, status);
+			break;
+		case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+			status = smccc(command, num,
+				       SDEI_EVENT_INFO_EV_TYPE, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->info.type, status);
+			status = smccc(command, num,
+				       SDEI_EVENT_INFO_EV_PRIORITY, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->info.priority, status);
+			status = smccc(command, num,
+				       SDEI_EVENT_INFO_EV_SIGNALED, 0, 0, 0);
+			if (is_error(status))
+				break;
+
+			WRITE_ONCE(state->info.signaled, status);
+			break;
+		case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
+			status = smccc(command, num,
+				       (uint64_t)sdei_event_handler,
+				       (uint64_t)state,
+				       SDEI_EVENT_REGISTER_RM_ANY, 0);
+			WRITE_ONCE(state->status, status);
+			break;
+		case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
+		case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
+		case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
+			status = smccc(command, num, 0, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			break;
+		case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+			status = smccc(command, num, (uint64_t)state, 0, 0, 0);
+			WRITE_ONCE(state->status, status);
+			if (is_error(status))
+				break;
+
+			if (!sdei_event_wait(state, 5))
+				WRITE_ONCE(state->status, SDEI_DENIED);
+
+			break;
+		case VCPU_COMMAND_EXIT:
+			WRITE_ONCE(state->status, SDEI_SUCCESS);
+			GUEST_DONE();
+			break;
+		default:
+			WRITE_ONCE(state->status, SDEI_INVALID_PARAMETERS);
+		}
+
+		last_command = command;
+		WRITE_ONCE(state->command_completed, true);
+	}
+}
+
+static void *vcpu_thread(void *arg)
+{
+	struct vcpu_state *state = arg;
+
+	vcpu_run(state->vm, state->vcpu_id);
+
+	return NULL;
+}
+
+static bool vcpu_wait(struct kvm_vm *vm, int timeout_in_seconds)
+{
+	unsigned long count, limit;
+	int i;
+
+	count = 0;
+	limit = (timeout_in_seconds * 1000000) / 50;
+	while (1) {
+		for (i = 0; i < NR_VCPUS; i++) {
+			sync_global_from_guest(vm, vcpu_states[i].state);
+			if (!vcpu_states[i].state.command_completed)
+				break;
+		}
+
+		if (i >= NR_VCPUS)
+			return true;
+
+		if (++count > limit)
+			return false;
+
+		usleep(50);
+	}
+
+	return false;
+}
+
+static void vcpu_send_command(struct kvm_vm *vm, uint64_t command)
+{
+	int i;
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		memset(&vcpu_states[i].state, 0,
+		       sizeof(vcpu_states[0].state));
+		vcpu_states[i].state.num = SDEI_EVENT_NUM;
+		vcpu_states[i].state.status = SDEI_SUCCESS;
+		vcpu_states[i].state.command = command;
+		vcpu_states[i].state.command_completed = false;
+
+		sync_global_to_guest(vm, vcpu_states[i].state);
+	}
+}
+
+static bool vcpu_check_state(struct kvm_vm *vm)
+{
+	int i, j, ret;
+
+	for (i = 0; i < NR_VCPUS; i++)
+		sync_global_from_guest(vm, vcpu_states[i].state);
+
+	for (i = 0; i < NR_VCPUS; i++) {
+		if (is_error(vcpu_states[i].state.status))
+			return false;
+
+		for (j = 0; j < NR_VCPUS; j++) {
+			ret = memcmp(&vcpu_states[i].state,
+				     &vcpu_states[j].state,
+				     sizeof(vcpu_states[0].state));
+			if (ret)
+				return false;
+		}
+	}
+
+	return true;
+}
+
+static void vcpu_dump_state(int index)
+{
+	struct sdei_state *state = &vcpu_states[0].state;
+
+	pr_info("--- %s\n", vcpu_commands[index].name);
+	switch (state->command) {
+	case SDEI_1_0_FN_SDEI_VERSION:
+		pr_info("    Version:              %ld.%ld (vendor: 0x%lx)\n",
+			SDEI_VERSION_MAJOR(state->version),
+			SDEI_VERSION_MINOR(state->version),
+			SDEI_VERSION_VENDOR(state->version));
+		break;
+	case SDEI_1_1_FN_SDEI_FEATURES:
+		pr_info("    Shared event slots:   %d\n",
+			state->feature.shared_slots);
+		pr_info("    Private event slots:  %d\n",
+			state->feature.private_slots);
+		pr_info("    Relative mode:        %s\n",
+			state->feature.relative_mode ? "Yes" : "No");
+			break;
+	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
+		pr_info("    Type:                 %s\n",
+			state->info.type == SDEI_EVENT_TYPE_SHARED ?
+			"Shared" : "Private");
+		pr_info("    Priority:             %s\n",
+			state->info.priority == SDEI_EVENT_PRIORITY_NORMAL ?
+			"Normal" : "Critical");
+		pr_info("    Signaled:             %s\n",
+			state->info.signaled ? "Yes" : "No");
+		break;
+	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
+		pr_info("    Handled:              %s\n",
+			state->signal.handled ? "Yes" : "No");
+		pr_info("    IRQ:                  %s\n",
+			state->signal.irq ? "Yes" : "No");
+		pr_info("    Status:               %s-%s-%s\n",
+			state->signal.status & (1 << SDEI_EVENT_STATUS_REGISTERED) ?
+			"Registered" : "x",
+			state->signal.status & (1 << SDEI_EVENT_STATUS_ENABLED) ?
+			"Enabled" : "x",
+			state->signal.status & (1 << SDEI_EVENT_STATUS_RUNNING) ?
+			"Running" : "x");
+		pr_info("    PC/PSTATE:            %016lx %016lx\n",
+			state->signal.pc, state->signal.pstate);
+		pr_info("    Regs:                 %016lx %016lx %016lx %016lx\n",
+			state->signal.regs[0], state->signal.regs[1],
+			state->signal.regs[2], state->signal.regs[3]);
+		break;
+	}
+
+	if (index == ARRAY_SIZE(vcpu_commands))
+		pr_info("\n");
+}
+
+int main(int argc, char **argv)
+{
+	struct kvm_vm *vm;
+	struct kvm_sdei_cmd cmd;
+	struct kvm_sdei_exposed_event_state exposed_event;
+	uint32_t vcpu_ids[NR_VCPUS];
+	int i, ret;
+
+	if (!kvm_check_cap(KVM_CAP_ARM_SDEI)) {
+		pr_info("SDEI not supported\n");
+		return 0;
+	}
+
+	/* Create VM */
+	for (i = 0; i < NR_VCPUS; i++) {
+		vcpu_states[i].vcpu_id = i;
+		vcpu_ids[i] = i;
+	}
+
+	vm = vm_create_default_with_vcpus(NR_VCPUS, 0, 0,
+					  guest_code, vcpu_ids);
+	vm_init_descriptor_tables(vm);
+	vm_install_exception_handler(vm, VECTOR_IRQ_CURRENT,
+				     guest_irq_handler);
+
+	ucall_init(vm, NULL);
+
+	/* Ensure the version is v1.0.0 */
+	cmd.cmd = KVM_SDEI_CMD_GET_VERSION;
+	cmd.version = 0;
+	vm_ioctl(vm, KVM_ARM_SDEI_COMMAND, &cmd);
+	if (cmd.version != 0x10000) {
+		pr_info("v%d.%d.%d doesn't match with v1.0.0\n",
+			(cmd.version & 0xFF0000) >> 16,
+			(cmd.version & 0xFF00) >> 8,
+			(cmd.version & 0xFF));
+		return 0;
+	}
+
+	/* Expose the default SDEI event */
+	exposed_event.num = SDEI_EVENT_NUM;
+	exposed_event.type = SDEI_EVENT_TYPE_PRIVATE;
+	exposed_event.priority = SDEI_EVENT_PRIORITY_NORMAL;
+	exposed_event.signaled = 1;
+	exposed_event.notifier = 0;
+	cmd.cmd = KVM_SDEI_CMD_SET_EXPOSED_EVENT;
+	cmd.count = 1;
+	cmd.exposed_event_state = &exposed_event;
+	vm_ioctl(vm, KVM_ARM_SDEI_COMMAND, &cmd);
+
+	/* Start the vCPUs */
+	vcpu_send_command(vm, VCPU_COMMAND_IDLE);
+	for (i = 0; i < NR_VCPUS; i++) {
+		vcpu_states[i].vm = vm;
+		vcpu_args_set(vm, i, 1, i);
+		vcpu_init_descriptor_tables(vm, i);
+
+		ret = pthread_create(&vcpu_states[i].thread, NULL,
+				     vcpu_thread, &vcpu_states[i]);
+		TEST_ASSERT(!ret, "Failed to create vCPU-%d pthread\n", i);
+	}
+
+	/* Wait the idle command to complete */
+	ret = vcpu_wait(vm, 5);
+	TEST_ASSERT(ret, "Timeout to execute IDLE command\n");
+
+	/* Start the tests */
+	pr_info("\n");
+	pr_info("    NR_VCPUS: %d    SDEI Event: 0x%08x\n\n",
+		NR_VCPUS, SDEI_EVENT_NUM);
+	for (i = 0; i < ARRAY_SIZE(vcpu_commands); i++) {
+		/*
+		 * We depends on SDEI_1_1_FN_SDEI_EVENT_SIGNAL hypercall
+		 * to inject SDEI event. The number of the injected event
+		 * must be zero. So we have to skip the corresponding test
+		 * if the SDEI event number isn't zero.
+		 */
+		if (SDEI_EVENT_NUM != 0x0 &&
+		    vcpu_commands[i].command == SDEI_1_1_FN_SDEI_EVENT_SIGNAL)
+			continue;
+
+		vcpu_send_command(vm, vcpu_commands[i].command);
+		ret = vcpu_wait(vm, 5);
+		if (!ret) {
+			pr_info("%s: Timeout\n", vcpu_commands[i].name);
+			return -1;
+		}
+
+		ret = vcpu_check_state(vm);
+		if (!ret) {
+			pr_info("%s: Fail\n", vcpu_commands[i].name);
+			return -1;
+		}
+
+		vcpu_dump_state(i);
+	}
+
+	/* Terminate the guests */
+	pr_info("\n    Result: OK\n\n");
+	vcpu_send_command(vm, VCPU_COMMAND_EXIT);
+	sleep(1);
+
+	return 0;
+}
-- 
2.23.0

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply related	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-22  8:06   ` Gavin Shan
@ 2022-03-22 18:04     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 18:04 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
> This supports SDEI_VERSION hypercall by returning v1.1, which is
> the specification version we're following. The vendor is set to
> 'KVM'.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  arch/arm64/kvm/sdei.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
> index 8a9b477b8977..5a3a64cd6e84 100644
> --- a/arch/arm64/kvm/sdei.c
> +++ b/arch/arm64/kvm/sdei.c
> @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
>  	return pending;
>  }
>  
> +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
> +{
> +	/* v1.1 and the vendor is KVM */
> +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
> +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
> +	       0x4b564d;

It looks like the SDEI specification states that the vendor-defined
version number is 32 bits. Could we just use one of the
ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?

ASCII 'KVM' is neat, but in reality guest software will just throw it in
a macro regardless. Might as well use one of the values we've already
trained it to use :-)

Also, it would appear that guest discovery of SDEI relies upon KVM
reporting a valid SDEI version. IMO, this patch should come at the very
end when KVM actually implements SDEI.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-22 18:04     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 18:04 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
> This supports SDEI_VERSION hypercall by returning v1.1, which is
> the specification version we're following. The vendor is set to
> 'KVM'.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  arch/arm64/kvm/sdei.c | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
> index 8a9b477b8977..5a3a64cd6e84 100644
> --- a/arch/arm64/kvm/sdei.c
> +++ b/arch/arm64/kvm/sdei.c
> @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
>  	return pending;
>  }
>  
> +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
> +{
> +	/* v1.1 and the vendor is KVM */
> +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
> +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
> +	       0x4b564d;

It looks like the SDEI specification states that the vendor-defined
version number is 32 bits. Could we just use one of the
ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?

ASCII 'KVM' is neat, but in reality guest software will just throw it in
a macro regardless. Might as well use one of the values we've already
trained it to use :-)

Also, it would appear that guest discovery of SDEI relies upon KVM
reporting a valid SDEI version. IMO, this patch should come at the very
end when KVM actually implements SDEI.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 00/22] Support SDEI Virtualization
  2022-03-22  8:06 ` Gavin Shan
@ 2022-03-22 18:13   ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 18:13 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Tue, Mar 22, 2022 at 04:06:48PM +0800, Gavin Shan wrote:
> This series intends to virtualize Software Delegated Exception Interface
> (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to
> deliver NMI-alike SDEI event to guest and it's needed by Async PF to
> deliver page-not-present notification from hypervisor to guest. The code
> and the required qemu changes can be found from:
> 
>    https://developer.arm.com/documentation/den0054/c
>    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
> 
> For the design and migration needs, please refer to the document in
> PATCH[21/22] in this series. The series is organized as below:
> 
>   PATCH[01]    Introduces template for smccc_get_argx()
>   PATCH[02]    Adds SDEI virtualization infrastructure
>   PATCH[03-17] Supports various SDEI hypercalls and event handling
>   PATCH[18-20] Adds ioctl commands to support migration and configuration
>                and exports SDEI capability
>   PATCH[21]    Adds SDEI document 
>   PATCH[22]    Adds SDEI selftest case
> 
> Testing
> =======
> 
> [1] The selftest case included in this series works fine. The default SDEI
>     event, whose number is zero, can be registered, enabled, raised. The
>     SDEI event handler can be invoked.
> 
>     [host]# pwd
>     /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
>     [root@virtlab-arm01 kvm]# ./aarch64/sdei 
> 
>         NR_VCPUS: 2    SDEI Event: 0x00000000
> 
>     --- VERSION
>         Version:              1.1 (vendor: 0x4b564d)
>     --- FEATURES
>         Shared event slots:   0
>         Private event slots:  0
>         Relative mode:        No
>     --- PRIVATE_RESET
>     --- SHARED_RESET
>     --- PE_UNMASK
>     --- EVENT_GET_INFO
>         Type:                 Private
>         Priority:             Normal
>         Signaled:             Yes
>     --- EVENT_REGISTER
>     --- EVENT_ENABLE
>     --- EVENT_SIGNAL
>         Handled:              Yes
>         IRQ:                  No
>         Status:               Registered-Enabled-Running
>         PC/PSTATE:            000000000040232c 00000000600003c5
>         Regs:                 0000000000000000 0000000000000000
>                               0000000000000000 0000000000000000
>     --- PE_MASK
>     --- EVENT_DISABLE
>     --- EVENT_UNREGISTER
> 
>         Result: OK
> 
> [2] There are additional patches in the following repositories to create
>     procfs entries, allowing to inject SDEI event from host side. The
>     SDEI client in the guest side registers the SDEI default event, whose
>     number is zero. Also, the QEMU exports SDEI ACPI table and supports
>     migration for SDEI.
> 
>     https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>     https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
> 
>     [2.1] Start the guests and migrate the source VM to the destination
>           VM.
> 
>     [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>             -accel kvm -machine virt,gic-version=host                     \
>             -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>             -m 1024M,slots=16,maxmem=64G                                  \
>                :                                                          \
>             -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>             -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>             -append earlycon=pl011,mmio,0x9000000                         \
>                :
> 
>     [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>             -accel kvm -machine virt,gic-version=host                     \
>             -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>             -m 1024M,slots=16,maxmem=64G                                  \
>                :                                                          \
>             -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>             -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>             -append earlycon=pl011,mmio,0x9000000                         \
>             -incoming tcp:0:4444                                          \
>                :
> 
>     [2.2] Check kernel log on the source VM. The SDEI service is enabled
>           and the default SDEI event (0x0) is enabled.
> 
>      [guest-src]# dmesg | grep -i sdei
>      ACPI: SDEI 0x000000005BC80000 000024 \
>                 (v00 BOCHS  BXPC     00000001 BXPC 00000001)
>      sdei: SDEIv1.1 (0x4b564d) detected in firmware.
>      SDEI TEST: Version 1.1, Vendor 0x4b564d
>      sdei_init: SDEI event (0x0) registered
>      sdei_init: SDEI event (0x0) enabled
> 
>  
>      (qemu) migrate -d tcp:localhost:4444
> 
>     [2.3] Migrate the source VM to the destination VM. Inject SDEI event
>           to the destination VM. The event is raised and handled.
> 
>     (qemu) migrate -d tcp:localhost:4444
> 
>     [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1
> 
>     [guest-dst]#
>     =========== SDEI Event (CPU#1) ===========
>     Event: 0000000000000000  Parameter: 00000000dabfdabf
>     PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
>     Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
>              ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 
>              0000000000000000 0000000000000000 0000000000000000 0000000000000000 
>              0000000000000000 0000000000000000 0000000000000000 0000000000000000 
>              0000000000000000 0000000000000000 0000000000000000 ffff800009399008 
>              ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 
>              0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 
>              0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 
>     Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
>              ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 
>              ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 
> 
> Changelog
> ========= 
> v5:

Next time can you include a link to the cover letter of the previous
patch set? It is extremely helpful for understanding the progress to
date and allows reviewers to see prior feedback.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 00/22] Support SDEI Virtualization
@ 2022-03-22 18:13   ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 18:13 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Tue, Mar 22, 2022 at 04:06:48PM +0800, Gavin Shan wrote:
> This series intends to virtualize Software Delegated Exception Interface
> (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to
> deliver NMI-alike SDEI event to guest and it's needed by Async PF to
> deliver page-not-present notification from hypervisor to guest. The code
> and the required qemu changes can be found from:
> 
>    https://developer.arm.com/documentation/den0054/c
>    https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>    https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
> 
> For the design and migration needs, please refer to the document in
> PATCH[21/22] in this series. The series is organized as below:
> 
>   PATCH[01]    Introduces template for smccc_get_argx()
>   PATCH[02]    Adds SDEI virtualization infrastructure
>   PATCH[03-17] Supports various SDEI hypercalls and event handling
>   PATCH[18-20] Adds ioctl commands to support migration and configuration
>                and exports SDEI capability
>   PATCH[21]    Adds SDEI document 
>   PATCH[22]    Adds SDEI selftest case
> 
> Testing
> =======
> 
> [1] The selftest case included in this series works fine. The default SDEI
>     event, whose number is zero, can be registered, enabled, raised. The
>     SDEI event handler can be invoked.
> 
>     [host]# pwd
>     /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
>     [root@virtlab-arm01 kvm]# ./aarch64/sdei 
> 
>         NR_VCPUS: 2    SDEI Event: 0x00000000
> 
>     --- VERSION
>         Version:              1.1 (vendor: 0x4b564d)
>     --- FEATURES
>         Shared event slots:   0
>         Private event slots:  0
>         Relative mode:        No
>     --- PRIVATE_RESET
>     --- SHARED_RESET
>     --- PE_UNMASK
>     --- EVENT_GET_INFO
>         Type:                 Private
>         Priority:             Normal
>         Signaled:             Yes
>     --- EVENT_REGISTER
>     --- EVENT_ENABLE
>     --- EVENT_SIGNAL
>         Handled:              Yes
>         IRQ:                  No
>         Status:               Registered-Enabled-Running
>         PC/PSTATE:            000000000040232c 00000000600003c5
>         Regs:                 0000000000000000 0000000000000000
>                               0000000000000000 0000000000000000
>     --- PE_MASK
>     --- EVENT_DISABLE
>     --- EVENT_UNREGISTER
> 
>         Result: OK
> 
> [2] There are additional patches in the following repositories to create
>     procfs entries, allowing to inject SDEI event from host side. The
>     SDEI client in the guest side registers the SDEI default event, whose
>     number is zero. Also, the QEMU exports SDEI ACPI table and supports
>     migration for SDEI.
> 
>     https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>     https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
> 
>     [2.1] Start the guests and migrate the source VM to the destination
>           VM.
> 
>     [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>             -accel kvm -machine virt,gic-version=host                     \
>             -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>             -m 1024M,slots=16,maxmem=64G                                  \
>                :                                                          \
>             -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>             -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>             -append earlycon=pl011,mmio,0x9000000                         \
>                :
> 
>     [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>             -accel kvm -machine virt,gic-version=host                     \
>             -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>             -m 1024M,slots=16,maxmem=64G                                  \
>                :                                                          \
>             -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>             -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>             -append earlycon=pl011,mmio,0x9000000                         \
>             -incoming tcp:0:4444                                          \
>                :
> 
>     [2.2] Check kernel log on the source VM. The SDEI service is enabled
>           and the default SDEI event (0x0) is enabled.
> 
>      [guest-src]# dmesg | grep -i sdei
>      ACPI: SDEI 0x000000005BC80000 000024 \
>                 (v00 BOCHS  BXPC     00000001 BXPC 00000001)
>      sdei: SDEIv1.1 (0x4b564d) detected in firmware.
>      SDEI TEST: Version 1.1, Vendor 0x4b564d
>      sdei_init: SDEI event (0x0) registered
>      sdei_init: SDEI event (0x0) enabled
> 
>  
>      (qemu) migrate -d tcp:localhost:4444
> 
>     [2.3] Migrate the source VM to the destination VM. Inject SDEI event
>           to the destination VM. The event is raised and handled.
> 
>     (qemu) migrate -d tcp:localhost:4444
> 
>     [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1
> 
>     [guest-dst]#
>     =========== SDEI Event (CPU#1) ===========
>     Event: 0000000000000000  Parameter: 00000000dabfdabf
>     PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
>     Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
>              ffff800016c28000 0000000000000000 0000000000000000 0000000000000000 
>              0000000000000000 0000000000000000 0000000000000000 0000000000000000 
>              0000000000000000 0000000000000000 0000000000000000 0000000000000000 
>              0000000000000000 0000000000000000 0000000000000000 ffff800009399008 
>              ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18 
>              0000000000000000 0000000000000000 ffff000000339d00 0000000000000000 
>              0000000000000000 ffff800009c7bde0 ffff800008cbb5c4 
>     Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001 
>              ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190 
>              ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000 
> 
> Changelog
> ========= 
> v5:

Next time can you include a link to the cover letter of the previous
patch set? It is extremely helpful for understanding the progress to
date and allows reviewers to see prior feedback.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
  2022-03-22  8:06   ` Gavin Shan
@ 2022-03-22 19:42     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 19:42 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Gavin,

On Tue, Mar 22, 2022 at 04:06:49PM +0800, Gavin Shan wrote:
> The inline functions used to get the SMCCC parameters have same
> layout. It means these functions can be presented by an unified
> template, to make the code simplified. Besides, this adds more
> similar inline functions like smccc_get_arg{4,5,6,7,8}() to get
> more SMCCC arguments, which are needed by SDEI virtualization
> support.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
>  1 file changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/include/kvm/arm_hypercalls.h b/include/kvm/arm_hypercalls.h
> index 0e2509d27910..d5144c852fe4 100644
> --- a/include/kvm/arm_hypercalls.h
> +++ b/include/kvm/arm_hypercalls.h
> @@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
>  	return vcpu_get_reg(vcpu, 0);
>  }
>  
> -static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
> -{
> -	return vcpu_get_reg(vcpu, 1);
> -}
> -
> -static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
> -{
> -	return vcpu_get_reg(vcpu, 2);
> +#define SMCCC_DECLARE_GET_ARG(reg)					\
> +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
> +{									\
> +	return vcpu_get_reg(vcpu, reg);					\
>  }
>  
> -static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
> -{
> -	return vcpu_get_reg(vcpu, 3);
> -}
> +SMCCC_DECLARE_GET_ARG(1)
> +SMCCC_DECLARE_GET_ARG(2)
> +SMCCC_DECLARE_GET_ARG(3)
> +SMCCC_DECLARE_GET_ARG(4)
> +SMCCC_DECLARE_GET_ARG(5)
> +SMCCC_DECLARE_GET_ARG(6)
> +SMCCC_DECLARE_GET_ARG(7)
> +SMCCC_DECLARE_GET_ARG(8)

Hmm. What if we specify a single inline function where the caller passes
the arg # as a parameter? We really just want to abstract away the
off-by-one difference between GP registers and SMCCC arguments.

Macros generally make me uneasy for template functions, but I may be in
the vocal minority on this topic :)

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
@ 2022-03-22 19:42     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 19:42 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Gavin,

On Tue, Mar 22, 2022 at 04:06:49PM +0800, Gavin Shan wrote:
> The inline functions used to get the SMCCC parameters have same
> layout. It means these functions can be presented by an unified
> template, to make the code simplified. Besides, this adds more
> similar inline functions like smccc_get_arg{4,5,6,7,8}() to get
> more SMCCC arguments, which are needed by SDEI virtualization
> support.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
>  1 file changed, 12 insertions(+), 12 deletions(-)
> 
> diff --git a/include/kvm/arm_hypercalls.h b/include/kvm/arm_hypercalls.h
> index 0e2509d27910..d5144c852fe4 100644
> --- a/include/kvm/arm_hypercalls.h
> +++ b/include/kvm/arm_hypercalls.h
> @@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
>  	return vcpu_get_reg(vcpu, 0);
>  }
>  
> -static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
> -{
> -	return vcpu_get_reg(vcpu, 1);
> -}
> -
> -static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
> -{
> -	return vcpu_get_reg(vcpu, 2);
> +#define SMCCC_DECLARE_GET_ARG(reg)					\
> +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
> +{									\
> +	return vcpu_get_reg(vcpu, reg);					\
>  }
>  
> -static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
> -{
> -	return vcpu_get_reg(vcpu, 3);
> -}
> +SMCCC_DECLARE_GET_ARG(1)
> +SMCCC_DECLARE_GET_ARG(2)
> +SMCCC_DECLARE_GET_ARG(3)
> +SMCCC_DECLARE_GET_ARG(4)
> +SMCCC_DECLARE_GET_ARG(5)
> +SMCCC_DECLARE_GET_ARG(6)
> +SMCCC_DECLARE_GET_ARG(7)
> +SMCCC_DECLARE_GET_ARG(8)

Hmm. What if we specify a single inline function where the caller passes
the arg # as a parameter? We really just want to abstract away the
off-by-one difference between GP registers and SMCCC arguments.

Macros generally make me uneasy for template functions, but I may be in
the vocal minority on this topic :)

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-22  8:06   ` Gavin Shan
@ 2022-03-22 22:43     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 22:43 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Gavin,

On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
> Software Delegated Exception Interface (SDEI) provides a mechanism for
> registering and servicing system events. Those system events are high
> priority events, which must be serviced immediately. It's going to be
> used by Asynchronous Page Fault (APF) to deliver notification from KVM
> to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

I'm guessing that you're using linked lists for stitching all of this
together because the specification provides for 24 bits of event
encoding. However, it seems that there will be a finite number of events
in KVM. So the APF stuff and a software signaled event.

Given that the number of events in KVM is rather small, would it make
more sense to do away with the overhead of linked lists and having the
state just represented in a static or allocated array? I think you can
cram all of the VM scoped event state into a single structure and hang
the implementation off of that.

> This introduces SDEI virtualization infrastructure where the SDEI events
> are registered and manipulated by the guest through hypercall. The SDEI
> event is delivered to one specific vCPU by KVM once it's raised. This
> introduces data structures to represent the needed objects to support
> the feature, which is highlighted as below. As those objects could be
> migrated between VMs, these data structures are partially exposed to
> user space.
> 
>    * kvm_sdei_exposed_event
>      The exposed events are determined and added by VMM through ioctl
>      interface. Only the exposed events can be registered from the
>      guest.
> 
>    * kvm_sdei_registered_event
>      The events that have been registered from the guest through the
>      SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.
> 
>    * kvm_sdei_vcpu_event
>      The events that have been delivered to the target vCPU.
> 
>    * kvm_sdei_vcpu
>      Used to save the preempted context when the SDEI event is serviced
>      and delivered. After the SDEI event handling is completed, the
>      execution is resumed from the preempted context.
> 
>    * kvm_sdei_kvm
>      Place holder for the exposed and registered events.

It might be a good idea to expand these a bit and move them into
comments on each of the structures.

> The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for
> now. They will be supported in the subsequent patches.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  arch/arm64/include/asm/kvm_host.h            |   3 +
>  arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
>  arch/arm64/include/uapi/asm/kvm.h            |   1 +
>  arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
>  arch/arm64/kvm/Makefile                      |   2 +-
>  arch/arm64/kvm/arm.c                         |   8 +
>  arch/arm64/kvm/hypercalls.c                  |  21 ++
>  arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
>  include/uapi/linux/arm_sdei.h                |   2 +
>  9 files changed, 523 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arm64/include/asm/kvm_sdei.h
>  create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
>  create mode 100644 arch/arm64/kvm/sdei.c
> 
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 031e3a2537fc..5d37e046a458 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -113,6 +113,8 @@ struct kvm_arch {
>  	/* Interrupt controller */
>  	struct vgic_dist	vgic;
>  
> +	struct kvm_sdei_kvm *sdei;
> +

nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough
:)

>  	/* Mandated version of PSCI */
>  	u32 psci_version;
>  
> @@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
>  	 * Anything that is not used directly from assembly code goes
>  	 * here.
>  	 */
> +	struct kvm_sdei_vcpu *sdei;
>

nit: put your scoping tokens at the beginning of a symbol name, so
'struct kvm_vcpu_sdei'.

[...]

> diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
> index 202b8c455724..3c20fee72bb4 100644
> --- a/arch/arm64/kvm/hypercalls.c
> +++ b/arch/arm64/kvm/hypercalls.c
> @@ -5,6 +5,7 @@
>  #include <linux/kvm_host.h>
>  
>  #include <asm/kvm_emulate.h>
> +#include <asm/kvm_sdei.h>
>  
>  #include <kvm/arm_hypercalls.h>
>  #include <kvm/arm_psci.h>
> @@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
>  	case ARM_SMCCC_TRNG_RND32:
>  	case ARM_SMCCC_TRNG_RND64:
>  		return kvm_trng_call(vcpu);
> +	case SDEI_1_0_FN_SDEI_VERSION:
> +	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
> +	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
> +	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
> +	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
> +	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
> +	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
> +	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
> +	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
> +	case SDEI_1_0_FN_SDEI_PE_MASK:
> +	case SDEI_1_0_FN_SDEI_PE_UNMASK:
> +	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
> +	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
> +	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
> +	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
> +	case SDEI_1_0_FN_SDEI_SHARED_RESET:
> +	case SDEI_1_1_FN_SDEI_FEATURES:

Consider only adding switch statements for hypercalls when they're
actually implemented.

Additionally, while this isn't directly related to your patch, I do have
a gripe about kvm_hvc_call_handler(). It is really ugly that we
enumerate the specific hypercalls we support, and otherwise fall through
to PSCI.

IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply
route a particular service range/service owner to the appropriate
handler. We can then terminate individual hypercalls in those handlers,
avoiding a catch-all switch such as this one is currently.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-22 22:43     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 22:43 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Gavin,

On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
> Software Delegated Exception Interface (SDEI) provides a mechanism for
> registering and servicing system events. Those system events are high
> priority events, which must be serviced immediately. It's going to be
> used by Asynchronous Page Fault (APF) to deliver notification from KVM
> to guest. It's noted that SDEI is defined by ARM DEN0054C specification.

I'm guessing that you're using linked lists for stitching all of this
together because the specification provides for 24 bits of event
encoding. However, it seems that there will be a finite number of events
in KVM. So the APF stuff and a software signaled event.

Given that the number of events in KVM is rather small, would it make
more sense to do away with the overhead of linked lists and having the
state just represented in a static or allocated array? I think you can
cram all of the VM scoped event state into a single structure and hang
the implementation off of that.

> This introduces SDEI virtualization infrastructure where the SDEI events
> are registered and manipulated by the guest through hypercall. The SDEI
> event is delivered to one specific vCPU by KVM once it's raised. This
> introduces data structures to represent the needed objects to support
> the feature, which is highlighted as below. As those objects could be
> migrated between VMs, these data structures are partially exposed to
> user space.
> 
>    * kvm_sdei_exposed_event
>      The exposed events are determined and added by VMM through ioctl
>      interface. Only the exposed events can be registered from the
>      guest.
> 
>    * kvm_sdei_registered_event
>      The events that have been registered from the guest through the
>      SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.
> 
>    * kvm_sdei_vcpu_event
>      The events that have been delivered to the target vCPU.
> 
>    * kvm_sdei_vcpu
>      Used to save the preempted context when the SDEI event is serviced
>      and delivered. After the SDEI event handling is completed, the
>      execution is resumed from the preempted context.
> 
>    * kvm_sdei_kvm
>      Place holder for the exposed and registered events.

It might be a good idea to expand these a bit and move them into
comments on each of the structures.

> The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for
> now. They will be supported in the subsequent patches.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  arch/arm64/include/asm/kvm_host.h            |   3 +
>  arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
>  arch/arm64/include/uapi/asm/kvm.h            |   1 +
>  arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
>  arch/arm64/kvm/Makefile                      |   2 +-
>  arch/arm64/kvm/arm.c                         |   8 +
>  arch/arm64/kvm/hypercalls.c                  |  21 ++
>  arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
>  include/uapi/linux/arm_sdei.h                |   2 +
>  9 files changed, 523 insertions(+), 1 deletion(-)
>  create mode 100644 arch/arm64/include/asm/kvm_sdei.h
>  create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
>  create mode 100644 arch/arm64/kvm/sdei.c
> 
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 031e3a2537fc..5d37e046a458 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -113,6 +113,8 @@ struct kvm_arch {
>  	/* Interrupt controller */
>  	struct vgic_dist	vgic;
>  
> +	struct kvm_sdei_kvm *sdei;
> +

nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough
:)

>  	/* Mandated version of PSCI */
>  	u32 psci_version;
>  
> @@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
>  	 * Anything that is not used directly from assembly code goes
>  	 * here.
>  	 */
> +	struct kvm_sdei_vcpu *sdei;
>

nit: put your scoping tokens at the beginning of a symbol name, so
'struct kvm_vcpu_sdei'.

[...]

> diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
> index 202b8c455724..3c20fee72bb4 100644
> --- a/arch/arm64/kvm/hypercalls.c
> +++ b/arch/arm64/kvm/hypercalls.c
> @@ -5,6 +5,7 @@
>  #include <linux/kvm_host.h>
>  
>  #include <asm/kvm_emulate.h>
> +#include <asm/kvm_sdei.h>
>  
>  #include <kvm/arm_hypercalls.h>
>  #include <kvm/arm_psci.h>
> @@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
>  	case ARM_SMCCC_TRNG_RND32:
>  	case ARM_SMCCC_TRNG_RND64:
>  		return kvm_trng_call(vcpu);
> +	case SDEI_1_0_FN_SDEI_VERSION:
> +	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
> +	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
> +	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
> +	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
> +	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
> +	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
> +	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
> +	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
> +	case SDEI_1_0_FN_SDEI_PE_MASK:
> +	case SDEI_1_0_FN_SDEI_PE_UNMASK:
> +	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
> +	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
> +	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
> +	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
> +	case SDEI_1_0_FN_SDEI_SHARED_RESET:
> +	case SDEI_1_1_FN_SDEI_FEATURES:

Consider only adding switch statements for hypercalls when they're
actually implemented.

Additionally, while this isn't directly related to your patch, I do have
a gripe about kvm_hvc_call_handler(). It is really ugly that we
enumerate the specific hypercalls we support, and otherwise fall through
to PSCI.

IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply
route a particular service range/service owner to the appropriate
handler. We can then terminate individual hypercalls in those handlers,
avoiding a catch-all switch such as this one is currently.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
  2022-03-22  8:07   ` Gavin Shan
@ 2022-03-22 23:06     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 23:06 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Gavin,

On Tue, Mar 22, 2022 at 04:07:03PM +0800, Gavin Shan wrote:
> This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest
> to inject SDEI event, whose number must be zero to the specified
> vCPU. As the routing mode and affinity isn't supported yet, the
> calling vCPU is assumed to be the target.
> 
> The SDEI event 0x0 is a private one, with normal priority. It's
> usually used for testing.

I don't know if that is actually the case. One real use that immediately
comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't
glean at how the guest may use a particular call, so at most we should
just point at the spec and state that event 0 is for software signaled
events.

> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 63 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
> index a24270378305..ba2ca65c871b 100644
> --- a/arch/arm64/kvm/sdei.c
> +++ b/arch/arm64/kvm/sdei.c
> @@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
> +{
> +	struct kvm *kvm = vcpu->kvm;
> +	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
> +	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
> +	struct kvm_sdei_exposed_event *exposed_event;
> +	struct kvm_sdei_registered_event *registered_event;
> +	unsigned long event_num = smccc_get_arg1(vcpu);
> +	int index;
> +	unsigned long ret = SDEI_SUCCESS;
> +
> +	/* @event_num must be zero */
> +	if (!kvm_sdei_is_default(event_num)) {

0 isn't KVM's default event. I'd argue KVM doesn't have a default event
to begin with. This has a precise definition coming from the spec. In
fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any
missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.

That goes for any values coming from the specification. KVM's
implementation details belong in a KVM header :)

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
@ 2022-03-22 23:06     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-22 23:06 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Gavin,

On Tue, Mar 22, 2022 at 04:07:03PM +0800, Gavin Shan wrote:
> This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest
> to inject SDEI event, whose number must be zero to the specified
> vCPU. As the routing mode and affinity isn't supported yet, the
> calling vCPU is assumed to be the target.
> 
> The SDEI event 0x0 is a private one, with normal priority. It's
> usually used for testing.

I don't know if that is actually the case. One real use that immediately
comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't
glean at how the guest may use a particular call, so at most we should
just point at the spec and state that event 0 is for software signaled
events.

> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
>  1 file changed, 63 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
> index a24270378305..ba2ca65c871b 100644
> --- a/arch/arm64/kvm/sdei.c
> +++ b/arch/arm64/kvm/sdei.c
> @@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
> +{
> +	struct kvm *kvm = vcpu->kvm;
> +	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
> +	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
> +	struct kvm_sdei_exposed_event *exposed_event;
> +	struct kvm_sdei_registered_event *registered_event;
> +	unsigned long event_num = smccc_get_arg1(vcpu);
> +	int index;
> +	unsigned long ret = SDEI_SUCCESS;
> +
> +	/* @event_num must be zero */
> +	if (!kvm_sdei_is_default(event_num)) {

0 isn't KVM's default event. I'd argue KVM doesn't have a default event
to begin with. This has a precise definition coming from the spec. In
fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any
missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.

That goes for any values coming from the specification. KVM's
implementation details belong in a KVM header :)

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
  2022-03-22 19:42     ` Oliver Upton
@ 2022-03-23 12:16       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:16 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/23/22 3:42 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:49PM +0800, Gavin Shan wrote:
>> The inline functions used to get the SMCCC parameters have same
>> layout. It means these functions can be presented by an unified
>> template, to make the code simplified. Besides, this adds more
>> similar inline functions like smccc_get_arg{4,5,6,7,8}() to get
>> more SMCCC arguments, which are needed by SDEI virtualization
>> support.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
>>   1 file changed, 12 insertions(+), 12 deletions(-)
>>
>> diff --git a/include/kvm/arm_hypercalls.h b/include/kvm/arm_hypercalls.h
>> index 0e2509d27910..d5144c852fe4 100644
>> --- a/include/kvm/arm_hypercalls.h
>> +++ b/include/kvm/arm_hypercalls.h
>> @@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
>>   	return vcpu_get_reg(vcpu, 0);
>>   }
>>   
>> -static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
>> -{
>> -	return vcpu_get_reg(vcpu, 1);
>> -}
>> -
>> -static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
>> -{
>> -	return vcpu_get_reg(vcpu, 2);
>> +#define SMCCC_DECLARE_GET_ARG(reg)					\
>> +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
>> +{									\
>> +	return vcpu_get_reg(vcpu, reg);					\
>>   }
>>   
>> -static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
>> -{
>> -	return vcpu_get_reg(vcpu, 3);
>> -}
>> +SMCCC_DECLARE_GET_ARG(1)
>> +SMCCC_DECLARE_GET_ARG(2)
>> +SMCCC_DECLARE_GET_ARG(3)
>> +SMCCC_DECLARE_GET_ARG(4)
>> +SMCCC_DECLARE_GET_ARG(5)
>> +SMCCC_DECLARE_GET_ARG(6)
>> +SMCCC_DECLARE_GET_ARG(7)
>> +SMCCC_DECLARE_GET_ARG(8)
> 
> Hmm. What if we specify a single inline function where the caller passes
> the arg # as a parameter? We really just want to abstract away the
> off-by-one difference between GP registers and SMCCC arguments.
> 
> Macros generally make me uneasy for template functions, but I may be in
> the vocal minority on this topic :)
> 

I think it's a good idea to have smccc_get_arg(unsigned char index).
However, it will cause more code changes because the following functions
have been used. Anyway, I think it's still worthy to pass @index to
differentiate the argument index. I will change it accordingly in
next respin.

    smccc_get_arg1()
    smccc_get_arg2()
    smccc_get_arg3()

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions
@ 2022-03-23 12:16       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:16 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/23/22 3:42 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:49PM +0800, Gavin Shan wrote:
>> The inline functions used to get the SMCCC parameters have same
>> layout. It means these functions can be presented by an unified
>> template, to make the code simplified. Besides, this adds more
>> similar inline functions like smccc_get_arg{4,5,6,7,8}() to get
>> more SMCCC arguments, which are needed by SDEI virtualization
>> support.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   include/kvm/arm_hypercalls.h | 24 ++++++++++++------------
>>   1 file changed, 12 insertions(+), 12 deletions(-)
>>
>> diff --git a/include/kvm/arm_hypercalls.h b/include/kvm/arm_hypercalls.h
>> index 0e2509d27910..d5144c852fe4 100644
>> --- a/include/kvm/arm_hypercalls.h
>> +++ b/include/kvm/arm_hypercalls.h
>> @@ -13,20 +13,20 @@ static inline u32 smccc_get_function(struct kvm_vcpu *vcpu)
>>   	return vcpu_get_reg(vcpu, 0);
>>   }
>>   
>> -static inline unsigned long smccc_get_arg1(struct kvm_vcpu *vcpu)
>> -{
>> -	return vcpu_get_reg(vcpu, 1);
>> -}
>> -
>> -static inline unsigned long smccc_get_arg2(struct kvm_vcpu *vcpu)
>> -{
>> -	return vcpu_get_reg(vcpu, 2);
>> +#define SMCCC_DECLARE_GET_ARG(reg)					\
>> +static inline unsigned long smccc_get_arg##reg(struct kvm_vcpu *vcpu)	\
>> +{									\
>> +	return vcpu_get_reg(vcpu, reg);					\
>>   }
>>   
>> -static inline unsigned long smccc_get_arg3(struct kvm_vcpu *vcpu)
>> -{
>> -	return vcpu_get_reg(vcpu, 3);
>> -}
>> +SMCCC_DECLARE_GET_ARG(1)
>> +SMCCC_DECLARE_GET_ARG(2)
>> +SMCCC_DECLARE_GET_ARG(3)
>> +SMCCC_DECLARE_GET_ARG(4)
>> +SMCCC_DECLARE_GET_ARG(5)
>> +SMCCC_DECLARE_GET_ARG(6)
>> +SMCCC_DECLARE_GET_ARG(7)
>> +SMCCC_DECLARE_GET_ARG(8)
> 
> Hmm. What if we specify a single inline function where the caller passes
> the arg # as a parameter? We really just want to abstract away the
> off-by-one difference between GP registers and SMCCC arguments.
> 
> Macros generally make me uneasy for template functions, but I may be in
> the vocal minority on this topic :)
> 

I think it's a good idea to have smccc_get_arg(unsigned char index).
However, it will cause more code changes because the following functions
have been used. Anyway, I think it's still worthy to pass @index to
differentiate the argument index. I will change it accordingly in
next respin.

    smccc_get_arg1()
    smccc_get_arg2()
    smccc_get_arg3()

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-22 22:43     ` Oliver Upton
@ 2022-03-23 12:40       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:40 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/23/22 6:43 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
>> Software Delegated Exception Interface (SDEI) provides a mechanism for
>> registering and servicing system events. Those system events are high
>> priority events, which must be serviced immediately. It's going to be
>> used by Asynchronous Page Fault (APF) to deliver notification from KVM
>> to guest. It's noted that SDEI is defined by ARM DEN0054C specification.
> 
> I'm guessing that you're using linked lists for stitching all of this
> together because the specification provides for 24 bits of event
> encoding. However, it seems that there will be a finite number of events
> in KVM. So the APF stuff and a software signaled event.
> 
> Given that the number of events in KVM is rather small, would it make
> more sense to do away with the overhead of linked lists and having the
> state just represented in a static or allocated array? I think you can
> cram all of the VM scoped event state into a single structure and hang
> the implementation off of that.
> 

Yes, the number of events in KVM is small. Including the events for Async
PF and the software signaled event, 8 events would be enough currently. In
the meanwhile, there are several types of objects for various events. Some
of them can be put into static array, while the left might need static array
of pointers to avoid the linked list:

    struct kvm_sdei_exposed_event/state         on struct kvm_arch
           size: 24 bytes
           static array, 8 entries
    struct kvm_sdei_registered_event/state      on struct kvm_arch
           size: 9KB
           static array of pointers, still need allocate them dynamically, 8 entries
    struct kvm_sdei_vcpu_event/state            on struct kvm_vcpu_arch
           size: 16 bytes
           static array, 8 entries


>> This introduces SDEI virtualization infrastructure where the SDEI events
>> are registered and manipulated by the guest through hypercall. The SDEI
>> event is delivered to one specific vCPU by KVM once it's raised. This
>> introduces data structures to represent the needed objects to support
>> the feature, which is highlighted as below. As those objects could be
>> migrated between VMs, these data structures are partially exposed to
>> user space.
>>
>>     * kvm_sdei_exposed_event
>>       The exposed events are determined and added by VMM through ioctl
>>       interface. Only the exposed events can be registered from the
>>       guest.
>>
>>     * kvm_sdei_registered_event
>>       The events that have been registered from the guest through the
>>       SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.
>>
>>     * kvm_sdei_vcpu_event
>>       The events that have been delivered to the target vCPU.
>>
>>     * kvm_sdei_vcpu
>>       Used to save the preempted context when the SDEI event is serviced
>>       and delivered. After the SDEI event handling is completed, the
>>       execution is resumed from the preempted context.
>>
>>     * kvm_sdei_kvm
>>       Place holder for the exposed and registered events.
> 
> It might be a good idea to expand these a bit and move them into
> comments on each of the structures.
> 

Sure, I will do in next respin.

>> The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for
>> now. They will be supported in the subsequent patches.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   arch/arm64/include/asm/kvm_host.h            |   3 +
>>   arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
>>   arch/arm64/include/uapi/asm/kvm.h            |   1 +
>>   arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
>>   arch/arm64/kvm/Makefile                      |   2 +-
>>   arch/arm64/kvm/arm.c                         |   8 +
>>   arch/arm64/kvm/hypercalls.c                  |  21 ++
>>   arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
>>   include/uapi/linux/arm_sdei.h                |   2 +
>>   9 files changed, 523 insertions(+), 1 deletion(-)
>>   create mode 100644 arch/arm64/include/asm/kvm_sdei.h
>>   create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
>>   create mode 100644 arch/arm64/kvm/sdei.c
>>
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>> index 031e3a2537fc..5d37e046a458 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
>> @@ -113,6 +113,8 @@ struct kvm_arch {
>>   	/* Interrupt controller */
>>   	struct vgic_dist	vgic;
>>   
>> +	struct kvm_sdei_kvm *sdei;
>> +
> 
> nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough
> :)
> 

Indeed, "struct kvm_sdei" is better :)

>>   	/* Mandated version of PSCI */
>>   	u32 psci_version;
>>   
>> @@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
>>   	 * Anything that is not used directly from assembly code goes
>>   	 * here.
>>   	 */
>> +	struct kvm_sdei_vcpu *sdei;
>>
> 
> nit: put your scoping tokens at the beginning of a symbol name, so
> 'struct kvm_vcpu_sdei'.
> 
> [...]
> 

Yep, "struct kvm_vcpu_sdei" is the one I will have in next respin :)

>> diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
>> index 202b8c455724..3c20fee72bb4 100644
>> --- a/arch/arm64/kvm/hypercalls.c
>> +++ b/arch/arm64/kvm/hypercalls.c
>> @@ -5,6 +5,7 @@
>>   #include <linux/kvm_host.h>
>>   
>>   #include <asm/kvm_emulate.h>
>> +#include <asm/kvm_sdei.h>
>>   
>>   #include <kvm/arm_hypercalls.h>
>>   #include <kvm/arm_psci.h>
>> @@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
>>   	case ARM_SMCCC_TRNG_RND32:
>>   	case ARM_SMCCC_TRNG_RND64:
>>   		return kvm_trng_call(vcpu);
>> +	case SDEI_1_0_FN_SDEI_VERSION:
>> +	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
>> +	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
>> +	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
>> +	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
>> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
>> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
>> +	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
>> +	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
>> +	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
>> +	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
>> +	case SDEI_1_0_FN_SDEI_PE_MASK:
>> +	case SDEI_1_0_FN_SDEI_PE_UNMASK:
>> +	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
>> +	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
>> +	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
>> +	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
>> +	case SDEI_1_0_FN_SDEI_SHARED_RESET:
>> +	case SDEI_1_1_FN_SDEI_FEATURES:
> 
> Consider only adding switch statements for hypercalls when they're
> actually implemented.
> 
> Additionally, while this isn't directly related to your patch, I do have
> a gripe about kvm_hvc_call_handler(). It is really ugly that we
> enumerate the specific hypercalls we support, and otherwise fall through
> to PSCI.
> 
> IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply
> route a particular service range/service owner to the appropriate
> handler. We can then terminate individual hypercalls in those handlers,
> avoiding a catch-all switch such as this one is currently.
> 

Yes, I agree. I can have a separate patch as preparatory work to
route the range of hypercalls to their owner for further handling.
In this way, we can route the range of SDEI hypercalls to its own
handler. I will figure it out in next respin.

Thanks,
Gavin



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-23 12:40       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:40 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/23/22 6:43 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
>> Software Delegated Exception Interface (SDEI) provides a mechanism for
>> registering and servicing system events. Those system events are high
>> priority events, which must be serviced immediately. It's going to be
>> used by Asynchronous Page Fault (APF) to deliver notification from KVM
>> to guest. It's noted that SDEI is defined by ARM DEN0054C specification.
> 
> I'm guessing that you're using linked lists for stitching all of this
> together because the specification provides for 24 bits of event
> encoding. However, it seems that there will be a finite number of events
> in KVM. So the APF stuff and a software signaled event.
> 
> Given that the number of events in KVM is rather small, would it make
> more sense to do away with the overhead of linked lists and having the
> state just represented in a static or allocated array? I think you can
> cram all of the VM scoped event state into a single structure and hang
> the implementation off of that.
> 

Yes, the number of events in KVM is small. Including the events for Async
PF and the software signaled event, 8 events would be enough currently. In
the meanwhile, there are several types of objects for various events. Some
of them can be put into static array, while the left might need static array
of pointers to avoid the linked list:

    struct kvm_sdei_exposed_event/state         on struct kvm_arch
           size: 24 bytes
           static array, 8 entries
    struct kvm_sdei_registered_event/state      on struct kvm_arch
           size: 9KB
           static array of pointers, still need allocate them dynamically, 8 entries
    struct kvm_sdei_vcpu_event/state            on struct kvm_vcpu_arch
           size: 16 bytes
           static array, 8 entries


>> This introduces SDEI virtualization infrastructure where the SDEI events
>> are registered and manipulated by the guest through hypercall. The SDEI
>> event is delivered to one specific vCPU by KVM once it's raised. This
>> introduces data structures to represent the needed objects to support
>> the feature, which is highlighted as below. As those objects could be
>> migrated between VMs, these data structures are partially exposed to
>> user space.
>>
>>     * kvm_sdei_exposed_event
>>       The exposed events are determined and added by VMM through ioctl
>>       interface. Only the exposed events can be registered from the
>>       guest.
>>
>>     * kvm_sdei_registered_event
>>       The events that have been registered from the guest through the
>>       SDEI_1_0_FN_SDEI_EVENT_REGISTER hypercall.
>>
>>     * kvm_sdei_vcpu_event
>>       The events that have been delivered to the target vCPU.
>>
>>     * kvm_sdei_vcpu
>>       Used to save the preempted context when the SDEI event is serviced
>>       and delivered. After the SDEI event handling is completed, the
>>       execution is resumed from the preempted context.
>>
>>     * kvm_sdei_kvm
>>       Place holder for the exposed and registered events.
> 
> It might be a good idea to expand these a bit and move them into
> comments on each of the structures.
> 

Sure, I will do in next respin.

>> The error of SDEI_NOT_SUPPORTED is returned for all SDEI hypercalls for
>> now. They will be supported in the subsequent patches.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   arch/arm64/include/asm/kvm_host.h            |   3 +
>>   arch/arm64/include/asm/kvm_sdei.h            | 171 +++++++++++++
>>   arch/arm64/include/uapi/asm/kvm.h            |   1 +
>>   arch/arm64/include/uapi/asm/kvm_sdei_state.h |  72 ++++++
>>   arch/arm64/kvm/Makefile                      |   2 +-
>>   arch/arm64/kvm/arm.c                         |   8 +
>>   arch/arm64/kvm/hypercalls.c                  |  21 ++
>>   arch/arm64/kvm/sdei.c                        | 244 +++++++++++++++++++
>>   include/uapi/linux/arm_sdei.h                |   2 +
>>   9 files changed, 523 insertions(+), 1 deletion(-)
>>   create mode 100644 arch/arm64/include/asm/kvm_sdei.h
>>   create mode 100644 arch/arm64/include/uapi/asm/kvm_sdei_state.h
>>   create mode 100644 arch/arm64/kvm/sdei.c
>>
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>> index 031e3a2537fc..5d37e046a458 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
>> @@ -113,6 +113,8 @@ struct kvm_arch {
>>   	/* Interrupt controller */
>>   	struct vgic_dist	vgic;
>>   
>> +	struct kvm_sdei_kvm *sdei;
>> +
> 
> nit: avoid repeating 'kvm'. struct kvm_sdei should be descriptive enough
> :)
> 

Indeed, "struct kvm_sdei" is better :)

>>   	/* Mandated version of PSCI */
>>   	u32 psci_version;
>>   
>> @@ -338,6 +340,7 @@ struct kvm_vcpu_arch {
>>   	 * Anything that is not used directly from assembly code goes
>>   	 * here.
>>   	 */
>> +	struct kvm_sdei_vcpu *sdei;
>>
> 
> nit: put your scoping tokens at the beginning of a symbol name, so
> 'struct kvm_vcpu_sdei'.
> 
> [...]
> 

Yep, "struct kvm_vcpu_sdei" is the one I will have in next respin :)

>> diff --git a/arch/arm64/kvm/hypercalls.c b/arch/arm64/kvm/hypercalls.c
>> index 202b8c455724..3c20fee72bb4 100644
>> --- a/arch/arm64/kvm/hypercalls.c
>> +++ b/arch/arm64/kvm/hypercalls.c
>> @@ -5,6 +5,7 @@
>>   #include <linux/kvm_host.h>
>>   
>>   #include <asm/kvm_emulate.h>
>> +#include <asm/kvm_sdei.h>
>>   
>>   #include <kvm/arm_hypercalls.h>
>>   #include <kvm/arm_psci.h>
>> @@ -151,6 +152,26 @@ int kvm_hvc_call_handler(struct kvm_vcpu *vcpu)
>>   	case ARM_SMCCC_TRNG_RND32:
>>   	case ARM_SMCCC_TRNG_RND64:
>>   		return kvm_trng_call(vcpu);
>> +	case SDEI_1_0_FN_SDEI_VERSION:
>> +	case SDEI_1_0_FN_SDEI_EVENT_REGISTER:
>> +	case SDEI_1_0_FN_SDEI_EVENT_ENABLE:
>> +	case SDEI_1_0_FN_SDEI_EVENT_DISABLE:
>> +	case SDEI_1_0_FN_SDEI_EVENT_CONTEXT:
>> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE:
>> +	case SDEI_1_0_FN_SDEI_EVENT_COMPLETE_AND_RESUME:
>> +	case SDEI_1_0_FN_SDEI_EVENT_UNREGISTER:
>> +	case SDEI_1_0_FN_SDEI_EVENT_STATUS:
>> +	case SDEI_1_0_FN_SDEI_EVENT_GET_INFO:
>> +	case SDEI_1_0_FN_SDEI_EVENT_ROUTING_SET:
>> +	case SDEI_1_0_FN_SDEI_PE_MASK:
>> +	case SDEI_1_0_FN_SDEI_PE_UNMASK:
>> +	case SDEI_1_0_FN_SDEI_INTERRUPT_BIND:
>> +	case SDEI_1_0_FN_SDEI_INTERRUPT_RELEASE:
>> +	case SDEI_1_1_FN_SDEI_EVENT_SIGNAL:
>> +	case SDEI_1_0_FN_SDEI_PRIVATE_RESET:
>> +	case SDEI_1_0_FN_SDEI_SHARED_RESET:
>> +	case SDEI_1_1_FN_SDEI_FEATURES:
> 
> Consider only adding switch statements for hypercalls when they're
> actually implemented.
> 
> Additionally, while this isn't directly related to your patch, I do have
> a gripe about kvm_hvc_call_handler(). It is really ugly that we
> enumerate the specific hypercalls we support, and otherwise fall through
> to PSCI.
> 
> IMO, a cleaner approach would be to have kvm_hvc_call_handler() simply
> route a particular service range/service owner to the appropriate
> handler. We can then terminate individual hypercalls in those handlers,
> avoiding a catch-all switch such as this one is currently.
> 

Yes, I agree. I can have a separate patch as preparatory work to
route the range of hypercalls to their owner for further handling.
In this way, we can route the range of SDEI hypercalls to its own
handler. I will figure it out in next respin.

Thanks,
Gavin


_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-22 18:04     ` Oliver Upton
@ 2022-03-23 12:46       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:46 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/23/22 2:04 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
>> This supports SDEI_VERSION hypercall by returning v1.1, which is
>> the specification version we're following. The vendor is set to
>> 'KVM'.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   arch/arm64/kvm/sdei.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
>> index 8a9b477b8977..5a3a64cd6e84 100644
>> --- a/arch/arm64/kvm/sdei.c
>> +++ b/arch/arm64/kvm/sdei.c
>> @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
>>   	return pending;
>>   }
>>   
>> +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
>> +{
>> +	/* v1.1 and the vendor is KVM */
>> +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
>> +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
>> +	       0x4b564d;
> 
> It looks like the SDEI specification states that the vendor-defined
> version number is 32 bits. Could we just use one of the
> ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
> 
> ASCII 'KVM' is neat, but in reality guest software will just throw it in
> a macro regardless. Might as well use one of the values we've already
> trained it to use :-)
> 
> Also, it would appear that guest discovery of SDEI relies upon KVM
> reporting a valid SDEI version. IMO, this patch should come at the very
> end when KVM actually implements SDEI.
> 

Yeah, I was sticky to the pattern of "KVM". However, I think it's good
to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
if you agree. Its first two characters are "VM" at least.

It's fine to return the version here because the SDEI capability isn't
exposed yet. It means no events have been exposed and able to be
registered. However, It's also fine to move this patch after the
following one:

[PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-23 12:46       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:46 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/23/22 2:04 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
>> This supports SDEI_VERSION hypercall by returning v1.1, which is
>> the specification version we're following. The vendor is set to
>> 'KVM'.
>>
>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   arch/arm64/kvm/sdei.c | 10 ++++++++++
>>   1 file changed, 10 insertions(+)
>>
>> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
>> index 8a9b477b8977..5a3a64cd6e84 100644
>> --- a/arch/arm64/kvm/sdei.c
>> +++ b/arch/arm64/kvm/sdei.c
>> @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
>>   	return pending;
>>   }
>>   
>> +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
>> +{
>> +	/* v1.1 and the vendor is KVM */
>> +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
>> +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
>> +	       0x4b564d;
> 
> It looks like the SDEI specification states that the vendor-defined
> version number is 32 bits. Could we just use one of the
> ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
> 
> ASCII 'KVM' is neat, but in reality guest software will just throw it in
> a macro regardless. Might as well use one of the values we've already
> trained it to use :-)
> 
> Also, it would appear that guest discovery of SDEI relies upon KVM
> reporting a valid SDEI version. IMO, this patch should come at the very
> end when KVM actually implements SDEI.
> 

Yeah, I was sticky to the pattern of "KVM". However, I think it's good
to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
if you agree. Its first two characters are "VM" at least.

It's fine to return the version here because the SDEI capability isn't
exposed yet. It means no events have been exposed and able to be
registered. However, It's also fine to move this patch after the
following one:

[PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
  2022-03-22 23:06     ` Oliver Upton
@ 2022-03-23 12:52       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:52 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/23/22 7:06 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:07:03PM +0800, Gavin Shan wrote:
>> This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest
>> to inject SDEI event, whose number must be zero to the specified
>> vCPU. As the routing mode and affinity isn't supported yet, the
>> calling vCPU is assumed to be the target.
>>
>> The SDEI event 0x0 is a private one, with normal priority. It's
>> usually used for testing.
> 
> I don't know if that is actually the case. One real use that immediately
> comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't
> glean at how the guest may use a particular call, so at most we should
> just point at the spec and state that event 0 is for software signaled
> events.
> 

Yes, I agree. I will amend the change log in next respin, to make it
clear. As you said, it really depend on how the event is used by
the guest. We can't assume its usage.

>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 63 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
>> index a24270378305..ba2ca65c871b 100644
>> --- a/arch/arm64/kvm/sdei.c
>> +++ b/arch/arm64/kvm/sdei.c
>> @@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
>>   	return 0;
>>   }
>>   
>> +static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
>> +{
>> +	struct kvm *kvm = vcpu->kvm;
>> +	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
>> +	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
>> +	struct kvm_sdei_exposed_event *exposed_event;
>> +	struct kvm_sdei_registered_event *registered_event;
>> +	unsigned long event_num = smccc_get_arg1(vcpu);
>> +	int index;
>> +	unsigned long ret = SDEI_SUCCESS;
>> +
>> +	/* @event_num must be zero */
>> +	if (!kvm_sdei_is_default(event_num)) {
> 
> 0 isn't KVM's default event. I'd argue KVM doesn't have a default event
> to begin with. This has a precise definition coming from the spec. In
> fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any
> missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.
> 
> That goes for any values coming from the specification. KVM's
> implementation details belong in a KVM header :)
> 

Indeed. include/uapi/linux/arm_sdei.h is the right place to define
the software signaled event number. The function to validate/check
on it is still part of kvm_sdei.h, but its name would be something
like kvm_sdei_is_sw_signaled() or kvm_sdei_is_signaled_event().

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall
@ 2022-03-23 12:52       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:52 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/23/22 7:06 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:07:03PM +0800, Gavin Shan wrote:
>> This supports SDEI_EVENT_SIGNAL hypercall. It's used by the guest
>> to inject SDEI event, whose number must be zero to the specified
>> vCPU. As the routing mode and affinity isn't supported yet, the
>> calling vCPU is assumed to be the target.
>>
>> The SDEI event 0x0 is a private one, with normal priority. It's
>> usually used for testing.
> 
> I don't know if that is actually the case. One real use that immediately
> comes to mind is doing an NMI on a wedged CPU. KVM probably shouldn't
> glean at how the guest may use a particular call, so at most we should
> just point at the spec and state that event 0 is for software signaled
> events.
> 

Yes, I agree. I will amend the change log in next respin, to make it
clear. As you said, it really depend on how the event is used by
the guest. We can't assume its usage.

>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   arch/arm64/kvm/sdei.c | 64 ++++++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 63 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
>> index a24270378305..ba2ca65c871b 100644
>> --- a/arch/arm64/kvm/sdei.c
>> +++ b/arch/arm64/kvm/sdei.c
>> @@ -726,6 +726,66 @@ static int do_inject_event(struct kvm_vcpu *vcpu,
>>   	return 0;
>>   }
>>   
>> +static unsigned long hypercall_signal(struct kvm_vcpu *vcpu)
>> +{
>> +	struct kvm *kvm = vcpu->kvm;
>> +	struct kvm_sdei_kvm *ksdei = kvm->arch.sdei;
>> +	struct kvm_sdei_vcpu *vsdei = vcpu->arch.sdei;
>> +	struct kvm_sdei_exposed_event *exposed_event;
>> +	struct kvm_sdei_registered_event *registered_event;
>> +	unsigned long event_num = smccc_get_arg1(vcpu);
>> +	int index;
>> +	unsigned long ret = SDEI_SUCCESS;
>> +
>> +	/* @event_num must be zero */
>> +	if (!kvm_sdei_is_default(event_num)) {
> 
> 0 isn't KVM's default event. I'd argue KVM doesn't have a default event
> to begin with. This has a precise definition coming from the spec. In
> fact, 'KVM_SDEI_DEFAULT_EVENT' should probably be eliminated, and any
> missing SDEI definitions should be added to include/uapi/linux/arm_sdei.h.
> 
> That goes for any values coming from the specification. KVM's
> implementation details belong in a KVM header :)
> 

Indeed. include/uapi/linux/arm_sdei.h is the right place to define
the software signaled event number. The function to validate/check
on it is still part of kvm_sdei.h, but its name would be something
like kvm_sdei_is_sw_signaled() or kvm_sdei_is_signaled_event().

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 00/22] Support SDEI Virtualization
  2022-03-22 18:13   ` Oliver Upton
@ 2022-03-23 12:57     ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:57 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, James Morse, Mark Rutland,
	Shannon Zhao

Hi Oliver,

On 3/23/22 2:13 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:48PM +0800, Gavin Shan wrote:
>> This series intends to virtualize Software Delegated Exception Interface
>> (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to
>> deliver NMI-alike SDEI event to guest and it's needed by Async PF to
>> deliver page-not-present notification from hypervisor to guest. The code
>> and the required qemu changes can be found from:
>>
>>     https://developer.arm.com/documentation/den0054/c
>>     https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>>     https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
>>
>> For the design and migration needs, please refer to the document in
>> PATCH[21/22] in this series. The series is organized as below:
>>
>>    PATCH[01]    Introduces template for smccc_get_argx()
>>    PATCH[02]    Adds SDEI virtualization infrastructure
>>    PATCH[03-17] Supports various SDEI hypercalls and event handling
>>    PATCH[18-20] Adds ioctl commands to support migration and configuration
>>                 and exports SDEI capability
>>    PATCH[21]    Adds SDEI document
>>    PATCH[22]    Adds SDEI selftest case
>>
>> Testing
>> =======
>>
>> [1] The selftest case included in this series works fine. The default SDEI
>>      event, whose number is zero, can be registered, enabled, raised. The
>>      SDEI event handler can be invoked.
>>
>>      [host]# pwd
>>      /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
>>      [root@virtlab-arm01 kvm]# ./aarch64/sdei
>>
>>          NR_VCPUS: 2    SDEI Event: 0x00000000
>>
>>      --- VERSION
>>          Version:              1.1 (vendor: 0x4b564d)
>>      --- FEATURES
>>          Shared event slots:   0
>>          Private event slots:  0
>>          Relative mode:        No
>>      --- PRIVATE_RESET
>>      --- SHARED_RESET
>>      --- PE_UNMASK
>>      --- EVENT_GET_INFO
>>          Type:                 Private
>>          Priority:             Normal
>>          Signaled:             Yes
>>      --- EVENT_REGISTER
>>      --- EVENT_ENABLE
>>      --- EVENT_SIGNAL
>>          Handled:              Yes
>>          IRQ:                  No
>>          Status:               Registered-Enabled-Running
>>          PC/PSTATE:            000000000040232c 00000000600003c5
>>          Regs:                 0000000000000000 0000000000000000
>>                                0000000000000000 0000000000000000
>>      --- PE_MASK
>>      --- EVENT_DISABLE
>>      --- EVENT_UNREGISTER
>>
>>          Result: OK
>>
>> [2] There are additional patches in the following repositories to create
>>      procfs entries, allowing to inject SDEI event from host side. The
>>      SDEI client in the guest side registers the SDEI default event, whose
>>      number is zero. Also, the QEMU exports SDEI ACPI table and supports
>>      migration for SDEI.
>>
>>      https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>>      https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
>>
>>      [2.1] Start the guests and migrate the source VM to the destination
>>            VM.
>>
>>      [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>>              -accel kvm -machine virt,gic-version=host                     \
>>              -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>>              -m 1024M,slots=16,maxmem=64G                                  \
>>                 :                                                          \
>>              -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>>              -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>>              -append earlycon=pl011,mmio,0x9000000                         \
>>                 :
>>
>>      [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>>              -accel kvm -machine virt,gic-version=host                     \
>>              -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>>              -m 1024M,slots=16,maxmem=64G                                  \
>>                 :                                                          \
>>              -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>>              -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>>              -append earlycon=pl011,mmio,0x9000000                         \
>>              -incoming tcp:0:4444                                          \
>>                 :
>>
>>      [2.2] Check kernel log on the source VM. The SDEI service is enabled
>>            and the default SDEI event (0x0) is enabled.
>>
>>       [guest-src]# dmesg | grep -i sdei
>>       ACPI: SDEI 0x000000005BC80000 000024 \
>>                  (v00 BOCHS  BXPC     00000001 BXPC 00000001)
>>       sdei: SDEIv1.1 (0x4b564d) detected in firmware.
>>       SDEI TEST: Version 1.1, Vendor 0x4b564d
>>       sdei_init: SDEI event (0x0) registered
>>       sdei_init: SDEI event (0x0) enabled
>>
>>   
>>       (qemu) migrate -d tcp:localhost:4444
>>
>>      [2.3] Migrate the source VM to the destination VM. Inject SDEI event
>>            to the destination VM. The event is raised and handled.
>>
>>      (qemu) migrate -d tcp:localhost:4444
>>
>>      [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1
>>
>>      [guest-dst]#
>>      =========== SDEI Event (CPU#1) ===========
>>      Event: 0000000000000000  Parameter: 00000000dabfdabf
>>      PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
>>      Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
>>               ffff800016c28000 0000000000000000 0000000000000000 0000000000000000
>>               0000000000000000 0000000000000000 0000000000000000 0000000000000000
>>               0000000000000000 0000000000000000 0000000000000000 0000000000000000
>>               0000000000000000 0000000000000000 0000000000000000 ffff800009399008
>>               ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18
>>               0000000000000000 0000000000000000 ffff000000339d00 0000000000000000
>>               0000000000000000 ffff800009c7bde0 ffff800008cbb5c4
>>      Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
>>               ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190
>>               ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000
>>
>> Changelog
>> =========
>> v5:
> 
> Next time can you include a link to the cover letter of the previous
> patch set? It is extremely helpful for understanding the progress to
> date and allows reviewers to see prior feedback.
> 

Yep, I will provide the link to the cover letter of the previous version.
I'm amending it this time:

     https://lore.kernel.org/all/20210815001352.81927-3-gshan@redhat.com/

Besides, I don't know what happened to my "git send-email". Some of the
recipients are skipped even I have put them into the cc list. Lets amend
it again to avoid resending the series.

Thanks,
Gavin



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 00/22] Support SDEI Virtualization
@ 2022-03-23 12:57     ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-23 12:57 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/23/22 2:13 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:06:48PM +0800, Gavin Shan wrote:
>> This series intends to virtualize Software Delegated Exception Interface
>> (SDEI), which is defined by DEN0054C (v1.1). It allows the hypervisor to
>> deliver NMI-alike SDEI event to guest and it's needed by Async PF to
>> deliver page-not-present notification from hypervisor to guest. The code
>> and the required qemu changes can be found from:
>>
>>     https://developer.arm.com/documentation/den0054/c
>>     https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>>     https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
>>
>> For the design and migration needs, please refer to the document in
>> PATCH[21/22] in this series. The series is organized as below:
>>
>>    PATCH[01]    Introduces template for smccc_get_argx()
>>    PATCH[02]    Adds SDEI virtualization infrastructure
>>    PATCH[03-17] Supports various SDEI hypercalls and event handling
>>    PATCH[18-20] Adds ioctl commands to support migration and configuration
>>                 and exports SDEI capability
>>    PATCH[21]    Adds SDEI document
>>    PATCH[22]    Adds SDEI selftest case
>>
>> Testing
>> =======
>>
>> [1] The selftest case included in this series works fine. The default SDEI
>>      event, whose number is zero, can be registered, enabled, raised. The
>>      SDEI event handler can be invoked.
>>
>>      [host]# pwd
>>      /home/gavin/sandbox/linux.main/tools/testing/selftests/kvm
>>      [root@virtlab-arm01 kvm]# ./aarch64/sdei
>>
>>          NR_VCPUS: 2    SDEI Event: 0x00000000
>>
>>      --- VERSION
>>          Version:              1.1 (vendor: 0x4b564d)
>>      --- FEATURES
>>          Shared event slots:   0
>>          Private event slots:  0
>>          Relative mode:        No
>>      --- PRIVATE_RESET
>>      --- SHARED_RESET
>>      --- PE_UNMASK
>>      --- EVENT_GET_INFO
>>          Type:                 Private
>>          Priority:             Normal
>>          Signaled:             Yes
>>      --- EVENT_REGISTER
>>      --- EVENT_ENABLE
>>      --- EVENT_SIGNAL
>>          Handled:              Yes
>>          IRQ:                  No
>>          Status:               Registered-Enabled-Running
>>          PC/PSTATE:            000000000040232c 00000000600003c5
>>          Regs:                 0000000000000000 0000000000000000
>>                                0000000000000000 0000000000000000
>>      --- PE_MASK
>>      --- EVENT_DISABLE
>>      --- EVENT_UNREGISTER
>>
>>          Result: OK
>>
>> [2] There are additional patches in the following repositories to create
>>      procfs entries, allowing to inject SDEI event from host side. The
>>      SDEI client in the guest side registers the SDEI default event, whose
>>      number is zero. Also, the QEMU exports SDEI ACPI table and supports
>>      migration for SDEI.
>>
>>      https://github.com/gwshan/linux    ("kvm/arm64_sdei")
>>      https://github.com/gwshan/qemu     ("kvm/arm64_sdei")
>>
>>      [2.1] Start the guests and migrate the source VM to the destination
>>            VM.
>>
>>      [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>>              -accel kvm -machine virt,gic-version=host                     \
>>              -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>>              -m 1024M,slots=16,maxmem=64G                                  \
>>                 :                                                          \
>>              -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>>              -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>>              -append earlycon=pl011,mmio,0x9000000                         \
>>                 :
>>
>>      [host]# /home/gavin/sandbox/qemu.main/build/qemu-system-aarch64       \
>>              -accel kvm -machine virt,gic-version=host                     \
>>              -cpu host -smp 6,sockets=2,cores=3,threads=1                  \
>>              -m 1024M,slots=16,maxmem=64G                                  \
>>                 :                                                          \
>>              -kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
>>              -initrd /home/gavin/sandbox/images/rootfs.cpio.xz             \
>>              -append earlycon=pl011,mmio,0x9000000                         \
>>              -incoming tcp:0:4444                                          \
>>                 :
>>
>>      [2.2] Check kernel log on the source VM. The SDEI service is enabled
>>            and the default SDEI event (0x0) is enabled.
>>
>>       [guest-src]# dmesg | grep -i sdei
>>       ACPI: SDEI 0x000000005BC80000 000024 \
>>                  (v00 BOCHS  BXPC     00000001 BXPC 00000001)
>>       sdei: SDEIv1.1 (0x4b564d) detected in firmware.
>>       SDEI TEST: Version 1.1, Vendor 0x4b564d
>>       sdei_init: SDEI event (0x0) registered
>>       sdei_init: SDEI event (0x0) enabled
>>
>>   
>>       (qemu) migrate -d tcp:localhost:4444
>>
>>      [2.3] Migrate the source VM to the destination VM. Inject SDEI event
>>            to the destination VM. The event is raised and handled.
>>
>>      (qemu) migrate -d tcp:localhost:4444
>>
>>      [host]# echo 0 > /proc/kvm/kvm-5360/vcpu-1
>>
>>      [guest-dst]#
>>      =========== SDEI Event (CPU#1) ===========
>>      Event: 0000000000000000  Parameter: 00000000dabfdabf
>>      PC: ffff800008cbb554  PSTATE: 00000000604000c5  SP: ffff800009c7bde0
>>      Regs:    00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
>>               ffff800016c28000 0000000000000000 0000000000000000 0000000000000000
>>               0000000000000000 0000000000000000 0000000000000000 0000000000000000
>>               0000000000000000 0000000000000000 0000000000000000 0000000000000000
>>               0000000000000000 0000000000000000 0000000000000000 ffff800009399008
>>               ffff8000097d9af0 ffff8000097d99f8 ffff8000093a8db8 ffff8000097d9b18
>>               0000000000000000 0000000000000000 ffff000000339d00 0000000000000000
>>               0000000000000000 ffff800009c7bde0 ffff800008cbb5c4
>>      Context: 00000000000016ee ffff00001ffd2e28 00000000000016ed 0000000000000001
>>               ffff800016c28000 03ffffffffffffff 000000024325db59 ffff8000097de190
>>               ffff00000033a790 ffff800008cbb814 0000000000000a30 0000000000000000
>>
>> Changelog
>> =========
>> v5:
> 
> Next time can you include a link to the cover letter of the previous
> patch set? It is extremely helpful for understanding the progress to
> date and allows reviewers to see prior feedback.
> 

Yep, I will provide the link to the cover letter of the previous version.
I'm amending it this time:

     https://lore.kernel.org/all/20210815001352.81927-3-gshan@redhat.com/

Besides, I don't know what happened to my "git send-email". Some of the
recipients are skipped even I have put them into the cc list. Lets amend
it again to avoid resending the series.

Thanks,
Gavin


_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-23 12:46       ` Gavin Shan
@ 2022-03-23 16:31         ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 16:31 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Wed, Mar 23, 2022 at 08:46:40PM +0800, Gavin Shan wrote:
> Hi Oliver,
> 
> On 3/23/22 2:04 AM, Oliver Upton wrote:
> > On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
> > > This supports SDEI_VERSION hypercall by returning v1.1, which is
> > > the specification version we're following. The vendor is set to
> > > 'KVM'.
> > > 
> > > Signed-off-by: Gavin Shan <gshan@redhat.com>
> > > ---
> > >   arch/arm64/kvm/sdei.c | 10 ++++++++++
> > >   1 file changed, 10 insertions(+)
> > > 
> > > diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
> > > index 8a9b477b8977..5a3a64cd6e84 100644
> > > --- a/arch/arm64/kvm/sdei.c
> > > +++ b/arch/arm64/kvm/sdei.c
> > > @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
> > >   	return pending;
> > >   }
> > > +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
> > > +{
> > > +	/* v1.1 and the vendor is KVM */
> > > +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
> > > +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
> > > +	       0x4b564d;
> > 
> > It looks like the SDEI specification states that the vendor-defined
> > version number is 32 bits. Could we just use one of the
> > ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
> > 
> > ASCII 'KVM' is neat, but in reality guest software will just throw it in
> > a macro regardless. Might as well use one of the values we've already
> > trained it to use :-)
> > 
> > Also, it would appear that guest discovery of SDEI relies upon KVM
> > reporting a valid SDEI version. IMO, this patch should come at the very
> > end when KVM actually implements SDEI.
> > 
> 
> Yeah, I was sticky to the pattern of "KVM". However, I think it's good
> to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> if you agree. Its first two characters are "VM" at least.

Sounds fine to me. The only other nit I'd say is we should define a
macro for it too, something like:

  #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-23 16:31         ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 16:31 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Wed, Mar 23, 2022 at 08:46:40PM +0800, Gavin Shan wrote:
> Hi Oliver,
> 
> On 3/23/22 2:04 AM, Oliver Upton wrote:
> > On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
> > > This supports SDEI_VERSION hypercall by returning v1.1, which is
> > > the specification version we're following. The vendor is set to
> > > 'KVM'.
> > > 
> > > Signed-off-by: Gavin Shan <gshan@redhat.com>
> > > ---
> > >   arch/arm64/kvm/sdei.c | 10 ++++++++++
> > >   1 file changed, 10 insertions(+)
> > > 
> > > diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
> > > index 8a9b477b8977..5a3a64cd6e84 100644
> > > --- a/arch/arm64/kvm/sdei.c
> > > +++ b/arch/arm64/kvm/sdei.c
> > > @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
> > >   	return pending;
> > >   }
> > > +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
> > > +{
> > > +	/* v1.1 and the vendor is KVM */
> > > +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
> > > +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
> > > +	       0x4b564d;
> > 
> > It looks like the SDEI specification states that the vendor-defined
> > version number is 32 bits. Could we just use one of the
> > ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
> > 
> > ASCII 'KVM' is neat, but in reality guest software will just throw it in
> > a macro regardless. Might as well use one of the values we've already
> > trained it to use :-)
> > 
> > Also, it would appear that guest discovery of SDEI relies upon KVM
> > reporting a valid SDEI version. IMO, this patch should come at the very
> > end when KVM actually implements SDEI.
> > 
> 
> Yeah, I was sticky to the pattern of "KVM". However, I think it's good
> to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> if you agree. Its first two characters are "VM" at least.

Sounds fine to me. The only other nit I'd say is we should define a
macro for it too, something like:

  #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-22  8:06   ` Gavin Shan
@ 2022-03-23 17:11     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 17:11 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Gavin,

More comments, didn't see exactly how all of these structures are
getting used.

On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:

[...]

> diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
> new file mode 100644
> index 000000000000..b14844230117
> --- /dev/null
> +++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
> @@ -0,0 +1,72 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Definitions of various KVM SDEI event states.
> + *
> + * Copyright (C) 2022 Red Hat, Inc.
> + *
> + * Author(s): Gavin Shan <gshan@redhat.com>
> + */
> +
> +#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
> +#define _UAPI__ASM_KVM_SDEI_STATE_H
> +
> +#ifndef __ASSEMBLY__
> +#include <linux/types.h>
> +
> +/*
> + * The software signaled event is the default one, which is
> + * defined in v1.1 specification.
> + */
> +#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF

Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event
number allocation")

> +#define KVM_SDEI_DEFAULT_EVENT	0
> +
> +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
> +#define KVM_SDEI_MAX_EVENTS	128

I would *strongly* recommend against having these limits. I find the
vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
ABI, which it definitely is not. Anything that deals with a vCPU should
be accessed through a vCPU FD (and thus agnostic to the maximum number
of vCPUs) to avoid such a complication.

> +struct kvm_sdei_exposed_event_state {
> +	__u64	num;
> +
> +	__u8	type;
> +	__u8	signaled;
> +	__u8	priority;
> +	__u8	padding[5];
> +	__u64	notifier;

Wait, isn't this a kernel function pointer!?

> +};
> +
> +struct kvm_sdei_registered_event_state {

You should fold these fields together with kvm_sdei_exposed_event_state
into a single 'kvm_sdei_event' structure:

> +	__u64	num;
> +
> +	__u8	route_mode;
> +	__u8	padding[3];
> +	__u64	route_affinity;

And these shouldn't be UAPI at the VM scope. Each of these properties
could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:

> +	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
> +	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
> +	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
> +	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
> +	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
> +};
> +
> +struct kvm_sdei_vcpu_event_state {
> +	__u64	num;
> +
> +	__u32	event_count;
> +	__u32	padding;
> +};
> +
> +struct kvm_sdei_vcpu_regs_state {
> +	__u64	regs[18];
> +	__u64	pc;
> +	__u64	pstate;
> +};
> +
> +struct kvm_sdei_vcpu_state {

Same goes here, I strongly recommend you try to expose this through the
KVM_{GET,SET}_ONE_REG interface if at all possible since it
significantly reduces the UAPI burden, both on KVM to maintain it and
VMMs to actually use it.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-23 17:11     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 17:11 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Gavin,

More comments, didn't see exactly how all of these structures are
getting used.

On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:

[...]

> diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
> new file mode 100644
> index 000000000000..b14844230117
> --- /dev/null
> +++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
> @@ -0,0 +1,72 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Definitions of various KVM SDEI event states.
> + *
> + * Copyright (C) 2022 Red Hat, Inc.
> + *
> + * Author(s): Gavin Shan <gshan@redhat.com>
> + */
> +
> +#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
> +#define _UAPI__ASM_KVM_SDEI_STATE_H
> +
> +#ifndef __ASSEMBLY__
> +#include <linux/types.h>
> +
> +/*
> + * The software signaled event is the default one, which is
> + * defined in v1.1 specification.
> + */
> +#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF

Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event
number allocation")

> +#define KVM_SDEI_DEFAULT_EVENT	0
> +
> +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
> +#define KVM_SDEI_MAX_EVENTS	128

I would *strongly* recommend against having these limits. I find the
vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
ABI, which it definitely is not. Anything that deals with a vCPU should
be accessed through a vCPU FD (and thus agnostic to the maximum number
of vCPUs) to avoid such a complication.

> +struct kvm_sdei_exposed_event_state {
> +	__u64	num;
> +
> +	__u8	type;
> +	__u8	signaled;
> +	__u8	priority;
> +	__u8	padding[5];
> +	__u64	notifier;

Wait, isn't this a kernel function pointer!?

> +};
> +
> +struct kvm_sdei_registered_event_state {

You should fold these fields together with kvm_sdei_exposed_event_state
into a single 'kvm_sdei_event' structure:

> +	__u64	num;
> +
> +	__u8	route_mode;
> +	__u8	padding[3];
> +	__u64	route_affinity;

And these shouldn't be UAPI at the VM scope. Each of these properties
could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:

> +	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
> +	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
> +	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
> +	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
> +	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
> +};
> +
> +struct kvm_sdei_vcpu_event_state {
> +	__u64	num;
> +
> +	__u32	event_count;
> +	__u32	padding;
> +};
> +
> +struct kvm_sdei_vcpu_regs_state {
> +	__u64	regs[18];
> +	__u64	pc;
> +	__u64	pstate;
> +};
> +
> +struct kvm_sdei_vcpu_state {

Same goes here, I strongly recommend you try to expose this through the
KVM_{GET,SET}_ONE_REG interface if at all possible since it
significantly reduces the UAPI burden, both on KVM to maintain it and
VMMs to actually use it.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
  2022-03-22  8:07   ` Gavin Shan
@ 2022-03-23 17:28     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 17:28 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Tue, Mar 22, 2022 at 04:07:06PM +0800, Gavin Shan wrote:
> This supports ioctl commands on VM to manage the various objects.
> It's primarily used by VMM to accomplish migration. The ioctl
> commands introduced by this are highlighted as below:
> 
>    * KVM_SDEI_CMD_GET_VERSION
>      Retrieve the version of current implementation. It's different
>      from the version of the followed SDEI specification. This version
>      is used to indicates what functionalities documented in the SDEI
>      specification have been supported or not supported.

Don't we need a way to set the version as well? KVM is very much
responsible for upholding ABI of older specs. So, if a VMM and guest
expect SDEI v1.1, we can't just forcibly raise it to something else
during a migration.

The PSCI implementation is a great example of how KVM has grown its
implementation in line with a specification, all the while preserving
backwards compatibility.

>    * KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
>      Return the total count of exposed events.
> 
>    * KVM_SDEI_CMD_GET_EXPOSED_EVENT
>    * KVM_SDEI_CMD_SET_EXPOSED_EVENT
>      Get or set exposed event
> 
>    * KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
>      Return the total count of registered events.
> 
>    * KVM_SDEI_CMD_GET_REGISTERED_EVENT
>    * KVM_SDEI_CMD_SET_REGISTERED_EVENT
>      Get or set registered event.

Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst

Additionally, we desperately need a better, generic way to save/restore
VM scoped state. IMO, we should only be adding ioctls if we are
affording userspace a meaningful interface. Every save/restore pair of
ioctls winds up wasting precious ioctl numbers and requires userspace
take a change to read/write an otherwise opaque value.

Marc had made some suggestions in this area already that Raghavendra
experimented with [1], and I think its time to meaningfully consider
our options. Basically, KVM_GET_REG_LIST needs to convey whether a
particular register is VM or vCPU state. We only need to save/restore a
VM state register once. That way, userspace doesn't have to care about
the underlying data and the next piece of VM state that comes along
doesn't require an ioctl nr nor VMM participation.

[1]: http://lore.kernel.org/r/20220224172559.4170192-3-rananta@google.com

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
@ 2022-03-23 17:28     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 17:28 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Tue, Mar 22, 2022 at 04:07:06PM +0800, Gavin Shan wrote:
> This supports ioctl commands on VM to manage the various objects.
> It's primarily used by VMM to accomplish migration. The ioctl
> commands introduced by this are highlighted as below:
> 
>    * KVM_SDEI_CMD_GET_VERSION
>      Retrieve the version of current implementation. It's different
>      from the version of the followed SDEI specification. This version
>      is used to indicates what functionalities documented in the SDEI
>      specification have been supported or not supported.

Don't we need a way to set the version as well? KVM is very much
responsible for upholding ABI of older specs. So, if a VMM and guest
expect SDEI v1.1, we can't just forcibly raise it to something else
during a migration.

The PSCI implementation is a great example of how KVM has grown its
implementation in line with a specification, all the while preserving
backwards compatibility.

>    * KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
>      Return the total count of exposed events.
> 
>    * KVM_SDEI_CMD_GET_EXPOSED_EVENT
>    * KVM_SDEI_CMD_SET_EXPOSED_EVENT
>      Get or set exposed event
> 
>    * KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
>      Return the total count of registered events.
> 
>    * KVM_SDEI_CMD_GET_REGISTERED_EVENT
>    * KVM_SDEI_CMD_SET_REGISTERED_EVENT
>      Get or set registered event.

Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst

Additionally, we desperately need a better, generic way to save/restore
VM scoped state. IMO, we should only be adding ioctls if we are
affording userspace a meaningful interface. Every save/restore pair of
ioctls winds up wasting precious ioctl numbers and requires userspace
take a change to read/write an otherwise opaque value.

Marc had made some suggestions in this area already that Raghavendra
experimented with [1], and I think its time to meaningfully consider
our options. Basically, KVM_GET_REG_LIST needs to convey whether a
particular register is VM or vCPU state. We only need to save/restore a
VM state register once. That way, userspace doesn't have to care about
the underlying data and the next piece of VM state that comes along
doesn't require an ioctl nr nor VMM participation.

[1]: http://lore.kernel.org/r/20220224172559.4170192-3-rananta@google.com

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
  2022-03-22  8:07   ` Gavin Shan
@ 2022-03-23 17:55     ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 17:55 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
> This supports ioctl commands on vCPU to manage the various object.
> It's primarily used by VMM to accomplish migration. The ioctl
> commands introduced by this are highlighted as below:
> 
>    * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
>      Return the total count of vCPU events, which have been queued
>      on the target vCPU.
> 
>    * KVM_SDEI_CMD_GET_VCPU_EVENT
>    * KVM_SDEI_CMD_SET_VCPU_EVENT
>      Get or set vCPU events.
> 
>    * KVM_SDEI_CMD_GET_VCPU_STATE
>    * KVM_SDEI_CMD_SET_VCPU_STATE
>      Get or set vCPU state.

All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
immediately. Just introduce new registers any time a new event comes
along. The only event we have at the end of this series is the
software-signaled event, with async PF coming later right?

Some special consideration is likely necessary to avoid adding a
register for every u64 chunk of data. I don't think we need to afford
userspace any illusion of granularity with these, and can probably lump
it all under one giant pseudoregister.

>    * KVM_SDEI_CMD_INJECT_EVENT
>      Inject SDEI event.

What events are we going to allow userspace to inject? IIUC, the
software-signaled event is an IPI and really under the control of the
guest. Async PF is entriely under KVM control.

I do agree that having some form of event injection would be great. VM
providers have found it useful to allow users to NMI their VMs when they
get wedged. I just believe that userspace should not be able to trigger
events that have a defined meaning and are under full KVM ownership.

IMO, unless the async PF changes need to go out to userspace, you could
probably skip event injection for now and only worry about SDEI within a
VM.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
@ 2022-03-23 17:55     ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-23 17:55 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
> This supports ioctl commands on vCPU to manage the various object.
> It's primarily used by VMM to accomplish migration. The ioctl
> commands introduced by this are highlighted as below:
> 
>    * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
>      Return the total count of vCPU events, which have been queued
>      on the target vCPU.
> 
>    * KVM_SDEI_CMD_GET_VCPU_EVENT
>    * KVM_SDEI_CMD_SET_VCPU_EVENT
>      Get or set vCPU events.
> 
>    * KVM_SDEI_CMD_GET_VCPU_STATE
>    * KVM_SDEI_CMD_SET_VCPU_STATE
>      Get or set vCPU state.

All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
immediately. Just introduce new registers any time a new event comes
along. The only event we have at the end of this series is the
software-signaled event, with async PF coming later right?

Some special consideration is likely necessary to avoid adding a
register for every u64 chunk of data. I don't think we need to afford
userspace any illusion of granularity with these, and can probably lump
it all under one giant pseudoregister.

>    * KVM_SDEI_CMD_INJECT_EVENT
>      Inject SDEI event.

What events are we going to allow userspace to inject? IIUC, the
software-signaled event is an IPI and really under the control of the
guest. Async PF is entriely under KVM control.

I do agree that having some form of event injection would be great. VM
providers have found it useful to allow users to NMI their VMs when they
get wedged. I just believe that userspace should not be able to trigger
events that have a defined meaning and are under full KVM ownership.

IMO, unless the async PF changes need to go out to userspace, you could
probably skip event injection for now and only worry about SDEI within a
VM.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-23 16:31         ` Oliver Upton
@ 2022-03-24  4:07           ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-24  4:07 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/24/22 12:31 AM, Oliver Upton wrote:
> On Wed, Mar 23, 2022 at 08:46:40PM +0800, Gavin Shan wrote:
>> On 3/23/22 2:04 AM, Oliver Upton wrote:
>>> On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
>>>> This supports SDEI_VERSION hypercall by returning v1.1, which is
>>>> the specification version we're following. The vendor is set to
>>>> 'KVM'.
>>>>
>>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>>> ---
>>>>    arch/arm64/kvm/sdei.c | 10 ++++++++++
>>>>    1 file changed, 10 insertions(+)
>>>>
>>>> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
>>>> index 8a9b477b8977..5a3a64cd6e84 100644
>>>> --- a/arch/arm64/kvm/sdei.c
>>>> +++ b/arch/arm64/kvm/sdei.c
>>>> @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
>>>>    	return pending;
>>>>    }
>>>> +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
>>>> +{
>>>> +	/* v1.1 and the vendor is KVM */
>>>> +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
>>>> +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
>>>> +	       0x4b564d;
>>>
>>> It looks like the SDEI specification states that the vendor-defined
>>> version number is 32 bits. Could we just use one of the
>>> ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
>>>
>>> ASCII 'KVM' is neat, but in reality guest software will just throw it in
>>> a macro regardless. Might as well use one of the values we've already
>>> trained it to use :-)
>>>
>>> Also, it would appear that guest discovery of SDEI relies upon KVM
>>> reporting a valid SDEI version. IMO, this patch should come at the very
>>> end when KVM actually implements SDEI.
>>>
>>
>> Yeah, I was sticky to the pattern of "KVM". However, I think it's good
>> to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
>> if you agree. Its first two characters are "VM" at least.
> 
> Sounds fine to me. The only other nit I'd say is we should define a
> macro for it too, something like:
> 
>    #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> 

Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h.
arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because
the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by
ABI.

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-24  4:07           ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-24  4:07 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/24/22 12:31 AM, Oliver Upton wrote:
> On Wed, Mar 23, 2022 at 08:46:40PM +0800, Gavin Shan wrote:
>> On 3/23/22 2:04 AM, Oliver Upton wrote:
>>> On Tue, Mar 22, 2022 at 04:06:51PM +0800, Gavin Shan wrote:
>>>> This supports SDEI_VERSION hypercall by returning v1.1, which is
>>>> the specification version we're following. The vendor is set to
>>>> 'KVM'.
>>>>
>>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>>> ---
>>>>    arch/arm64/kvm/sdei.c | 10 ++++++++++
>>>>    1 file changed, 10 insertions(+)
>>>>
>>>> diff --git a/arch/arm64/kvm/sdei.c b/arch/arm64/kvm/sdei.c
>>>> index 8a9b477b8977..5a3a64cd6e84 100644
>>>> --- a/arch/arm64/kvm/sdei.c
>>>> +++ b/arch/arm64/kvm/sdei.c
>>>> @@ -118,6 +118,14 @@ static bool remove_all_vcpu_events(struct kvm_vcpu *vcpu,
>>>>    	return pending;
>>>>    }
>>>> +static unsigned long hypercall_version(struct kvm_vcpu *vcpu)
>>>> +{
>>>> +	/* v1.1 and the vendor is KVM */
>>>> +	return (1UL << SDEI_VERSION_MAJOR_SHIFT) |
>>>> +	       (1UL << SDEI_VERSION_MINOR_SHIFT) |
>>>> +	       0x4b564d;
>>>
>>> It looks like the SDEI specification states that the vendor-defined
>>> version number is 32 bits. Could we just use one of the
>>> ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_{0,3} values instead?
>>>
>>> ASCII 'KVM' is neat, but in reality guest software will just throw it in
>>> a macro regardless. Might as well use one of the values we've already
>>> trained it to use :-)
>>>
>>> Also, it would appear that guest discovery of SDEI relies upon KVM
>>> reporting a valid SDEI version. IMO, this patch should come at the very
>>> end when KVM actually implements SDEI.
>>>
>>
>> Yeah, I was sticky to the pattern of "KVM". However, I think it's good
>> to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
>> if you agree. Its first two characters are "VM" at least.
> 
> Sounds fine to me. The only other nit I'd say is we should define a
> macro for it too, something like:
> 
>    #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> 

Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h.
arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because
the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by
ABI.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-23 17:11     ` Oliver Upton
@ 2022-03-24  6:54       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-24  6:54 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/24/22 1:11 AM, Oliver Upton wrote:
> More comments, didn't see exactly how all of these structures are
> getting used.
> 

Ok, thanks for your review and comments.

> On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
> 
> [...]
> 
>> diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
>> new file mode 100644
>> index 000000000000..b14844230117
>> --- /dev/null
>> +++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
>> @@ -0,0 +1,72 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Definitions of various KVM SDEI event states.
>> + *
>> + * Copyright (C) 2022 Red Hat, Inc.
>> + *
>> + * Author(s): Gavin Shan <gshan@redhat.com>
>> + */
>> +
>> +#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
>> +#define _UAPI__ASM_KVM_SDEI_STATE_H
>> +
>> +#ifndef __ASSEMBLY__
>> +#include <linux/types.h>
>> +
>> +/*
>> + * The software signaled event is the default one, which is
>> + * defined in v1.1 specification.
>> + */
>> +#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF
> 
> Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event
> number allocation")
> 

Yes, bit 31 of the event number should be zero. So this is invalid
event number, used by struct kvm_sdei_vcpu_state::critical_num
and normal_num to indicate if there is event being handled on the
corresponding vcpu. When those fields are set to KVM_SDEI_INVALID_EVENT,
no event is being handled on the vcpu.

>> +#define KVM_SDEI_DEFAULT_EVENT	0
>> +
>> +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
>> +#define KVM_SDEI_MAX_EVENTS	128
> 
> I would *strongly* recommend against having these limits. I find the
> vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
> ABI, which it definitely is not. Anything that deals with a vCPU should
> be accessed through a vCPU FD (and thus agnostic to the maximum number
> of vCPUs) to avoid such a complication.
> 

For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled
event. As you suggested on PATCH[15/22], we can't assume its usage.
I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h

For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to
kvm_sdei.h after static arrays to hold the data structures or their
pointers are used, as you suggested early for this patch (PATCH[02/22]).

There are two types of (SDEI) events: shared and private. For the private
event, it can be registered independently from the vcpus. It also means
the address and argument for the entry points, corresponding to @ep_address
and @ep_arg in struct kvm_sdei_registered_event, can be different on
the individual vcpus. However, all the registered/enabled states and
the entry point address and argument are same on all vcpus for the shared
event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to
represent both shared and private event.

If the data belongs to particular vcpu should be accessed through the
vcpu fd, then we need to split or reorganize the data struct as below.

     /*
      * The events are exposed through ioctl interface or similar
      * mechanism (synthetic system registers?) before they can be
      * registered. struct kvm_sdei_exposed_event instance is reserved
      * from the kvm's static array on receiving the ioctl command
      * from VMM.
      */
     struct kvm_sdei_exposed_event {
         __u32   num;

         __u8    type;
         __u8    signaled;
         __u8    priority;
         __u8    padding;
     };

     /*
      * The struct kvm_sdei_registered_event instance is allocated or
      * reserved from the static array. For the shared event, the instance
      * is linked to kvm, but it will be allocated or reserved from vcpu's
      * static array and linked to the vcpu if it's a private event.
      *
      * The instance is only allocated and reserved upon SDEI_EVENT_REGISTER
      * hypercall.
      */
     struct kvm_sdei_registered_event {
         __u32   num

#define KVM_SDEI_EVENT_STATE_REGISTERED         (1 << 0)
#define KVM_SDEI_EVENT_STATE_ENABLE             (1 << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1 << 2)
         __u8    state;
         __u8	route_mode;
         __u8    padding[2];
         __u64   route_affinity;
         __u64	ep_address;
         __u64	ep_arg;
         __u64   notifier;
     }

>> +struct kvm_sdei_exposed_event_state {
>> +	__u64	num;
>> +
>> +	__u8	type;
>> +	__u8	signaled;
>> +	__u8	priority;
>> +	__u8	padding[5];
>> +	__u64	notifier;
> 
> Wait, isn't this a kernel function pointer!?
> 

Yeah, it is a kernel function pointer, used by Async PF to know if
the corresponding event has been handled or not. Async PF can cancel
the previously injected event for performance concerns. Either Async PF
or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF,
SDEI is responsible for its migration.

>> +};
>> +
>> +struct kvm_sdei_registered_event_state {
> 
> You should fold these fields together with kvm_sdei_exposed_event_state
> into a single 'kvm_sdei_event' structure:
> 

@route_mode and @route_affinity can't be configured or modified until
the event is registered. Besides, they're only valid to the shared
events. For private events, they don't have the routing needs. It means
those two fields would be part of struct kvm_sdei_registered_event instead
of kvm_sdei_exposed_event.


>> +	__u64	num;
>> +
>> +	__u8	route_mode;
>> +	__u8	padding[3];
>> +	__u64	route_affinity;
> 
> And these shouldn't be UAPI at the VM scope. Each of these properties
> could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
> 

They're accessed through vcpu or kvm fd depending on what type the event
is. For the VM-owned shared event, they're accessed through KVM fd. For the
vcpu-owned private event, they're accessed through vcpu fd.

I'm not sure if I catch the idea to have a synthetic register and I'm to
confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED"
system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation
defined registers can be adopted, I don't think we need to expose anything
through ABI. All the operations and the needed data can be passed through
the system registers.

     SYS_REG_SDEI_COMMAND
         Receives commands like to expose event, register event and change
         vcpu state etc.
     SYS_REG_SDEI_DATA
         The needed data corresponding to the received command.

However, I'm not positive that synthetic register can be used here. When
Mark Rutland review "PATCH[RFC v1] Async PF support", the implementation
defined registers can't be used in a very limited way. That time, a set
of implementation defined registers are defined to identify the asynchronous
page faults and access to the control data block. However, the idea was
rejected. Later on, Marc recommended SDEI for Async PF.

https://www.spinics.net/lists/kvm-arm/msg40315.html


>> +	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
>> +	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
>> +	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
>> +	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
>> +	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
>> +};
>> +
>> +struct kvm_sdei_vcpu_event_state {
>> +	__u64	num;
>> +
>> +	__u32	event_count;
>> +	__u32	padding;
>> +};
>> +
>> +struct kvm_sdei_vcpu_regs_state {
>> +	__u64	regs[18];
>> +	__u64	pc;
>> +	__u64	pstate;
>> +};
>> +
>> +struct kvm_sdei_vcpu_state {
> 
> Same goes here, I strongly recommend you try to expose this through the
> KVM_{GET,SET}_ONE_REG interface if at all possible since it
> significantly reduces the UAPI burden, both on KVM to maintain it and
> VMMs to actually use it.
> 

Yeah, it's much convenient to use the implementation defined register here.
However, I'm not positive if we can do this. Please see the details I
provided above :)

Thanks,
Gavin



^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-24  6:54       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-24  6:54 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/24/22 1:11 AM, Oliver Upton wrote:
> More comments, didn't see exactly how all of these structures are
> getting used.
> 

Ok, thanks for your review and comments.

> On Tue, Mar 22, 2022 at 04:06:50PM +0800, Gavin Shan wrote:
> 
> [...]
> 
>> diff --git a/arch/arm64/include/uapi/asm/kvm_sdei_state.h b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
>> new file mode 100644
>> index 000000000000..b14844230117
>> --- /dev/null
>> +++ b/arch/arm64/include/uapi/asm/kvm_sdei_state.h
>> @@ -0,0 +1,72 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Definitions of various KVM SDEI event states.
>> + *
>> + * Copyright (C) 2022 Red Hat, Inc.
>> + *
>> + * Author(s): Gavin Shan <gshan@redhat.com>
>> + */
>> +
>> +#ifndef _UAPI__ASM_KVM_SDEI_STATE_H
>> +#define _UAPI__ASM_KVM_SDEI_STATE_H
>> +
>> +#ifndef __ASSEMBLY__
>> +#include <linux/types.h>
>> +
>> +/*
>> + * The software signaled event is the default one, which is
>> + * defined in v1.1 specification.
>> + */
>> +#define KVM_SDEI_INVALID_EVENT	0xFFFFFFFF
> 
> Isn't the constraint that bit 31 must be zero? (DEN 0054C 4.4 "Event
> number allocation")
> 

Yes, bit 31 of the event number should be zero. So this is invalid
event number, used by struct kvm_sdei_vcpu_state::critical_num
and normal_num to indicate if there is event being handled on the
corresponding vcpu. When those fields are set to KVM_SDEI_INVALID_EVENT,
no event is being handled on the vcpu.

>> +#define KVM_SDEI_DEFAULT_EVENT	0
>> +
>> +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
>> +#define KVM_SDEI_MAX_EVENTS	128
> 
> I would *strongly* recommend against having these limits. I find the
> vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
> ABI, which it definitely is not. Anything that deals with a vCPU should
> be accessed through a vCPU FD (and thus agnostic to the maximum number
> of vCPUs) to avoid such a complication.
> 

For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled
event. As you suggested on PATCH[15/22], we can't assume its usage.
I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h

For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to
kvm_sdei.h after static arrays to hold the data structures or their
pointers are used, as you suggested early for this patch (PATCH[02/22]).

There are two types of (SDEI) events: shared and private. For the private
event, it can be registered independently from the vcpus. It also means
the address and argument for the entry points, corresponding to @ep_address
and @ep_arg in struct kvm_sdei_registered_event, can be different on
the individual vcpus. However, all the registered/enabled states and
the entry point address and argument are same on all vcpus for the shared
event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to
represent both shared and private event.

If the data belongs to particular vcpu should be accessed through the
vcpu fd, then we need to split or reorganize the data struct as below.

     /*
      * The events are exposed through ioctl interface or similar
      * mechanism (synthetic system registers?) before they can be
      * registered. struct kvm_sdei_exposed_event instance is reserved
      * from the kvm's static array on receiving the ioctl command
      * from VMM.
      */
     struct kvm_sdei_exposed_event {
         __u32   num;

         __u8    type;
         __u8    signaled;
         __u8    priority;
         __u8    padding;
     };

     /*
      * The struct kvm_sdei_registered_event instance is allocated or
      * reserved from the static array. For the shared event, the instance
      * is linked to kvm, but it will be allocated or reserved from vcpu's
      * static array and linked to the vcpu if it's a private event.
      *
      * The instance is only allocated and reserved upon SDEI_EVENT_REGISTER
      * hypercall.
      */
     struct kvm_sdei_registered_event {
         __u32   num

#define KVM_SDEI_EVENT_STATE_REGISTERED         (1 << 0)
#define KVM_SDEI_EVENT_STATE_ENABLE             (1 << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1 << 2)
         __u8    state;
         __u8	route_mode;
         __u8    padding[2];
         __u64   route_affinity;
         __u64	ep_address;
         __u64	ep_arg;
         __u64   notifier;
     }

>> +struct kvm_sdei_exposed_event_state {
>> +	__u64	num;
>> +
>> +	__u8	type;
>> +	__u8	signaled;
>> +	__u8	priority;
>> +	__u8	padding[5];
>> +	__u64	notifier;
> 
> Wait, isn't this a kernel function pointer!?
> 

Yeah, it is a kernel function pointer, used by Async PF to know if
the corresponding event has been handled or not. Async PF can cancel
the previously injected event for performance concerns. Either Async PF
or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF,
SDEI is responsible for its migration.

>> +};
>> +
>> +struct kvm_sdei_registered_event_state {
> 
> You should fold these fields together with kvm_sdei_exposed_event_state
> into a single 'kvm_sdei_event' structure:
> 

@route_mode and @route_affinity can't be configured or modified until
the event is registered. Besides, they're only valid to the shared
events. For private events, they don't have the routing needs. It means
those two fields would be part of struct kvm_sdei_registered_event instead
of kvm_sdei_exposed_event.


>> +	__u64	num;
>> +
>> +	__u8	route_mode;
>> +	__u8	padding[3];
>> +	__u64	route_affinity;
> 
> And these shouldn't be UAPI at the VM scope. Each of these properties
> could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
> 

They're accessed through vcpu or kvm fd depending on what type the event
is. For the VM-owned shared event, they're accessed through KVM fd. For the
vcpu-owned private event, they're accessed through vcpu fd.

I'm not sure if I catch the idea to have a synthetic register and I'm to
confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED"
system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation
defined registers can be adopted, I don't think we need to expose anything
through ABI. All the operations and the needed data can be passed through
the system registers.

     SYS_REG_SDEI_COMMAND
         Receives commands like to expose event, register event and change
         vcpu state etc.
     SYS_REG_SDEI_DATA
         The needed data corresponding to the received command.

However, I'm not positive that synthetic register can be used here. When
Mark Rutland review "PATCH[RFC v1] Async PF support", the implementation
defined registers can't be used in a very limited way. That time, a set
of implementation defined registers are defined to identify the asynchronous
page faults and access to the control data block. However, the idea was
rejected. Later on, Marc recommended SDEI for Async PF.

https://www.spinics.net/lists/kvm-arm/msg40315.html


>> +	__u64	ep_address[KVM_SDEI_MAX_VCPUS];
>> +	__u64	ep_arg[KVM_SDEI_MAX_VCPUS];
>> +	__u64	registered[KVM_SDEI_MAX_VCPUS/64];
>> +	__u64	enabled[KVM_SDEI_MAX_VCPUS/64];
>> +	__u64	unregister_pending[KVM_SDEI_MAX_VCPUS/64];
>> +};
>> +
>> +struct kvm_sdei_vcpu_event_state {
>> +	__u64	num;
>> +
>> +	__u32	event_count;
>> +	__u32	padding;
>> +};
>> +
>> +struct kvm_sdei_vcpu_regs_state {
>> +	__u64	regs[18];
>> +	__u64	pc;
>> +	__u64	pstate;
>> +};
>> +
>> +struct kvm_sdei_vcpu_state {
> 
> Same goes here, I strongly recommend you try to expose this through the
> KVM_{GET,SET}_ONE_REG interface if at all possible since it
> significantly reduces the UAPI burden, both on KVM to maintain it and
> VMMs to actually use it.
> 

Yeah, it's much convenient to use the implementation defined register here.
However, I'm not positive if we can do this. Please see the details I
provided above :)

Thanks,
Gavin


_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-24  4:07           ` Gavin Shan
@ 2022-03-24  7:48             ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-24  7:48 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:

[...]

> > > Yeah, I was sticky to the pattern of "KVM". However, I think it's good
> > > to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> > > if you agree. Its first two characters are "VM" at least.
> > 
> > Sounds fine to me. The only other nit I'd say is we should define a
> > macro for it too, something like:
> > 
> >    #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> > 
> 
> Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h.
> arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because
> the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by
> ABI.

The argument could definitely be made that our vendor ID should be
promoted to UAPI. Even though linux is the only known user of our
vendor-specific hypercalls, nothing is stopping other software from
using them. Beyond that, these values should really never change anyway.

It isn't a big deal if you add it to internal headers, either, as the
only known consumer will be the kernel.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-24  7:48             ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-24  7:48 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:

[...]

> > > Yeah, I was sticky to the pattern of "KVM". However, I think it's good
> > > to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> > > if you agree. Its first two characters are "VM" at least.
> > 
> > Sounds fine to me. The only other nit I'd say is we should define a
> > macro for it too, something like:
> > 
> >    #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
> > 
> 
> Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h.
> arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because
> the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by
> ABI.

The argument could definitely be made that our vendor ID should be
promoted to UAPI. Even though linux is the only known user of our
vendor-specific hypercalls, nothing is stopping other software from
using them. Beyond that, these values should really never change anyway.

It isn't a big deal if you add it to internal headers, either, as the
only known consumer will be the kernel.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-24  6:54       ` Gavin Shan
@ 2022-03-24  9:04         ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-24  9:04 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Thu, Mar 24, 2022 at 02:54:00PM +0800, Gavin Shan wrote:
> > > +#define KVM_SDEI_DEFAULT_EVENT	0
> > > +
> > > +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
> > > +#define KVM_SDEI_MAX_EVENTS	128
> > 
> > I would *strongly* recommend against having these limits. I find the
> > vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
> > ABI, which it definitely is not. Anything that deals with a vCPU should
> > be accessed through a vCPU FD (and thus agnostic to the maximum number
> > of vCPUs) to avoid such a complication.
> > 
> 
> For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled
> event. As you suggested on PATCH[15/22], we can't assume its usage.
> I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h
> 
> For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to
> kvm_sdei.h after static arrays to hold the data structures or their
> pointers are used, as you suggested early for this patch (PATCH[02/22]).
> 
> There are two types of (SDEI) events: shared and private. For the private
> event, it can be registered independently from the vcpus. It also means
> the address and argument for the entry points, corresponding to @ep_address
> and @ep_arg in struct kvm_sdei_registered_event, can be different on
> the individual vcpus. However, all the registered/enabled states and
> the entry point address and argument are same on all vcpus for the shared
> event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to
> represent both shared and private event.

You're providing a great deal of abstraction around the SDEI
specification, but I really am unconvinced that KVM needs that. This
series needs to add support for a single SDEI event (event 0) and async
PF to follow. Since we are going to support a static set of events under
KVM I believe a lot of the complexity in this design should fade away.

> > > +struct kvm_sdei_exposed_event_state {
> > > +	__u64	num;
> > > +
> > > +	__u8	type;
> > > +	__u8	signaled;
> > > +	__u8	priority;
> > > +	__u8	padding[5];
> > > +	__u64	notifier;
> > 
> > Wait, isn't this a kernel function pointer!?
> > 
> 
> Yeah, it is a kernel function pointer, used by Async PF to know if
> the corresponding event has been handled or not. Async PF can cancel
> the previously injected event for performance concerns. Either Async PF
> or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF,
> SDEI is responsible for its migration.

But this is a UAPI header, why would we even consider giving userspace a
window into the kernel like this? Couldn't userspace crash the kernel by
writing whatever it wants to this field, knowing that it will call it as
a function pointer?

Security aside, there's no guarantee that a function winds up at the
same address between compiler versions or kernel versions.

Overall, I don't think that userspace should have the ability to add
arbitrary SDEI events. KVM takes ownership of its own vendor-specific
ABI in patch 3/22 by declaring its vendor identifier, so any new events
we support must remain within KVM. There is going to be some state that
will need to be migrated for KVM's SDEI events, that ought to be
surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.

Given that there isn't much userspace involvement to make SDEI
work, do you think it would be possible to drop the proposed UAPI from
your series and work on enabling software-signaled SDEI events within
KVM first? By this I mean a VM running under KVM shouldn't require any
ioctls to make it work.

In so doing, we can discover exactly what the mechanics look like in KVM
and only then talk about the necessary UAPI to migrate state. One piece
of the mechanics that is top of mind which I'd like to see addressed is
the use of linked lists and the preallocations that have been made in
structures. KVM will know how many events exist at compile time, so we
can represent these statically.

> > > +};
> > > +
> > > +struct kvm_sdei_registered_event_state {
> > 
> > You should fold these fields together with kvm_sdei_exposed_event_state
> > into a single 'kvm_sdei_event' structure:
> > 
> 
> @route_mode and @route_affinity can't be configured or modified until
> the event is registered. Besides, they're only valid to the shared
> events. For private events, they don't have the routing needs. It means
> those two fields would be part of struct kvm_sdei_registered_event instead
> of kvm_sdei_exposed_event.
> 
> 
> > > +	__u64	num;
> > > +
> > > +	__u8	route_mode;
> > > +	__u8	padding[3];
> > > +	__u64	route_affinity;
> > 
> > And these shouldn't be UAPI at the VM scope. Each of these properties
> > could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
> > 
> 
> They're accessed through vcpu or kvm fd depending on what type the event
> is. For the VM-owned shared event, they're accessed through KVM fd. For the
> vcpu-owned private event, they're accessed through vcpu fd.

Some of the state that you represent in struct kvm_sdei_registered_event_state
is really per-vCPU state. Any time that there's data available at a vCPU
granularity userspace should access it with a vCPU FD.

> I'm not sure if I catch the idea to have a synthetic register and I'm to
> confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED"
> system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation
> defined registers can be adopted, I don't think we need to expose anything
> through ABI. All the operations and the needed data can be passed through
> the system registers.

No, I'm not talking about the guest-facing interface. I'm talking about
what gets exposed to userspace to migrate the VM's state. For parts of
the guest that do not map to an architectural construct, we've defined
the concept of a firmware pseudo-register. What that really means is we
expose a register to userspace that is inaccessible from the guest which
migrates KVM's nonarchitected state. We are abusing the fact that VMMs
will save/restore whatever registers are reported on KVM_GET_REG_LIST to
trick it into migrating KVM state, and it has worked quite nicely to
avoid adding new ioctls for new features. It also means that older VMMs
are capable of utilizing new KVM features, so long as they obey the
prescribed rules.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-24  9:04         ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-24  9:04 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Thu, Mar 24, 2022 at 02:54:00PM +0800, Gavin Shan wrote:
> > > +#define KVM_SDEI_DEFAULT_EVENT	0
> > > +
> > > +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
> > > +#define KVM_SDEI_MAX_EVENTS	128
> > 
> > I would *strongly* recommend against having these limits. I find the
> > vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
> > ABI, which it definitely is not. Anything that deals with a vCPU should
> > be accessed through a vCPU FD (and thus agnostic to the maximum number
> > of vCPUs) to avoid such a complication.
> > 
> 
> For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled
> event. As you suggested on PATCH[15/22], we can't assume its usage.
> I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h
> 
> For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to
> kvm_sdei.h after static arrays to hold the data structures or their
> pointers are used, as you suggested early for this patch (PATCH[02/22]).
> 
> There are two types of (SDEI) events: shared and private. For the private
> event, it can be registered independently from the vcpus. It also means
> the address and argument for the entry points, corresponding to @ep_address
> and @ep_arg in struct kvm_sdei_registered_event, can be different on
> the individual vcpus. However, all the registered/enabled states and
> the entry point address and argument are same on all vcpus for the shared
> event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to
> represent both shared and private event.

You're providing a great deal of abstraction around the SDEI
specification, but I really am unconvinced that KVM needs that. This
series needs to add support for a single SDEI event (event 0) and async
PF to follow. Since we are going to support a static set of events under
KVM I believe a lot of the complexity in this design should fade away.

> > > +struct kvm_sdei_exposed_event_state {
> > > +	__u64	num;
> > > +
> > > +	__u8	type;
> > > +	__u8	signaled;
> > > +	__u8	priority;
> > > +	__u8	padding[5];
> > > +	__u64	notifier;
> > 
> > Wait, isn't this a kernel function pointer!?
> > 
> 
> Yeah, it is a kernel function pointer, used by Async PF to know if
> the corresponding event has been handled or not. Async PF can cancel
> the previously injected event for performance concerns. Either Async PF
> or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF,
> SDEI is responsible for its migration.

But this is a UAPI header, why would we even consider giving userspace a
window into the kernel like this? Couldn't userspace crash the kernel by
writing whatever it wants to this field, knowing that it will call it as
a function pointer?

Security aside, there's no guarantee that a function winds up at the
same address between compiler versions or kernel versions.

Overall, I don't think that userspace should have the ability to add
arbitrary SDEI events. KVM takes ownership of its own vendor-specific
ABI in patch 3/22 by declaring its vendor identifier, so any new events
we support must remain within KVM. There is going to be some state that
will need to be migrated for KVM's SDEI events, that ought to be
surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.

Given that there isn't much userspace involvement to make SDEI
work, do you think it would be possible to drop the proposed UAPI from
your series and work on enabling software-signaled SDEI events within
KVM first? By this I mean a VM running under KVM shouldn't require any
ioctls to make it work.

In so doing, we can discover exactly what the mechanics look like in KVM
and only then talk about the necessary UAPI to migrate state. One piece
of the mechanics that is top of mind which I'd like to see addressed is
the use of linked lists and the preallocations that have been made in
structures. KVM will know how many events exist at compile time, so we
can represent these statically.

> > > +};
> > > +
> > > +struct kvm_sdei_registered_event_state {
> > 
> > You should fold these fields together with kvm_sdei_exposed_event_state
> > into a single 'kvm_sdei_event' structure:
> > 
> 
> @route_mode and @route_affinity can't be configured or modified until
> the event is registered. Besides, they're only valid to the shared
> events. For private events, they don't have the routing needs. It means
> those two fields would be part of struct kvm_sdei_registered_event instead
> of kvm_sdei_exposed_event.
> 
> 
> > > +	__u64	num;
> > > +
> > > +	__u8	route_mode;
> > > +	__u8	padding[3];
> > > +	__u64	route_affinity;
> > 
> > And these shouldn't be UAPI at the VM scope. Each of these properties
> > could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
> > 
> 
> They're accessed through vcpu or kvm fd depending on what type the event
> is. For the VM-owned shared event, they're accessed through KVM fd. For the
> vcpu-owned private event, they're accessed through vcpu fd.

Some of the state that you represent in struct kvm_sdei_registered_event_state
is really per-vCPU state. Any time that there's data available at a vCPU
granularity userspace should access it with a vCPU FD.

> I'm not sure if I catch the idea to have a synthetic register and I'm to
> confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED"
> system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation
> defined registers can be adopted, I don't think we need to expose anything
> through ABI. All the operations and the needed data can be passed through
> the system registers.

No, I'm not talking about the guest-facing interface. I'm talking about
what gets exposed to userspace to migrate the VM's state. For parts of
the guest that do not map to an architectural construct, we've defined
the concept of a firmware pseudo-register. What that really means is we
expose a register to userspace that is inaccessible from the guest which
migrates KVM's nonarchitected state. We are abusing the fact that VMMs
will save/restore whatever registers are reported on KVM_GET_REG_LIST to
trick it into migrating KVM state, and it has worked quite nicely to
avoid adding new ioctls for new features. It also means that older VMMs
are capable of utilizing new KVM features, so long as they obey the
prescribed rules.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
  2022-03-24  9:04         ` Oliver Upton
@ 2022-03-25  6:07           ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  6:07 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/24/22 5:04 PM, Oliver Upton wrote:
> On Thu, Mar 24, 2022 at 02:54:00PM +0800, Gavin Shan wrote:
>>>> +#define KVM_SDEI_DEFAULT_EVENT	0
>>>> +
>>>> +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
>>>> +#define KVM_SDEI_MAX_EVENTS	128
>>>
>>> I would *strongly* recommend against having these limits. I find the
>>> vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
>>> ABI, which it definitely is not. Anything that deals with a vCPU should
>>> be accessed through a vCPU FD (and thus agnostic to the maximum number
>>> of vCPUs) to avoid such a complication.
>>>
>>
>> For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled
>> event. As you suggested on PATCH[15/22], we can't assume its usage.
>> I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h
>>
>> For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to
>> kvm_sdei.h after static arrays to hold the data structures or their
>> pointers are used, as you suggested early for this patch (PATCH[02/22]).
>>
>> There are two types of (SDEI) events: shared and private. For the private
>> event, it can be registered independently from the vcpus. It also means
>> the address and argument for the entry points, corresponding to @ep_address
>> and @ep_arg in struct kvm_sdei_registered_event, can be different on
>> the individual vcpus. However, all the registered/enabled states and
>> the entry point address and argument are same on all vcpus for the shared
>> event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to
>> represent both shared and private event.
> 
> You're providing a great deal of abstraction around the SDEI
> specification, but I really am unconvinced that KVM needs that. This
> series needs to add support for a single SDEI event (event 0) and async
> PF to follow. Since we are going to support a static set of events under
> KVM I believe a lot of the complexity in this design should fade away.
> 

Yeah, I think we can drop the functionality to support the shared
event since both events are exposed by KVM as private events. Please
refer below reply for more details.

>>>> +struct kvm_sdei_exposed_event_state {
>>>> +	__u64	num;
>>>> +
>>>> +	__u8	type;
>>>> +	__u8	signaled;
>>>> +	__u8	priority;
>>>> +	__u8	padding[5];
>>>> +	__u64	notifier;
>>>
>>> Wait, isn't this a kernel function pointer!?
>>>
>>
>> Yeah, it is a kernel function pointer, used by Async PF to know if
>> the corresponding event has been handled or not. Async PF can cancel
>> the previously injected event for performance concerns. Either Async PF
>> or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF,
>> SDEI is responsible for its migration.
> 
> But this is a UAPI header, why would we even consider giving userspace a
> window into the kernel like this? Couldn't userspace crash the kernel by
> writing whatever it wants to this field, knowing that it will call it as
> a function pointer?
> 
> Security aside, there's no guarantee that a function winds up at the
> same address between compiler versions or kernel versions.
> 
> Overall, I don't think that userspace should have the ability to add
> arbitrary SDEI events. KVM takes ownership of its own vendor-specific
> ABI in patch 3/22 by declaring its vendor identifier, so any new events
> we support must remain within KVM. There is going to be some state that
> will need to be migrated for KVM's SDEI events, that ought to be
> surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.
> 
> Given that there isn't much userspace involvement to make SDEI
> work, do you think it would be possible to drop the proposed UAPI from
> your series and work on enabling software-signaled SDEI events within
> KVM first? By this I mean a VM running under KVM shouldn't require any
> ioctls to make it work.
> 
> In so doing, we can discover exactly what the mechanics look like in KVM
> and only then talk about the necessary UAPI to migrate state. One piece
> of the mechanics that is top of mind which I'd like to see addressed is
> the use of linked lists and the preallocations that have been made in
> structures. KVM will know how many events exist at compile time, so we
> can represent these statically.
> 

For @notifier, it is set on the kernel of source VM and migrated to the
kernel of destination VM. It's really concerned with security, I will
drop this in next respin.

I plan to make the following changes for next revision (v6) as below.
Please let me know if there are more things to be covered:

(1) Drop the code to support the shared (SDEI) events since the needed
     events, either software signaled or Async PF event, are all private.
     After that, the private event is the only type to be supported. When
     the shared event becomes unsupported, we can simply return SDEI_NOT_SUPPORTED
     for hypercall SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.

(2) Drop the code to support migration for now. It means the data
     structures defined in arch/arm64/include/uapi/asm/kvm_sdei_state.h
     will be combined to arch/arm64/include/asm/kvm_sdei.h. Besides,
     all the ioctl commands, including KVM_CAP_ARM_SDEI will be dropped
     either.

(3) As we discussed before, the linked lists will be replaced by the
     static arrays or preallocation. The data structures will be amended
     like below.

     (3.1) In arch/arm64/include/asm/kvm_sdei.h
  
     struct kvm_sdei_exposed_event {
         unsigned int                     num;
         unsigned char                    type;           /* reserved, should be 'private' */
         unsigned char                    signaled;
         unsigned char                    priority;
     };

     struct kvm_sdei_registered_event {
         struct kvm_sdei_exposed_event    *exposed_event;
         unsigned long                    ep_address;
         unsigned long                    ep_arg;
#define KVM_SDEI_EVENT_STATE_REGISTERED         (1UL << 0)
#define KVM_SDEI_EVENT_STATE_ENABLED            (1UL << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1UL << 2)
         unsigned long                    state;

         unsigned long                    event_count;  /* number of events pending for hanlding */
     };
        
     struct kvm_sdei_context {
         struct kvm_sdei_registered_event *registered_event;
         unsigned long                    regs[18];
         unsigned long                    pc;
         unsigned long                    pstate;
     };

     /* We even needn't a lock */
     struct kvm_sdei_vcpu {
         unsigned char                    masked;
         struct kvm_sdei_registered_event *registered_event;
         struct kvm_sdei_context          context[SDEI_EVENT_PRIORITY_CRITICAL + 1];
     };

     /* struct kvm_sdei won't be existing any more */

     (3.2) In arch/arm64/kvm/sdei.c

     static struct kvm_sdei_exposed_event exposed_events[] = {
         { .num      = SDEI_EVENT_SW_SIGNALED,    /* defined in include/uapi/linux/arm_sdei.h */
           .type     = SDEI_EVENT_TYPE_PRIVATE,
           .signaled = 1,
           .priority = SDEI_EVENT_PRIORITY_NORMAL,
         },
         {
           .num      = SDEI_EVENT_ASYNC_PF,       /* defined in include/asm/kvm_sdei.h */
           .type     = SDEI_EVENT_TYPE_PRIVATE,
           .signaled = 1,
           .priority = SDEI_EVENT_PRIORITY_CRITICAL,
         },
     };

     In kvm_sdei_create_vcpu(), the events are pre-allocated. They will be destroy
     in kvm_sdei_destroy_vcpu().
  
     struct kvm_vcpu_arch::sdei =
            kzalloc(sizeof(struct kvm_sdei_vcpu), GFP_KERNEL_ACCOUNT);
     struct kvm_sdei_vcpu::registered_event =
            kcalloc(sizeof(struct kvm_sdei_exposed_event), ARRAY_SIZE(exposed_events), GFP_KERNEL_ACCOUNT);

     In kvm_sdei_hypercall(), SDEI_NOT_SUPPORTED will be returned on hypercall
     SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.

(4) Stuff to be considered or discussed in future: migration and mechanism.
     We probably don't want to support the shared event. The function of a
     event is to notify from host to guest kernel. In this regard, the shared
     event can be replaced by the private one. With above changes in (1)/(2)/(3),
     we don't have any VM-scoped properties or states. It means all the properties
     and states are associated with VCPU. It will help to adopt 'firmware-pseudo'
     registers and KVM_{GET,SET}_ONE_REG during the migration in the future.

>>>> +};
>>>> +
>>>> +struct kvm_sdei_registered_event_state {
>>>
>>> You should fold these fields together with kvm_sdei_exposed_event_state
>>> into a single 'kvm_sdei_event' structure:
>>>
>>
>> @route_mode and @route_affinity can't be configured or modified until
>> the event is registered. Besides, they're only valid to the shared
>> events. For private events, they don't have the routing needs. It means
>> those two fields would be part of struct kvm_sdei_registered_event instead
>> of kvm_sdei_exposed_event.
>>
>>
>>>> +	__u64	num;
>>>> +
>>>> +	__u8	route_mode;
>>>> +	__u8	padding[3];
>>>> +	__u64	route_affinity;
>>>
>>> And these shouldn't be UAPI at the VM scope. Each of these properties
>>> could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
>>>
>>
>> They're accessed through vcpu or kvm fd depending on what type the event
>> is. For the VM-owned shared event, they're accessed through KVM fd. For the
>> vcpu-owned private event, they're accessed through vcpu fd.
> 
> Some of the state that you represent in struct kvm_sdei_registered_event_state
> is really per-vCPU state. Any time that there's data available at a vCPU
> granularity userspace should access it with a vCPU FD.
> 

Yeah, I used "struct kvm_sdei_registered_event" to represent the shared and
private event. For the shared event, they are VM scoped. However, they're
VCPU scoped for the private event. As I suggested above, all the states are
VCPU scoped when the private event becomes the only supported one.

>> I'm not sure if I catch the idea to have a synthetic register and I'm to
>> confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED"
>> system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation
>> defined registers can be adopted, I don't think we need to expose anything
>> through ABI. All the operations and the needed data can be passed through
>> the system registers.
> 
> No, I'm not talking about the guest-facing interface. I'm talking about
> what gets exposed to userspace to migrate the VM's state. For parts of
> the guest that do not map to an architectural construct, we've defined
> the concept of a firmware pseudo-register. What that really means is we
> expose a register to userspace that is inaccessible from the guest which
> migrates KVM's nonarchitected state. We are abusing the fact that VMMs
> will save/restore whatever registers are reported on KVM_GET_REG_LIST to
> trick it into migrating KVM state, and it has worked quite nicely to
> avoid adding new ioctls for new features. It also means that older VMMs
> are capable of utilizing new KVM features, so long as they obey the
> prescribed rules.
> 

Thanks for the details about the 'firmware pseudo register'. Oliver, is
there exiting one in current KVM implementation? I would like to see how
it's being used. It's definitely a good idea. Those non-architectural
CPU properties can be mapped and migrated in a natural way. I'm not
sure if we had similar mechanism or 'firmware pseudo registers' for
the VM scoped properties?

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure
@ 2022-03-25  6:07           ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  6:07 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/24/22 5:04 PM, Oliver Upton wrote:
> On Thu, Mar 24, 2022 at 02:54:00PM +0800, Gavin Shan wrote:
>>>> +#define KVM_SDEI_DEFAULT_EVENT	0
>>>> +
>>>> +#define KVM_SDEI_MAX_VCPUS	512	/* Aligned to 64 */
>>>> +#define KVM_SDEI_MAX_EVENTS	128
>>>
>>> I would *strongly* recommend against having these limits. I find the
>>> vCPU limit especially concerning, because we're making KVM_MAX_VCPUS
>>> ABI, which it definitely is not. Anything that deals with a vCPU should
>>> be accessed through a vCPU FD (and thus agnostic to the maximum number
>>> of vCPUs) to avoid such a complication.
>>>
>>
>> For KVM_SDEI_DEFAULT_EVENT, which corresponds to the software signaled
>> event. As you suggested on PATCH[15/22], we can't assume its usage.
>> I will define it with SDEI_SW_SIGNALED_EVENT in uapi/linux/arm_sdei.h
>>
>> For KVM_SDEI_MAX_EVENTS, it will be moved from this header file to
>> kvm_sdei.h after static arrays to hold the data structures or their
>> pointers are used, as you suggested early for this patch (PATCH[02/22]).
>>
>> There are two types of (SDEI) events: shared and private. For the private
>> event, it can be registered independently from the vcpus. It also means
>> the address and argument for the entry points, corresponding to @ep_address
>> and @ep_arg in struct kvm_sdei_registered_event, can be different on
>> the individual vcpus. However, all the registered/enabled states and
>> the entry point address and argument are same on all vcpus for the shared
>> event. KVM_SDEI_MAX_VCPUS was introduced to use same data structure to
>> represent both shared and private event.
> 
> You're providing a great deal of abstraction around the SDEI
> specification, but I really am unconvinced that KVM needs that. This
> series needs to add support for a single SDEI event (event 0) and async
> PF to follow. Since we are going to support a static set of events under
> KVM I believe a lot of the complexity in this design should fade away.
> 

Yeah, I think we can drop the functionality to support the shared
event since both events are exposed by KVM as private events. Please
refer below reply for more details.

>>>> +struct kvm_sdei_exposed_event_state {
>>>> +	__u64	num;
>>>> +
>>>> +	__u8	type;
>>>> +	__u8	signaled;
>>>> +	__u8	priority;
>>>> +	__u8	padding[5];
>>>> +	__u64	notifier;
>>>
>>> Wait, isn't this a kernel function pointer!?
>>>
>>
>> Yeah, it is a kernel function pointer, used by Async PF to know if
>> the corresponding event has been handled or not. Async PF can cancel
>> the previously injected event for performance concerns. Either Async PF
>> or SDEI needs to migrate it. To keep SDEI transparent enough to Async PF,
>> SDEI is responsible for its migration.
> 
> But this is a UAPI header, why would we even consider giving userspace a
> window into the kernel like this? Couldn't userspace crash the kernel by
> writing whatever it wants to this field, knowing that it will call it as
> a function pointer?
> 
> Security aside, there's no guarantee that a function winds up at the
> same address between compiler versions or kernel versions.
> 
> Overall, I don't think that userspace should have the ability to add
> arbitrary SDEI events. KVM takes ownership of its own vendor-specific
> ABI in patch 3/22 by declaring its vendor identifier, so any new events
> we support must remain within KVM. There is going to be some state that
> will need to be migrated for KVM's SDEI events, that ought to be
> surfaced to userspace through the KVM_{GET,SET}_ONE_REG ioctls.
> 
> Given that there isn't much userspace involvement to make SDEI
> work, do you think it would be possible to drop the proposed UAPI from
> your series and work on enabling software-signaled SDEI events within
> KVM first? By this I mean a VM running under KVM shouldn't require any
> ioctls to make it work.
> 
> In so doing, we can discover exactly what the mechanics look like in KVM
> and only then talk about the necessary UAPI to migrate state. One piece
> of the mechanics that is top of mind which I'd like to see addressed is
> the use of linked lists and the preallocations that have been made in
> structures. KVM will know how many events exist at compile time, so we
> can represent these statically.
> 

For @notifier, it is set on the kernel of source VM and migrated to the
kernel of destination VM. It's really concerned with security, I will
drop this in next respin.

I plan to make the following changes for next revision (v6) as below.
Please let me know if there are more things to be covered:

(1) Drop the code to support the shared (SDEI) events since the needed
     events, either software signaled or Async PF event, are all private.
     After that, the private event is the only type to be supported. When
     the shared event becomes unsupported, we can simply return SDEI_NOT_SUPPORTED
     for hypercall SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.

(2) Drop the code to support migration for now. It means the data
     structures defined in arch/arm64/include/uapi/asm/kvm_sdei_state.h
     will be combined to arch/arm64/include/asm/kvm_sdei.h. Besides,
     all the ioctl commands, including KVM_CAP_ARM_SDEI will be dropped
     either.

(3) As we discussed before, the linked lists will be replaced by the
     static arrays or preallocation. The data structures will be amended
     like below.

     (3.1) In arch/arm64/include/asm/kvm_sdei.h
  
     struct kvm_sdei_exposed_event {
         unsigned int                     num;
         unsigned char                    type;           /* reserved, should be 'private' */
         unsigned char                    signaled;
         unsigned char                    priority;
     };

     struct kvm_sdei_registered_event {
         struct kvm_sdei_exposed_event    *exposed_event;
         unsigned long                    ep_address;
         unsigned long                    ep_arg;
#define KVM_SDEI_EVENT_STATE_REGISTERED         (1UL << 0)
#define KVM_SDEI_EVENT_STATE_ENABLED            (1UL << 1)
#define KVM_SDEI_EVENT_STATE_UNREGISTER_PENDING (1UL << 2)
         unsigned long                    state;

         unsigned long                    event_count;  /* number of events pending for hanlding */
     };
        
     struct kvm_sdei_context {
         struct kvm_sdei_registered_event *registered_event;
         unsigned long                    regs[18];
         unsigned long                    pc;
         unsigned long                    pstate;
     };

     /* We even needn't a lock */
     struct kvm_sdei_vcpu {
         unsigned char                    masked;
         struct kvm_sdei_registered_event *registered_event;
         struct kvm_sdei_context          context[SDEI_EVENT_PRIORITY_CRITICAL + 1];
     };

     /* struct kvm_sdei won't be existing any more */

     (3.2) In arch/arm64/kvm/sdei.c

     static struct kvm_sdei_exposed_event exposed_events[] = {
         { .num      = SDEI_EVENT_SW_SIGNALED,    /* defined in include/uapi/linux/arm_sdei.h */
           .type     = SDEI_EVENT_TYPE_PRIVATE,
           .signaled = 1,
           .priority = SDEI_EVENT_PRIORITY_NORMAL,
         },
         {
           .num      = SDEI_EVENT_ASYNC_PF,       /* defined in include/asm/kvm_sdei.h */
           .type     = SDEI_EVENT_TYPE_PRIVATE,
           .signaled = 1,
           .priority = SDEI_EVENT_PRIORITY_CRITICAL,
         },
     };

     In kvm_sdei_create_vcpu(), the events are pre-allocated. They will be destroy
     in kvm_sdei_destroy_vcpu().
  
     struct kvm_vcpu_arch::sdei =
            kzalloc(sizeof(struct kvm_sdei_vcpu), GFP_KERNEL_ACCOUNT);
     struct kvm_sdei_vcpu::registered_event =
            kcalloc(sizeof(struct kvm_sdei_exposed_event), ARRAY_SIZE(exposed_events), GFP_KERNEL_ACCOUNT);

     In kvm_sdei_hypercall(), SDEI_NOT_SUPPORTED will be returned on hypercall
     SDEI_EVENT_ROUTING_SET and SDEI_SHARED_RESET.

(4) Stuff to be considered or discussed in future: migration and mechanism.
     We probably don't want to support the shared event. The function of a
     event is to notify from host to guest kernel. In this regard, the shared
     event can be replaced by the private one. With above changes in (1)/(2)/(3),
     we don't have any VM-scoped properties or states. It means all the properties
     and states are associated with VCPU. It will help to adopt 'firmware-pseudo'
     registers and KVM_{GET,SET}_ONE_REG during the migration in the future.

>>>> +};
>>>> +
>>>> +struct kvm_sdei_registered_event_state {
>>>
>>> You should fold these fields together with kvm_sdei_exposed_event_state
>>> into a single 'kvm_sdei_event' structure:
>>>
>>
>> @route_mode and @route_affinity can't be configured or modified until
>> the event is registered. Besides, they're only valid to the shared
>> events. For private events, they don't have the routing needs. It means
>> those two fields would be part of struct kvm_sdei_registered_event instead
>> of kvm_sdei_exposed_event.
>>
>>
>>>> +	__u64	num;
>>>> +
>>>> +	__u8	route_mode;
>>>> +	__u8	padding[3];
>>>> +	__u64	route_affinity;
>>>
>>> And these shouldn't be UAPI at the VM scope. Each of these properties
>>> could be accessed via a synthetic/'pseudo-firmware' register on a vCPU FD:
>>>
>>
>> They're accessed through vcpu or kvm fd depending on what type the event
>> is. For the VM-owned shared event, they're accessed through KVM fd. For the
>> vcpu-owned private event, they're accessed through vcpu fd.
> 
> Some of the state that you represent in struct kvm_sdei_registered_event_state
> is really per-vCPU state. Any time that there's data available at a vCPU
> granularity userspace should access it with a vCPU FD.
> 

Yeah, I used "struct kvm_sdei_registered_event" to represent the shared and
private event. For the shared event, they are VM scoped. However, they're
VCPU scoped for the private event. As I suggested above, all the states are
VCPU scoped when the private event becomes the only supported one.

>> I'm not sure if I catch the idea to have a synthetic register and I'm to
>> confirm. If I'm correct, you're talking about the "IMPLEMENTATION DEFINED"
>> system register, whose OP0 and CRn are 0B11 and 0B1x11. If two implementation
>> defined registers can be adopted, I don't think we need to expose anything
>> through ABI. All the operations and the needed data can be passed through
>> the system registers.
> 
> No, I'm not talking about the guest-facing interface. I'm talking about
> what gets exposed to userspace to migrate the VM's state. For parts of
> the guest that do not map to an architectural construct, we've defined
> the concept of a firmware pseudo-register. What that really means is we
> expose a register to userspace that is inaccessible from the guest which
> migrates KVM's nonarchitected state. We are abusing the fact that VMMs
> will save/restore whatever registers are reported on KVM_GET_REG_LIST to
> trick it into migrating KVM state, and it has worked quite nicely to
> avoid adding new ioctls for new features. It also means that older VMMs
> are capable of utilizing new KVM features, so long as they obey the
> prescribed rules.
> 

Thanks for the details about the 'firmware pseudo register'. Oliver, is
there exiting one in current KVM implementation? I would like to see how
it's being used. It's definitely a good idea. Those non-architectural
CPU properties can be mapped and migrated in a natural way. I'm not
sure if we had similar mechanism or 'firmware pseudo registers' for
the VM scoped properties?

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
  2022-03-24  7:48             ` Oliver Upton
@ 2022-03-25  6:11               ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  6:11 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/24/22 3:48 PM, Oliver Upton wrote:
> On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:
> 
> [...]
> 
>>>> Yeah, I was sticky to the pattern of "KVM". However, I think it's good
>>>> to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
>>>> if you agree. Its first two characters are "VM" at least.
>>>
>>> Sounds fine to me. The only other nit I'd say is we should define a
>>> macro for it too, something like:
>>>
>>>     #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
>>>
>>
>> Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h.
>> arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because
>> the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by
>> ABI.
> 
> The argument could definitely be made that our vendor ID should be
> promoted to UAPI. Even though linux is the only known user of our
> vendor-specific hypercalls, nothing is stopping other software from
> using them. Beyond that, these values should really never change anyway.
> 
> It isn't a big deal if you add it to internal headers, either, as the
> only known consumer will be the kernel.
> 

Sure. Lets add it into include/asm/kvm_sdei.h in next respin. We can
expose it when it is needed. For now, I do think Linux is the only
user.

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall
@ 2022-03-25  6:11               ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  6:11 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/24/22 3:48 PM, Oliver Upton wrote:
> On Thu, Mar 24, 2022 at 12:07:34PM +0800, Gavin Shan wrote:
> 
> [...]
> 
>>>> Yeah, I was sticky to the pattern of "KVM". However, I think it's good
>>>> to reuse the existing one. Lets use ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
>>>> if you agree. Its first two characters are "VM" at least.
>>>
>>> Sounds fine to me. The only other nit I'd say is we should define a
>>> macro for it too, something like:
>>>
>>>     #define KVM_SDEI_VENDOR	ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2
>>>
>>
>> Agreed, and the macro will be put into arch/arm64/include/asm/kvm_sdei.h.
>> arch/arm64/include/uapi/asm/kvm_sdei_state.h isn't the right place because
>> the dependent macro ARM_SMCCC_VENDOR_HYP_UID_KVM_REG_2 isn't exposed by
>> ABI.
> 
> The argument could definitely be made that our vendor ID should be
> promoted to UAPI. Even though linux is the only known user of our
> vendor-specific hypercalls, nothing is stopping other software from
> using them. Beyond that, these values should really never change anyway.
> 
> It isn't a big deal if you add it to internal headers, either, as the
> only known consumer will be the kernel.
> 

Sure. Lets add it into include/asm/kvm_sdei.h in next respin. We can
expose it when it is needed. For now, I do think Linux is the only
user.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
  2022-03-23 17:28     ` Oliver Upton
@ 2022-03-25  6:59       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  6:59 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/24/22 1:28 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:07:06PM +0800, Gavin Shan wrote:
>> This supports ioctl commands on VM to manage the various objects.
>> It's primarily used by VMM to accomplish migration. The ioctl
>> commands introduced by this are highlighted as below:
>>
>>     * KVM_SDEI_CMD_GET_VERSION
>>       Retrieve the version of current implementation. It's different
>>       from the version of the followed SDEI specification. This version
>>       is used to indicates what functionalities documented in the SDEI
>>       specification have been supported or not supported.
> 
> Don't we need a way to set the version as well? KVM is very much
> responsible for upholding ABI of older specs. So, if a VMM and guest
> expect SDEI v1.1, we can't just forcibly raise it to something else
> during a migration.
> 
> The PSCI implementation is a great example of how KVM has grown its
> implementation in line with a specification, all the while preserving
> backwards compatibility.
> 

The only information feed by VMM is the exposed events. The events
can't be registered from guest kernel, and raised from host to guest
kernel until it's exposed by VMM. Besides, the exposed events will
be defined staticly in host/KVM as we discussed on PATCH[02/22]. We
also discussed to eliminate those ioctl commands. So I think we needn't
to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a
concern to host itself if the migration can be done through the
firmware pseudo system registers since the migration compatibility
is the only concern to VMM (QEMU).

Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is
picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems
QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable
it.

>>     * KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
>>       Return the total count of exposed events.
>>
>>     * KVM_SDEI_CMD_GET_EXPOSED_EVENT
>>     * KVM_SDEI_CMD_SET_EXPOSED_EVENT
>>       Get or set exposed event
>>
>>     * KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
>>       Return the total count of registered events.
>>
>>     * KVM_SDEI_CMD_GET_REGISTERED_EVENT
>>     * KVM_SDEI_CMD_SET_REGISTERED_EVENT
>>       Get or set registered event.
> 
> Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst
> 
> Additionally, we desperately need a better, generic way to save/restore
> VM scoped state. IMO, we should only be adding ioctls if we are
> affording userspace a meaningful interface. Every save/restore pair of
> ioctls winds up wasting precious ioctl numbers and requires userspace
> take a change to read/write an otherwise opaque value.
> 
> Marc had made some suggestions in this area already that Raghavendra
> experimented with [1], and I think its time to meaningfully consider
> our options. Basically, KVM_GET_REG_LIST needs to convey whether a
> particular register is VM or vCPU state. We only need to save/restore a
> VM state register once. That way, userspace doesn't have to care about
> the underlying data and the next piece of VM state that comes along
> doesn't require an ioctl nr nor VMM participation.
> 
> [1]: http://lore.kernel.org/r/20220224172559.4170192-3-rananta@google.com
> 

Thanks for the pointer to Raghavendra's series. The firmware pseudo
system registers have been classified into VM and VCPU scoped in the
series. I think it fits the SDEI migration requirements very well.
The shared events can even be migrated through the VM scoped firmware
pseudo system registers. However, I don't plan to support it in next
revision (v6) as currently needed events are all private. I may
spend more time to go through Raghavendra's series later.

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
@ 2022-03-25  6:59       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  6:59 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/24/22 1:28 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:07:06PM +0800, Gavin Shan wrote:
>> This supports ioctl commands on VM to manage the various objects.
>> It's primarily used by VMM to accomplish migration. The ioctl
>> commands introduced by this are highlighted as below:
>>
>>     * KVM_SDEI_CMD_GET_VERSION
>>       Retrieve the version of current implementation. It's different
>>       from the version of the followed SDEI specification. This version
>>       is used to indicates what functionalities documented in the SDEI
>>       specification have been supported or not supported.
> 
> Don't we need a way to set the version as well? KVM is very much
> responsible for upholding ABI of older specs. So, if a VMM and guest
> expect SDEI v1.1, we can't just forcibly raise it to something else
> during a migration.
> 
> The PSCI implementation is a great example of how KVM has grown its
> implementation in line with a specification, all the while preserving
> backwards compatibility.
> 

The only information feed by VMM is the exposed events. The events
can't be registered from guest kernel, and raised from host to guest
kernel until it's exposed by VMM. Besides, the exposed events will
be defined staticly in host/KVM as we discussed on PATCH[02/22]. We
also discussed to eliminate those ioctl commands. So I think we needn't
to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a
concern to host itself if the migration can be done through the
firmware pseudo system registers since the migration compatibility
is the only concern to VMM (QEMU).

Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is
picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems
QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable
it.

>>     * KVM_SDEI_CMD_GET_EXPOSED_EVENT_COUNT
>>       Return the total count of exposed events.
>>
>>     * KVM_SDEI_CMD_GET_EXPOSED_EVENT
>>     * KVM_SDEI_CMD_SET_EXPOSED_EVENT
>>       Get or set exposed event
>>
>>     * KVM_SDEI_CMD_GET_REGISTERED_EVENT_COUNT
>>       Return the total count of registered events.
>>
>>     * KVM_SDEI_CMD_GET_REGISTERED_EVENT
>>     * KVM_SDEI_CMD_SET_REGISTERED_EVENT
>>       Get or set registered event.
> 
> Any new UAPI needs to be documented in Documentation/virt/kvm/api.rst
> 
> Additionally, we desperately need a better, generic way to save/restore
> VM scoped state. IMO, we should only be adding ioctls if we are
> affording userspace a meaningful interface. Every save/restore pair of
> ioctls winds up wasting precious ioctl numbers and requires userspace
> take a change to read/write an otherwise opaque value.
> 
> Marc had made some suggestions in this area already that Raghavendra
> experimented with [1], and I think its time to meaningfully consider
> our options. Basically, KVM_GET_REG_LIST needs to convey whether a
> particular register is VM or vCPU state. We only need to save/restore a
> VM state register once. That way, userspace doesn't have to care about
> the underlying data and the next piece of VM state that comes along
> doesn't require an ioctl nr nor VMM participation.
> 
> [1]: http://lore.kernel.org/r/20220224172559.4170192-3-rananta@google.com
> 

Thanks for the pointer to Raghavendra's series. The firmware pseudo
system registers have been classified into VM and VCPU scoped in the
series. I think it fits the SDEI migration requirements very well.
The shared events can even be migrated through the VM scoped firmware
pseudo system registers. However, I don't plan to support it in next
revision (v6) as currently needed events are all private. I may
spend more time to go through Raghavendra's series later.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
  2022-03-25  6:59       ` Gavin Shan
@ 2022-03-25  7:35         ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-25  7:35 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Fri, Mar 25, 2022 at 02:59:52PM +0800, Gavin Shan wrote:
> > The PSCI implementation is a great example of how KVM has grown its
> > implementation in line with a specification, all the while preserving
> > backwards compatibility.
> > 
> 
> The only information feed by VMM is the exposed events. The events
> can't be registered from guest kernel, and raised from host to guest
> kernel until it's exposed by VMM.

I would suggest assuming that all SDEI events are exposed by default in
KVM. We will not require a VMM change to enable events individually.

> Besides, the exposed events will
> be defined staticly in host/KVM as we discussed on PATCH[02/22]. We
> also discussed to eliminate those ioctl commands. So I think we needn't
> to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a
> concern to host itself if the migration can be done through the
> firmware pseudo system registers since the migration compatibility
> is the only concern to VMM (QEMU).

This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version,
I'd recommend taking a look at how we handle that register in KVM.

> Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is
> picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems
> QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable
> it.

As far as how it interacts with KVM, QEMU looks fine. The name of the
KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that
KVM will enable it highest supported PSCI version. If the feature bit is
cleared then you only get PSCIv0.1

However, the DT node that QEMU sets up looks a bit crusty. The
properties for conveying PSCI function IDs were only ever necessary for
PSCIv0.1. The only property of interest any more is 'method', to convey
the SMCCC conduit instruction.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
@ 2022-03-25  7:35         ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-25  7:35 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Fri, Mar 25, 2022 at 02:59:52PM +0800, Gavin Shan wrote:
> > The PSCI implementation is a great example of how KVM has grown its
> > implementation in line with a specification, all the while preserving
> > backwards compatibility.
> > 
> 
> The only information feed by VMM is the exposed events. The events
> can't be registered from guest kernel, and raised from host to guest
> kernel until it's exposed by VMM.

I would suggest assuming that all SDEI events are exposed by default in
KVM. We will not require a VMM change to enable events individually.

> Besides, the exposed events will
> be defined staticly in host/KVM as we discussed on PATCH[02/22]. We
> also discussed to eliminate those ioctl commands. So I think we needn't
> to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a
> concern to host itself if the migration can be done through the
> firmware pseudo system registers since the migration compatibility
> is the only concern to VMM (QEMU).

This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version,
I'd recommend taking a look at how we handle that register in KVM.

> Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is
> picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems
> QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable
> it.

As far as how it interacts with KVM, QEMU looks fine. The name of the
KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that
KVM will enable it highest supported PSCI version. If the feature bit is
cleared then you only get PSCIv0.1

However, the DT node that QEMU sets up looks a bit crusty. The
properties for conveying PSCI function IDs were only ever necessary for
PSCIv0.1. The only property of interest any more is 'method', to convey
the SMCCC conduit instruction.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
  2022-03-23 17:55     ` Oliver Upton
@ 2022-03-25  7:59       ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  7:59 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/24/22 1:55 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
>> This supports ioctl commands on vCPU to manage the various object.
>> It's primarily used by VMM to accomplish migration. The ioctl
>> commands introduced by this are highlighted as below:
>>
>>     * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
>>       Return the total count of vCPU events, which have been queued
>>       on the target vCPU.
>>
>>     * KVM_SDEI_CMD_GET_VCPU_EVENT
>>     * KVM_SDEI_CMD_SET_VCPU_EVENT
>>       Get or set vCPU events.
>>
>>     * KVM_SDEI_CMD_GET_VCPU_STATE
>>     * KVM_SDEI_CMD_SET_VCPU_STATE
>>       Get or set vCPU state.
> 
> All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
> immediately. Just introduce new registers any time a new event comes
> along. The only event we have at the end of this series is the
> software-signaled event, with async PF coming later right?
> 
> Some special consideration is likely necessary to avoid adding a
> register for every u64 chunk of data. I don't think we need to afford
> userspace any illusion of granularity with these, and can probably lump
> it all under one giant pseudoregister.
> 

Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're
correct we're only concerned by software signaled event and the one for
Async PF.

I didn't look into Raghavendra's series deeply. Actually, a lump of
registers can be avoid after 2048 byte size is specified in its
encoding. I think 2048 bytes are enough for now since there are
only two supported events.

In the future, we probably have varied number of SDEI events to
be migrated. In that case, we need to add a new bit to the encoding
of the pseudo system register, so that VMM (QEMU) can support
variable sized system register and keep reading and writing on
these registers on migration:

     PSEUDO_SDEI_ADDR: 64-bits in width
     PSEUDO_SDEI_DATA: has varied size

PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private)
registered events on VM and vCPU, VCPU state.

PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be
written. (2) Flags to indicate current block of information is the
last one or not.

>>     * KVM_SDEI_CMD_INJECT_EVENT
>>       Inject SDEI event.
> 
> What events are we going to allow userspace to inject? IIUC, the
> software-signaled event is an IPI and really under the control of the
> guest. Async PF is entriely under KVM control.
> 
> I do agree that having some form of event injection would be great. VM
> providers have found it useful to allow users to NMI their VMs when they
> get wedged. I just believe that userspace should not be able to trigger
> events that have a defined meaning and are under full KVM ownership.
> 
> IMO, unless the async PF changes need to go out to userspace, you could
> probably skip event injection for now and only worry about SDEI within a
> VM.
> 

I was overthinking on the usage of SDEI. I had the assumption that SDEI
may be used by emulated devices to inject SDEI events in VMM. It can
even be done through PSEUDO_SDEI_{ADDR, DATA} in future. For now, the
software signaled and Async PF events are concerned, Async PF event
is always raised by host/KVM. So we needn't it for now and I will drop
this functionality, actually the whole ioctl commands and migration
support in next respin, as you suggested :)

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
@ 2022-03-25  7:59       ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25  7:59 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/24/22 1:55 AM, Oliver Upton wrote:
> On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
>> This supports ioctl commands on vCPU to manage the various object.
>> It's primarily used by VMM to accomplish migration. The ioctl
>> commands introduced by this are highlighted as below:
>>
>>     * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
>>       Return the total count of vCPU events, which have been queued
>>       on the target vCPU.
>>
>>     * KVM_SDEI_CMD_GET_VCPU_EVENT
>>     * KVM_SDEI_CMD_SET_VCPU_EVENT
>>       Get or set vCPU events.
>>
>>     * KVM_SDEI_CMD_GET_VCPU_STATE
>>     * KVM_SDEI_CMD_SET_VCPU_STATE
>>       Get or set vCPU state.
> 
> All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
> immediately. Just introduce new registers any time a new event comes
> along. The only event we have at the end of this series is the
> software-signaled event, with async PF coming later right?
> 
> Some special consideration is likely necessary to avoid adding a
> register for every u64 chunk of data. I don't think we need to afford
> userspace any illusion of granularity with these, and can probably lump
> it all under one giant pseudoregister.
> 

Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're
correct we're only concerned by software signaled event and the one for
Async PF.

I didn't look into Raghavendra's series deeply. Actually, a lump of
registers can be avoid after 2048 byte size is specified in its
encoding. I think 2048 bytes are enough for now since there are
only two supported events.

In the future, we probably have varied number of SDEI events to
be migrated. In that case, we need to add a new bit to the encoding
of the pseudo system register, so that VMM (QEMU) can support
variable sized system register and keep reading and writing on
these registers on migration:

     PSEUDO_SDEI_ADDR: 64-bits in width
     PSEUDO_SDEI_DATA: has varied size

PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
(2) The information to be read/written, for example the (shared/private)
registered events on VM and vCPU, VCPU state.

PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be
written. (2) Flags to indicate current block of information is the
last one or not.

>>     * KVM_SDEI_CMD_INJECT_EVENT
>>       Inject SDEI event.
> 
> What events are we going to allow userspace to inject? IIUC, the
> software-signaled event is an IPI and really under the control of the
> guest. Async PF is entriely under KVM control.
> 
> I do agree that having some form of event injection would be great. VM
> providers have found it useful to allow users to NMI their VMs when they
> get wedged. I just believe that userspace should not be able to trigger
> events that have a defined meaning and are under full KVM ownership.
> 
> IMO, unless the async PF changes need to go out to userspace, you could
> probably skip event injection for now and only worry about SDEI within a
> VM.
> 

I was overthinking on the usage of SDEI. I had the assumption that SDEI
may be used by emulated devices to inject SDEI events in VMM. It can
even be done through PSEUDO_SDEI_{ADDR, DATA} in future. For now, the
software signaled and Async PF events are concerned, Async PF event
is always raised by host/KVM. So we needn't it for now and I will drop
this functionality, actually the whole ioctl commands and migration
support in next respin, as you suggested :)

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
  2022-03-25  7:59       ` Gavin Shan
@ 2022-03-25  8:37         ` Oliver Upton
  -1 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-25  8:37 UTC (permalink / raw)
  To: Gavin Shan
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

On Fri, Mar 25, 2022 at 03:59:50PM +0800, Gavin Shan wrote:
> Hi Oliver,
> 
> On 3/24/22 1:55 AM, Oliver Upton wrote:
> > On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
> > > This supports ioctl commands on vCPU to manage the various object.
> > > It's primarily used by VMM to accomplish migration. The ioctl
> > > commands introduced by this are highlighted as below:
> > > 
> > >     * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
> > >       Return the total count of vCPU events, which have been queued
> > >       on the target vCPU.
> > > 
> > >     * KVM_SDEI_CMD_GET_VCPU_EVENT
> > >     * KVM_SDEI_CMD_SET_VCPU_EVENT
> > >       Get or set vCPU events.
> > > 
> > >     * KVM_SDEI_CMD_GET_VCPU_STATE
> > >     * KVM_SDEI_CMD_SET_VCPU_STATE
> > >       Get or set vCPU state.
> > 
> > All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
> > immediately. Just introduce new registers any time a new event comes
> > along. The only event we have at the end of this series is the
> > software-signaled event, with async PF coming later right?
> > 
> > Some special consideration is likely necessary to avoid adding a
> > register for every u64 chunk of data. I don't think we need to afford
> > userspace any illusion of granularity with these, and can probably lump
> > it all under one giant pseudoregister.
> > 
> 
> Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're
> correct we're only concerned by software signaled event and the one for
> Async PF.
> 
> I didn't look into Raghavendra's series deeply. Actually, a lump of
> registers can be avoid after 2048 byte size is specified in its
> encoding. I think 2048 bytes are enough for now since there are
> only two supported events.

When I had said 'one giant pseudoregister' I actually meant 'one
pseudoregister per event', not all of SDEI into a single
structure. Since most of this is in flux now, it is hard to point
out what/how we should migrate from conversation alone.

And on the topic of Raghavendra's series, I do not believe it is
required anymore here w/ the removal of shared events, which I'm
strongly in favor of.

Let's delve deeper into migration on the next series :)

> In the future, we probably have varied number of SDEI events to
> be migrated. In that case, we need to add a new bit to the encoding
> of the pseudo system register, so that VMM (QEMU) can support
> variable sized system register and keep reading and writing on
> these registers on migration:
> 
>     PSEUDO_SDEI_ADDR: 64-bits in width
>     PSEUDO_SDEI_DATA: has varied size
> 
> PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
> (2) The information to be read/written, for example the (shared/private)
> registered events on VM and vCPU, VCPU state.
> 
> PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be
> written. (2) Flags to indicate current block of information is the
> last one or not.

I don't think we have sufficient encoding space in the register ID
to allow for arbitrary length registers. Any new registers for SDEI will
need to fit into one of the predefined sizes. Note that we've already
conditioned userspace to handle registers this way and anything else is
an ABI change.

--
Thanks,
Oliver

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
@ 2022-03-25  8:37         ` Oliver Upton
  0 siblings, 0 replies; 98+ messages in thread
From: Oliver Upton @ 2022-03-25  8:37 UTC (permalink / raw)
  To: Gavin Shan
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

On Fri, Mar 25, 2022 at 03:59:50PM +0800, Gavin Shan wrote:
> Hi Oliver,
> 
> On 3/24/22 1:55 AM, Oliver Upton wrote:
> > On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
> > > This supports ioctl commands on vCPU to manage the various object.
> > > It's primarily used by VMM to accomplish migration. The ioctl
> > > commands introduced by this are highlighted as below:
> > > 
> > >     * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
> > >       Return the total count of vCPU events, which have been queued
> > >       on the target vCPU.
> > > 
> > >     * KVM_SDEI_CMD_GET_VCPU_EVENT
> > >     * KVM_SDEI_CMD_SET_VCPU_EVENT
> > >       Get or set vCPU events.
> > > 
> > >     * KVM_SDEI_CMD_GET_VCPU_STATE
> > >     * KVM_SDEI_CMD_SET_VCPU_STATE
> > >       Get or set vCPU state.
> > 
> > All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
> > immediately. Just introduce new registers any time a new event comes
> > along. The only event we have at the end of this series is the
> > software-signaled event, with async PF coming later right?
> > 
> > Some special consideration is likely necessary to avoid adding a
> > register for every u64 chunk of data. I don't think we need to afford
> > userspace any illusion of granularity with these, and can probably lump
> > it all under one giant pseudoregister.
> > 
> 
> Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're
> correct we're only concerned by software signaled event and the one for
> Async PF.
> 
> I didn't look into Raghavendra's series deeply. Actually, a lump of
> registers can be avoid after 2048 byte size is specified in its
> encoding. I think 2048 bytes are enough for now since there are
> only two supported events.

When I had said 'one giant pseudoregister' I actually meant 'one
pseudoregister per event', not all of SDEI into a single
structure. Since most of this is in flux now, it is hard to point
out what/how we should migrate from conversation alone.

And on the topic of Raghavendra's series, I do not believe it is
required anymore here w/ the removal of shared events, which I'm
strongly in favor of.

Let's delve deeper into migration on the next series :)

> In the future, we probably have varied number of SDEI events to
> be migrated. In that case, we need to add a new bit to the encoding
> of the pseudo system register, so that VMM (QEMU) can support
> variable sized system register and keep reading and writing on
> these registers on migration:
> 
>     PSEUDO_SDEI_ADDR: 64-bits in width
>     PSEUDO_SDEI_DATA: has varied size
> 
> PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
> (2) The information to be read/written, for example the (shared/private)
> registered events on VM and vCPU, VCPU state.
> 
> PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be
> written. (2) Flags to indicate current block of information is the
> last one or not.

I don't think we have sufficient encoding space in the register ID
to allow for arbitrary length registers. Any new registers for SDEI will
need to fit into one of the predefined sizes. Note that we've already
conditioned userspace to handle registers this way and anything else is
an ABI change.

--
Thanks,
Oliver
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
  2022-03-25  7:35         ` Oliver Upton
@ 2022-03-25 10:14           ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25 10:14 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/25/22 3:35 PM, Oliver Upton wrote:
> On Fri, Mar 25, 2022 at 02:59:52PM +0800, Gavin Shan wrote:
>>> The PSCI implementation is a great example of how KVM has grown its
>>> implementation in line with a specification, all the while preserving
>>> backwards compatibility.
>>>
>>
>> The only information feed by VMM is the exposed events. The events
>> can't be registered from guest kernel, and raised from host to guest
>> kernel until it's exposed by VMM.
> 
> I would suggest assuming that all SDEI events are exposed by default in
> KVM. We will not require a VMM change to enable events individually.
> 

Ok, it was exactly what I did in v4, but the event is exposed
by VMM in v5. In v6, it will be staticly defined again :)

>> Besides, the exposed events will
>> be defined staticly in host/KVM as we discussed on PATCH[02/22]. We
>> also discussed to eliminate those ioctl commands. So I think we needn't
>> to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a
>> concern to host itself if the migration can be done through the
>> firmware pseudo system registers since the migration compatibility
>> is the only concern to VMM (QEMU).
> 
> This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version,
> I'd recommend taking a look at how we handle that register in KVM.
> 

Ok. I will do necessary investigation :)

>> Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is
>> picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems
>> QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable
>> it.
> 
> As far as how it interacts with KVM, QEMU looks fine. The name of the
> KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that
> KVM will enable it highest supported PSCI version. If the feature bit is
> cleared then you only get PSCIv0.1
> 
> However, the DT node that QEMU sets up looks a bit crusty. The
> properties for conveying PSCI function IDs were only ever necessary for
> PSCIv0.1. The only property of interest any more is 'method', to convey
> the SMCCC conduit instruction.
> 

Ok, Thanks again for the further information about PSCI implementation.
I will go through the code when I have free cycyles :)

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM
@ 2022-03-25 10:14           ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25 10:14 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/25/22 3:35 PM, Oliver Upton wrote:
> On Fri, Mar 25, 2022 at 02:59:52PM +0800, Gavin Shan wrote:
>>> The PSCI implementation is a great example of how KVM has grown its
>>> implementation in line with a specification, all the while preserving
>>> backwards compatibility.
>>>
>>
>> The only information feed by VMM is the exposed events. The events
>> can't be registered from guest kernel, and raised from host to guest
>> kernel until it's exposed by VMM.
> 
> I would suggest assuming that all SDEI events are exposed by default in
> KVM. We will not require a VMM change to enable events individually.
> 

Ok, it was exactly what I did in v4, but the event is exposed
by VMM in v5. In v6, it will be staticly defined again :)

>> Besides, the exposed events will
>> be defined staticly in host/KVM as we discussed on PATCH[02/22]. We
>> also discussed to eliminate those ioctl commands. So I think we needn't
>> to add KVM_SDEI_CMD_SET_VERSION. Further more, the version is only a
>> concern to host itself if the migration can be done through the
>> firmware pseudo system registers since the migration compatibility
>> is the only concern to VMM (QEMU).
> 
> This all needs to work just like the KVM_REG_ARM_PSCI_VERSION version,
> I'd recommend taking a look at how we handle that register in KVM.
> 

Ok. I will do necessary investigation :)

>> Yes, Currently, 0.1/0.2/1.0 versions are supported by PSCI. 0.1 is
>> picked until VMM asks for 0.2 and 1.0 explicitly. However, it seems
>> QEMU isn't using 1.0 PSCI yet and maybe more patch is needed to enable
>> it.
> 
> As far as how it interacts with KVM, QEMU looks fine. The name of the
> KVM_ARM_VCPU_PSCI_0_2 bit is quite frustrating. It actually implies that
> KVM will enable it highest supported PSCI version. If the feature bit is
> cleared then you only get PSCIv0.1
> 
> However, the DT node that QEMU sets up looks a bit crusty. The
> properties for conveying PSCI function IDs were only ever necessary for
> PSCIv0.1. The only property of interest any more is 'method', to convey
> the SMCCC conduit instruction.
> 

Ok, Thanks again for the further information about PSCI implementation.
I will go through the code when I have free cycyles :)

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
  2022-03-25  8:37         ` Oliver Upton
@ 2022-03-25 10:23           ` Gavin Shan
  -1 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25 10:23 UTC (permalink / raw)
  To: Oliver Upton
  Cc: kvmarm, maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will

Hi Oliver,

On 3/25/22 4:37 PM, Oliver Upton wrote:
> On Fri, Mar 25, 2022 at 03:59:50PM +0800, Gavin Shan wrote:
>> On 3/24/22 1:55 AM, Oliver Upton wrote:
>>> On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
>>>> This supports ioctl commands on vCPU to manage the various object.
>>>> It's primarily used by VMM to accomplish migration. The ioctl
>>>> commands introduced by this are highlighted as below:
>>>>
>>>>      * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
>>>>        Return the total count of vCPU events, which have been queued
>>>>        on the target vCPU.
>>>>
>>>>      * KVM_SDEI_CMD_GET_VCPU_EVENT
>>>>      * KVM_SDEI_CMD_SET_VCPU_EVENT
>>>>        Get or set vCPU events.
>>>>
>>>>      * KVM_SDEI_CMD_GET_VCPU_STATE
>>>>      * KVM_SDEI_CMD_SET_VCPU_STATE
>>>>        Get or set vCPU state.
>>>
>>> All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
>>> immediately. Just introduce new registers any time a new event comes
>>> along. The only event we have at the end of this series is the
>>> software-signaled event, with async PF coming later right?
>>>
>>> Some special consideration is likely necessary to avoid adding a
>>> register for every u64 chunk of data. I don't think we need to afford
>>> userspace any illusion of granularity with these, and can probably lump
>>> it all under one giant pseudoregister.
>>>
>>
>> Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're
>> correct we're only concerned by software signaled event and the one for
>> Async PF.
>>
>> I didn't look into Raghavendra's series deeply. Actually, a lump of
>> registers can be avoid after 2048 byte size is specified in its
>> encoding. I think 2048 bytes are enough for now since there are
>> only two supported events.
> 
> When I had said 'one giant pseudoregister' I actually meant 'one
> pseudoregister per event', not all of SDEI into a single
> structure. Since most of this is in flux now, it is hard to point
> out what/how we should migrate from conversation alone.
> 
> And on the topic of Raghavendra's series, I do not believe it is
> required anymore here w/ the removal of shared events, which I'm
> strongly in favor of.
> 
> Let's delve deeper into migration on the next series :)
> 

Ok, Thanks for your clarification about 'one giant pseudoregister'.
Lets have more discussion about the migration on next revision.
To be more clear, I plan to implement the base functionality,
where only the private event is supported. After it reaches into
mergeable or merged, we can post the add-on series to support
migration.

>> In the future, we probably have varied number of SDEI events to
>> be migrated. In that case, we need to add a new bit to the encoding
>> of the pseudo system register, so that VMM (QEMU) can support
>> variable sized system register and keep reading and writing on
>> these registers on migration:
>>
>>      PSEUDO_SDEI_ADDR: 64-bits in width
>>      PSEUDO_SDEI_DATA: has varied size
>>
>> PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
>> (2) The information to be read/written, for example the (shared/private)
>> registered events on VM and vCPU, VCPU state.
>>
>> PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be
>> written. (2) Flags to indicate current block of information is the
>> last one or not.
> 
> I don't think we have sufficient encoding space in the register ID
> to allow for arbitrary length registers. Any new registers for SDEI will
> need to fit into one of the predefined sizes. Note that we've already
> conditioned userspace to handle registers this way and anything else is
> an ABI change.
> 

Ok, I think we need padding to structures of the event, to make the
event object aligned to 64-bytes aligned, and VCPU state to 512-bytes
aligned. 64-bytes and 128-bytes registers have been supported.

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 98+ messages in thread

* Re: [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU
@ 2022-03-25 10:23           ` Gavin Shan
  0 siblings, 0 replies; 98+ messages in thread
From: Gavin Shan @ 2022-03-25 10:23 UTC (permalink / raw)
  To: Oliver Upton
  Cc: maz, linux-kernel, eauger, shan.gavin, Jonathan.Cameron,
	pbonzini, vkuznets, will, kvmarm

Hi Oliver,

On 3/25/22 4:37 PM, Oliver Upton wrote:
> On Fri, Mar 25, 2022 at 03:59:50PM +0800, Gavin Shan wrote:
>> On 3/24/22 1:55 AM, Oliver Upton wrote:
>>> On Tue, Mar 22, 2022 at 04:07:07PM +0800, Gavin Shan wrote:
>>>> This supports ioctl commands on vCPU to manage the various object.
>>>> It's primarily used by VMM to accomplish migration. The ioctl
>>>> commands introduced by this are highlighted as below:
>>>>
>>>>      * KVM_SDEI_CMD_GET_VCPU_EVENT_COUNT
>>>>        Return the total count of vCPU events, which have been queued
>>>>        on the target vCPU.
>>>>
>>>>      * KVM_SDEI_CMD_GET_VCPU_EVENT
>>>>      * KVM_SDEI_CMD_SET_VCPU_EVENT
>>>>        Get or set vCPU events.
>>>>
>>>>      * KVM_SDEI_CMD_GET_VCPU_STATE
>>>>      * KVM_SDEI_CMD_SET_VCPU_STATE
>>>>        Get or set vCPU state.
>>>
>>> All of this GET/SET stuff can probably be added to KVM_{GET,SET}_ONE_REG
>>> immediately. Just introduce new registers any time a new event comes
>>> along. The only event we have at the end of this series is the
>>> software-signaled event, with async PF coming later right?
>>>
>>> Some special consideration is likely necessary to avoid adding a
>>> register for every u64 chunk of data. I don't think we need to afford
>>> userspace any illusion of granularity with these, and can probably lump
>>> it all under one giant pseudoregister.
>>>
>>
>> Yes, KVM_{GET,SET}_ONE_REG is the ideal interface for migration. You're
>> correct we're only concerned by software signaled event and the one for
>> Async PF.
>>
>> I didn't look into Raghavendra's series deeply. Actually, a lump of
>> registers can be avoid after 2048 byte size is specified in its
>> encoding. I think 2048 bytes are enough for now since there are
>> only two supported events.
> 
> When I had said 'one giant pseudoregister' I actually meant 'one
> pseudoregister per event', not all of SDEI into a single
> structure. Since most of this is in flux now, it is hard to point
> out what/how we should migrate from conversation alone.
> 
> And on the topic of Raghavendra's series, I do not believe it is
> required anymore here w/ the removal of shared events, which I'm
> strongly in favor of.
> 
> Let's delve deeper into migration on the next series :)
> 

Ok, Thanks for your clarification about 'one giant pseudoregister'.
Lets have more discussion about the migration on next revision.
To be more clear, I plan to implement the base functionality,
where only the private event is supported. After it reaches into
mergeable or merged, we can post the add-on series to support
migration.

>> In the future, we probably have varied number of SDEI events to
>> be migrated. In that case, we need to add a new bit to the encoding
>> of the pseudo system register, so that VMM (QEMU) can support
>> variable sized system register and keep reading and writing on
>> these registers on migration:
>>
>>      PSEUDO_SDEI_ADDR: 64-bits in width
>>      PSEUDO_SDEI_DATA: has varied size
>>
>> PSEUDO_SDEI_ADDR is used to (1) Indicate the size of PSEUDO_SDEI_DATA
>> (2) The information to be read/written, for example the (shared/private)
>> registered events on VM and vCPU, VCPU state.
>>
>> PSEUDO_SDEI_DATA is used to (1) Retrieved information or that to be
>> written. (2) Flags to indicate current block of information is the
>> last one or not.
> 
> I don't think we have sufficient encoding space in the register ID
> to allow for arbitrary length registers. Any new registers for SDEI will
> need to fit into one of the predefined sizes. Note that we've already
> conditioned userspace to handle registers this way and anything else is
> an ABI change.
> 

Ok, I think we need padding to structures of the event, to make the
event object aligned to 64-bytes aligned, and VCPU state to 512-bytes
aligned. 64-bytes and 128-bytes registers have been supported.

Thanks,
Gavin

_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 98+ messages in thread

end of thread, other threads:[~2022-03-25 10:23 UTC | newest]

Thread overview: 98+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-03-22  8:06 [PATCH v5 00/22] Support SDEI Virtualization Gavin Shan
2022-03-22  8:06 ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 01/22] KVM: arm64: Introduce template for inline functions Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22 19:42   ` Oliver Upton
2022-03-22 19:42     ` Oliver Upton
2022-03-23 12:16     ` Gavin Shan
2022-03-23 12:16       ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 02/22] KVM: arm64: Add SDEI virtualization infrastructure Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22 22:43   ` Oliver Upton
2022-03-22 22:43     ` Oliver Upton
2022-03-23 12:40     ` Gavin Shan
2022-03-23 12:40       ` Gavin Shan
2022-03-23 17:11   ` Oliver Upton
2022-03-23 17:11     ` Oliver Upton
2022-03-24  6:54     ` Gavin Shan
2022-03-24  6:54       ` Gavin Shan
2022-03-24  9:04       ` Oliver Upton
2022-03-24  9:04         ` Oliver Upton
2022-03-25  6:07         ` Gavin Shan
2022-03-25  6:07           ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 03/22] KVM: arm64: Support SDEI_VERSION hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22 18:04   ` Oliver Upton
2022-03-22 18:04     ` Oliver Upton
2022-03-23 12:46     ` Gavin Shan
2022-03-23 12:46       ` Gavin Shan
2022-03-23 16:31       ` Oliver Upton
2022-03-23 16:31         ` Oliver Upton
2022-03-24  4:07         ` Gavin Shan
2022-03-24  4:07           ` Gavin Shan
2022-03-24  7:48           ` Oliver Upton
2022-03-24  7:48             ` Oliver Upton
2022-03-25  6:11             ` Gavin Shan
2022-03-25  6:11               ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 04/22] KVM: arm64: Support SDEI_EVENT_REGISTER hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 05/22] KVM: arm64: Support SDEI_EVENT_{ENABLE, DISABLE} hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 06/22] KVM: arm64: Support SDEI_EVENT_CONTEXT hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 07/22] KVM: arm64: Support SDEI_EVENT_UNREGISTER hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 08/22] KVM: arm64: Support SDEI_EVENT_STATUS hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 09/22] KVM: arm64: Support SDEI_EVENT_GET_INFO hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 10/22] KVM: arm64: Support SDEI_EVENT_ROUTING_SET hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:06 ` [PATCH v5 11/22] KVM: arm64: Support SDEI_PE_{MASK, UNMASK} hypercall Gavin Shan
2022-03-22  8:06   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 12/22] KVM: arm64: Support SDEI_{PRIVATE, SHARED}_RESET Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 13/22] KVM: arm64: Support SDEI_FEATURES hypercall Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 14/22] KVM: arm64: Support SDEI event injection, delivery and cancellation Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 15/22] KVM: arm64: Support SDEI_EVENT_SIGNAL hypercall Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22 23:06   ` Oliver Upton
2022-03-22 23:06     ` Oliver Upton
2022-03-23 12:52     ` Gavin Shan
2022-03-23 12:52       ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE,COMPLETE_AND_RESUME} hypercall Gavin Shan
2022-03-22  8:07   ` [PATCH v5 16/22] KVM: arm64: Support SDEI_EVENT_{COMPLETE, COMPLETE_AND_RESUME} hypercall Gavin Shan
2022-03-22  8:07 ` [PATCH v5 17/22] KVM: arm64: Support SDEI event notifier Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 18/22] KVM: arm64: Support SDEI ioctl commands on VM Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-23 17:28   ` Oliver Upton
2022-03-23 17:28     ` Oliver Upton
2022-03-25  6:59     ` Gavin Shan
2022-03-25  6:59       ` Gavin Shan
2022-03-25  7:35       ` Oliver Upton
2022-03-25  7:35         ` Oliver Upton
2022-03-25 10:14         ` Gavin Shan
2022-03-25 10:14           ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 19/22] KVM: arm64: Support SDEI ioctl commands on vCPU Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-23 17:55   ` Oliver Upton
2022-03-23 17:55     ` Oliver Upton
2022-03-25  7:59     ` Gavin Shan
2022-03-25  7:59       ` Gavin Shan
2022-03-25  8:37       ` Oliver Upton
2022-03-25  8:37         ` Oliver Upton
2022-03-25 10:23         ` Gavin Shan
2022-03-25 10:23           ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 20/22] KVM: arm64: Export SDEI capability Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 21/22] KVM: arm64: Add SDEI document Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22  8:07 ` [PATCH v5 22/22] KVM: selftests: Add SDEI test case Gavin Shan
2022-03-22  8:07   ` Gavin Shan
2022-03-22 18:13 ` [PATCH v5 00/22] Support SDEI Virtualization Oliver Upton
2022-03-22 18:13   ` Oliver Upton
2022-03-23 12:57   ` Gavin Shan
2022-03-23 12:57     ` Gavin Shan

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.