All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
@ 2016-03-26  2:13 ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:13 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

Hi,

this is a rebased and reworked version of the ITS emulation support.
It allows KVM guests which use an emulated GICv3 to use LPIs as well,
though in the moment this is limited to emulated PCI devices.
It is now based on and requires the new VGIC implementation [1].

You can find all of this code (and the prerequisites) in the
its-emul/v4 branch of my repository [2].

Cheers,
Andre.

Changelog v3..v4:
- adapting to new VGIC (changes in IRQ injection mechanism)

Changelog v2..v3:
- adapt to 4.3-rc and Christoffer's timer rework
- adapt spin locks on handling PROPBASER/PENDBASER registers
- rework locking in ITS command handling (dropping dist where needed)
- only clear LPI pending bit if LPI could actually be queued
- simplify GICR_CTLR handling
- properly free ITTEs (including our pending bitmap)
- fix corner cases with unmapped collections
- keep retire_lr() around
- rename vgic_handle_base_register to vgic_reg64_access()
- use kcalloc instead of kmalloc
- minor fixes, renames and added comments

Changelog v1..v2
- fix issues when using non-ITS GICv3 emulation
- streamline frame address initialization (new patch 05/15)
- preallocate buffer memory for reading from guest's memory
- move locking into the actual command handlers
-   preallocate memory for new structures if needed
- use non-atomic __set_bit() and __clear_bit() when under the lock
- add INT command handler to allow LPI injection from the guest
- rewrite CWRITER handler to align with new locking scheme
- remove unneeded CONFIG_HAVE_KVM_MSI #ifdefs
- check memory table size against our LPI limit (65536 interrupts)
- observe initial gap of 1024 interrupts in pending table
- use term "configuration table" to be in line with the spec
- clarify and extend documentation on API extensions
- introduce new KVM_CAP_MSI_DEVID capability to advertise device ID requirement
- update, fix and add many comments
- minor style changes as requested by reviewers

---------------

The GICv3 ITS (Interrupt Translation Service) is a part of the
ARM GICv3 interrupt controller [4] used for implementing MSIs.
It specifies a new kind of interrupts (LPIs), which are mapped to
establish a connection between a device, its MSI payload value and
the target processor the IRQ is eventually delivered to.
In order to allow using MSIs in an ARM64 KVM guest, we emulate this
ITS widget in the kernel.
The ITS works by reading commands written by software (from the guest
in our case) into a (guest allocated) memory region and establishing
the mapping between a device, the MSI payload and the target CPU.
We parse these commands and update our internal data structures to
reflect those changes. On an MSI injection we iterate those
structures to learn the LPI number we have to inject.
For the time being we use simple lists to hold the data, this is
good enough for the small number of entries each of the components
currently have. Should this become a performance bottleneck in the
future, those can be extended to arrays or trees if needed.

Most of the code lives in a separate source file (its-emul.c), though
there are some changes necessary in the existing VGIC files.

For the time being this series gives us the ability to use emulated
PCI devices that can use MSIs in the guest. Those have to be
triggered by letting the userland device emulation simulate the MSI
write with the KVM_SIGNAL_MSI ioctl. This will be translated into
the proper LPI by the ITS emulation and injected into the guest in
the usual way (just with a higher IRQ number).

This series is based on 4.5 plus the KVM queue for 4.6 and the new VGIC
emulation code and can be found at the its-emul/v4 branch of this
repository [2].
For this to be used you need a GICv3 host machine (a fast model would
do), though it does not rely on any host ITS bits (neither in hardware
or software).

To test this you can use the kvmtool patches available in the "its"
branch here [3].
Start a guest with: "$ lkvm run --irqchip=gicv3-its --force-pci"
and see the ITS being used for instance by the virtio devices.

[1]: https://lists.cs.columbia.edu/pipermail/kvmarm/2016-March/019191.html
[2]: git://linux-arm.org/linux-ap.git
     http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4
[3]: git://linux-arm.org/kvmtool.git
     http://www.linux-arm.org/git?p=kvmtool.git;a=log;h=refs/heads/its
[4]: http://arminfo.emea.arm.com/help/topic/com.arm.doc.ihi0069a/IHI0069A_gic_architecture_specification.pdf

Andre Przywara (12):
  KVM: extend struct kvm_msi to hold a 32-bit device ID
  KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
  KVM: arm64: Introduce new MMIO region for the ITS base address
  KVM: arm64: handle ITS related GICv3 redistributor registers
  KVM: arm64: introduce ITS emulation file with stub functions
  KVM: arm64: implement basic ITS register handlers
  KVM: arm64: add data structures to model ITS interrupt translation
  KVM: arm64: connect LPIs to the VGIC emulation
  KVM: arm64: sync LPI configuration and pending tables
  KVM: arm64: implement ITS command queue command handlers
  KVM: arm64: implement MSI injection in ITS emulation
  KVM: arm64: enable ITS emulation as a virtual MSI controller

 Documentation/virtual/kvm/api.txt              |   14 +-
 Documentation/virtual/kvm/devices/arm-vgic.txt |    9 +
 arch/arm/include/asm/kvm_host.h                |    2 +-
 arch/arm/kvm/arm.c                             |    2 +-
 arch/arm64/include/asm/kvm_host.h              |    2 +-
 arch/arm64/include/uapi/asm/kvm.h              |    2 +
 arch/arm64/kvm/Kconfig                         |    1 +
 arch/arm64/kvm/Makefile                        |    1 +
 arch/arm64/kvm/reset.c                         |   12 +-
 include/kvm/vgic/vgic.h                        |   33 +
 include/linux/irqchip/arm-gic-v3.h             |   28 +-
 include/uapi/linux/kvm.h                       |    5 +-
 virt/kvm/arm/vgic.c                            |    5 +
 virt/kvm/arm/vgic/its-emul.c                   | 1146 ++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.c                       |   18 +
 virt/kvm/arm/vgic/vgic.h                       |   33 +
 virt/kvm/arm/vgic/vgic_init.c                  |    9 +
 virt/kvm/arm/vgic/vgic_kvm_device.c            |    7 +
 virt/kvm/arm/vgic/vgic_mmio.c                  |   54 +-
 19 files changed, 1365 insertions(+), 18 deletions(-)
 create mode 100644 virt/kvm/arm/vgic/its-emul.c

-- 
2.7.3

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
@ 2016-03-26  2:13 ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:13 UTC (permalink / raw)
  To: linux-arm-kernel

Hi,

this is a rebased and reworked version of the ITS emulation support.
It allows KVM guests which use an emulated GICv3 to use LPIs as well,
though in the moment this is limited to emulated PCI devices.
It is now based on and requires the new VGIC implementation [1].

You can find all of this code (and the prerequisites) in the
its-emul/v4 branch of my repository [2].

Cheers,
Andre.

Changelog v3..v4:
- adapting to new VGIC (changes in IRQ injection mechanism)

Changelog v2..v3:
- adapt to 4.3-rc and Christoffer's timer rework
- adapt spin locks on handling PROPBASER/PENDBASER registers
- rework locking in ITS command handling (dropping dist where needed)
- only clear LPI pending bit if LPI could actually be queued
- simplify GICR_CTLR handling
- properly free ITTEs (including our pending bitmap)
- fix corner cases with unmapped collections
- keep retire_lr() around
- rename vgic_handle_base_register to vgic_reg64_access()
- use kcalloc instead of kmalloc
- minor fixes, renames and added comments

Changelog v1..v2
- fix issues when using non-ITS GICv3 emulation
- streamline frame address initialization (new patch 05/15)
- preallocate buffer memory for reading from guest's memory
- move locking into the actual command handlers
-   preallocate memory for new structures if needed
- use non-atomic __set_bit() and __clear_bit() when under the lock
- add INT command handler to allow LPI injection from the guest
- rewrite CWRITER handler to align with new locking scheme
- remove unneeded CONFIG_HAVE_KVM_MSI #ifdefs
- check memory table size against our LPI limit (65536 interrupts)
- observe initial gap of 1024 interrupts in pending table
- use term "configuration table" to be in line with the spec
- clarify and extend documentation on API extensions
- introduce new KVM_CAP_MSI_DEVID capability to advertise device ID requirement
- update, fix and add many comments
- minor style changes as requested by reviewers

---------------

The GICv3 ITS (Interrupt Translation Service) is a part of the
ARM GICv3 interrupt controller [4] used for implementing MSIs.
It specifies a new kind of interrupts (LPIs), which are mapped to
establish a connection between a device, its MSI payload value and
the target processor the IRQ is eventually delivered to.
In order to allow using MSIs in an ARM64 KVM guest, we emulate this
ITS widget in the kernel.
The ITS works by reading commands written by software (from the guest
in our case) into a (guest allocated) memory region and establishing
the mapping between a device, the MSI payload and the target CPU.
We parse these commands and update our internal data structures to
reflect those changes. On an MSI injection we iterate those
structures to learn the LPI number we have to inject.
For the time being we use simple lists to hold the data, this is
good enough for the small number of entries each of the components
currently have. Should this become a performance bottleneck in the
future, those can be extended to arrays or trees if needed.

Most of the code lives in a separate source file (its-emul.c), though
there are some changes necessary in the existing VGIC files.

For the time being this series gives us the ability to use emulated
PCI devices that can use MSIs in the guest. Those have to be
triggered by letting the userland device emulation simulate the MSI
write with the KVM_SIGNAL_MSI ioctl. This will be translated into
the proper LPI by the ITS emulation and injected into the guest in
the usual way (just with a higher IRQ number).

This series is based on 4.5 plus the KVM queue for 4.6 and the new VGIC
emulation code and can be found at the its-emul/v4 branch of this
repository [2].
For this to be used you need a GICv3 host machine (a fast model would
do), though it does not rely on any host ITS bits (neither in hardware
or software).

To test this you can use the kvmtool patches available in the "its"
branch here [3].
Start a guest with: "$ lkvm run --irqchip=gicv3-its --force-pci"
and see the ITS being used for instance by the virtio devices.

[1]: https://lists.cs.columbia.edu/pipermail/kvmarm/2016-March/019191.html
[2]: git://linux-arm.org/linux-ap.git
     http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-emul/v4
[3]: git://linux-arm.org/kvmtool.git
     http://www.linux-arm.org/git?p=kvmtool.git;a=log;h=refs/heads/its
[4]: http://arminfo.emea.arm.com/help/topic/com.arm.doc.ihi0069a/IHI0069A_gic_architecture_specification.pdf

Andre Przywara (12):
  KVM: extend struct kvm_msi to hold a 32-bit device ID
  KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
  KVM: arm64: Introduce new MMIO region for the ITS base address
  KVM: arm64: handle ITS related GICv3 redistributor registers
  KVM: arm64: introduce ITS emulation file with stub functions
  KVM: arm64: implement basic ITS register handlers
  KVM: arm64: add data structures to model ITS interrupt translation
  KVM: arm64: connect LPIs to the VGIC emulation
  KVM: arm64: sync LPI configuration and pending tables
  KVM: arm64: implement ITS command queue command handlers
  KVM: arm64: implement MSI injection in ITS emulation
  KVM: arm64: enable ITS emulation as a virtual MSI controller

 Documentation/virtual/kvm/api.txt              |   14 +-
 Documentation/virtual/kvm/devices/arm-vgic.txt |    9 +
 arch/arm/include/asm/kvm_host.h                |    2 +-
 arch/arm/kvm/arm.c                             |    2 +-
 arch/arm64/include/asm/kvm_host.h              |    2 +-
 arch/arm64/include/uapi/asm/kvm.h              |    2 +
 arch/arm64/kvm/Kconfig                         |    1 +
 arch/arm64/kvm/Makefile                        |    1 +
 arch/arm64/kvm/reset.c                         |   12 +-
 include/kvm/vgic/vgic.h                        |   33 +
 include/linux/irqchip/arm-gic-v3.h             |   28 +-
 include/uapi/linux/kvm.h                       |    5 +-
 virt/kvm/arm/vgic.c                            |    5 +
 virt/kvm/arm/vgic/its-emul.c                   | 1146 ++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.c                       |   18 +
 virt/kvm/arm/vgic/vgic.h                       |   33 +
 virt/kvm/arm/vgic/vgic_init.c                  |    9 +
 virt/kvm/arm/vgic/vgic_kvm_device.c            |    7 +
 virt/kvm/arm/vgic/vgic_mmio.c                  |   54 +-
 19 files changed, 1365 insertions(+), 18 deletions(-)
 create mode 100644 virt/kvm/arm/vgic/its-emul.c

-- 
2.7.3

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:13   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:13 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

The ARM GICv3 ITS MSI controller requires a device ID to be able to
assign the proper interrupt vector. On real hardware, this ID is
sampled from the bus. To be able to emulate an ITS controller, extend
the KVM MSI interface to let userspace provide such a device ID. For
PCI devices, the device ID is simply the 16-bit bus-device-function
triplet, which should be easily available to the userland tool.

Also there is a new KVM capability which advertises whether the
current VM requires a device ID to be set along with the MSI data.
This flag is still reported as not available everywhere, later we will
enable it when ITS emulation is used.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Reviewed-by: Eric Auger <eric.auger@linaro.org>
---
 Documentation/virtual/kvm/api.txt | 12 ++++++++++--
 include/uapi/linux/kvm.h          |  5 ++++-
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index cb2ef0b..8f7351d 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2163,10 +2163,18 @@ struct kvm_msi {
 	__u32 address_hi;
 	__u32 data;
 	__u32 flags;
-	__u8  pad[16];
+	__u32 devid;
+	__u8  pad[12];
 };
 
-No flags are defined so far. The corresponding field must be 0.
+flags: KVM_MSI_VALID_DEVID: devid contains a valid value
+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
+       for the device that wrote the MSI message.
+       For PCI, this is usually a BFD identifier in the lower 16 bits.
+
+The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
+the device ID. If this capability is not set, userland cannot rely on
+the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
 
 
 4.71 KVM_CREATE_PIT2
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 50f44a2..6a02871 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
 #define KVM_CAP_S390_RI 124
 #define KVM_CAP_ARM_PMU_V3 125
 #define KVM_CAP_VCPU_ATTRIBUTES 126
+#define KVM_CAP_MSI_DEVID 127
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1010,12 +1011,14 @@ struct kvm_one_reg {
 	__u64 addr;
 };
 
+#define KVM_MSI_VALID_DEVID	(1U << 0)
 struct kvm_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
 	__u32 flags;
-	__u8  pad[16];
+	__u32 devid;
+	__u8  pad[12];
 };
 
 struct kvm_arm_device_addr {
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
@ 2016-03-26  2:13   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:13 UTC (permalink / raw)
  To: linux-arm-kernel

The ARM GICv3 ITS MSI controller requires a device ID to be able to
assign the proper interrupt vector. On real hardware, this ID is
sampled from the bus. To be able to emulate an ITS controller, extend
the KVM MSI interface to let userspace provide such a device ID. For
PCI devices, the device ID is simply the 16-bit bus-device-function
triplet, which should be easily available to the userland tool.

Also there is a new KVM capability which advertises whether the
current VM requires a device ID to be set along with the MSI data.
This flag is still reported as not available everywhere, later we will
enable it when ITS emulation is used.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Reviewed-by: Eric Auger <eric.auger@linaro.org>
---
 Documentation/virtual/kvm/api.txt | 12 ++++++++++--
 include/uapi/linux/kvm.h          |  5 ++++-
 2 files changed, 14 insertions(+), 3 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index cb2ef0b..8f7351d 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2163,10 +2163,18 @@ struct kvm_msi {
 	__u32 address_hi;
 	__u32 data;
 	__u32 flags;
-	__u8  pad[16];
+	__u32 devid;
+	__u8  pad[12];
 };
 
-No flags are defined so far. The corresponding field must be 0.
+flags: KVM_MSI_VALID_DEVID: devid contains a valid value
+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
+       for the device that wrote the MSI message.
+       For PCI, this is usually a BFD identifier in the lower 16 bits.
+
+The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
+the device ID. If this capability is not set, userland cannot rely on
+the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
 
 
 4.71 KVM_CREATE_PIT2
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 50f44a2..6a02871 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
 #define KVM_CAP_S390_RI 124
 #define KVM_CAP_ARM_PMU_V3 125
 #define KVM_CAP_VCPU_ATTRIBUTES 126
+#define KVM_CAP_MSI_DEVID 127
 
 #ifdef KVM_CAP_IRQ_ROUTING
 
@@ -1010,12 +1011,14 @@ struct kvm_one_reg {
 	__u64 addr;
 };
 
+#define KVM_MSI_VALID_DEVID	(1U << 0)
 struct kvm_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
 	__u32 flags;
-	__u8  pad[16];
+	__u32 devid;
+	__u8  pad[12];
 };
 
 struct kvm_arm_device_addr {
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: linux-arm-kernel, kvmarm, kvm

KVM capabilities can be a per-VM property, though ARM/ARM64 currently
does not pass on the VM pointer to the architecture specific
capability handlers.
Add a "struct kvm*" parameter to those function to later allow proper
per-VM capability reporting.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 arch/arm/include/asm/kvm_host.h   | 2 +-
 arch/arm/kvm/arm.c                | 2 +-
 arch/arm64/include/asm/kvm_host.h | 2 +-
 arch/arm64/kvm/reset.c            | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 3850701..494b004 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -265,7 +265,7 @@ static inline void __cpu_init_stage2(void)
 	kvm_call_hyp(__init_stage2_translation);
 }
 
-static inline int kvm_arch_dev_ioctl_check_extension(long ext)
+static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
 {
 	return 0;
 }
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 5b9d9e3..e2e9be2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -205,7 +205,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = KVM_MAX_VCPUS;
 		break;
 	default:
-		r = kvm_arch_dev_ioctl_check_extension(ext);
+		r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
 		break;
 	}
 	return r;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 71fa6fe..2cdd7ae 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -46,7 +46,7 @@
 
 int __attribute_const__ kvm_target_cpu(void);
 int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
-int kvm_arch_dev_ioctl_check_extension(long ext);
+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
 
 struct kvm_arch {
 	/* The VMID generation used for the virt. memory system */
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 9677bf0..2ba7c8d 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -63,7 +63,7 @@ static bool cpu_has_32bit_el1(void)
  * We currently assume that the number of HW registers is uniform
  * across all CPUs (see cpuinfo_sanity_check).
  */
-int kvm_arch_dev_ioctl_check_extension(long ext)
+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
 {
 	int r;
 
-- 
2.7.3


^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

KVM capabilities can be a per-VM property, though ARM/ARM64 currently
does not pass on the VM pointer to the architecture specific
capability handlers.
Add a "struct kvm*" parameter to those function to later allow proper
per-VM capability reporting.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 arch/arm/include/asm/kvm_host.h   | 2 +-
 arch/arm/kvm/arm.c                | 2 +-
 arch/arm64/include/asm/kvm_host.h | 2 +-
 arch/arm64/kvm/reset.c            | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 3850701..494b004 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -265,7 +265,7 @@ static inline void __cpu_init_stage2(void)
 	kvm_call_hyp(__init_stage2_translation);
 }
 
-static inline int kvm_arch_dev_ioctl_check_extension(long ext)
+static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
 {
 	return 0;
 }
diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
index 5b9d9e3..e2e9be2 100644
--- a/arch/arm/kvm/arm.c
+++ b/arch/arm/kvm/arm.c
@@ -205,7 +205,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = KVM_MAX_VCPUS;
 		break;
 	default:
-		r = kvm_arch_dev_ioctl_check_extension(ext);
+		r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
 		break;
 	}
 	return r;
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 71fa6fe..2cdd7ae 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -46,7 +46,7 @@
 
 int __attribute_const__ kvm_target_cpu(void);
 int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
-int kvm_arch_dev_ioctl_check_extension(long ext);
+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
 
 struct kvm_arch {
 	/* The VMID generation used for the virt. memory system */
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 9677bf0..2ba7c8d 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -63,7 +63,7 @@ static bool cpu_has_32bit_el1(void)
  * We currently assume that the number of HW registers is uniform
  * across all CPUs (see cpuinfo_sanity_check).
  */
-int kvm_arch_dev_ioctl_check_extension(long ext)
+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
 {
 	int r;
 
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

The ARM GICv3 ITS controller requires a separate register frame to
cover ITS specific registers. Add a new VGIC address type and store
the address in a field in the vgic_dist structure.
Provide a function to check whether userland has provided the address,
so ITS functionality can be guarded by that check.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
 arch/arm64/include/uapi/asm/kvm.h              |  2 ++
 include/kvm/vgic/vgic.h                        |  5 +++++
 virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
 virt/kvm/arm/vgic/vgic_init.c                  |  1 +
 virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
 6 files changed, 34 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
index 59541d4..087e2d9 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
@@ -39,6 +39,15 @@ Groups:
       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
       This address needs to be 64K aligned.
 
+    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
+      Base address in the guest physical address space of the GICv3 ITS
+      control register frame. The ITS allows MSI(-X) interrupts to be
+      injected into guests. This extension is optional, if the kernel
+      does not support the ITS, the call returns -ENODEV.
+      This memory is solely for the guest to access the ITS control
+      registers and does not cover the ITS translation register.
+      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
+      This address needs to be 64K aligned and the region covers 64 KByte.
 
   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
   Attributes:
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index f209ea1..c2b257d 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -87,9 +87,11 @@ struct kvm_regs {
 /* Supported VGICv3 address types  */
 #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
 #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
+#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
 
 #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
 #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
+#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
 
 #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
 #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index 7c1d145..11344e6 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -135,6 +135,9 @@ struct vgic_dist {
 		gpa_t			vgic_redist_base;
 	};
 
+	/* The base address of the ITS control register frame */
+	gpa_t			vgic_its_base;
+
 	/* distributor enabled */
 	u32			enabled;
 
@@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
 	return kvm_vgic_global_state.max_gic_vcpus;
 }
 
+bool vgic_has_its(struct kvm *kvm);
+
 #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 65395af..1de2478 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
 
 	return map_is_active;
 }
+
+bool vgic_has_its(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+
+	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
+		return false;
+
+	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
+}
diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index ac655b5..2301e03 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
+	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
 
 out_unlock:
 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
index 7f78a16..3ec2ac3 100644
--- a/virt/kvm/arm/vgic/vgic_kvm_device.c
+++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
@@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
 		block_size = KVM_VGIC_V3_REDIST_SIZE;
 		alignment = SZ_64K;
 		break;
+	case KVM_VGIC_V3_ADDR_TYPE_ITS:
+		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
+		addr_ptr = &vgic->vgic_its_base;
+		block_size = KVM_VGIC_V3_ITS_SIZE;
+		alignment = SZ_64K;
+		break;
 #endif
 	default:
 		r = -ENODEV;
@@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
 		switch (attr->attr) {
 		case KVM_VGIC_V3_ADDR_TYPE_DIST:
 		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
+		case KVM_VGIC_V3_ADDR_TYPE_ITS:
 			return 0;
 		}
 		break;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

The ARM GICv3 ITS controller requires a separate register frame to
cover ITS specific registers. Add a new VGIC address type and store
the address in a field in the vgic_dist structure.
Provide a function to check whether userland has provided the address,
so ITS functionality can be guarded by that check.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
 arch/arm64/include/uapi/asm/kvm.h              |  2 ++
 include/kvm/vgic/vgic.h                        |  5 +++++
 virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
 virt/kvm/arm/vgic/vgic_init.c                  |  1 +
 virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
 6 files changed, 34 insertions(+)

diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
index 59541d4..087e2d9 100644
--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
@@ -39,6 +39,15 @@ Groups:
       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
       This address needs to be 64K aligned.
 
+    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
+      Base address in the guest physical address space of the GICv3 ITS
+      control register frame. The ITS allows MSI(-X) interrupts to be
+      injected into guests. This extension is optional, if the kernel
+      does not support the ITS, the call returns -ENODEV.
+      This memory is solely for the guest to access the ITS control
+      registers and does not cover the ITS translation register.
+      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
+      This address needs to be 64K aligned and the region covers 64 KByte.
 
   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
   Attributes:
diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
index f209ea1..c2b257d 100644
--- a/arch/arm64/include/uapi/asm/kvm.h
+++ b/arch/arm64/include/uapi/asm/kvm.h
@@ -87,9 +87,11 @@ struct kvm_regs {
 /* Supported VGICv3 address types  */
 #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
 #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
+#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
 
 #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
 #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
+#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
 
 #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
 #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index 7c1d145..11344e6 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -135,6 +135,9 @@ struct vgic_dist {
 		gpa_t			vgic_redist_base;
 	};
 
+	/* The base address of the ITS control register frame */
+	gpa_t			vgic_its_base;
+
 	/* distributor enabled */
 	u32			enabled;
 
@@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
 	return kvm_vgic_global_state.max_gic_vcpus;
 }
 
+bool vgic_has_its(struct kvm *kvm);
+
 #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 65395af..1de2478 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
 
 	return map_is_active;
 }
+
+bool vgic_has_its(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+
+	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
+		return false;
+
+	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
+}
diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index ac655b5..2301e03 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
+	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
 
 out_unlock:
 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
index 7f78a16..3ec2ac3 100644
--- a/virt/kvm/arm/vgic/vgic_kvm_device.c
+++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
@@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
 		block_size = KVM_VGIC_V3_REDIST_SIZE;
 		alignment = SZ_64K;
 		break;
+	case KVM_VGIC_V3_ADDR_TYPE_ITS:
+		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
+		addr_ptr = &vgic->vgic_its_base;
+		block_size = KVM_VGIC_V3_ITS_SIZE;
+		alignment = SZ_64K;
+		break;
 #endif
 	default:
 		r = -ENODEV;
@@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
 		switch (attr->attr) {
 		case KVM_VGIC_V3_ADDR_TYPE_DIST:
 		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
+		case KVM_VGIC_V3_ADDR_TYPE_ITS:
 			return 0;
 		}
 		break;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: linux-arm-kernel, kvmarm, kvm

In the GICv3 redistributor there are the PENDBASER and PROPBASER
registers which we did not emulate so far, as they only make sense
when having an ITS. In preparation for that emulate those MMIO
accesses by storing the 64-bit data written into it into a variable
which we later read in the ITS emulation.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h       |  8 ++++++++
 virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
 2 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index 11344e6..8ea5dd7 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -145,6 +145,14 @@ struct vgic_dist {
 
 	struct vgic_io_device	*dist_iodevs;
 	struct vgic_io_device	*redist_iodevs;
+
+	/* Address of LPI configuration table shared by all redistributors */
+	u64			propbaser;
+
+	/* Addresses of LPI pending tables per redistributor */
+	u64			*pendbaser;
+
+	bool			lpis_enabled;
 };
 
 struct vgic_v2_cpu_if {
diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
index 7d5d630..252b9aff 100644
--- a/virt/kvm/arm/vgic/vgic_mmio.c
+++ b/virt/kvm/arm/vgic/vgic_mmio.c
@@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
 				       struct kvm_io_device *this,
 				       gpa_t addr, int len, void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+	write_mask64(dist->propbaser, addr & 7, len, val);
 	return 0;
 }
 
@@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
 				        struct kvm_io_device *this,
 				        gpa_t addr, int len, const void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+	/* Storing a value with LPIs already enabled is undefined */
+	if (!dist->lpis_enabled)
+		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
 	return 0;
 }
 
@@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
 				       struct kvm_io_device *this,
 				       gpa_t addr, int len, void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
+
+	write_mask64(pendbaser, addr & 7, len, val);
 	return 0;
 }
 
@@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
 				        struct kvm_io_device *this,
 				        gpa_t addr, int len, const void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
+
+	/* Storing a value with LPIs already enabled is undefined */
+	if (!dist->lpis_enabled)
+		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
 	return 0;
 }
 
-- 
2.7.3


^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

In the GICv3 redistributor there are the PENDBASER and PROPBASER
registers which we did not emulate so far, as they only make sense
when having an ITS. In preparation for that emulate those MMIO
accesses by storing the 64-bit data written into it into a variable
which we later read in the ITS emulation.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h       |  8 ++++++++
 virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
 2 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index 11344e6..8ea5dd7 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -145,6 +145,14 @@ struct vgic_dist {
 
 	struct vgic_io_device	*dist_iodevs;
 	struct vgic_io_device	*redist_iodevs;
+
+	/* Address of LPI configuration table shared by all redistributors */
+	u64			propbaser;
+
+	/* Addresses of LPI pending tables per redistributor */
+	u64			*pendbaser;
+
+	bool			lpis_enabled;
 };
 
 struct vgic_v2_cpu_if {
diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
index 7d5d630..252b9aff 100644
--- a/virt/kvm/arm/vgic/vgic_mmio.c
+++ b/virt/kvm/arm/vgic/vgic_mmio.c
@@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
 				       struct kvm_io_device *this,
 				       gpa_t addr, int len, void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+	write_mask64(dist->propbaser, addr & 7, len, val);
 	return 0;
 }
 
@@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
 				        struct kvm_io_device *this,
 				        gpa_t addr, int len, const void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
+	/* Storing a value with LPIs already enabled is undefined */
+	if (!dist->lpis_enabled)
+		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
 	return 0;
 }
 
@@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
 				       struct kvm_io_device *this,
 				       gpa_t addr, int len, void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
+
+	write_mask64(pendbaser, addr & 7, len, val);
 	return 0;
 }
 
@@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
 				        struct kvm_io_device *this,
 				        gpa_t addr, int len, const void *val)
 {
-	/* TODO: implement */
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
+
+	/* Storing a value with LPIs already enabled is undefined */
+	if (!dist->lpis_enabled)
+		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
 	return 0;
 }
 
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: linux-arm-kernel, kvmarm, kvm

The ARM GICv3 ITS emulation code goes into a separate file, but
needs to be connected to the GICv3 emulation, of which it is an
option.
Introduce the skeleton with function stubs to be filled later.
Introduce the basic ITS data structure and initialize it, but don't
return any success yet, as we are not yet ready for the show.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 arch/arm64/kvm/Makefile            |  1 +
 include/kvm/vgic/vgic.h            |  6 +++
 include/linux/irqchip/arm-gic-v3.h |  1 +
 virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h           | 14 +++++++
 virt/kvm/arm/vgic/vgic_init.c      |  6 +++
 virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
 7 files changed, 126 insertions(+), 4 deletions(-)
 create mode 100644 virt/kvm/arm/vgic/its-emul.c

diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 2f5d431..3bec10e 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
 else
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index 8ea5dd7..c79bed5 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -112,6 +112,11 @@ struct vgic_io_device {
 	struct kvm_io_device dev;
 };
 
+struct vgic_its {
+	bool			enabled;
+	spinlock_t		lock;
+};
+
 struct vgic_dist {
 	bool			in_kernel;
 	bool			ready;
@@ -153,6 +158,7 @@ struct vgic_dist {
 	u64			*pendbaser;
 
 	bool			lpis_enabled;
+	struct vgic_its		its;
 };
 
 struct vgic_v2_cpu_if {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index d5d798b..a813c3e 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -177,6 +177,7 @@
 #define GITS_CWRITER			0x0088
 #define GITS_CREADR			0x0090
 #define GITS_BASER			0x0100
+#define GITS_IDREGS_BASE		0xffd0
 #define GITS_PIDR2			GICR_PIDR2
 
 #define GITS_TRANSLATER			0x10040
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
new file mode 100644
index 0000000..49dd5e4
--- /dev/null
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -0,0 +1,84 @@
+/*
+ * GICv3 ITS emulation
+ *
+ * Copyright (C) 2015 ARM Ltd.
+ * Author: Andre Przywara <andre.przywara@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/cpu.h>
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+#include <linux/interrupt.h>
+
+#include <linux/irqchip/arm-gic-v3.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_arm.h>
+#include <asm/kvm_mmu.h>
+
+#include "vgic.h"
+#include "vgic_mmio.h"
+
+struct vgic_register_region its_registers[] = {
+	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
+	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
+};
+
+/* This is called on setting the LPI enable bit in the redistributor. */
+void vgic_enable_lpis(struct kvm_vcpu *vcpu)
+{
+}
+
+int vits_init(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+	struct vgic_io_device *regions;
+	int ret, i;
+
+	spin_lock_init(&its->lock);
+
+	regions = kmalloc_array(ARRAY_SIZE(its_registers),
+				sizeof(struct vgic_io_device), GFP_KERNEL);
+
+	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
+		regions[i].base_addr = dist->vgic_its_base;
+
+		ret = kvm_vgic_register_mmio_region(kvm, NULL,
+						    &its_registers[i],
+						    &regions[i], 0, false);
+	}
+
+	if (ret)
+		return ret;
+
+	its->enabled = false;
+
+	return -ENXIO;
+}
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index db9dfd7..4e7dcb8 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
 int vgic_v3_probe(struct device_node *vgic_node);
 int vgic_v3_map_resources(struct kvm *kvm);
 int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
+
+int vits_init(struct kvm *kvm);
+void vgic_enable_lpis(struct kvm_vcpu *vcpu);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
 					       u64 mpidr)
@@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
 {
 	return -ENODEV;
 }
+
+int vits_init(struct kvm *kvm)
+{
+	return 0;
+}
+
+static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
+{
+	return;
+}
 #endif
 
 void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
@@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
 
 int vgic_lazy_init(struct kvm *kvm);
 int vgic_init(struct kvm *kvm);
+int vits_init(struct kvm *kvm);
 void kvm_register_vgic_device(unsigned long type);
 
 #endif
diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index 2301e03..dcfb93d 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
 	if (ret)
 		goto out;
 
+	if (vgic_has_its(kvm)) {
+		ret = vits_init(kvm);
+		if (ret)
+			goto out;
+	}
+
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_init(vcpu);
 
diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
index 252b9aff..086555e 100644
--- a/virt/kvm/arm/vgic/vgic_mmio.c
+++ b/virt/kvm/arm/vgic/vgic_mmio.c
@@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
 				   struct kvm_io_device *this,
 				   gpa_t addr, int len, void *val)
 {
-	/* TODO: implement for ITS support */
-	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
+
+	write_mask32(reg, addr & 3, len, val);
+	return 0;
 }
 
 static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
 				    struct kvm_io_device *this,
 				    gpa_t addr, int len, const void *val)
 {
-	/* TODO: implement for ITS support */
-	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
+
+	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
+	    (reg & GICR_CTLR_ENABLE_LPIS)) {
+		/* Eventually do something */
+	}
+
+	return 0;
 }
 
 static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
-- 
2.7.3


^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

The ARM GICv3 ITS emulation code goes into a separate file, but
needs to be connected to the GICv3 emulation, of which it is an
option.
Introduce the skeleton with function stubs to be filled later.
Introduce the basic ITS data structure and initialize it, but don't
return any success yet, as we are not yet ready for the show.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 arch/arm64/kvm/Makefile            |  1 +
 include/kvm/vgic/vgic.h            |  6 +++
 include/linux/irqchip/arm-gic-v3.h |  1 +
 virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h           | 14 +++++++
 virt/kvm/arm/vgic/vgic_init.c      |  6 +++
 virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
 7 files changed, 126 insertions(+), 4 deletions(-)
 create mode 100644 virt/kvm/arm/vgic/its-emul.c

diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 2f5d431..3bec10e 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
 else
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index 8ea5dd7..c79bed5 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -112,6 +112,11 @@ struct vgic_io_device {
 	struct kvm_io_device dev;
 };
 
+struct vgic_its {
+	bool			enabled;
+	spinlock_t		lock;
+};
+
 struct vgic_dist {
 	bool			in_kernel;
 	bool			ready;
@@ -153,6 +158,7 @@ struct vgic_dist {
 	u64			*pendbaser;
 
 	bool			lpis_enabled;
+	struct vgic_its		its;
 };
 
 struct vgic_v2_cpu_if {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index d5d798b..a813c3e 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -177,6 +177,7 @@
 #define GITS_CWRITER			0x0088
 #define GITS_CREADR			0x0090
 #define GITS_BASER			0x0100
+#define GITS_IDREGS_BASE		0xffd0
 #define GITS_PIDR2			GICR_PIDR2
 
 #define GITS_TRANSLATER			0x10040
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
new file mode 100644
index 0000000..49dd5e4
--- /dev/null
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -0,0 +1,84 @@
+/*
+ * GICv3 ITS emulation
+ *
+ * Copyright (C) 2015 ARM Ltd.
+ * Author: Andre Przywara <andre.przywara@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/cpu.h>
+#include <linux/kvm.h>
+#include <linux/kvm_host.h>
+#include <linux/interrupt.h>
+
+#include <linux/irqchip/arm-gic-v3.h>
+
+#include <asm/kvm_emulate.h>
+#include <asm/kvm_arm.h>
+#include <asm/kvm_mmu.h>
+
+#include "vgic.h"
+#include "vgic_mmio.h"
+
+struct vgic_register_region its_registers[] = {
+	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
+	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
+		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
+};
+
+/* This is called on setting the LPI enable bit in the redistributor. */
+void vgic_enable_lpis(struct kvm_vcpu *vcpu)
+{
+}
+
+int vits_init(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+	struct vgic_io_device *regions;
+	int ret, i;
+
+	spin_lock_init(&its->lock);
+
+	regions = kmalloc_array(ARRAY_SIZE(its_registers),
+				sizeof(struct vgic_io_device), GFP_KERNEL);
+
+	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
+		regions[i].base_addr = dist->vgic_its_base;
+
+		ret = kvm_vgic_register_mmio_region(kvm, NULL,
+						    &its_registers[i],
+						    &regions[i], 0, false);
+	}
+
+	if (ret)
+		return ret;
+
+	its->enabled = false;
+
+	return -ENXIO;
+}
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index db9dfd7..4e7dcb8 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
 int vgic_v3_probe(struct device_node *vgic_node);
 int vgic_v3_map_resources(struct kvm *kvm);
 int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
+
+int vits_init(struct kvm *kvm);
+void vgic_enable_lpis(struct kvm_vcpu *vcpu);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
 					       u64 mpidr)
@@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
 {
 	return -ENODEV;
 }
+
+int vits_init(struct kvm *kvm)
+{
+	return 0;
+}
+
+static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
+{
+	return;
+}
 #endif
 
 void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
@@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
 
 int vgic_lazy_init(struct kvm *kvm);
 int vgic_init(struct kvm *kvm);
+int vits_init(struct kvm *kvm);
 void kvm_register_vgic_device(unsigned long type);
 
 #endif
diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index 2301e03..dcfb93d 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
 	if (ret)
 		goto out;
 
+	if (vgic_has_its(kvm)) {
+		ret = vits_init(kvm);
+		if (ret)
+			goto out;
+	}
+
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_init(vcpu);
 
diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
index 252b9aff..086555e 100644
--- a/virt/kvm/arm/vgic/vgic_mmio.c
+++ b/virt/kvm/arm/vgic/vgic_mmio.c
@@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
 				   struct kvm_io_device *this,
 				   gpa_t addr, int len, void *val)
 {
-	/* TODO: implement for ITS support */
-	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
+
+	write_mask32(reg, addr & 3, len, val);
+	return 0;
 }
 
 static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
 				    struct kvm_io_device *this,
 				    gpa_t addr, int len, const void *val)
 {
-	/* TODO: implement for ITS support */
-	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
+
+	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
+	    (reg & GICR_CTLR_ENABLE_LPIS)) {
+		/* Eventually do something */
+	}
+
+	return 0;
 }
 
 static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: linux-arm-kernel, kvmarm, kvm

Add emulation for some basic MMIO registers used in the ITS emulation.
This includes:
- GITS_{CTLR,TYPER,IIDR}
- ID registers
- GITS_{CBASER,CREADR,CWRITER}
  those implement the ITS command buffer handling

Most of the handlers are pretty straight forward, but CWRITER goes
some extra miles to allow fine grained locking. The idea here
is to let only the first instance iterate through the command ring
buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
by that first instance and handled as well. The ITS lock is thus only
hold for very small periods of time and is dropped before the actual
command handler is called.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h            |   3 +
 include/linux/irqchip/arm-gic-v3.h |   8 ++
 virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
 virt/kvm/arm/vgic/vgic.h           |   6 +
 virt/kvm/arm/vgic/vgic_init.c      |   2 +
 5 files changed, 284 insertions(+), 7 deletions(-)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index c79bed5..bafea11 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -115,6 +115,9 @@ struct vgic_io_device {
 struct vgic_its {
 	bool			enabled;
 	spinlock_t		lock;
+	u64			cbaser;
+	int			creadr;
+	int			cwriter;
 };
 
 struct vgic_dist {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index a813c3e..7011b98 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -179,15 +179,23 @@
 #define GITS_BASER			0x0100
 #define GITS_IDREGS_BASE		0xffd0
 #define GITS_PIDR2			GICR_PIDR2
+#define GITS_PIDR4			0xffd0
+#define GITS_CIDR0			0xfff0
+#define GITS_CIDR1			0xfff4
+#define GITS_CIDR2			0xfff8
+#define GITS_CIDR3			0xfffc
 
 #define GITS_TRANSLATER			0x10040
 
 #define GITS_CTLR_ENABLE		(1U << 0)
 #define GITS_CTLR_QUIESCENT		(1U << 31)
 
+#define GITS_TYPER_PLPIS		(1UL << 0)
+#define GITS_TYPER_IDBITS_SHIFT		8
 #define GITS_TYPER_DEVBITS_SHIFT	13
 #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
 #define GITS_TYPER_PTA			(1UL << 19)
+#define GITS_TYPER_HWCOLLCNT_SHIFT	24
 
 #define GITS_CBASER_VALID		(1UL << 63)
 #define GITS_CBASER_nCnB		(0UL << 59)
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index 49dd5e4..de8d360 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -31,23 +31,263 @@
 #include "vgic.h"
 #include "vgic_mmio.h"
 
+#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
+
+static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
+				   struct kvm_io_device *this,
+				   gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	u32 reg;
+
+	reg = GITS_CTLR_QUIESCENT;
+	if (its->enabled)
+		reg |= GITS_CTLR_ENABLE;
+
+	write_mask32(reg, addr & 3, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
+				    struct kvm_io_device *this,
+				    gpa_t addr, int len, const void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+
+        if (addr - iodev->base_addr == 0)
+		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
+				    struct kvm_io_device *this,
+				    gpa_t addr, int len, void *val)
+{
+	u64 reg = GITS_TYPER_PLPIS;
+
+	/*
+	 * We use linear CPU numbers for redistributor addressing,
+	 * so GITS_TYPER.PTA is 0.
+	 * To avoid memory waste on the guest side, we keep the
+	 * number of IDBits and DevBits low for the time being.
+	 * This could later be made configurable by userland.
+	 * Since we have all collections in linked list, we claim
+	 * that we can hold all of the collection tables in our
+	 * own memory and that the ITT entry size is 1 byte (the
+	 * smallest possible one).
+	 */
+	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
+	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
+	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
+
+	write_mask64(reg, addr & 7, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
+				   struct kvm_io_device *this,
+				   gpa_t addr, int len, void *val)
+{
+	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
+
+	write_mask32(reg, addr & 3, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
+				     struct kvm_io_device *this,
+				     gpa_t addr, int len, void *val)
+{
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+	u32 reg = 0;
+	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
+
+	switch (idreg) {
+	case GITS_PIDR2:
+		reg = GIC_PIDR2_ARCH_GICv3;
+		break;
+	case GITS_PIDR4:
+		/* This is a 64K software visible page */
+		reg = 0x40;
+		break;
+	/* Those are the ID registers for (any) GIC. */
+	case GITS_CIDR0:
+		reg = 0x0d;
+		break;
+	case GITS_CIDR1:
+		reg = 0xf0;
+		break;
+	case GITS_CIDR2:
+		reg = 0x05;
+		break;
+	case GITS_CIDR3:
+		reg = 0xb1;
+		break;
+	}
+
+	write_mask32(reg, addr & 3, len, val);
+
+	return 0;
+}
+
+/*
+ * This function is called with both the ITS and the distributor lock dropped,
+ * so the actual command handlers must take the respective locks when needed.
+ */
+static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
+{
+	return -ENODEV;
+}
+
+static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
+				    struct kvm_io_device *this,
+				    gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+
+	write_mask64(its->cbaser, addr & 7, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
+				      struct kvm_io_device *this,
+				      gpa_t addr, int len, const void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+
+	if (its->enabled)
+		return 0;
+
+	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
+	its->creadr = 0;
+
+	return 0;
+}
+
+static int its_cmd_buffer_size(struct kvm *kvm)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+
+	return ((its->cbaser & 0xff) + 1) << 12;
+}
+
+static gpa_t its_cmd_buffer_base(struct kvm *kvm)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+
+	return BASER_BASE_ADDRESS(its->cbaser);
+}
+
+/*
+ * By writing to CWRITER the guest announces new commands to be processed.
+ * Since we cannot read from guest memory inside the ITS spinlock, we
+ * iterate over the command buffer (with the lock dropped) until the read
+ * pointer matches the write pointer. Other VCPUs writing this register in the
+ * meantime will just update the write pointer, leaving the command
+ * processing to the first instance of the function.
+ */
+static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
+				       struct kvm_io_device *this,
+				       gpa_t addr, int len, const void *val)
+{
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
+	u64 cmd_buf[4];
+	u32 reg;
+	bool finished;
+
+	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
+	reg &= 0xfffe0;
+	if (reg > its_cmd_buffer_size(vcpu->kvm))
+		return 0;
+
+	spin_lock(&its->lock);
+
+	/*
+	 * If there is still another VCPU handling commands, let this
+	 * one pick up the new CWRITER and process "our" new commands as well.
+	 */
+	finished = (its->cwriter != its->creadr);
+	its->cwriter = reg;
+
+	spin_unlock(&its->lock);
+
+	while (!finished) {
+		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
+					 cmd_buf, 32);
+		if (ret) {
+			/*
+			 * Gah, we are screwed. Reset CWRITER to that command
+			 * that we have finished processing and return.
+			 */
+			spin_lock(&its->lock);
+			its->cwriter = its->creadr;
+			spin_unlock(&its->lock);
+			break;
+		}
+		vits_handle_command(vcpu, cmd_buf);
+
+		spin_lock(&its->lock);
+		its->creadr += 32;
+		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
+			its->creadr = 0;
+		finished = (its->creadr == its->cwriter);
+		spin_unlock(&its->lock);
+	}
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
+				      struct kvm_io_device *this,
+				      gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	u64 reg = its->cwriter & 0xfffe0;
+
+	write_mask64(reg, addr & 7, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
+				     struct kvm_io_device *this,
+				     gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	u64 reg = its->creadr & 0xfffe0;
+
+	write_mask64(reg, addr & 7, len, val);
+
+	return 0;
+}
+
 struct vgic_register_region its_registers[] = {
 	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
 	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
 	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
 	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
 	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
 	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
 	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
 		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
 	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
+		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
 };
 
 /* This is called on setting the LPI enable bit in the redistributor. */
@@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_its *its = &dist->its;
+	int nr_vcpus = atomic_read(&kvm->online_vcpus);
 	struct vgic_io_device *regions;
 	int ret, i;
 
+	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
+	if (!dist->pendbaser)
+		return -ENOMEM;
+
 	spin_lock_init(&its->lock);
 
 	regions = kmalloc_array(ARRAY_SIZE(its_registers),
@@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
 
 	return -ENXIO;
 }
+
+void vits_destroy(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+
+	if (!vgic_has_its(kvm))
+		return;
+
+	kfree(dist->pendbaser);
+
+	its->enabled = false;
+}
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 4e7dcb8..08f97d1 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
 
 int vits_init(struct kvm *kvm);
 void vgic_enable_lpis(struct kvm_vcpu *vcpu);
+void vits_destroy(struct kvm *kvm);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
 					       u64 mpidr)
@@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
 {
 	return;
 }
+
+static inline void vits_destroy(struct kvm *kvm)
+{
+	return;
+}
 #endif
 
 void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index dcfb93d..e4459e3 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
 
 	kvm_vgic_dist_destroy(kvm);
 
+	vits_destroy(kvm);
+
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_destroy(vcpu);
 }
-- 
2.7.3


^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

Add emulation for some basic MMIO registers used in the ITS emulation.
This includes:
- GITS_{CTLR,TYPER,IIDR}
- ID registers
- GITS_{CBASER,CREADR,CWRITER}
  those implement the ITS command buffer handling

Most of the handlers are pretty straight forward, but CWRITER goes
some extra miles to allow fine grained locking. The idea here
is to let only the first instance iterate through the command ring
buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
by that first instance and handled as well. The ITS lock is thus only
hold for very small periods of time and is dropped before the actual
command handler is called.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h            |   3 +
 include/linux/irqchip/arm-gic-v3.h |   8 ++
 virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
 virt/kvm/arm/vgic/vgic.h           |   6 +
 virt/kvm/arm/vgic/vgic_init.c      |   2 +
 5 files changed, 284 insertions(+), 7 deletions(-)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index c79bed5..bafea11 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -115,6 +115,9 @@ struct vgic_io_device {
 struct vgic_its {
 	bool			enabled;
 	spinlock_t		lock;
+	u64			cbaser;
+	int			creadr;
+	int			cwriter;
 };
 
 struct vgic_dist {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index a813c3e..7011b98 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -179,15 +179,23 @@
 #define GITS_BASER			0x0100
 #define GITS_IDREGS_BASE		0xffd0
 #define GITS_PIDR2			GICR_PIDR2
+#define GITS_PIDR4			0xffd0
+#define GITS_CIDR0			0xfff0
+#define GITS_CIDR1			0xfff4
+#define GITS_CIDR2			0xfff8
+#define GITS_CIDR3			0xfffc
 
 #define GITS_TRANSLATER			0x10040
 
 #define GITS_CTLR_ENABLE		(1U << 0)
 #define GITS_CTLR_QUIESCENT		(1U << 31)
 
+#define GITS_TYPER_PLPIS		(1UL << 0)
+#define GITS_TYPER_IDBITS_SHIFT		8
 #define GITS_TYPER_DEVBITS_SHIFT	13
 #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
 #define GITS_TYPER_PTA			(1UL << 19)
+#define GITS_TYPER_HWCOLLCNT_SHIFT	24
 
 #define GITS_CBASER_VALID		(1UL << 63)
 #define GITS_CBASER_nCnB		(0UL << 59)
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index 49dd5e4..de8d360 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -31,23 +31,263 @@
 #include "vgic.h"
 #include "vgic_mmio.h"
 
+#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
+
+static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
+				   struct kvm_io_device *this,
+				   gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	u32 reg;
+
+	reg = GITS_CTLR_QUIESCENT;
+	if (its->enabled)
+		reg |= GITS_CTLR_ENABLE;
+
+	write_mask32(reg, addr & 3, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
+				    struct kvm_io_device *this,
+				    gpa_t addr, int len, const void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+
+        if (addr - iodev->base_addr == 0)
+		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
+				    struct kvm_io_device *this,
+				    gpa_t addr, int len, void *val)
+{
+	u64 reg = GITS_TYPER_PLPIS;
+
+	/*
+	 * We use linear CPU numbers for redistributor addressing,
+	 * so GITS_TYPER.PTA is 0.
+	 * To avoid memory waste on the guest side, we keep the
+	 * number of IDBits and DevBits low for the time being.
+	 * This could later be made configurable by userland.
+	 * Since we have all collections in linked list, we claim
+	 * that we can hold all of the collection tables in our
+	 * own memory and that the ITT entry size is 1 byte (the
+	 * smallest possible one).
+	 */
+	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
+	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
+	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
+
+	write_mask64(reg, addr & 7, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
+				   struct kvm_io_device *this,
+				   gpa_t addr, int len, void *val)
+{
+	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
+
+	write_mask32(reg, addr & 3, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
+				     struct kvm_io_device *this,
+				     gpa_t addr, int len, void *val)
+{
+	struct vgic_io_device *iodev = container_of(this,
+						    struct vgic_io_device, dev);
+	u32 reg = 0;
+	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
+
+	switch (idreg) {
+	case GITS_PIDR2:
+		reg = GIC_PIDR2_ARCH_GICv3;
+		break;
+	case GITS_PIDR4:
+		/* This is a 64K software visible page */
+		reg = 0x40;
+		break;
+	/* Those are the ID registers for (any) GIC. */
+	case GITS_CIDR0:
+		reg = 0x0d;
+		break;
+	case GITS_CIDR1:
+		reg = 0xf0;
+		break;
+	case GITS_CIDR2:
+		reg = 0x05;
+		break;
+	case GITS_CIDR3:
+		reg = 0xb1;
+		break;
+	}
+
+	write_mask32(reg, addr & 3, len, val);
+
+	return 0;
+}
+
+/*
+ * This function is called with both the ITS and the distributor lock dropped,
+ * so the actual command handlers must take the respective locks when needed.
+ */
+static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
+{
+	return -ENODEV;
+}
+
+static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
+				    struct kvm_io_device *this,
+				    gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+
+	write_mask64(its->cbaser, addr & 7, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
+				      struct kvm_io_device *this,
+				      gpa_t addr, int len, const void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+
+	if (its->enabled)
+		return 0;
+
+	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
+	its->creadr = 0;
+
+	return 0;
+}
+
+static int its_cmd_buffer_size(struct kvm *kvm)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+
+	return ((its->cbaser & 0xff) + 1) << 12;
+}
+
+static gpa_t its_cmd_buffer_base(struct kvm *kvm)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+
+	return BASER_BASE_ADDRESS(its->cbaser);
+}
+
+/*
+ * By writing to CWRITER the guest announces new commands to be processed.
+ * Since we cannot read from guest memory inside the ITS spinlock, we
+ * iterate over the command buffer (with the lock dropped) until the read
+ * pointer matches the write pointer. Other VCPUs writing this register in the
+ * meantime will just update the write pointer, leaving the command
+ * processing to the first instance of the function.
+ */
+static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
+				       struct kvm_io_device *this,
+				       gpa_t addr, int len, const void *val)
+{
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
+	u64 cmd_buf[4];
+	u32 reg;
+	bool finished;
+
+	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
+	reg &= 0xfffe0;
+	if (reg > its_cmd_buffer_size(vcpu->kvm))
+		return 0;
+
+	spin_lock(&its->lock);
+
+	/*
+	 * If there is still another VCPU handling commands, let this
+	 * one pick up the new CWRITER and process "our" new commands as well.
+	 */
+	finished = (its->cwriter != its->creadr);
+	its->cwriter = reg;
+
+	spin_unlock(&its->lock);
+
+	while (!finished) {
+		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
+					 cmd_buf, 32);
+		if (ret) {
+			/*
+			 * Gah, we are screwed. Reset CWRITER to that command
+			 * that we have finished processing and return.
+			 */
+			spin_lock(&its->lock);
+			its->cwriter = its->creadr;
+			spin_unlock(&its->lock);
+			break;
+		}
+		vits_handle_command(vcpu, cmd_buf);
+
+		spin_lock(&its->lock);
+		its->creadr += 32;
+		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
+			its->creadr = 0;
+		finished = (its->creadr == its->cwriter);
+		spin_unlock(&its->lock);
+	}
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
+				      struct kvm_io_device *this,
+				      gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	u64 reg = its->cwriter & 0xfffe0;
+
+	write_mask64(reg, addr & 7, len, val);
+
+	return 0;
+}
+
+static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
+				     struct kvm_io_device *this,
+				     gpa_t addr, int len, void *val)
+{
+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
+	u64 reg = its->creadr & 0xfffe0;
+
+	write_mask64(reg, addr & 7, len, val);
+
+	return 0;
+}
+
 struct vgic_register_region its_registers[] = {
 	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
 	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
 	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
+		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
 	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
 	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
 	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
+		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
 	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
 		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
 	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
-		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
+		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
 };
 
 /* This is called on setting the LPI enable bit in the redistributor. */
@@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_its *its = &dist->its;
+	int nr_vcpus = atomic_read(&kvm->online_vcpus);
 	struct vgic_io_device *regions;
 	int ret, i;
 
+	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
+	if (!dist->pendbaser)
+		return -ENOMEM;
+
 	spin_lock_init(&its->lock);
 
 	regions = kmalloc_array(ARRAY_SIZE(its_registers),
@@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
 
 	return -ENXIO;
 }
+
+void vits_destroy(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+
+	if (!vgic_has_its(kvm))
+		return;
+
+	kfree(dist->pendbaser);
+
+	its->enabled = false;
+}
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 4e7dcb8..08f97d1 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
 
 int vits_init(struct kvm *kvm);
 void vgic_enable_lpis(struct kvm_vcpu *vcpu);
+void vits_destroy(struct kvm *kvm);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
 					       u64 mpidr)
@@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
 {
 	return;
 }
+
+static inline void vits_destroy(struct kvm *kvm)
+{
+	return;
+}
 #endif
 
 void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
index dcfb93d..e4459e3 100644
--- a/virt/kvm/arm/vgic/vgic_init.c
+++ b/virt/kvm/arm/vgic/vgic_init.c
@@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
 
 	kvm_vgic_dist_destroy(kvm);
 
+	vits_destroy(kvm);
+
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_destroy(vcpu);
 }
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 07/12] KVM: arm64: add data structures to model ITS interrupt translation
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

The GICv3 Interrupt Translation Service (ITS) uses tables in memory
to allow a sophisticated interrupt routing. It features device tables,
an interrupt table per device and a table connecting "collections" to
actual CPUs (aka. redistributors in the GICv3 lingo).
Since the interrupt numbers for the LPIs are allocated quite sparsely
and the range can be quite huge (8192 LPIs being the minimum), using
bitmaps or arrays for storing information is a waste of memory.
We use linked lists instead, which we iterate linearily. This works
very well with the actual number of LPIs/MSIs in the guest being
quite low. Should the number of LPIs exceed the number where iterating
through lists seems acceptable, we can later revisit this and use more
efficient data structures.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h      |  3 ++
 virt/kvm/arm/vgic/its-emul.c | 66 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index bafea11..ecf3260 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -22,6 +22,7 @@
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <kvm/iodev.h>
+#include <linux/list.h>
 
 #define VGIC_V3_MAX_CPUS	255
 #define VGIC_V2_MAX_CPUS	8
@@ -118,6 +119,8 @@ struct vgic_its {
 	u64			cbaser;
 	int			creadr;
 	int			cwriter;
+	struct list_head	device_list;
+	struct list_head	collection_list;
 };
 
 struct vgic_dist {
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index de8d360..c0334ff 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -21,6 +21,7 @@
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <linux/interrupt.h>
+#include <linux/list.h>
 
 #include <linux/irqchip/arm-gic-v3.h>
 
@@ -31,6 +32,34 @@
 #include "vgic.h"
 #include "vgic_mmio.h"
 
+struct its_device {
+	struct list_head dev_list;
+
+	/* the head for the list of ITTEs */
+	struct list_head itt;
+	u32 device_id;
+};
+
+#define COLLECTION_NOT_MAPPED ((u32)-1)
+
+struct its_collection {
+	struct list_head coll_list;
+
+	u32 collection_id;
+	u32 target_addr;
+};
+
+#define its_is_collection_mapped(coll) ((coll) && \
+				((coll)->target_addr != COLLECTION_NOT_MAPPED))
+
+struct its_itte {
+	struct list_head itte_list;
+
+	struct its_collection *collection;
+	u32 lpi;
+	u32 event_id;
+};
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
 
 static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
@@ -137,6 +166,12 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static void its_free_itte(struct its_itte *itte)
+{
+	list_del(&itte->itte_list);
+	kfree(itte);
+}
+
 /*
  * This function is called with both the ITS and the distributor lock dropped,
  * so the actual command handlers must take the respective locks when needed.
@@ -309,6 +344,9 @@ int vits_init(struct kvm *kvm)
 
 	spin_lock_init(&its->lock);
 
+	INIT_LIST_HEAD(&its->device_list);
+	INIT_LIST_HEAD(&its->collection_list);
+
 	regions = kmalloc_array(ARRAY_SIZE(its_registers),
 				sizeof(struct vgic_io_device), GFP_KERNEL);
 
@@ -332,11 +370,39 @@ void vits_destroy(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_its *its = &dist->its;
+	struct its_device *dev;
+	struct its_itte *itte;
+	struct list_head *dev_cur, *dev_temp;
+	struct list_head *cur, *temp;
 
 	if (!vgic_has_its(kvm))
 		return;
 
+	/*
+	 * We may end up here without the lists ever having been initialized.
+	 * Check this and bail out early to avoid dereferencing a NULL pointer.
+	 */
+	if (!its->device_list.next)
+		return;
+
+	spin_lock(&its->lock);
+	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
+		dev = container_of(dev_cur, struct its_device, dev_list);
+		list_for_each_safe(cur, temp, &dev->itt) {
+			itte = (container_of(cur, struct its_itte, itte_list));
+			its_free_itte(itte);
+		}
+		list_del(dev_cur);
+		kfree(dev);
+	}
+
+	list_for_each_safe(cur, temp, &its->collection_list) {
+		list_del(cur);
+		kfree(container_of(cur, struct its_collection, coll_list));
+	}
+
 	kfree(dist->pendbaser);
 
 	its->enabled = false;
+	spin_unlock(&its->lock);
 }
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 07/12] KVM: arm64: add data structures to model ITS interrupt translation
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

The GICv3 Interrupt Translation Service (ITS) uses tables in memory
to allow a sophisticated interrupt routing. It features device tables,
an interrupt table per device and a table connecting "collections" to
actual CPUs (aka. redistributors in the GICv3 lingo).
Since the interrupt numbers for the LPIs are allocated quite sparsely
and the range can be quite huge (8192 LPIs being the minimum), using
bitmaps or arrays for storing information is a waste of memory.
We use linked lists instead, which we iterate linearily. This works
very well with the actual number of LPIs/MSIs in the guest being
quite low. Should the number of LPIs exceed the number where iterating
through lists seems acceptable, we can later revisit this and use more
efficient data structures.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h      |  3 ++
 virt/kvm/arm/vgic/its-emul.c | 66 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 69 insertions(+)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index bafea11..ecf3260 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -22,6 +22,7 @@
 #include <linux/spinlock.h>
 #include <linux/types.h>
 #include <kvm/iodev.h>
+#include <linux/list.h>
 
 #define VGIC_V3_MAX_CPUS	255
 #define VGIC_V2_MAX_CPUS	8
@@ -118,6 +119,8 @@ struct vgic_its {
 	u64			cbaser;
 	int			creadr;
 	int			cwriter;
+	struct list_head	device_list;
+	struct list_head	collection_list;
 };
 
 struct vgic_dist {
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index de8d360..c0334ff 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -21,6 +21,7 @@
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <linux/interrupt.h>
+#include <linux/list.h>
 
 #include <linux/irqchip/arm-gic-v3.h>
 
@@ -31,6 +32,34 @@
 #include "vgic.h"
 #include "vgic_mmio.h"
 
+struct its_device {
+	struct list_head dev_list;
+
+	/* the head for the list of ITTEs */
+	struct list_head itt;
+	u32 device_id;
+};
+
+#define COLLECTION_NOT_MAPPED ((u32)-1)
+
+struct its_collection {
+	struct list_head coll_list;
+
+	u32 collection_id;
+	u32 target_addr;
+};
+
+#define its_is_collection_mapped(coll) ((coll) && \
+				((coll)->target_addr != COLLECTION_NOT_MAPPED))
+
+struct its_itte {
+	struct list_head itte_list;
+
+	struct its_collection *collection;
+	u32 lpi;
+	u32 event_id;
+};
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
 
 static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
@@ -137,6 +166,12 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+static void its_free_itte(struct its_itte *itte)
+{
+	list_del(&itte->itte_list);
+	kfree(itte);
+}
+
 /*
  * This function is called with both the ITS and the distributor lock dropped,
  * so the actual command handlers must take the respective locks when needed.
@@ -309,6 +344,9 @@ int vits_init(struct kvm *kvm)
 
 	spin_lock_init(&its->lock);
 
+	INIT_LIST_HEAD(&its->device_list);
+	INIT_LIST_HEAD(&its->collection_list);
+
 	regions = kmalloc_array(ARRAY_SIZE(its_registers),
 				sizeof(struct vgic_io_device), GFP_KERNEL);
 
@@ -332,11 +370,39 @@ void vits_destroy(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
 	struct vgic_its *its = &dist->its;
+	struct its_device *dev;
+	struct its_itte *itte;
+	struct list_head *dev_cur, *dev_temp;
+	struct list_head *cur, *temp;
 
 	if (!vgic_has_its(kvm))
 		return;
 
+	/*
+	 * We may end up here without the lists ever having been initialized.
+	 * Check this and bail out early to avoid dereferencing a NULL pointer.
+	 */
+	if (!its->device_list.next)
+		return;
+
+	spin_lock(&its->lock);
+	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
+		dev = container_of(dev_cur, struct its_device, dev_list);
+		list_for_each_safe(cur, temp, &dev->itt) {
+			itte = (container_of(cur, struct its_itte, itte_list));
+			its_free_itte(itte);
+		}
+		list_del(dev_cur);
+		kfree(dev);
+	}
+
+	list_for_each_safe(cur, temp, &its->collection_list) {
+		list_del(cur);
+		kfree(container_of(cur, struct its_collection, coll_list));
+	}
+
 	kfree(dist->pendbaser);
 
 	its->enabled = false;
+	spin_unlock(&its->lock);
 }
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

LPIs are dynamically created (mapped) at guest runtime and their
actual numbers can be quite high, but is mostly assigned using a very
sparse allocation scheme. So arrays are not an ideal data structure
to hold the information. We use our equivalent of the "Interrupt
Translation Table Entry" (ITTE) to hold the vgic_irq struct for a
virtual LPI embedded in in the ITTE.
Connect the VGIC core code via an accessor function to help it get the
struct vgic_irq for a certain LPI.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 virt/kvm/arm/vgic/its-emul.c | 29 +++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  8 +++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index c0334ff..1188e9a 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -55,11 +55,29 @@ struct its_collection {
 struct its_itte {
 	struct list_head itte_list;
 
+	struct vgic_irq irq;
 	struct its_collection *collection;
 	u32 lpi;
 	u32 event_id;
 };
 
+/* To be used as an iterator this macro misses the enclosing parentheses */
+#define for_each_lpi(dev, itte, kvm) \
+	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
+		list_for_each_entry(itte, &(dev)->itt, itte_list)
+
+static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
+{
+	struct its_device *device;
+	struct its_itte *itte;
+
+	for_each_lpi(device, itte, kvm) {
+		if (itte->lpi == lpi)
+			return itte;
+	}
+	return NULL;
+}
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
 
 static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
@@ -166,6 +184,17 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
+{
+	struct its_itte *itte;
+
+	itte = find_itte_by_lpi(kvm, intid);
+	if (!itte)
+		return NULL;
+
+	return &itte->irq;
+}
+
 static void its_free_itte(struct its_itte *itte)
 {
 	list_del(&itte->itte_list);
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 08f97d1..160c511 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
 
 int vits_init(struct kvm *kvm);
 void vgic_enable_lpis(struct kvm_vcpu *vcpu);
+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
 void vits_destroy(struct kvm *kvm);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
@@ -129,7 +130,7 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
 	return -ENODEV;
 }
 
-int vits_init(struct kvm *kvm)
+static inline int vits_init(struct kvm *kvm)
 {
 	return 0;
 }
@@ -139,6 +140,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
 	return;
 }
 
+static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
+{
+	return NULL;
+}
+
 static inline void vits_destroy(struct kvm *kvm)
 {
 	return;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

LPIs are dynamically created (mapped) at guest runtime and their
actual numbers can be quite high, but is mostly assigned using a very
sparse allocation scheme. So arrays are not an ideal data structure
to hold the information. We use our equivalent of the "Interrupt
Translation Table Entry" (ITTE) to hold the vgic_irq struct for a
virtual LPI embedded in in the ITTE.
Connect the VGIC core code via an accessor function to help it get the
struct vgic_irq for a certain LPI.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 virt/kvm/arm/vgic/its-emul.c | 29 +++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  8 +++++++-
 2 files changed, 36 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index c0334ff..1188e9a 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -55,11 +55,29 @@ struct its_collection {
 struct its_itte {
 	struct list_head itte_list;
 
+	struct vgic_irq irq;
 	struct its_collection *collection;
 	u32 lpi;
 	u32 event_id;
 };
 
+/* To be used as an iterator this macro misses the enclosing parentheses */
+#define for_each_lpi(dev, itte, kvm) \
+	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
+		list_for_each_entry(itte, &(dev)->itt, itte_list)
+
+static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
+{
+	struct its_device *device;
+	struct its_itte *itte;
+
+	for_each_lpi(device, itte, kvm) {
+		if (itte->lpi == lpi)
+			return itte;
+	}
+	return NULL;
+}
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
 
 static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
@@ -166,6 +184,17 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
+{
+	struct its_itte *itte;
+
+	itte = find_itte_by_lpi(kvm, intid);
+	if (!itte)
+		return NULL;
+
+	return &itte->irq;
+}
+
 static void its_free_itte(struct its_itte *itte)
 {
 	list_del(&itte->itte_list);
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 08f97d1..160c511 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
 
 int vits_init(struct kvm *kvm);
 void vgic_enable_lpis(struct kvm_vcpu *vcpu);
+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
 void vits_destroy(struct kvm *kvm);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
@@ -129,7 +130,7 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
 	return -ENODEV;
 }
 
-int vits_init(struct kvm *kvm)
+static inline int vits_init(struct kvm *kvm)
 {
 	return 0;
 }
@@ -139,6 +140,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
 	return;
 }
 
+static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
+{
+	return NULL;
+}
+
 static inline void vits_destroy(struct kvm *kvm)
 {
 	return;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: linux-arm-kernel, kvmarm, kvm

The LPI configuration and pending tables of the GICv3 LPIs are held
in tables in (guest) memory. To achieve reasonable performance, we
cache this data in our own data structures, so we need to sync those
two views from time to time. This behaviour is well described in the
GICv3 spec and is also exercised by hardware, so the sync points are
well known.

Provide functions that read the guest memory and store the
information from the configuration and pending tables in the kernel.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h      |   2 +
 virt/kvm/arm/vgic/its-emul.c | 135 +++++++++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |   1 +
 3 files changed, 138 insertions(+)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index ecf3260..ea98133 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -121,6 +121,8 @@ struct vgic_its {
 	int			cwriter;
 	struct list_head	device_list;
 	struct list_head	collection_list;
+	/* memory used for buffering guest's memory */
+	void			*buffer_page;
 };
 
 struct vgic_dist {
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index 1188e9a..d82ba9b 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -78,8 +78,130 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
 	return NULL;
 }
 
+#define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
+#define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
+
+/* stores the priority and enable bit for a given LPI */
+static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
+{
+	spin_lock(&itte->irq.irq_lock);
+	itte->irq.priority = LPI_PROP_PRIORITY(prop);
+	itte->irq.enabled = LPI_PROP_ENABLE_BIT(prop);
+
+	vgic_queue_irq(kvm, &itte->irq);
+}
+
+#define GIC_LPI_OFFSET 8192
+
+/* We scan the table in chunks the size of the smallest page size */
+#define CHUNK_SIZE 4096U
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
 
+static int nr_idbits_propbase(u64 propbaser)
+{
+	int nr_idbits = (1U << (propbaser & 0x1f)) + 1;
+
+	return max(nr_idbits, INTERRUPT_ID_BITS_ITS);
+}
+
+/*
+ * Scan the whole LPI configuration table and put the LPI configuration
+ * data in our own data structures. This relies on the LPI being
+ * mapped before.
+ */
+static bool its_update_lpis_configuration(struct kvm *kvm, u64 prop_base_reg)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u8 *prop = dist->its.buffer_page;
+	u32 tsize;
+	gpa_t propbase;
+	int lpi = GIC_LPI_OFFSET;
+	struct its_itte *itte;
+	struct its_device *device;
+	int ret;
+
+	propbase = BASER_BASE_ADDRESS(prop_base_reg);
+	tsize = nr_idbits_propbase(prop_base_reg);
+
+	while (tsize > 0) {
+		int chunksize = min(tsize, CHUNK_SIZE);
+
+		ret = kvm_read_guest(kvm, propbase, prop, chunksize);
+		if (ret)
+			return false;
+
+		spin_lock(&dist->its.lock);
+		/*
+		 * Updating the status for all allocated LPIs. We catch
+		 * those LPIs that get disabled. We really don't care
+		 * about unmapped LPIs, as they need to be updated
+		 * later manually anyway once they get mapped.
+		 */
+		for_each_lpi(device, itte, kvm) {
+			if (itte->lpi < lpi || itte->lpi >= lpi + chunksize)
+				continue;
+
+			update_lpi_config(kvm, itte, prop[itte->lpi - lpi]);
+		}
+		spin_unlock(&dist->its.lock);
+		tsize -= chunksize;
+		lpi += chunksize;
+		propbase += chunksize;
+	}
+
+	return true;
+}
+
+/*
+ * Scan the whole LPI pending table and sync the pending bit in there
+ * with our own data structures. This relies on the LPI being
+ * mapped before.
+ */
+static bool its_sync_lpi_pending_table(struct kvm_vcpu *vcpu, u64 base_addr_reg)
+{
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	unsigned long *pendmask = dist->its.buffer_page;
+	u32 nr_lpis = 1U << INTERRUPT_ID_BITS_ITS;
+	gpa_t pendbase;
+	int lpi = 0;
+	struct its_itte *itte;
+	struct its_device *device;
+	int ret;
+	int lpi_bit, nr_bits;
+
+	pendbase = BASER_BASE_ADDRESS(base_addr_reg);
+
+	while (nr_lpis > 0) {
+		nr_bits = min(nr_lpis, CHUNK_SIZE * 8);
+
+		ret = kvm_read_guest(vcpu->kvm, pendbase, pendmask,
+				     nr_bits / 8);
+		if (ret)
+			return false;
+
+		spin_lock(&dist->its.lock);
+		for_each_lpi(device, itte, vcpu->kvm) {
+			lpi_bit = itte->lpi - lpi;
+			if (lpi_bit < 0 || lpi_bit >= nr_bits)
+				continue;
+
+			if (!test_bit(lpi_bit, pendmask))
+				continue;
+
+			spin_lock(&itte->irq.irq_lock);
+			itte->irq.pending = true;
+			vgic_queue_irq(vcpu->kvm, &itte->irq);
+		}
+		spin_unlock(&dist->its.lock);
+		nr_lpis -= nr_bits;
+		lpi += nr_bits;
+		pendbase += nr_bits / 8;
+	}
+
+	return true;
+}
+
 static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
 				   struct kvm_io_device *this,
 				   gpa_t addr, int len, void *val)
@@ -357,6 +479,14 @@ struct vgic_register_region its_registers[] = {
 /* This is called on setting the LPI enable bit in the redistributor. */
 void vgic_enable_lpis(struct kvm_vcpu *vcpu)
 {
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	u64 prop_base_reg, pend_base_reg;
+
+	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
+	prop_base_reg = dist->propbaser;
+
+	its_update_lpis_configuration(vcpu->kvm, prop_base_reg);
+	its_sync_lpi_pending_table(vcpu, pend_base_reg);
 }
 
 int vits_init(struct kvm *kvm)
@@ -371,6 +501,10 @@ int vits_init(struct kvm *kvm)
 	if (!dist->pendbaser)
 		return -ENOMEM;
 
+	its->buffer_page = kmalloc(CHUNK_SIZE, GFP_KERNEL);
+	if (!its->buffer_page)
+		return -ENOMEM;
+
 	spin_lock_init(&its->lock);
 
 	INIT_LIST_HEAD(&its->device_list);
@@ -430,6 +564,7 @@ void vits_destroy(struct kvm *kvm)
 		kfree(container_of(cur, struct its_collection, coll_list));
 	}
 
+	kfree(its->buffer_page);
 	kfree(dist->pendbaser);
 
 	its->enabled = false;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 160c511..a7218b0 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -23,6 +23,7 @@
 #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
 
 #define INTERRUPT_ID_BITS_SPIS	10
+#define INTERRUPT_ID_BITS_ITS	16
 
 struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
 			      u32 intid);
-- 
2.7.3


^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

The LPI configuration and pending tables of the GICv3 LPIs are held
in tables in (guest) memory. To achieve reasonable performance, we
cache this data in our own data structures, so we need to sync those
two views from time to time. This behaviour is well described in the
GICv3 spec and is also exercised by hardware, so the sync points are
well known.

Provide functions that read the guest memory and store the
information from the configuration and pending tables in the kernel.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/kvm/vgic/vgic.h      |   2 +
 virt/kvm/arm/vgic/its-emul.c | 135 +++++++++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |   1 +
 3 files changed, 138 insertions(+)

diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index ecf3260..ea98133 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -121,6 +121,8 @@ struct vgic_its {
 	int			cwriter;
 	struct list_head	device_list;
 	struct list_head	collection_list;
+	/* memory used for buffering guest's memory */
+	void			*buffer_page;
 };
 
 struct vgic_dist {
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index 1188e9a..d82ba9b 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -78,8 +78,130 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
 	return NULL;
 }
 
+#define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
+#define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
+
+/* stores the priority and enable bit for a given LPI */
+static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
+{
+	spin_lock(&itte->irq.irq_lock);
+	itte->irq.priority = LPI_PROP_PRIORITY(prop);
+	itte->irq.enabled = LPI_PROP_ENABLE_BIT(prop);
+
+	vgic_queue_irq(kvm, &itte->irq);
+}
+
+#define GIC_LPI_OFFSET 8192
+
+/* We scan the table in chunks the size of the smallest page size */
+#define CHUNK_SIZE 4096U
+
 #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
 
+static int nr_idbits_propbase(u64 propbaser)
+{
+	int nr_idbits = (1U << (propbaser & 0x1f)) + 1;
+
+	return max(nr_idbits, INTERRUPT_ID_BITS_ITS);
+}
+
+/*
+ * Scan the whole LPI configuration table and put the LPI configuration
+ * data in our own data structures. This relies on the LPI being
+ * mapped before.
+ */
+static bool its_update_lpis_configuration(struct kvm *kvm, u64 prop_base_reg)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u8 *prop = dist->its.buffer_page;
+	u32 tsize;
+	gpa_t propbase;
+	int lpi = GIC_LPI_OFFSET;
+	struct its_itte *itte;
+	struct its_device *device;
+	int ret;
+
+	propbase = BASER_BASE_ADDRESS(prop_base_reg);
+	tsize = nr_idbits_propbase(prop_base_reg);
+
+	while (tsize > 0) {
+		int chunksize = min(tsize, CHUNK_SIZE);
+
+		ret = kvm_read_guest(kvm, propbase, prop, chunksize);
+		if (ret)
+			return false;
+
+		spin_lock(&dist->its.lock);
+		/*
+		 * Updating the status for all allocated LPIs. We catch
+		 * those LPIs that get disabled. We really don't care
+		 * about unmapped LPIs, as they need to be updated
+		 * later manually anyway once they get mapped.
+		 */
+		for_each_lpi(device, itte, kvm) {
+			if (itte->lpi < lpi || itte->lpi >= lpi + chunksize)
+				continue;
+
+			update_lpi_config(kvm, itte, prop[itte->lpi - lpi]);
+		}
+		spin_unlock(&dist->its.lock);
+		tsize -= chunksize;
+		lpi += chunksize;
+		propbase += chunksize;
+	}
+
+	return true;
+}
+
+/*
+ * Scan the whole LPI pending table and sync the pending bit in there
+ * with our own data structures. This relies on the LPI being
+ * mapped before.
+ */
+static bool its_sync_lpi_pending_table(struct kvm_vcpu *vcpu, u64 base_addr_reg)
+{
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	unsigned long *pendmask = dist->its.buffer_page;
+	u32 nr_lpis = 1U << INTERRUPT_ID_BITS_ITS;
+	gpa_t pendbase;
+	int lpi = 0;
+	struct its_itte *itte;
+	struct its_device *device;
+	int ret;
+	int lpi_bit, nr_bits;
+
+	pendbase = BASER_BASE_ADDRESS(base_addr_reg);
+
+	while (nr_lpis > 0) {
+		nr_bits = min(nr_lpis, CHUNK_SIZE * 8);
+
+		ret = kvm_read_guest(vcpu->kvm, pendbase, pendmask,
+				     nr_bits / 8);
+		if (ret)
+			return false;
+
+		spin_lock(&dist->its.lock);
+		for_each_lpi(device, itte, vcpu->kvm) {
+			lpi_bit = itte->lpi - lpi;
+			if (lpi_bit < 0 || lpi_bit >= nr_bits)
+				continue;
+
+			if (!test_bit(lpi_bit, pendmask))
+				continue;
+
+			spin_lock(&itte->irq.irq_lock);
+			itte->irq.pending = true;
+			vgic_queue_irq(vcpu->kvm, &itte->irq);
+		}
+		spin_unlock(&dist->its.lock);
+		nr_lpis -= nr_bits;
+		lpi += nr_bits;
+		pendbase += nr_bits / 8;
+	}
+
+	return true;
+}
+
 static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
 				   struct kvm_io_device *this,
 				   gpa_t addr, int len, void *val)
@@ -357,6 +479,14 @@ struct vgic_register_region its_registers[] = {
 /* This is called on setting the LPI enable bit in the redistributor. */
 void vgic_enable_lpis(struct kvm_vcpu *vcpu)
 {
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+	u64 prop_base_reg, pend_base_reg;
+
+	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
+	prop_base_reg = dist->propbaser;
+
+	its_update_lpis_configuration(vcpu->kvm, prop_base_reg);
+	its_sync_lpi_pending_table(vcpu, pend_base_reg);
 }
 
 int vits_init(struct kvm *kvm)
@@ -371,6 +501,10 @@ int vits_init(struct kvm *kvm)
 	if (!dist->pendbaser)
 		return -ENOMEM;
 
+	its->buffer_page = kmalloc(CHUNK_SIZE, GFP_KERNEL);
+	if (!its->buffer_page)
+		return -ENOMEM;
+
 	spin_lock_init(&its->lock);
 
 	INIT_LIST_HEAD(&its->device_list);
@@ -430,6 +564,7 @@ void vits_destroy(struct kvm *kvm)
 		kfree(container_of(cur, struct its_collection, coll_list));
 	}
 
+	kfree(its->buffer_page);
 	kfree(dist->pendbaser);
 
 	its->enabled = false;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index 160c511..a7218b0 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -23,6 +23,7 @@
 #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
 
 #define INTERRUPT_ID_BITS_SPIS	10
+#define INTERRUPT_ID_BITS_ITS	16
 
 struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
 			      u32 intid);
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

The connection between a device, an event ID, the LPI number and the
allocated CPU is stored in in-memory tables in a GICv3, but their
format is not specified by the spec. Instead software uses a command
queue in a ring buffer to let the ITS implementation use their own
format.
Implement handlers for the various ITS commands and let them store
the requested relation into our own data structures.
To avoid kmallocs inside the ITS spinlock, we preallocate possibly
needed memory outside of the lock and free that if it turns out to
be not needed (mostly error handling).
Error handling is very basic at this point, as we don't have a good
way of communicating errors to the guest (usually a SError).
The INT command handler is missing at this point, as we gain the
capability of actually injecting MSIs into the guest only later on.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/linux/irqchip/arm-gic-v3.h |  19 +-
 virt/kvm/arm/vgic/its-emul.c       | 514 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 531 insertions(+), 2 deletions(-)

diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 7011b98..c9aa8d8 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -254,7 +254,10 @@
  */
 #define GITS_CMD_MAPD			0x08
 #define GITS_CMD_MAPC			0x09
-#define GITS_CMD_MAPVI			0x0a
+#define GITS_CMD_MAPTI			0x0a
+/* older GIC documentation used MAPVI for this command */
+#define GITS_CMD_MAPVI			GITS_CMD_MAPTI
+#define GITS_CMD_MAPI			0x0b
 #define GITS_CMD_MOVI			0x01
 #define GITS_CMD_DISCARD		0x0f
 #define GITS_CMD_INV			0x0c
@@ -265,6 +268,20 @@
 #define GITS_CMD_SYNC			0x05
 
 /*
+ * ITS error numbers
+ */
+#define E_ITS_MOVI_UNMAPPED_INTERRUPT           0x010107
+#define E_ITS_MOVI_UNMAPPED_COLLECTION          0x010109
+#define E_ITS_CLEAR_UNMAPPED_INTERRUPT          0x010507
+#define E_ITS_MAPC_PROCNUM_OOR                  0x010902
+#define E_ITS_MAPTI_UNMAPPED_DEVICE             0x010a04
+#define E_ITS_MAPTI_PHYSICALID_OOR              0x010a06
+#define E_ITS_INV_UNMAPPED_INTERRUPT            0x010c07
+#define E_ITS_INVALL_UNMAPPED_COLLECTION        0x010d09
+#define E_ITS_MOVALL_PROCNUM_OOR                0x010e01
+#define E_ITS_DISCARD_UNMAPPED_INTERRUPT        0x010f07
+
+/*
  * CPU interface registers
  */
 #define ICC_CTLR_EL1_EOImode_drop_dir	(0U << 1)
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index d82ba9b..dcd0cac 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -22,6 +22,7 @@
 #include <linux/kvm_host.h>
 #include <linux/interrupt.h>
 #include <linux/list.h>
+#include <linux/slab.h>
 
 #include <linux/irqchip/arm-gic-v3.h>
 
@@ -61,6 +62,34 @@ struct its_itte {
 	u32 event_id;
 };
 
+static struct its_device *find_its_device(struct kvm *kvm, u32 device_id)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	struct its_device *device;
+
+	list_for_each_entry(device, &its->device_list, dev_list)
+		if (device_id == device->device_id)
+			return device;
+
+	return NULL;
+}
+
+static struct its_itte *find_itte(struct kvm *kvm, u32 device_id, u32 event_id)
+{
+	struct its_device *device;
+	struct its_itte *itte;
+
+	device = find_its_device(kvm, device_id);
+	if (device == NULL)
+		return NULL;
+
+	list_for_each_entry(itte, &device->itt, itte_list)
+		if (itte->event_id == event_id)
+			return itte;
+
+	return NULL;
+}
+
 /* To be used as an iterator this macro misses the enclosing parentheses */
 #define for_each_lpi(dev, itte, kvm) \
 	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
@@ -78,6 +107,19 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
 	return NULL;
 }
 
+static struct its_collection *find_collection(struct kvm *kvm, int coll_id)
+{
+	struct its_collection *collection;
+
+	list_for_each_entry(collection, &kvm->arch.vgic.its.collection_list,
+			    coll_list) {
+		if (coll_id == collection->collection_id)
+			return collection;
+	}
+
+	return NULL;
+}
+
 #define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
 #define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
 
@@ -91,6 +133,29 @@ static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
 	vgic_queue_irq(kvm, &itte->irq);
 }
 
+/*
+ * Finds all LPIs which are mapped to this collection and updates the
+ * struct irq's target_vcpu field accordingly.
+ * Needs to be called whenever either the collection for a LPIs has
+ * changed or the collection itself got retargetted.
+ */
+static void update_affinity(struct kvm *kvm, struct its_collection *coll)
+{
+	struct its_device *device;
+	struct its_itte *itte;
+	struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, coll->target_addr);
+
+	for_each_lpi(device, itte, kvm) {
+		if (!itte->collection ||
+		    coll->collection_id != itte->collection->collection_id)
+			continue;
+
+		spin_lock(&itte->irq.irq_lock);
+		itte->irq.target_vcpu = vcpu;
+		spin_unlock(&itte->irq.irq_lock);
+	}
+}
+
 #define GIC_LPI_OFFSET 8192
 
 /* We scan the table in chunks the size of the smallest page size */
@@ -323,13 +388,460 @@ static void its_free_itte(struct its_itte *itte)
 	kfree(itte);
 }
 
+static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
+{
+	return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT_ULL(size) - 1);
+}
+
+#define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
+#define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
+#define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
+#define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
+#define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
+#define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
+#define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
+
+/* The DISCARD command frees an Interrupt Translation Table Entry (ITTE). */
+static int vits_cmd_handle_discard(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 device_id;
+	u32 event_id;
+	struct its_itte *itte;
+	int ret = E_ITS_DISCARD_UNMAPPED_INTERRUPT;
+
+	device_id = its_cmd_get_deviceid(its_cmd);
+	event_id = its_cmd_get_id(its_cmd);
+
+	spin_lock(&its->lock);
+	itte = find_itte(kvm, device_id, event_id);
+	if (itte && itte->collection) {
+		/*
+		 * Though the spec talks about removing the pending state, we
+		 * don't bother here since we clear the ITTE anyway and the
+		 * pending state is a property of the ITTE struct.
+		 */
+		its_free_itte(itte);
+		ret = 0;
+	}
+
+	spin_unlock(&its->lock);
+	return ret;
+}
+
+/* The MOVI command moves an ITTE to a different collection. */
+static int vits_cmd_handle_movi(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 device_id = its_cmd_get_deviceid(its_cmd);
+	u32 event_id = its_cmd_get_id(its_cmd);
+	u32 coll_id = its_cmd_get_collection(its_cmd);
+	struct its_itte *itte;
+	struct its_collection *collection;
+	int ret;
+
+	spin_lock(&its->lock);
+	itte = find_itte(kvm, device_id, event_id);
+	if (!itte) {
+		ret = E_ITS_MOVI_UNMAPPED_INTERRUPT;
+		goto out_unlock;
+	}
+	if (!its_is_collection_mapped(itte->collection)) {
+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
+		goto out_unlock;
+	}
+
+	collection = find_collection(kvm, coll_id);
+	if (!its_is_collection_mapped(collection)) {
+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
+		goto out_unlock;
+	}
+
+	itte->collection = collection;
+	update_affinity(kvm, collection);
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return ret;
+}
+
+static void vits_init_collection(struct kvm *kvm,
+				 struct its_collection *collection,
+				 u32 coll_id)
+{
+	collection->collection_id = coll_id;
+	collection->target_addr = COLLECTION_NOT_MAPPED;
+
+	list_add_tail(&collection->coll_list,
+		&kvm->arch.vgic.its.collection_list);
+}
+
+/* The MAPTI and MAPI commands map LPIs to ITTEs. */
+static int vits_cmd_handle_mapi(struct kvm *kvm, u64 *its_cmd, u8 cmd)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 device_id = its_cmd_get_deviceid(its_cmd);
+	u32 event_id = its_cmd_get_id(its_cmd);
+	u32 coll_id = its_cmd_get_collection(its_cmd);
+	struct its_itte *itte, *new_itte;
+	struct its_device *device;
+	struct its_collection *collection, *new_coll;
+	int lpi_nr;
+	int ret = 0;
+
+	/* Preallocate possibly needed memory here outside of the lock */
+	new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
+	new_itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
+
+	spin_lock(&dist->its.lock);
+
+	device = find_its_device(kvm, device_id);
+	if (!device) {
+		ret = E_ITS_MAPTI_UNMAPPED_DEVICE;
+		goto out_unlock;
+	}
+
+	collection = find_collection(kvm, coll_id);
+	if (!collection && !new_coll) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}
+
+	if (cmd == GITS_CMD_MAPTI)
+		lpi_nr = its_cmd_get_physical_id(its_cmd);
+	else
+		lpi_nr = event_id;
+	if (lpi_nr < GIC_LPI_OFFSET ||
+	    lpi_nr >= nr_idbits_propbase(dist->propbaser)) {
+		ret = E_ITS_MAPTI_PHYSICALID_OOR;
+		goto out_unlock;
+	}
+
+	itte = find_itte(kvm, device_id, event_id);
+	if (!itte) {
+		if (!new_itte) {
+			ret = -ENOMEM;
+			goto out_unlock;
+		}
+		itte = new_itte;
+
+		itte->event_id	= event_id;
+		list_add_tail(&itte->itte_list, &device->itt);
+	} else {
+		kfree(new_itte);
+	}
+
+	if (!collection) {
+		collection = new_coll;
+		vits_init_collection(kvm, collection, coll_id);
+	} else {
+		kfree(new_coll);
+	}
+
+	itte->collection = collection;
+	itte->lpi = lpi_nr;
+	itte->irq.intid = lpi_nr;
+	INIT_LIST_HEAD(&itte->irq.ap_list);
+	spin_lock_init(&itte->irq.irq_lock);
+	itte->irq.vcpu = NULL;
+	update_affinity(kvm, collection);
+
+out_unlock:
+	spin_unlock(&dist->its.lock);
+	if (ret) {
+		kfree(new_coll);
+		kfree(new_itte);
+	}
+	return ret;
+}
+
+static void vits_unmap_device(struct kvm *kvm, struct its_device *device)
+{
+	struct its_itte *itte, *temp;
+
+	/*
+	 * The spec says that unmapping a device with still valid
+	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
+	 * since we cannot leave the memory unreferenced.
+	 */
+	list_for_each_entry_safe(itte, temp, &device->itt, itte_list)
+		its_free_itte(itte);
+
+	list_del(&device->dev_list);
+	kfree(device);
+}
+
+/* MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs). */
+static int vits_cmd_handle_mapd(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	bool valid = its_cmd_get_validbit(its_cmd);
+	u32 device_id = its_cmd_get_deviceid(its_cmd);
+	struct its_device *device, *new_device = NULL;
+
+	/* We preallocate memory outside of the lock here */
+	if (valid) {
+		new_device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
+		if (!new_device)
+			return -ENOMEM;
+	}
+
+	spin_lock(&its->lock);
+
+	device = find_its_device(kvm, device_id);
+	if (device)
+		vits_unmap_device(kvm, device);
+
+	/*
+	 * The spec does not say whether unmapping a not-mapped device
+	 * is an error, so we are done in any case.
+	 */
+	if (!valid)
+		goto out_unlock;
+
+	device = new_device;
+
+	device->device_id = device_id;
+	INIT_LIST_HEAD(&device->itt);
+
+	list_add_tail(&device->dev_list,
+		      &kvm->arch.vgic.its.device_list);
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return 0;
+}
+
+/* The MAPC command maps collection IDs to redistributors. */
+static int vits_cmd_handle_mapc(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u16 coll_id;
+	u32 target_addr;
+	struct its_collection *collection, *new_coll = NULL;
+	bool valid;
+
+	valid = its_cmd_get_validbit(its_cmd);
+	coll_id = its_cmd_get_collection(its_cmd);
+	target_addr = its_cmd_get_target_addr(its_cmd);
+
+	if (target_addr >= atomic_read(&kvm->online_vcpus))
+		return E_ITS_MAPC_PROCNUM_OOR;
+
+	/* We preallocate memory outside of the lock here */
+	if (valid) {
+		new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
+		if (!new_coll)
+			return -ENOMEM;
+	}
+
+	spin_lock(&its->lock);
+	collection = find_collection(kvm, coll_id);
+
+	if (!valid) {
+		struct its_device *device;
+		struct its_itte *itte;
+		/*
+		 * Clearing the mapping for that collection ID removes the
+		 * entry from the list. If there wasn't any before, we can
+		 * go home early.
+		 */
+		if (!collection)
+			goto out_unlock;
+
+		for_each_lpi(device, itte, kvm)
+			if (itte->collection &&
+			    itte->collection->collection_id == coll_id)
+				itte->collection = NULL;
+
+		list_del(&collection->coll_list);
+		kfree(collection);
+	} else {
+		if (!collection)
+			collection = new_coll;
+		else
+			kfree(new_coll);
+
+		vits_init_collection(kvm, collection, coll_id);
+		collection->target_addr = target_addr;
+		update_affinity(kvm, collection);
+	}
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return 0;
+}
+
+/* The CLEAR command removes the pending state for a particular LPI. */
+static int vits_cmd_handle_clear(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 device_id;
+	u32 event_id;
+	struct its_itte *itte;
+	int ret = 0;
+
+	device_id = its_cmd_get_deviceid(its_cmd);
+	event_id = its_cmd_get_id(its_cmd);
+
+	spin_lock(&its->lock);
+
+	itte = find_itte(kvm, device_id, event_id);
+	if (!itte) {
+		ret = E_ITS_CLEAR_UNMAPPED_INTERRUPT;
+		goto out_unlock;
+	}
+
+	itte->irq.pending = false;
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return ret;
+}
+
+/* The INV command syncs the configuration bits from the memory tables. */
+static int vits_cmd_handle_inv(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 device_id;
+	u32 event_id;
+	struct its_itte *itte, *new_itte;
+	gpa_t propbase;
+	int ret;
+	u8 prop;
+
+	device_id = its_cmd_get_deviceid(its_cmd);
+	event_id = its_cmd_get_id(its_cmd);
+
+	spin_lock(&dist->its.lock);
+	itte = find_itte(kvm, device_id, event_id);
+	spin_unlock(&dist->its.lock);
+	if (!itte)
+		return E_ITS_INV_UNMAPPED_INTERRUPT;
+
+	/*
+	 * We cannot read from guest memory inside the spinlock, so we
+	 * need to re-read our tables to learn whether the LPI number we are
+	 * using is still valid.
+	 */
+	do {
+		propbase = BASER_BASE_ADDRESS(dist->propbaser);
+		ret = kvm_read_guest(kvm, propbase + itte->lpi - GIC_LPI_OFFSET,
+				     &prop, 1);
+		if (ret)
+			return ret;
+
+		spin_lock(&dist->its.lock);
+		new_itte = find_itte(kvm, device_id, event_id);
+		if (new_itte->lpi != itte->lpi) {
+			itte = new_itte;
+			spin_unlock(&dist->its.lock);
+			continue;
+		}
+		update_lpi_config(kvm, itte, prop);
+		spin_unlock(&dist->its.lock);
+	} while (0);
+	return 0;
+}
+
+/* The INVALL command requests flushing of all IRQ data in this collection. */
+static int vits_cmd_handle_invall(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u64 prop_base_reg, pend_base_reg;
+	u32 coll_id = its_cmd_get_collection(its_cmd);
+	struct its_collection *collection;
+	struct kvm_vcpu *vcpu;
+
+	collection = find_collection(kvm, coll_id);
+	if (!its_is_collection_mapped(collection))
+		return E_ITS_INVALL_UNMAPPED_COLLECTION;
+
+	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
+
+	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
+	prop_base_reg = dist->propbaser;
+
+	its_update_lpis_configuration(kvm, prop_base_reg);
+	its_sync_lpi_pending_table(vcpu, pend_base_reg);
+
+	return 0;
+}
+
+/* The MOVALL command moves all IRQs from one redistributor to another. */
+static int vits_cmd_handle_movall(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 target1_addr = its_cmd_get_target_addr(its_cmd);
+	u32 target2_addr = its_cmd_mask_field(its_cmd, 3, 16, 32);
+	struct its_collection *collection;
+
+	if (target1_addr >= atomic_read(&kvm->online_vcpus) ||
+	    target2_addr >= atomic_read(&kvm->online_vcpus))
+		return E_ITS_MOVALL_PROCNUM_OOR;
+
+	if (target1_addr == target2_addr)
+		return 0;
+
+	spin_lock(&its->lock);
+	list_for_each_entry(collection, &its->collection_list,
+			    coll_list) {
+		if (collection && collection->target_addr == target1_addr)
+			collection->target_addr = target2_addr;
+		update_affinity(kvm, collection);
+	}
+
+	spin_unlock(&its->lock);
+	return 0;
+}
+
 /*
  * This function is called with both the ITS and the distributor lock dropped,
  * so the actual command handlers must take the respective locks when needed.
  */
 static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
 {
-	return -ENODEV;
+	u8 cmd = its_cmd_get_command(its_cmd);
+	int ret = -ENODEV;
+
+	switch (cmd) {
+	case GITS_CMD_MAPD:
+		ret = vits_cmd_handle_mapd(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_MAPC:
+		ret = vits_cmd_handle_mapc(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_MAPI:
+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
+		break;
+	case GITS_CMD_MAPTI:
+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
+		break;
+	case GITS_CMD_MOVI:
+		ret = vits_cmd_handle_movi(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_DISCARD:
+		ret = vits_cmd_handle_discard(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_CLEAR:
+		ret = vits_cmd_handle_clear(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_MOVALL:
+		ret = vits_cmd_handle_movall(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_INV:
+		ret = vits_cmd_handle_inv(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_INVALL:
+		ret = vits_cmd_handle_invall(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_SYNC:
+		/* we ignore this command: we are in sync all of the time */
+		ret = 0;
+		break;
+	}
+
+	return ret;
 }
 
 static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

The connection between a device, an event ID, the LPI number and the
allocated CPU is stored in in-memory tables in a GICv3, but their
format is not specified by the spec. Instead software uses a command
queue in a ring buffer to let the ITS implementation use their own
format.
Implement handlers for the various ITS commands and let them store
the requested relation into our own data structures.
To avoid kmallocs inside the ITS spinlock, we preallocate possibly
needed memory outside of the lock and free that if it turns out to
be not needed (mostly error handling).
Error handling is very basic at this point, as we don't have a good
way of communicating errors to the guest (usually a SError).
The INT command handler is missing at this point, as we gain the
capability of actually injecting MSIs into the guest only later on.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 include/linux/irqchip/arm-gic-v3.h |  19 +-
 virt/kvm/arm/vgic/its-emul.c       | 514 ++++++++++++++++++++++++++++++++++++-
 2 files changed, 531 insertions(+), 2 deletions(-)

diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 7011b98..c9aa8d8 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -254,7 +254,10 @@
  */
 #define GITS_CMD_MAPD			0x08
 #define GITS_CMD_MAPC			0x09
-#define GITS_CMD_MAPVI			0x0a
+#define GITS_CMD_MAPTI			0x0a
+/* older GIC documentation used MAPVI for this command */
+#define GITS_CMD_MAPVI			GITS_CMD_MAPTI
+#define GITS_CMD_MAPI			0x0b
 #define GITS_CMD_MOVI			0x01
 #define GITS_CMD_DISCARD		0x0f
 #define GITS_CMD_INV			0x0c
@@ -265,6 +268,20 @@
 #define GITS_CMD_SYNC			0x05
 
 /*
+ * ITS error numbers
+ */
+#define E_ITS_MOVI_UNMAPPED_INTERRUPT           0x010107
+#define E_ITS_MOVI_UNMAPPED_COLLECTION          0x010109
+#define E_ITS_CLEAR_UNMAPPED_INTERRUPT          0x010507
+#define E_ITS_MAPC_PROCNUM_OOR                  0x010902
+#define E_ITS_MAPTI_UNMAPPED_DEVICE             0x010a04
+#define E_ITS_MAPTI_PHYSICALID_OOR              0x010a06
+#define E_ITS_INV_UNMAPPED_INTERRUPT            0x010c07
+#define E_ITS_INVALL_UNMAPPED_COLLECTION        0x010d09
+#define E_ITS_MOVALL_PROCNUM_OOR                0x010e01
+#define E_ITS_DISCARD_UNMAPPED_INTERRUPT        0x010f07
+
+/*
  * CPU interface registers
  */
 #define ICC_CTLR_EL1_EOImode_drop_dir	(0U << 1)
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index d82ba9b..dcd0cac 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -22,6 +22,7 @@
 #include <linux/kvm_host.h>
 #include <linux/interrupt.h>
 #include <linux/list.h>
+#include <linux/slab.h>
 
 #include <linux/irqchip/arm-gic-v3.h>
 
@@ -61,6 +62,34 @@ struct its_itte {
 	u32 event_id;
 };
 
+static struct its_device *find_its_device(struct kvm *kvm, u32 device_id)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	struct its_device *device;
+
+	list_for_each_entry(device, &its->device_list, dev_list)
+		if (device_id == device->device_id)
+			return device;
+
+	return NULL;
+}
+
+static struct its_itte *find_itte(struct kvm *kvm, u32 device_id, u32 event_id)
+{
+	struct its_device *device;
+	struct its_itte *itte;
+
+	device = find_its_device(kvm, device_id);
+	if (device == NULL)
+		return NULL;
+
+	list_for_each_entry(itte, &device->itt, itte_list)
+		if (itte->event_id == event_id)
+			return itte;
+
+	return NULL;
+}
+
 /* To be used as an iterator this macro misses the enclosing parentheses */
 #define for_each_lpi(dev, itte, kvm) \
 	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
@@ -78,6 +107,19 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
 	return NULL;
 }
 
+static struct its_collection *find_collection(struct kvm *kvm, int coll_id)
+{
+	struct its_collection *collection;
+
+	list_for_each_entry(collection, &kvm->arch.vgic.its.collection_list,
+			    coll_list) {
+		if (coll_id == collection->collection_id)
+			return collection;
+	}
+
+	return NULL;
+}
+
 #define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
 #define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
 
@@ -91,6 +133,29 @@ static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
 	vgic_queue_irq(kvm, &itte->irq);
 }
 
+/*
+ * Finds all LPIs which are mapped to this collection and updates the
+ * struct irq's target_vcpu field accordingly.
+ * Needs to be called whenever either the collection for a LPIs has
+ * changed or the collection itself got retargetted.
+ */
+static void update_affinity(struct kvm *kvm, struct its_collection *coll)
+{
+	struct its_device *device;
+	struct its_itte *itte;
+	struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, coll->target_addr);
+
+	for_each_lpi(device, itte, kvm) {
+		if (!itte->collection ||
+		    coll->collection_id != itte->collection->collection_id)
+			continue;
+
+		spin_lock(&itte->irq.irq_lock);
+		itte->irq.target_vcpu = vcpu;
+		spin_unlock(&itte->irq.irq_lock);
+	}
+}
+
 #define GIC_LPI_OFFSET 8192
 
 /* We scan the table in chunks the size of the smallest page size */
@@ -323,13 +388,460 @@ static void its_free_itte(struct its_itte *itte)
 	kfree(itte);
 }
 
+static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
+{
+	return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT_ULL(size) - 1);
+}
+
+#define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
+#define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
+#define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
+#define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
+#define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
+#define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
+#define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
+
+/* The DISCARD command frees an Interrupt Translation Table Entry (ITTE). */
+static int vits_cmd_handle_discard(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 device_id;
+	u32 event_id;
+	struct its_itte *itte;
+	int ret = E_ITS_DISCARD_UNMAPPED_INTERRUPT;
+
+	device_id = its_cmd_get_deviceid(its_cmd);
+	event_id = its_cmd_get_id(its_cmd);
+
+	spin_lock(&its->lock);
+	itte = find_itte(kvm, device_id, event_id);
+	if (itte && itte->collection) {
+		/*
+		 * Though the spec talks about removing the pending state, we
+		 * don't bother here since we clear the ITTE anyway and the
+		 * pending state is a property of the ITTE struct.
+		 */
+		its_free_itte(itte);
+		ret = 0;
+	}
+
+	spin_unlock(&its->lock);
+	return ret;
+}
+
+/* The MOVI command moves an ITTE to a different collection. */
+static int vits_cmd_handle_movi(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 device_id = its_cmd_get_deviceid(its_cmd);
+	u32 event_id = its_cmd_get_id(its_cmd);
+	u32 coll_id = its_cmd_get_collection(its_cmd);
+	struct its_itte *itte;
+	struct its_collection *collection;
+	int ret;
+
+	spin_lock(&its->lock);
+	itte = find_itte(kvm, device_id, event_id);
+	if (!itte) {
+		ret = E_ITS_MOVI_UNMAPPED_INTERRUPT;
+		goto out_unlock;
+	}
+	if (!its_is_collection_mapped(itte->collection)) {
+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
+		goto out_unlock;
+	}
+
+	collection = find_collection(kvm, coll_id);
+	if (!its_is_collection_mapped(collection)) {
+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
+		goto out_unlock;
+	}
+
+	itte->collection = collection;
+	update_affinity(kvm, collection);
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return ret;
+}
+
+static void vits_init_collection(struct kvm *kvm,
+				 struct its_collection *collection,
+				 u32 coll_id)
+{
+	collection->collection_id = coll_id;
+	collection->target_addr = COLLECTION_NOT_MAPPED;
+
+	list_add_tail(&collection->coll_list,
+		&kvm->arch.vgic.its.collection_list);
+}
+
+/* The MAPTI and MAPI commands map LPIs to ITTEs. */
+static int vits_cmd_handle_mapi(struct kvm *kvm, u64 *its_cmd, u8 cmd)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 device_id = its_cmd_get_deviceid(its_cmd);
+	u32 event_id = its_cmd_get_id(its_cmd);
+	u32 coll_id = its_cmd_get_collection(its_cmd);
+	struct its_itte *itte, *new_itte;
+	struct its_device *device;
+	struct its_collection *collection, *new_coll;
+	int lpi_nr;
+	int ret = 0;
+
+	/* Preallocate possibly needed memory here outside of the lock */
+	new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
+	new_itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
+
+	spin_lock(&dist->its.lock);
+
+	device = find_its_device(kvm, device_id);
+	if (!device) {
+		ret = E_ITS_MAPTI_UNMAPPED_DEVICE;
+		goto out_unlock;
+	}
+
+	collection = find_collection(kvm, coll_id);
+	if (!collection && !new_coll) {
+		ret = -ENOMEM;
+		goto out_unlock;
+	}
+
+	if (cmd == GITS_CMD_MAPTI)
+		lpi_nr = its_cmd_get_physical_id(its_cmd);
+	else
+		lpi_nr = event_id;
+	if (lpi_nr < GIC_LPI_OFFSET ||
+	    lpi_nr >= nr_idbits_propbase(dist->propbaser)) {
+		ret = E_ITS_MAPTI_PHYSICALID_OOR;
+		goto out_unlock;
+	}
+
+	itte = find_itte(kvm, device_id, event_id);
+	if (!itte) {
+		if (!new_itte) {
+			ret = -ENOMEM;
+			goto out_unlock;
+		}
+		itte = new_itte;
+
+		itte->event_id	= event_id;
+		list_add_tail(&itte->itte_list, &device->itt);
+	} else {
+		kfree(new_itte);
+	}
+
+	if (!collection) {
+		collection = new_coll;
+		vits_init_collection(kvm, collection, coll_id);
+	} else {
+		kfree(new_coll);
+	}
+
+	itte->collection = collection;
+	itte->lpi = lpi_nr;
+	itte->irq.intid = lpi_nr;
+	INIT_LIST_HEAD(&itte->irq.ap_list);
+	spin_lock_init(&itte->irq.irq_lock);
+	itte->irq.vcpu = NULL;
+	update_affinity(kvm, collection);
+
+out_unlock:
+	spin_unlock(&dist->its.lock);
+	if (ret) {
+		kfree(new_coll);
+		kfree(new_itte);
+	}
+	return ret;
+}
+
+static void vits_unmap_device(struct kvm *kvm, struct its_device *device)
+{
+	struct its_itte *itte, *temp;
+
+	/*
+	 * The spec says that unmapping a device with still valid
+	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
+	 * since we cannot leave the memory unreferenced.
+	 */
+	list_for_each_entry_safe(itte, temp, &device->itt, itte_list)
+		its_free_itte(itte);
+
+	list_del(&device->dev_list);
+	kfree(device);
+}
+
+/* MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs). */
+static int vits_cmd_handle_mapd(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	bool valid = its_cmd_get_validbit(its_cmd);
+	u32 device_id = its_cmd_get_deviceid(its_cmd);
+	struct its_device *device, *new_device = NULL;
+
+	/* We preallocate memory outside of the lock here */
+	if (valid) {
+		new_device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
+		if (!new_device)
+			return -ENOMEM;
+	}
+
+	spin_lock(&its->lock);
+
+	device = find_its_device(kvm, device_id);
+	if (device)
+		vits_unmap_device(kvm, device);
+
+	/*
+	 * The spec does not say whether unmapping a not-mapped device
+	 * is an error, so we are done in any case.
+	 */
+	if (!valid)
+		goto out_unlock;
+
+	device = new_device;
+
+	device->device_id = device_id;
+	INIT_LIST_HEAD(&device->itt);
+
+	list_add_tail(&device->dev_list,
+		      &kvm->arch.vgic.its.device_list);
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return 0;
+}
+
+/* The MAPC command maps collection IDs to redistributors. */
+static int vits_cmd_handle_mapc(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u16 coll_id;
+	u32 target_addr;
+	struct its_collection *collection, *new_coll = NULL;
+	bool valid;
+
+	valid = its_cmd_get_validbit(its_cmd);
+	coll_id = its_cmd_get_collection(its_cmd);
+	target_addr = its_cmd_get_target_addr(its_cmd);
+
+	if (target_addr >= atomic_read(&kvm->online_vcpus))
+		return E_ITS_MAPC_PROCNUM_OOR;
+
+	/* We preallocate memory outside of the lock here */
+	if (valid) {
+		new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
+		if (!new_coll)
+			return -ENOMEM;
+	}
+
+	spin_lock(&its->lock);
+	collection = find_collection(kvm, coll_id);
+
+	if (!valid) {
+		struct its_device *device;
+		struct its_itte *itte;
+		/*
+		 * Clearing the mapping for that collection ID removes the
+		 * entry from the list. If there wasn't any before, we can
+		 * go home early.
+		 */
+		if (!collection)
+			goto out_unlock;
+
+		for_each_lpi(device, itte, kvm)
+			if (itte->collection &&
+			    itte->collection->collection_id == coll_id)
+				itte->collection = NULL;
+
+		list_del(&collection->coll_list);
+		kfree(collection);
+	} else {
+		if (!collection)
+			collection = new_coll;
+		else
+			kfree(new_coll);
+
+		vits_init_collection(kvm, collection, coll_id);
+		collection->target_addr = target_addr;
+		update_affinity(kvm, collection);
+	}
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return 0;
+}
+
+/* The CLEAR command removes the pending state for a particular LPI. */
+static int vits_cmd_handle_clear(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 device_id;
+	u32 event_id;
+	struct its_itte *itte;
+	int ret = 0;
+
+	device_id = its_cmd_get_deviceid(its_cmd);
+	event_id = its_cmd_get_id(its_cmd);
+
+	spin_lock(&its->lock);
+
+	itte = find_itte(kvm, device_id, event_id);
+	if (!itte) {
+		ret = E_ITS_CLEAR_UNMAPPED_INTERRUPT;
+		goto out_unlock;
+	}
+
+	itte->irq.pending = false;
+
+out_unlock:
+	spin_unlock(&its->lock);
+	return ret;
+}
+
+/* The INV command syncs the configuration bits from the memory tables. */
+static int vits_cmd_handle_inv(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 device_id;
+	u32 event_id;
+	struct its_itte *itte, *new_itte;
+	gpa_t propbase;
+	int ret;
+	u8 prop;
+
+	device_id = its_cmd_get_deviceid(its_cmd);
+	event_id = its_cmd_get_id(its_cmd);
+
+	spin_lock(&dist->its.lock);
+	itte = find_itte(kvm, device_id, event_id);
+	spin_unlock(&dist->its.lock);
+	if (!itte)
+		return E_ITS_INV_UNMAPPED_INTERRUPT;
+
+	/*
+	 * We cannot read from guest memory inside the spinlock, so we
+	 * need to re-read our tables to learn whether the LPI number we are
+	 * using is still valid.
+	 */
+	do {
+		propbase = BASER_BASE_ADDRESS(dist->propbaser);
+		ret = kvm_read_guest(kvm, propbase + itte->lpi - GIC_LPI_OFFSET,
+				     &prop, 1);
+		if (ret)
+			return ret;
+
+		spin_lock(&dist->its.lock);
+		new_itte = find_itte(kvm, device_id, event_id);
+		if (new_itte->lpi != itte->lpi) {
+			itte = new_itte;
+			spin_unlock(&dist->its.lock);
+			continue;
+		}
+		update_lpi_config(kvm, itte, prop);
+		spin_unlock(&dist->its.lock);
+	} while (0);
+	return 0;
+}
+
+/* The INVALL command requests flushing of all IRQ data in this collection. */
+static int vits_cmd_handle_invall(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u64 prop_base_reg, pend_base_reg;
+	u32 coll_id = its_cmd_get_collection(its_cmd);
+	struct its_collection *collection;
+	struct kvm_vcpu *vcpu;
+
+	collection = find_collection(kvm, coll_id);
+	if (!its_is_collection_mapped(collection))
+		return E_ITS_INVALL_UNMAPPED_COLLECTION;
+
+	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
+
+	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
+	prop_base_reg = dist->propbaser;
+
+	its_update_lpis_configuration(kvm, prop_base_reg);
+	its_sync_lpi_pending_table(vcpu, pend_base_reg);
+
+	return 0;
+}
+
+/* The MOVALL command moves all IRQs from one redistributor to another. */
+static int vits_cmd_handle_movall(struct kvm *kvm, u64 *its_cmd)
+{
+	struct vgic_its *its = &kvm->arch.vgic.its;
+	u32 target1_addr = its_cmd_get_target_addr(its_cmd);
+	u32 target2_addr = its_cmd_mask_field(its_cmd, 3, 16, 32);
+	struct its_collection *collection;
+
+	if (target1_addr >= atomic_read(&kvm->online_vcpus) ||
+	    target2_addr >= atomic_read(&kvm->online_vcpus))
+		return E_ITS_MOVALL_PROCNUM_OOR;
+
+	if (target1_addr == target2_addr)
+		return 0;
+
+	spin_lock(&its->lock);
+	list_for_each_entry(collection, &its->collection_list,
+			    coll_list) {
+		if (collection && collection->target_addr == target1_addr)
+			collection->target_addr = target2_addr;
+		update_affinity(kvm, collection);
+	}
+
+	spin_unlock(&its->lock);
+	return 0;
+}
+
 /*
  * This function is called with both the ITS and the distributor lock dropped,
  * so the actual command handlers must take the respective locks when needed.
  */
 static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
 {
-	return -ENODEV;
+	u8 cmd = its_cmd_get_command(its_cmd);
+	int ret = -ENODEV;
+
+	switch (cmd) {
+	case GITS_CMD_MAPD:
+		ret = vits_cmd_handle_mapd(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_MAPC:
+		ret = vits_cmd_handle_mapc(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_MAPI:
+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
+		break;
+	case GITS_CMD_MAPTI:
+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
+		break;
+	case GITS_CMD_MOVI:
+		ret = vits_cmd_handle_movi(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_DISCARD:
+		ret = vits_cmd_handle_discard(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_CLEAR:
+		ret = vits_cmd_handle_clear(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_MOVALL:
+		ret = vits_cmd_handle_movall(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_INV:
+		ret = vits_cmd_handle_inv(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_INVALL:
+		ret = vits_cmd_handle_invall(vcpu->kvm, its_cmd);
+		break;
+	case GITS_CMD_SYNC:
+		/* we ignore this command: we are in sync all of the time */
+		ret = 0;
+		break;
+	}
+
+	return ret;
 }
 
 static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 11/12] KVM: arm64: implement MSI injection in ITS emulation
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: linux-arm-kernel, kvmarm, kvm

When userland wants to inject a MSI into the guest, we have to use
our data structures to find the LPI number and the VCPU to receive
the interrupt.
Use the wrapper functions to iterate the linked lists and find the
proper Interrupt Translation Table Entry. Then set the pending bit
in this ITTE to be later picked up by the LR handling code. Kick
the VCPU which is meant to handle this interrupt.
We provide a VGIC emulation model specific routine for the actual
MSI injection. The wrapper functions return an error for models not
(yet) implementing MSIs (like the GICv2 emulation).
We also provide the handler for the ITS "INT" command, which allows a
guest to trigger an MSI via the ITS command queue.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 virt/kvm/arm/vgic/its-emul.c | 61 ++++++++++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  6 +++++
 2 files changed, 67 insertions(+)

diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index dcd0cac..166551d 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -371,6 +371,51 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+/*
+ * Translates an incoming MSI request into the redistributor (=VCPU) and
+ * the associated LPI number. Sets the LPI pending bit and also marks the
+ * VCPU as having a pending interrupt.
+ */
+int vits_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+	struct its_itte *itte;
+	bool inject = false;
+	int ret = 0;
+
+	if (!vgic_has_its(kvm))
+		return -ENODEV;
+
+	if (!(msi->flags & KVM_MSI_VALID_DEVID))
+		return -EINVAL;
+
+	spin_lock(&its->lock);
+
+	if (!its->enabled || !dist->lpis_enabled) {
+		ret = -EAGAIN;
+		goto out_unlock;
+	}
+
+	itte = find_itte(kvm, msi->devid, msi->data);
+	/* Triggering an unmapped IRQ gets silently dropped. */
+	if (!itte || !its_is_collection_mapped(itte->collection))
+		goto out_unlock;
+
+	inject = true;
+
+out_unlock:
+	spin_unlock(&its->lock);
+
+	if (inject) {
+		spin_lock(&itte->irq.irq_lock);
+		itte->irq.pending = true;
+		vgic_queue_irq(kvm, &itte->irq);
+	}
+
+	return ret;
+}
+
 struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
 {
 	struct its_itte *itte;
@@ -795,6 +840,19 @@ static int vits_cmd_handle_movall(struct kvm *kvm, u64 *its_cmd)
 	return 0;
 }
 
+/* The INT command injects the LPI associated with that DevID/EvID pair. */
+static int vits_cmd_handle_int(struct kvm *kvm, u64 *its_cmd)
+{
+	struct kvm_msi msi = {
+		.data = its_cmd_get_id(its_cmd),
+		.devid = its_cmd_get_deviceid(its_cmd),
+		.flags = KVM_MSI_VALID_DEVID,
+	};
+
+	vits_inject_msi(kvm, &msi);
+	return 0;
+}
+
 /*
  * This function is called with both the ITS and the distributor lock dropped,
  * so the actual command handlers must take the respective locks when needed.
@@ -829,6 +887,9 @@ static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
 	case GITS_CMD_MOVALL:
 		ret = vits_cmd_handle_movall(vcpu->kvm, its_cmd);
 		break;
+	case GITS_CMD_INT:
+		ret = vits_cmd_handle_int(vcpu->kvm, its_cmd);
+		break;
 	case GITS_CMD_INV:
 		ret = vits_cmd_handle_inv(vcpu->kvm, its_cmd);
 		break;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index a7218b0..223c778 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -65,6 +65,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
 int vits_init(struct kvm *kvm);
 void vgic_enable_lpis(struct kvm_vcpu *vcpu);
 struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
+int vits_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
 void vits_destroy(struct kvm *kvm);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
@@ -146,6 +147,11 @@ static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
 	return NULL;
 }
 
+static inline int vits_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	return -ENODEV;
+}
+
 static inline void vits_destroy(struct kvm *kvm)
 {
 	return;
-- 
2.7.3


^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 11/12] KVM: arm64: implement MSI injection in ITS emulation
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

When userland wants to inject a MSI into the guest, we have to use
our data structures to find the LPI number and the VCPU to receive
the interrupt.
Use the wrapper functions to iterate the linked lists and find the
proper Interrupt Translation Table Entry. Then set the pending bit
in this ITTE to be later picked up by the LR handling code. Kick
the VCPU which is meant to handle this interrupt.
We provide a VGIC emulation model specific routine for the actual
MSI injection. The wrapper functions return an error for models not
(yet) implementing MSIs (like the GICv2 emulation).
We also provide the handler for the ITS "INT" command, which allows a
guest to trigger an MSI via the ITS command queue.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 virt/kvm/arm/vgic/its-emul.c | 61 ++++++++++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h     |  6 +++++
 2 files changed, 67 insertions(+)

diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index dcd0cac..166551d 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -371,6 +371,51 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
 	return 0;
 }
 
+/*
+ * Translates an incoming MSI request into the redistributor (=VCPU) and
+ * the associated LPI number. Sets the LPI pending bit and also marks the
+ * VCPU as having a pending interrupt.
+ */
+int vits_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct vgic_its *its = &dist->its;
+	struct its_itte *itte;
+	bool inject = false;
+	int ret = 0;
+
+	if (!vgic_has_its(kvm))
+		return -ENODEV;
+
+	if (!(msi->flags & KVM_MSI_VALID_DEVID))
+		return -EINVAL;
+
+	spin_lock(&its->lock);
+
+	if (!its->enabled || !dist->lpis_enabled) {
+		ret = -EAGAIN;
+		goto out_unlock;
+	}
+
+	itte = find_itte(kvm, msi->devid, msi->data);
+	/* Triggering an unmapped IRQ gets silently dropped. */
+	if (!itte || !its_is_collection_mapped(itte->collection))
+		goto out_unlock;
+
+	inject = true;
+
+out_unlock:
+	spin_unlock(&its->lock);
+
+	if (inject) {
+		spin_lock(&itte->irq.irq_lock);
+		itte->irq.pending = true;
+		vgic_queue_irq(kvm, &itte->irq);
+	}
+
+	return ret;
+}
+
 struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
 {
 	struct its_itte *itte;
@@ -795,6 +840,19 @@ static int vits_cmd_handle_movall(struct kvm *kvm, u64 *its_cmd)
 	return 0;
 }
 
+/* The INT command injects the LPI associated with that DevID/EvID pair. */
+static int vits_cmd_handle_int(struct kvm *kvm, u64 *its_cmd)
+{
+	struct kvm_msi msi = {
+		.data = its_cmd_get_id(its_cmd),
+		.devid = its_cmd_get_deviceid(its_cmd),
+		.flags = KVM_MSI_VALID_DEVID,
+	};
+
+	vits_inject_msi(kvm, &msi);
+	return 0;
+}
+
 /*
  * This function is called with both the ITS and the distributor lock dropped,
  * so the actual command handlers must take the respective locks when needed.
@@ -829,6 +887,9 @@ static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
 	case GITS_CMD_MOVALL:
 		ret = vits_cmd_handle_movall(vcpu->kvm, its_cmd);
 		break;
+	case GITS_CMD_INT:
+		ret = vits_cmd_handle_int(vcpu->kvm, its_cmd);
+		break;
 	case GITS_CMD_INV:
 		ret = vits_cmd_handle_inv(vcpu->kvm, its_cmd);
 		break;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index a7218b0..223c778 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -65,6 +65,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
 int vits_init(struct kvm *kvm);
 void vgic_enable_lpis(struct kvm_vcpu *vcpu);
 struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
+int vits_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
 void vits_destroy(struct kvm *kvm);
 #else
 static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
@@ -146,6 +147,11 @@ static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
 	return NULL;
 }
 
+static inline int vits_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	return -ENODEV;
+}
+
 static inline void vits_destroy(struct kvm *kvm)
 {
 	return;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 12/12] KVM: arm64: enable ITS emulation as a virtual MSI controller
  2016-03-26  2:13 ` Andre Przywara
@ 2016-03-26  2:14   ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: Christoffer Dall, Marc Zyngier, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

If userspace has provided a base address for the ITS register frame,
we enable the bits that advertise LPIs in the GICv3.
When the guest has enabled LPIs and the ITS, we enable the emulation
part by initializing the ITS data structures and trapping on ITS
register frame accesses by the guest.
Also we enable the KVM_SIGNAL_MSI feature to allow userland to inject
MSIs into the guest. Not having enabled the ITS emulation will lead
to a -ENODEV when trying to inject a MSI.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Documentation/virtual/kvm/api.txt |  2 +-
 arch/arm64/kvm/Kconfig            |  1 +
 arch/arm64/kvm/reset.c            | 10 ++++++++++
 include/kvm/vgic/vgic.h           |  6 ++++++
 virt/kvm/arm/vgic.c               |  5 +++++
 virt/kvm/arm/vgic/its-emul.c      |  3 ++-
 virt/kvm/arm/vgic/vgic.c          |  8 ++++++++
 virt/kvm/arm/vgic/vgic_mmio.c     | 12 ++++++++++--
 8 files changed, 43 insertions(+), 4 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 8f7351d..536f19b 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2150,7 +2150,7 @@ after pausing the vcpu, but before it is resumed.
 4.71 KVM_SIGNAL_MSI
 
 Capability: KVM_CAP_SIGNAL_MSI
-Architectures: x86
+Architectures: x86 arm64
 Type: vm ioctl
 Parameters: struct kvm_msi (in)
 Returns: >0 on delivery, 0 if guest blocked the MSI, and -1 on error
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 3f0e1ce..71c9ebc 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -37,6 +37,7 @@ config KVM
 	select HAVE_KVM_IRQFD
 	select KVM_ARM_VGIC_V3
 	select KVM_ARM_PMU if HW_PERF_EVENTS
+	select HAVE_KVM_MSI
 	---help---
 	  Support hosting virtualized guest machines.
 	  We don't support KVM with 16K page tables yet, due to the multiple
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 2ba7c8d..43c3836 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -84,6 +84,16 @@ int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_VCPU_ATTRIBUTES:
 		r = 1;
 		break;
+	case KVM_CAP_MSI_DEVID:
+#ifdef CONFIG_KVM_NEW_VGIC
+		if (!kvm)
+			r = -EINVAL;
+		else
+			r = kvm->arch.vgic.msis_require_devid;
+#else
+		r = -EINVAL;
+#endif
+		break;
 	default:
 		r = 0;
 	}
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index ea98133..c50890f 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -115,6 +115,7 @@ struct vgic_io_device {
 
 struct vgic_its {
 	bool			enabled;
+	struct vgic_io_device	iodev;
 	spinlock_t		lock;
 	u64			cbaser;
 	int			creadr;
@@ -133,6 +134,9 @@ struct vgic_dist {
 	/* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
 	u32			vgic_model;
 
+	/* Do injected MSIs require an additional device ID? */
+	bool			msis_require_devid;
+
 	int			nr_spis;
 
 	/* TODO: Consider moving to global state */
@@ -280,4 +284,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
 
 bool vgic_has_its(struct kvm *kvm);
 
+int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
+
 #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 079b0a7..3f01f7b 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2470,3 +2470,8 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 {
 	return 0;
 }
+
+int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	return -ENODEV;
+}
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index 166551d..f9d289a 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -1098,8 +1098,9 @@ int vits_init(struct kvm *kvm)
 		return ret;
 
 	its->enabled = false;
+	dist->msis_require_devid = true;
 
-	return -ENXIO;
+	return 0;
 }
 
 void vits_destroy(struct kvm *kvm)
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 1de2478..82bfb33 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -622,3 +622,11 @@ bool vgic_has_its(struct kvm *kvm)
 
 	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
 }
+
+int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	if (vgic_has_its(kvm))
+		return vits_inject_msi(kvm, msi);
+	else
+		return -ENODEV;
+}
diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
index 086555e..065934f 100644
--- a/virt/kvm/arm/vgic/vgic_mmio.c
+++ b/virt/kvm/arm/vgic/vgic_mmio.c
@@ -619,7 +619,12 @@ static int vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
 	case GICD_TYPER:
 		value = vcpu->kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
 		value = (value >> 5) - 1;
-		value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
+		if (vgic_has_its(vcpu->kvm)) {
+			value |= (INTERRUPT_ID_BITS_ITS - 1) << 19;
+			value |= GICD_TYPER_LPIS;
+		} else {
+			value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
+		}
 		break;
 	case GICD_IIDR:
 		value = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
@@ -742,7 +747,8 @@ static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
 
 	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
 	    (reg & GICR_CTLR_ENABLE_LPIS)) {
-		/* Eventually do something */
+		vgic_enable_lpis(vcpu);
+		dist->lpis_enabled = true;
 	}
 
 	return 0;
@@ -772,6 +778,8 @@ static int vgic_mmio_read_v3r_typer(struct kvm_vcpu *vcpu,
 	value |= ((target_vcpu_id & 0xffff) << 8);
 	if (target_vcpu_id == atomic_read(&vcpu->kvm->online_vcpus) - 1)
 		value |= GICR_TYPER_LAST;
+	if (vgic_has_its(vcpu->kvm))
+		value |= GICR_TYPER_PLPIS;
 
 	write_mask64(value, addr & 7, len, val);
 	return 0;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* [PATCH v4 12/12] KVM: arm64: enable ITS emulation as a virtual MSI controller
@ 2016-03-26  2:14   ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-03-26  2:14 UTC (permalink / raw)
  To: linux-arm-kernel

If userspace has provided a base address for the ITS register frame,
we enable the bits that advertise LPIs in the GICv3.
When the guest has enabled LPIs and the ITS, we enable the emulation
part by initializing the ITS data structures and trapping on ITS
register frame accesses by the guest.
Also we enable the KVM_SIGNAL_MSI feature to allow userland to inject
MSIs into the guest. Not having enabled the ITS emulation will lead
to a -ENODEV when trying to inject a MSI.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 Documentation/virtual/kvm/api.txt |  2 +-
 arch/arm64/kvm/Kconfig            |  1 +
 arch/arm64/kvm/reset.c            | 10 ++++++++++
 include/kvm/vgic/vgic.h           |  6 ++++++
 virt/kvm/arm/vgic.c               |  5 +++++
 virt/kvm/arm/vgic/its-emul.c      |  3 ++-
 virt/kvm/arm/vgic/vgic.c          |  8 ++++++++
 virt/kvm/arm/vgic/vgic_mmio.c     | 12 ++++++++++--
 8 files changed, 43 insertions(+), 4 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 8f7351d..536f19b 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2150,7 +2150,7 @@ after pausing the vcpu, but before it is resumed.
 4.71 KVM_SIGNAL_MSI
 
 Capability: KVM_CAP_SIGNAL_MSI
-Architectures: x86
+Architectures: x86 arm64
 Type: vm ioctl
 Parameters: struct kvm_msi (in)
 Returns: >0 on delivery, 0 if guest blocked the MSI, and -1 on error
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 3f0e1ce..71c9ebc 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -37,6 +37,7 @@ config KVM
 	select HAVE_KVM_IRQFD
 	select KVM_ARM_VGIC_V3
 	select KVM_ARM_PMU if HW_PERF_EVENTS
+	select HAVE_KVM_MSI
 	---help---
 	  Support hosting virtualized guest machines.
 	  We don't support KVM with 16K page tables yet, due to the multiple
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 2ba7c8d..43c3836 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -84,6 +84,16 @@ int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
 	case KVM_CAP_VCPU_ATTRIBUTES:
 		r = 1;
 		break;
+	case KVM_CAP_MSI_DEVID:
+#ifdef CONFIG_KVM_NEW_VGIC
+		if (!kvm)
+			r = -EINVAL;
+		else
+			r = kvm->arch.vgic.msis_require_devid;
+#else
+		r = -EINVAL;
+#endif
+		break;
 	default:
 		r = 0;
 	}
diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
index ea98133..c50890f 100644
--- a/include/kvm/vgic/vgic.h
+++ b/include/kvm/vgic/vgic.h
@@ -115,6 +115,7 @@ struct vgic_io_device {
 
 struct vgic_its {
 	bool			enabled;
+	struct vgic_io_device	iodev;
 	spinlock_t		lock;
 	u64			cbaser;
 	int			creadr;
@@ -133,6 +134,9 @@ struct vgic_dist {
 	/* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
 	u32			vgic_model;
 
+	/* Do injected MSIs require an additional device ID? */
+	bool			msis_require_devid;
+
 	int			nr_spis;
 
 	/* TODO: Consider moving to global state */
@@ -280,4 +284,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
 
 bool vgic_has_its(struct kvm *kvm);
 
+int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
+
 #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 079b0a7..3f01f7b 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2470,3 +2470,8 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 {
 	return 0;
 }
+
+int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	return -ENODEV;
+}
diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
index 166551d..f9d289a 100644
--- a/virt/kvm/arm/vgic/its-emul.c
+++ b/virt/kvm/arm/vgic/its-emul.c
@@ -1098,8 +1098,9 @@ int vits_init(struct kvm *kvm)
 		return ret;
 
 	its->enabled = false;
+	dist->msis_require_devid = true;
 
-	return -ENXIO;
+	return 0;
 }
 
 void vits_destroy(struct kvm *kvm)
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 1de2478..82bfb33 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -622,3 +622,11 @@ bool vgic_has_its(struct kvm *kvm)
 
 	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
 }
+
+int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	if (vgic_has_its(kvm))
+		return vits_inject_msi(kvm, msi);
+	else
+		return -ENODEV;
+}
diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
index 086555e..065934f 100644
--- a/virt/kvm/arm/vgic/vgic_mmio.c
+++ b/virt/kvm/arm/vgic/vgic_mmio.c
@@ -619,7 +619,12 @@ static int vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
 	case GICD_TYPER:
 		value = vcpu->kvm->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
 		value = (value >> 5) - 1;
-		value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
+		if (vgic_has_its(vcpu->kvm)) {
+			value |= (INTERRUPT_ID_BITS_ITS - 1) << 19;
+			value |= GICD_TYPER_LPIS;
+		} else {
+			value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
+		}
 		break;
 	case GICD_IIDR:
 		value = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
@@ -742,7 +747,8 @@ static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
 
 	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
 	    (reg & GICR_CTLR_ENABLE_LPIS)) {
-		/* Eventually do something */
+		vgic_enable_lpis(vcpu);
+		dist->lpis_enabled = true;
 	}
 
 	return 0;
@@ -772,6 +778,8 @@ static int vgic_mmio_read_v3r_typer(struct kvm_vcpu *vcpu,
 	value |= ((target_vcpu_id & 0xffff) << 8);
 	if (target_vcpu_id == atomic_read(&vcpu->kvm->online_vcpus) - 1)
 		value |= GICR_TYPER_LAST;
+	if (vgic_has_its(vcpu->kvm))
+		value |= GICR_TYPER_PLPIS;
 
 	write_mask64(value, addr & 7, len, val);
 	return 0;
-- 
2.7.3

^ permalink raw reply related	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
  2016-03-26  2:13   ` Andre Przywara
@ 2016-04-03  9:15     ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03  9:15 UTC (permalink / raw)
  To: Andre Przywara; +Cc: Marc Zyngier, kvm, kvmarm, linux-arm-kernel

On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
> The ARM GICv3 ITS MSI controller requires a device ID to be able to
> assign the proper interrupt vector. On real hardware, this ID is
> sampled from the bus. To be able to emulate an ITS controller, extend
> the KVM MSI interface to let userspace provide such a device ID. For
> PCI devices, the device ID is simply the 16-bit bus-device-function
> triplet, which should be easily available to the userland tool.
> 
> Also there is a new KVM capability which advertises whether the
> current VM requires a device ID to be set along with the MSI data.
> This flag is still reported as not available everywhere, later we will
> enable it when ITS emulation is used.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> Reviewed-by: Eric Auger <eric.auger@linaro.org>
> ---
>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
>  include/uapi/linux/kvm.h          |  5 ++++-
>  2 files changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index cb2ef0b..8f7351d 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -2163,10 +2163,18 @@ struct kvm_msi {
>  	__u32 address_hi;
>  	__u32 data;
>  	__u32 flags;
> -	__u8  pad[16];
> +	__u32 devid;

Are we imposing any unfortunate restrictions for other architectures by
using a u32 over a u64 for the device ID?

-Christoffer

> +	__u8  pad[12];
>  };
>  
> -No flags are defined so far. The corresponding field must be 0.
> +flags: KVM_MSI_VALID_DEVID: devid contains a valid value
> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
> +       for the device that wrote the MSI message.
> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
> +
> +The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
> +the device ID. If this capability is not set, userland cannot rely on
> +the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
>  
>  
>  4.71 KVM_CREATE_PIT2
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 50f44a2..6a02871 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
>  #define KVM_CAP_S390_RI 124
>  #define KVM_CAP_ARM_PMU_V3 125
>  #define KVM_CAP_VCPU_ATTRIBUTES 126
> +#define KVM_CAP_MSI_DEVID 127
>  
>  #ifdef KVM_CAP_IRQ_ROUTING
>  
> @@ -1010,12 +1011,14 @@ struct kvm_one_reg {
>  	__u64 addr;
>  };
>  
> +#define KVM_MSI_VALID_DEVID	(1U << 0)
>  struct kvm_msi {
>  	__u32 address_lo;
>  	__u32 address_hi;
>  	__u32 data;
>  	__u32 flags;
> -	__u8  pad[16];
> +	__u32 devid;
> +	__u8  pad[12];
>  };
>  
>  struct kvm_arm_device_addr {
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
@ 2016-04-03  9:15     ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03  9:15 UTC (permalink / raw)
  To: linux-arm-kernel

On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
> The ARM GICv3 ITS MSI controller requires a device ID to be able to
> assign the proper interrupt vector. On real hardware, this ID is
> sampled from the bus. To be able to emulate an ITS controller, extend
> the KVM MSI interface to let userspace provide such a device ID. For
> PCI devices, the device ID is simply the 16-bit bus-device-function
> triplet, which should be easily available to the userland tool.
> 
> Also there is a new KVM capability which advertises whether the
> current VM requires a device ID to be set along with the MSI data.
> This flag is still reported as not available everywhere, later we will
> enable it when ITS emulation is used.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> Reviewed-by: Eric Auger <eric.auger@linaro.org>
> ---
>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
>  include/uapi/linux/kvm.h          |  5 ++++-
>  2 files changed, 14 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index cb2ef0b..8f7351d 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -2163,10 +2163,18 @@ struct kvm_msi {
>  	__u32 address_hi;
>  	__u32 data;
>  	__u32 flags;
> -	__u8  pad[16];
> +	__u32 devid;

Are we imposing any unfortunate restrictions for other architectures by
using a u32 over a u64 for the device ID?

-Christoffer

> +	__u8  pad[12];
>  };
>  
> -No flags are defined so far. The corresponding field must be 0.
> +flags: KVM_MSI_VALID_DEVID: devid contains a valid value
> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
> +       for the device that wrote the MSI message.
> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
> +
> +The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
> +the device ID. If this capability is not set, userland cannot rely on
> +the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
>  
>  
>  4.71 KVM_CREATE_PIT2
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 50f44a2..6a02871 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
>  #define KVM_CAP_S390_RI 124
>  #define KVM_CAP_ARM_PMU_V3 125
>  #define KVM_CAP_VCPU_ATTRIBUTES 126
> +#define KVM_CAP_MSI_DEVID 127
>  
>  #ifdef KVM_CAP_IRQ_ROUTING
>  
> @@ -1010,12 +1011,14 @@ struct kvm_one_reg {
>  	__u64 addr;
>  };
>  
> +#define KVM_MSI_VALID_DEVID	(1U << 0)
>  struct kvm_msi {
>  	__u32 address_lo;
>  	__u32 address_hi;
>  	__u32 data;
>  	__u32 flags;
> -	__u8  pad[16];
> +	__u32 devid;
> +	__u8  pad[12];
>  };
>  
>  struct kvm_arm_device_addr {
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-03 10:08     ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: Andre Przywara; +Cc: Marc Zyngier, Eric Auger, linux-arm-kernel, kvmarm, kvm

On Sat, Mar 26, 2016 at 02:14:04AM +0000, Andre Przywara wrote:
> Add emulation for some basic MMIO registers used in the ITS emulation.
> This includes:
> - GITS_{CTLR,TYPER,IIDR}
> - ID registers
> - GITS_{CBASER,CREADR,CWRITER}
>   those implement the ITS command buffer handling
> 
> Most of the handlers are pretty straight forward, but CWRITER goes
> some extra miles to allow fine grained locking. The idea here
> is to let only the first instance iterate through the command ring
> buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
> by that first instance and handled as well. The ITS lock is thus only
> hold for very small periods of time and is dropped before the actual

s/hold/held/

> command handler is called.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h            |   3 +
>  include/linux/irqchip/arm-gic-v3.h |   8 ++
>  virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h           |   6 +
>  virt/kvm/arm/vgic/vgic_init.c      |   2 +
>  5 files changed, 284 insertions(+), 7 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c79bed5..bafea11 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -115,6 +115,9 @@ struct vgic_io_device {
>  struct vgic_its {
>  	bool			enabled;
>  	spinlock_t		lock;

exactly what does this lock protect?

> +	u64			cbaser;
> +	int			creadr;
> +	int			cwriter;
>  };
>  
>  struct vgic_dist {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index a813c3e..7011b98 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -179,15 +179,23 @@
>  #define GITS_BASER			0x0100
>  #define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
> +#define GITS_PIDR4			0xffd0
> +#define GITS_CIDR0			0xfff0
> +#define GITS_CIDR1			0xfff4
> +#define GITS_CIDR2			0xfff8
> +#define GITS_CIDR3			0xfffc
>  
>  #define GITS_TRANSLATER			0x10040
>  
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
> +#define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
>  #define GITS_TYPER_PTA			(1UL << 19)
> +#define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
>  #define GITS_CBASER_VALID		(1UL << 63)
>  #define GITS_CBASER_nCnB		(0UL << 59)
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 49dd5e4..de8d360 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -31,23 +31,263 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> +
> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u32 reg;
> +
> +	reg = GITS_CTLR_QUIESCENT;
> +	if (its->enabled)
> +		reg |= GITS_CTLR_ENABLE;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +
> +        if (addr - iodev->base_addr == 0)
> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	u64 reg = GITS_TYPER_PLPIS;
> +
> +	/*
> +	 * We use linear CPU numbers for redistributor addressing,
> +	 * so GITS_TYPER.PTA is 0.
> +	 * To avoid memory waste on the guest side, we keep the
> +	 * number of IDBits and DevBits low for the time being.
> +	 * This could later be made configurable by userland.
> +	 * Since we have all collections in linked list, we claim

s/linked list/linked lists/

is the point here not more specifically that we hold the data in kernel
data structures not visible by guest memory, rather than addressing
which specific data structure we use in the kernel?

> +	 * that we can hold all of the collection tables in our
> +	 * own memory and that the ITT entry size is 1 byte (the
> +	 * smallest possible one).
> +	 */
> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u32 reg = 0;
> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
> +
> +	switch (idreg) {
> +	case GITS_PIDR2:
> +		reg = GIC_PIDR2_ARCH_GICv3;
> +		break;
> +	case GITS_PIDR4:
> +		/* This is a 64K software visible page */
> +		reg = 0x40;
> +		break;
> +	/* Those are the ID registers for (any) GIC. */
> +	case GITS_CIDR0:
> +		reg = 0x0d;
> +		break;
> +	case GITS_CIDR1:
> +		reg = 0xf0;
> +		break;
> +	case GITS_CIDR2:
> +		reg = 0x05;
> +		break;
> +	case GITS_CIDR3:
> +		reg = 0xb1;
> +		break;
> +	}
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +/*
> + * This function is called with both the ITS and the distributor lock dropped,
> + * so the actual command handlers must take the respective locks when needed.
> + */

I don't understand this comment.  Are these requirements for the locking
situation when entering this function or is explaining when this
function is called?

> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> +{
> +	return -ENODEV;
> +}
> +
> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	write_mask64(its->cbaser, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	if (its->enabled)
> +		return 0;
> +
> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
> +	its->creadr = 0;
> +
> +	return 0;
> +}
> +
> +static int its_cmd_buffer_size(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return ((its->cbaser & 0xff) + 1) << 12;
> +}
> +
> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return BASER_BASE_ADDRESS(its->cbaser);
> +}
> +
> +/*
> + * By writing to CWRITER the guest announces new commands to be processed.
> + * Since we cannot read from guest memory inside the ITS spinlock, we
> + * iterate over the command buffer (with the lock dropped) until the read
> + * pointer matches the write pointer. Other VCPUs writing this register in the
> + * meantime will just update the write pointer, leaving the command
> + * processing to the first instance of the function.
> + */
> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
> +				       struct kvm_io_device *this,
> +				       gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
> +	u64 cmd_buf[4];
> +	u32 reg;
> +	bool finished;
> +
> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
> +	reg &= 0xfffe0;
> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
> +		return 0;
> +
> +	spin_lock(&its->lock);
> +
> +	/*
> +	 * If there is still another VCPU handling commands, let this
> +	 * one pick up the new CWRITER and process "our" new commands as well.
> +	 */
> +	finished = (its->cwriter != its->creadr);
> +	its->cwriter = reg;
> +
> +	spin_unlock(&its->lock);
> +
> +	while (!finished) {
> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
> +					 cmd_buf, 32);
> +		if (ret) {
> +			/*
> +			 * Gah, we are screwed. Reset CWRITER to that command

s/that/the/

> +			 * that we have finished processing and return.
> +			 */

what does this situation mean? That the guest programmed bogus values
into GITS_BASERn, or?  (Is that allowed?)

> +			spin_lock(&its->lock);
> +			its->cwriter = its->creadr;
> +			spin_unlock(&its->lock);
> +			break;
> +		}
> +		vits_handle_command(vcpu, cmd_buf);
> +
> +		spin_lock(&its->lock);
> +		its->creadr += 32;
> +		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
> +			its->creadr = 0;
> +		finished = (its->creadr == its->cwriter);
> +		spin_unlock(&its->lock);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->cwriter & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->creadr & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
>  struct vgic_register_region its_registers[] = {
>  	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>  		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
>  };
>  
>  /* This is called on setting the LPI enable bit in the redistributor. */
> @@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	int nr_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct vgic_io_device *regions;
>  	int ret, i;
>  
> +	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
> +	if (!dist->pendbaser)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> @@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
>  
>  	return -ENXIO;
>  }
> +
> +void vits_destroy(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +
> +	if (!vgic_has_its(kvm))
> +		return;
> +
> +	kfree(dist->pendbaser);
> +
> +	its->enabled = false;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 4e7dcb8..08f97d1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
>  	return;
>  }
> +
> +static inline void vits_destroy(struct kvm *kvm)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index dcfb93d..e4459e3 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
>  
>  	kvm_vgic_dist_destroy(kvm);
>  
> +	vits_destroy(kvm);
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_destroy(vcpu);
>  }
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-04-03 10:08     ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: linux-arm-kernel

On Sat, Mar 26, 2016 at 02:14:04AM +0000, Andre Przywara wrote:
> Add emulation for some basic MMIO registers used in the ITS emulation.
> This includes:
> - GITS_{CTLR,TYPER,IIDR}
> - ID registers
> - GITS_{CBASER,CREADR,CWRITER}
>   those implement the ITS command buffer handling
> 
> Most of the handlers are pretty straight forward, but CWRITER goes
> some extra miles to allow fine grained locking. The idea here
> is to let only the first instance iterate through the command ring
> buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
> by that first instance and handled as well. The ITS lock is thus only
> hold for very small periods of time and is dropped before the actual

s/hold/held/

> command handler is called.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h            |   3 +
>  include/linux/irqchip/arm-gic-v3.h |   8 ++
>  virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h           |   6 +
>  virt/kvm/arm/vgic/vgic_init.c      |   2 +
>  5 files changed, 284 insertions(+), 7 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c79bed5..bafea11 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -115,6 +115,9 @@ struct vgic_io_device {
>  struct vgic_its {
>  	bool			enabled;
>  	spinlock_t		lock;

exactly what does this lock protect?

> +	u64			cbaser;
> +	int			creadr;
> +	int			cwriter;
>  };
>  
>  struct vgic_dist {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index a813c3e..7011b98 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -179,15 +179,23 @@
>  #define GITS_BASER			0x0100
>  #define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
> +#define GITS_PIDR4			0xffd0
> +#define GITS_CIDR0			0xfff0
> +#define GITS_CIDR1			0xfff4
> +#define GITS_CIDR2			0xfff8
> +#define GITS_CIDR3			0xfffc
>  
>  #define GITS_TRANSLATER			0x10040
>  
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
> +#define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
>  #define GITS_TYPER_PTA			(1UL << 19)
> +#define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
>  #define GITS_CBASER_VALID		(1UL << 63)
>  #define GITS_CBASER_nCnB		(0UL << 59)
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 49dd5e4..de8d360 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -31,23 +31,263 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> +
> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u32 reg;
> +
> +	reg = GITS_CTLR_QUIESCENT;
> +	if (its->enabled)
> +		reg |= GITS_CTLR_ENABLE;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +
> +        if (addr - iodev->base_addr == 0)
> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	u64 reg = GITS_TYPER_PLPIS;
> +
> +	/*
> +	 * We use linear CPU numbers for redistributor addressing,
> +	 * so GITS_TYPER.PTA is 0.
> +	 * To avoid memory waste on the guest side, we keep the
> +	 * number of IDBits and DevBits low for the time being.
> +	 * This could later be made configurable by userland.
> +	 * Since we have all collections in linked list, we claim

s/linked list/linked lists/

is the point here not more specifically that we hold the data in kernel
data structures not visible by guest memory, rather than addressing
which specific data structure we use in the kernel?

> +	 * that we can hold all of the collection tables in our
> +	 * own memory and that the ITT entry size is 1 byte (the
> +	 * smallest possible one).
> +	 */
> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u32 reg = 0;
> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
> +
> +	switch (idreg) {
> +	case GITS_PIDR2:
> +		reg = GIC_PIDR2_ARCH_GICv3;
> +		break;
> +	case GITS_PIDR4:
> +		/* This is a 64K software visible page */
> +		reg = 0x40;
> +		break;
> +	/* Those are the ID registers for (any) GIC. */
> +	case GITS_CIDR0:
> +		reg = 0x0d;
> +		break;
> +	case GITS_CIDR1:
> +		reg = 0xf0;
> +		break;
> +	case GITS_CIDR2:
> +		reg = 0x05;
> +		break;
> +	case GITS_CIDR3:
> +		reg = 0xb1;
> +		break;
> +	}
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +/*
> + * This function is called with both the ITS and the distributor lock dropped,
> + * so the actual command handlers must take the respective locks when needed.
> + */

I don't understand this comment.  Are these requirements for the locking
situation when entering this function or is explaining when this
function is called?

> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> +{
> +	return -ENODEV;
> +}
> +
> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	write_mask64(its->cbaser, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	if (its->enabled)
> +		return 0;
> +
> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
> +	its->creadr = 0;
> +
> +	return 0;
> +}
> +
> +static int its_cmd_buffer_size(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return ((its->cbaser & 0xff) + 1) << 12;
> +}
> +
> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return BASER_BASE_ADDRESS(its->cbaser);
> +}
> +
> +/*
> + * By writing to CWRITER the guest announces new commands to be processed.
> + * Since we cannot read from guest memory inside the ITS spinlock, we
> + * iterate over the command buffer (with the lock dropped) until the read
> + * pointer matches the write pointer. Other VCPUs writing this register in the
> + * meantime will just update the write pointer, leaving the command
> + * processing to the first instance of the function.
> + */
> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
> +				       struct kvm_io_device *this,
> +				       gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
> +	u64 cmd_buf[4];
> +	u32 reg;
> +	bool finished;
> +
> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
> +	reg &= 0xfffe0;
> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
> +		return 0;
> +
> +	spin_lock(&its->lock);
> +
> +	/*
> +	 * If there is still another VCPU handling commands, let this
> +	 * one pick up the new CWRITER and process "our" new commands as well.
> +	 */
> +	finished = (its->cwriter != its->creadr);
> +	its->cwriter = reg;
> +
> +	spin_unlock(&its->lock);
> +
> +	while (!finished) {
> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
> +					 cmd_buf, 32);
> +		if (ret) {
> +			/*
> +			 * Gah, we are screwed. Reset CWRITER to that command

s/that/the/

> +			 * that we have finished processing and return.
> +			 */

what does this situation mean? That the guest programmed bogus values
into GITS_BASERn, or?  (Is that allowed?)

> +			spin_lock(&its->lock);
> +			its->cwriter = its->creadr;
> +			spin_unlock(&its->lock);
> +			break;
> +		}
> +		vits_handle_command(vcpu, cmd_buf);
> +
> +		spin_lock(&its->lock);
> +		its->creadr += 32;
> +		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
> +			its->creadr = 0;
> +		finished = (its->creadr == its->cwriter);
> +		spin_unlock(&its->lock);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->cwriter & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->creadr & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
>  struct vgic_register_region its_registers[] = {
>  	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>  		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
>  };
>  
>  /* This is called on setting the LPI enable bit in the redistributor. */
> @@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	int nr_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct vgic_io_device *regions;
>  	int ret, i;
>  
> +	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
> +	if (!dist->pendbaser)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> @@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
>  
>  	return -ENXIO;
>  }
> +
> +void vits_destroy(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +
> +	if (!vgic_has_its(kvm))
> +		return;
> +
> +	kfree(dist->pendbaser);
> +
> +	its->enabled = false;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 4e7dcb8..08f97d1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
>  	return;
>  }
> +
> +static inline void vits_destroy(struct kvm *kvm)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index dcfb93d..e4459e3 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
>  
>  	kvm_vgic_dist_destroy(kvm);
>  
> +	vits_destroy(kvm);
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_destroy(vcpu);
>  }
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-03 10:08     ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: Andre Przywara; +Cc: Marc Zyngier, kvm, kvmarm, linux-arm-kernel

On Sat, Mar 26, 2016 at 02:14:00AM +0000, Andre Przywara wrote:
> KVM capabilities can be a per-VM property, though ARM/ARM64 currently
> does not pass on the VM pointer to the architecture specific
> capability handlers.
> Add a "struct kvm*" parameter to those function to later allow proper
> per-VM capability reporting.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
@ 2016-04-03 10:08     ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: linux-arm-kernel

On Sat, Mar 26, 2016 at 02:14:00AM +0000, Andre Przywara wrote:
> KVM capabilities can be a per-VM property, though ARM/ARM64 currently
> does not pass on the VM pointer to the architecture specific
> capability handlers.
> Add a "struct kvm*" parameter to those function to later allow proper
> per-VM capability reporting.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-03 10:08     ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: Andre Przywara; +Cc: Marc Zyngier, Eric Auger, linux-arm-kernel, kvmarm, kvm

On Sat, Mar 26, 2016 at 02:14:01AM +0000, Andre Przywara wrote:
> The ARM GICv3 ITS controller requires a separate register frame to
> cover ITS specific registers. Add a new VGIC address type and store
> the address in a field in the vgic_dist structure.
> Provide a function to check whether userland has provided the address,
> so ITS functionality can be guarded by that check.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>  arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>  include/kvm/vgic/vgic.h                        |  5 +++++
>  virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>  virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>  virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>  6 files changed, 34 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
> index 59541d4..087e2d9 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -39,6 +39,15 @@ Groups:
>        Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>        This address needs to be 64K aligned.
>  
> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
> +      Base address in the guest physical address space of the GICv3 ITS
> +      control register frame. The ITS allows MSI(-X) interrupts to be
> +      injected into guests. This extension is optional, if the kernel

s/optional, if/optional. If/

> +      does not support the ITS, the call returns -ENODEV.
> +      This memory is solely for the guest to access the ITS control

s/memory/address region/ ?

> +      registers and does not cover the ITS translation register.
> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
> +      This address needs to be 64K aligned and the region covers 64 KByte.

s/KByte/KBytes/
...or you could you 64K both places in this line.

>  
>    KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>    Attributes:
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index f209ea1..c2b257d 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -87,9 +87,11 @@ struct kvm_regs {
>  /* Supported VGICv3 address types  */
>  #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>  #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>  
>  #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>  #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>  
>  #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>  #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 7c1d145..11344e6 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -135,6 +135,9 @@ struct vgic_dist {
>  		gpa_t			vgic_redist_base;
>  	};
>  
> +	/* The base address of the ITS control register frame */
> +	gpa_t			vgic_its_base;
> +
>  	/* distributor enabled */
>  	u32			enabled;
>  
> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>  	return kvm_vgic_global_state.max_gic_vcpus;
>  }
>  
> +bool vgic_has_its(struct kvm *kvm);
> +
>  #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
> index 65395af..1de2478 100644
> --- a/virt/kvm/arm/vgic/vgic.c
> +++ b/virt/kvm/arm/vgic/vgic.c
> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>  
>  	return map_is_active;
>  }
> +
> +bool vgic_has_its(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +
> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
> +		return false;
> +
> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
> +}
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index ac655b5..2301e03 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>  	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>  	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>  	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>  
>  out_unlock:
>  	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
> index 7f78a16..3ec2ac3 100644
> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>  		block_size = KVM_VGIC_V3_REDIST_SIZE;
>  		alignment = SZ_64K;
>  		break;
> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
> +		addr_ptr = &vgic->vgic_its_base;
> +		block_size = KVM_VGIC_V3_ITS_SIZE;
> +		alignment = SZ_64K;
> +		break;
>  #endif
>  	default:
>  		r = -ENODEV;
> @@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>  		switch (attr->attr) {
>  		case KVM_VGIC_V3_ADDR_TYPE_DIST:
>  		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
> +		case KVM_VGIC_V3_ADDR_TYPE_ITS:
>  			return 0;
>  		}
>  		break;
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-04-03 10:08     ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: linux-arm-kernel

On Sat, Mar 26, 2016 at 02:14:01AM +0000, Andre Przywara wrote:
> The ARM GICv3 ITS controller requires a separate register frame to
> cover ITS specific registers. Add a new VGIC address type and store
> the address in a field in the vgic_dist structure.
> Provide a function to check whether userland has provided the address,
> so ITS functionality can be guarded by that check.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>  arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>  include/kvm/vgic/vgic.h                        |  5 +++++
>  virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>  virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>  virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>  6 files changed, 34 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
> index 59541d4..087e2d9 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -39,6 +39,15 @@ Groups:
>        Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>        This address needs to be 64K aligned.
>  
> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
> +      Base address in the guest physical address space of the GICv3 ITS
> +      control register frame. The ITS allows MSI(-X) interrupts to be
> +      injected into guests. This extension is optional, if the kernel

s/optional, if/optional. If/

> +      does not support the ITS, the call returns -ENODEV.
> +      This memory is solely for the guest to access the ITS control

s/memory/address region/ ?

> +      registers and does not cover the ITS translation register.
> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
> +      This address needs to be 64K aligned and the region covers 64 KByte.

s/KByte/KBytes/
...or you could you 64K both places in this line.

>  
>    KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>    Attributes:
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index f209ea1..c2b257d 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -87,9 +87,11 @@ struct kvm_regs {
>  /* Supported VGICv3 address types  */
>  #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>  #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>  
>  #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>  #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>  
>  #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>  #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 7c1d145..11344e6 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -135,6 +135,9 @@ struct vgic_dist {
>  		gpa_t			vgic_redist_base;
>  	};
>  
> +	/* The base address of the ITS control register frame */
> +	gpa_t			vgic_its_base;
> +
>  	/* distributor enabled */
>  	u32			enabled;
>  
> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>  	return kvm_vgic_global_state.max_gic_vcpus;
>  }
>  
> +bool vgic_has_its(struct kvm *kvm);
> +
>  #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
> index 65395af..1de2478 100644
> --- a/virt/kvm/arm/vgic/vgic.c
> +++ b/virt/kvm/arm/vgic/vgic.c
> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>  
>  	return map_is_active;
>  }
> +
> +bool vgic_has_its(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +
> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
> +		return false;
> +
> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
> +}
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index ac655b5..2301e03 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>  	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>  	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>  	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>  
>  out_unlock:
>  	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
> index 7f78a16..3ec2ac3 100644
> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>  		block_size = KVM_VGIC_V3_REDIST_SIZE;
>  		alignment = SZ_64K;
>  		break;
> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
> +		addr_ptr = &vgic->vgic_its_base;
> +		block_size = KVM_VGIC_V3_ITS_SIZE;
> +		alignment = SZ_64K;
> +		break;
>  #endif
>  	default:
>  		r = -ENODEV;
> @@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>  		switch (attr->attr) {
>  		case KVM_VGIC_V3_ADDR_TYPE_DIST:
>  		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
> +		case KVM_VGIC_V3_ADDR_TYPE_ITS:
>  			return 0;
>  		}
>  		break;
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-03 10:08     ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: Andre Przywara; +Cc: Marc Zyngier, kvm, kvmarm, linux-arm-kernel

On Sat, Mar 26, 2016 at 02:14:02AM +0000, Andre Przywara wrote:
> In the GICv3 redistributor there are the PENDBASER and PROPBASER
> registers which we did not emulate so far, as they only make sense
> when having an ITS. In preparation for that emulate those MMIO
> accesses by storing the 64-bit data written into it into a variable
> which we later read in the ITS emulation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h       |  8 ++++++++
>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 11344e6..8ea5dd7 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -145,6 +145,14 @@ struct vgic_dist {
>  
>  	struct vgic_io_device	*dist_iodevs;
>  	struct vgic_io_device	*redist_iodevs;
> +
> +	/* Address of LPI configuration table shared by all redistributors */
> +	u64			propbaser;

is this an architectural requirement or an implementation decision?

> +
> +	/* Addresses of LPI pending tables per redistributor */
> +	u64			*pendbaser;

have we considered having an array of redistributors and putting the
field in each one of those instead?

> +
> +	bool			lpis_enabled;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 7d5d630..252b9aff 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	write_mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	write_mask64(pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-04-03 10:08     ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-04-03 10:08 UTC (permalink / raw)
  To: linux-arm-kernel

On Sat, Mar 26, 2016 at 02:14:02AM +0000, Andre Przywara wrote:
> In the GICv3 redistributor there are the PENDBASER and PROPBASER
> registers which we did not emulate so far, as they only make sense
> when having an ITS. In preparation for that emulate those MMIO
> accesses by storing the 64-bit data written into it into a variable
> which we later read in the ITS emulation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h       |  8 ++++++++
>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 11344e6..8ea5dd7 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -145,6 +145,14 @@ struct vgic_dist {
>  
>  	struct vgic_io_device	*dist_iodevs;
>  	struct vgic_io_device	*redist_iodevs;
> +
> +	/* Address of LPI configuration table shared by all redistributors */
> +	u64			propbaser;

is this an architectural requirement or an implementation decision?

> +
> +	/* Addresses of LPI pending tables per redistributor */
> +	u64			*pendbaser;

have we considered having an array of redistributors and putting the
field in each one of those instead?

> +
> +	bool			lpis_enabled;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 7d5d630..252b9aff 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	write_mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	write_mask64(pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> -- 
> 2.7.3
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-04-03 10:08     ` Christoffer Dall
@ 2016-04-05 12:47       ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 12:47 UTC (permalink / raw)
  To: Christoffer Dall, Andre Przywara
  Cc: Marc Zyngier, kvmarm, linux-arm-kernel, kvm

Hi Andre,
On 04/03/2016 12:08 PM, Christoffer Dall wrote:
> On Sat, Mar 26, 2016 at 02:14:01AM +0000, Andre Przywara wrote:
>> The ARM GICv3 ITS controller requires a separate register frame to
>> cover ITS specific registers. Add a new VGIC address type and store
>> the address in a field in the vgic_dist structure.
>> Provide a function to check whether userland has provided the address,
>> so ITS functionality can be guarded by that check.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>>  arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>>  include/kvm/vgic/vgic.h                        |  5 +++++
>>  virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>>  virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>>  virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>>  6 files changed, 34 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> index 59541d4..087e2d9 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> @@ -39,6 +39,15 @@ Groups:
>>        Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>        This address needs to be 64K aligned.
>>  
>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>> +      Base address in the guest physical address space of the GICv3 ITS
>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>> +      injected into guests. This extension is optional, if the kernel
> 
> s/optional, if/optional. If/
> 
>> +      does not support the ITS, the call returns -ENODEV.
>> +      This memory is solely for the guest to access the ITS control
> 
> s/memory/address region/ ?
> 
>> +      registers and does not cover the ITS translation register.
>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>> +      This address needs to be 64K aligned and the region covers 64 KByte.
> 
> s/KByte/KBytes/
> ...or you could you 64K both places in this line.
> 
>>  
>>    KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>    Attributes:
>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>> index f209ea1..c2b257d 100644
>> --- a/arch/arm64/include/uapi/asm/kvm.h
>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>> @@ -87,9 +87,11 @@ struct kvm_regs {
>>  /* Supported VGICv3 address types  */
>>  #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>>  #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>  
>>  #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>>  #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>  
>>  #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>>  #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 7c1d145..11344e6 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -135,6 +135,9 @@ struct vgic_dist {
>>  		gpa_t			vgic_redist_base;
>>  	};
>>  
>> +	/* The base address of the ITS control register frame */
>> +	gpa_t			vgic_its_base;
>> +
>>  	/* distributor enabled */
>>  	u32			enabled;
>>  
>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>  	return kvm_vgic_global_state.max_gic_vcpus;
>>  }
>>  
>> +bool vgic_has_its(struct kvm *kvm);
is it mandated to expose this function outside of the vgic subsystem?
why not declaring it in /virt/kvm/arm/vgic/vgic.h?
>> +
>>  #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index 65395af..1de2478 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>  
>>  	return map_is_active;
>>  }
>> +
>> +bool vgic_has_its(struct kvm *kvm)
I would implement this function vgic_init.c or vgic_kvm_device.c leaving
vgic.c for pure vgic state mgt.

Eric
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +
>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>> +		return false;
>> +
>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index ac655b5..2301e03 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>>  	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>>  	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>>  	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>  
>>  out_unlock:
>>  	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> index 7f78a16..3ec2ac3 100644
>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>>  		block_size = KVM_VGIC_V3_REDIST_SIZE;
>>  		alignment = SZ_64K;
>>  		break;
>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>> +		addr_ptr = &vgic->vgic_its_base;
>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>> +		alignment = SZ_64K;
>> +		break;
>>  #endif
>>  	default:
>>  		r = -ENODEV;
>> @@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>>  		switch (attr->attr) {
>>  		case KVM_VGIC_V3_ADDR_TYPE_DIST:
>>  		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
>> +		case KVM_VGIC_V3_ADDR_TYPE_ITS:
>>  			return 0;
>>  		}
>>  		break;
>> -- 
>> 2.7.3
>>

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-04-05 12:47       ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 12:47 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,
On 04/03/2016 12:08 PM, Christoffer Dall wrote:
> On Sat, Mar 26, 2016 at 02:14:01AM +0000, Andre Przywara wrote:
>> The ARM GICv3 ITS controller requires a separate register frame to
>> cover ITS specific registers. Add a new VGIC address type and store
>> the address in a field in the vgic_dist structure.
>> Provide a function to check whether userland has provided the address,
>> so ITS functionality can be guarded by that check.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>>  arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>>  include/kvm/vgic/vgic.h                        |  5 +++++
>>  virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>>  virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>>  virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>>  6 files changed, 34 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> index 59541d4..087e2d9 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> @@ -39,6 +39,15 @@ Groups:
>>        Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>        This address needs to be 64K aligned.
>>  
>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>> +      Base address in the guest physical address space of the GICv3 ITS
>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>> +      injected into guests. This extension is optional, if the kernel
> 
> s/optional, if/optional. If/
> 
>> +      does not support the ITS, the call returns -ENODEV.
>> +      This memory is solely for the guest to access the ITS control
> 
> s/memory/address region/ ?
> 
>> +      registers and does not cover the ITS translation register.
>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>> +      This address needs to be 64K aligned and the region covers 64 KByte.
> 
> s/KByte/KBytes/
> ...or you could you 64K both places in this line.
> 
>>  
>>    KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>    Attributes:
>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>> index f209ea1..c2b257d 100644
>> --- a/arch/arm64/include/uapi/asm/kvm.h
>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>> @@ -87,9 +87,11 @@ struct kvm_regs {
>>  /* Supported VGICv3 address types  */
>>  #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>>  #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>  
>>  #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>>  #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>  
>>  #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>>  #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 7c1d145..11344e6 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -135,6 +135,9 @@ struct vgic_dist {
>>  		gpa_t			vgic_redist_base;
>>  	};
>>  
>> +	/* The base address of the ITS control register frame */
>> +	gpa_t			vgic_its_base;
>> +
>>  	/* distributor enabled */
>>  	u32			enabled;
>>  
>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>  	return kvm_vgic_global_state.max_gic_vcpus;
>>  }
>>  
>> +bool vgic_has_its(struct kvm *kvm);
is it mandated to expose this function outside of the vgic subsystem?
why not declaring it in /virt/kvm/arm/vgic/vgic.h?
>> +
>>  #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index 65395af..1de2478 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>  
>>  	return map_is_active;
>>  }
>> +
>> +bool vgic_has_its(struct kvm *kvm)
I would implement this function vgic_init.c or vgic_kvm_device.c leaving
vgic.c for pure vgic state mgt.

Eric
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +
>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>> +		return false;
>> +
>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index ac655b5..2301e03 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>>  	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>>  	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>>  	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>  
>>  out_unlock:
>>  	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> index 7f78a16..3ec2ac3 100644
>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>>  		block_size = KVM_VGIC_V3_REDIST_SIZE;
>>  		alignment = SZ_64K;
>>  		break;
>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>> +		addr_ptr = &vgic->vgic_its_base;
>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>> +		alignment = SZ_64K;
>> +		break;
>>  #endif
>>  	default:
>>  		r = -ENODEV;
>> @@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
>>  		switch (attr->attr) {
>>  		case KVM_VGIC_V3_ADDR_TYPE_DIST:
>>  		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
>> +		case KVM_VGIC_V3_ADDR_TYPE_ITS:
>>  			return 0;
>>  		}
>>  		break;
>> -- 
>> 2.7.3
>>

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-05 12:48     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 12:48 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm

On 03/26/2016 03:14 AM, Andre Przywara wrote:
> KVM capabilities can be a per-VM property, though ARM/ARM64 currently
> does not pass on the VM pointer to the architecture specific
> capability handlers.
> Add a "struct kvm*" parameter to those function to later allow proper
> per-VM capability reporting.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Reviewed-by: Eric Auger <eric.auger@linaro.org>
> ---
>  arch/arm/include/asm/kvm_host.h   | 2 +-
>  arch/arm/kvm/arm.c                | 2 +-
>  arch/arm64/include/asm/kvm_host.h | 2 +-
>  arch/arm64/kvm/reset.c            | 2 +-
>  4 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> index 3850701..494b004 100644
> --- a/arch/arm/include/asm/kvm_host.h
> +++ b/arch/arm/include/asm/kvm_host.h
> @@ -265,7 +265,7 @@ static inline void __cpu_init_stage2(void)
>  	kvm_call_hyp(__init_stage2_translation);
>  }
>  
> -static inline int kvm_arch_dev_ioctl_check_extension(long ext)
> +static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
>  {
>  	return 0;
>  }
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index 5b9d9e3..e2e9be2 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -205,7 +205,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>  		r = KVM_MAX_VCPUS;
>  		break;
>  	default:
> -		r = kvm_arch_dev_ioctl_check_extension(ext);
> +		r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
>  		break;
>  	}
>  	return r;
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 71fa6fe..2cdd7ae 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -46,7 +46,7 @@
>  
>  int __attribute_const__ kvm_target_cpu(void);
>  int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
> -int kvm_arch_dev_ioctl_check_extension(long ext);
> +int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
>  
>  struct kvm_arch {
>  	/* The VMID generation used for the virt. memory system */
> diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
> index 9677bf0..2ba7c8d 100644
> --- a/arch/arm64/kvm/reset.c
> +++ b/arch/arm64/kvm/reset.c
> @@ -63,7 +63,7 @@ static bool cpu_has_32bit_el1(void)
>   * We currently assume that the number of HW registers is uniform
>   * across all CPUs (see cpuinfo_sanity_check).
>   */
> -int kvm_arch_dev_ioctl_check_extension(long ext)
> +int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
>  {
>  	int r;
>  
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
@ 2016-04-05 12:48     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 12:48 UTC (permalink / raw)
  To: linux-arm-kernel

On 03/26/2016 03:14 AM, Andre Przywara wrote:
> KVM capabilities can be a per-VM property, though ARM/ARM64 currently
> does not pass on the VM pointer to the architecture specific
> capability handlers.
> Add a "struct kvm*" parameter to those function to later allow proper
> per-VM capability reporting.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Reviewed-by: Eric Auger <eric.auger@linaro.org>
> ---
>  arch/arm/include/asm/kvm_host.h   | 2 +-
>  arch/arm/kvm/arm.c                | 2 +-
>  arch/arm64/include/asm/kvm_host.h | 2 +-
>  arch/arm64/kvm/reset.c            | 2 +-
>  4 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> index 3850701..494b004 100644
> --- a/arch/arm/include/asm/kvm_host.h
> +++ b/arch/arm/include/asm/kvm_host.h
> @@ -265,7 +265,7 @@ static inline void __cpu_init_stage2(void)
>  	kvm_call_hyp(__init_stage2_translation);
>  }
>  
> -static inline int kvm_arch_dev_ioctl_check_extension(long ext)
> +static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
>  {
>  	return 0;
>  }
> diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
> index 5b9d9e3..e2e9be2 100644
> --- a/arch/arm/kvm/arm.c
> +++ b/arch/arm/kvm/arm.c
> @@ -205,7 +205,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
>  		r = KVM_MAX_VCPUS;
>  		break;
>  	default:
> -		r = kvm_arch_dev_ioctl_check_extension(ext);
> +		r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
>  		break;
>  	}
>  	return r;
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 71fa6fe..2cdd7ae 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -46,7 +46,7 @@
>  
>  int __attribute_const__ kvm_target_cpu(void);
>  int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
> -int kvm_arch_dev_ioctl_check_extension(long ext);
> +int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
>  
>  struct kvm_arch {
>  	/* The VMID generation used for the virt. memory system */
> diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
> index 9677bf0..2ba7c8d 100644
> --- a/arch/arm64/kvm/reset.c
> +++ b/arch/arm64/kvm/reset.c
> @@ -63,7 +63,7 @@ static bool cpu_has_32bit_el1(void)
>   * We currently assume that the number of HW registers is uniform
>   * across all CPUs (see cpuinfo_sanity_check).
>   */
> -int kvm_arch_dev_ioctl_check_extension(long ext)
> +int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
>  {
>  	int r;
>  
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-04-03 10:08     ` Christoffer Dall
@ 2016-04-05 12:55       ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 12:55 UTC (permalink / raw)
  To: Christoffer Dall, Andre Przywara
  Cc: Marc Zyngier, linux-arm-kernel, kvmarm, kvm

Hi Christoffer,
On 04/03/2016 12:08 PM, Christoffer Dall wrote:
> On Sat, Mar 26, 2016 at 02:14:02AM +0000, Andre Przywara wrote:
>> In the GICv3 redistributor there are the PENDBASER and PROPBASER
>> registers which we did not emulate so far, as they only make sense
>> when having an ITS. In preparation for that emulate those MMIO
>> accesses by storing the 64-bit data written into it into a variable
>> which we later read in the ITS emulation.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  include/kvm/vgic/vgic.h       |  8 ++++++++
>>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 11344e6..8ea5dd7 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -145,6 +145,14 @@ struct vgic_dist {
>>  
>>  	struct vgic_io_device	*dist_iodevs;
>>  	struct vgic_io_device	*redist_iodevs;
>> +
>> +	/* Address of LPI configuration table shared by all redistributors */
>> +	u64			propbaser;
> 
> is this an architectural requirement or an implementation decision?
Looks like it is an architectural requirement:
GIC archi spec, 4.8.9 Configuring the Configuration and Pending Tables
for  states:
"Each redistributor shares a table containing the LPI configuration
(group, priority and enables) and has a separate table containing the
pending state for each LPI."

Eric
> 
>> +
>> +	/* Addresses of LPI pending tables per redistributor */
>> +	u64			*pendbaser;
> 
> have we considered having an array of redistributors and putting the
> field in each one of those instead?
> 
>> +
>> +	bool			lpis_enabled;
>>  };
>>  
>>  struct vgic_v2_cpu_if {
>> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
>> index 7d5d630..252b9aff 100644
>> --- a/virt/kvm/arm/vgic/vgic_mmio.c
>> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
>> @@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
>>  				       struct kvm_io_device *this,
>>  				       gpa_t addr, int len, void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +
>> +	write_mask64(dist->propbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> @@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
>>  				        struct kvm_io_device *this,
>>  				        gpa_t addr, int len, const void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +
>> +	/* Storing a value with LPIs already enabled is undefined */
>> +	if (!dist->lpis_enabled)
>> +		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> @@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
>>  				       struct kvm_io_device *this,
>>  				       gpa_t addr, int len, void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
>> +
>> +	write_mask64(pendbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> @@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
>>  				        struct kvm_io_device *this,
>>  				        gpa_t addr, int len, const void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
>> +
>> +	/* Storing a value with LPIs already enabled is undefined */
>> +	if (!dist->lpis_enabled)
>> +		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> -- 
>> 2.7.3
>>


^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-04-05 12:55       ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 12:55 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Christoffer,
On 04/03/2016 12:08 PM, Christoffer Dall wrote:
> On Sat, Mar 26, 2016 at 02:14:02AM +0000, Andre Przywara wrote:
>> In the GICv3 redistributor there are the PENDBASER and PROPBASER
>> registers which we did not emulate so far, as they only make sense
>> when having an ITS. In preparation for that emulate those MMIO
>> accesses by storing the 64-bit data written into it into a variable
>> which we later read in the ITS emulation.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  include/kvm/vgic/vgic.h       |  8 ++++++++
>>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 11344e6..8ea5dd7 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -145,6 +145,14 @@ struct vgic_dist {
>>  
>>  	struct vgic_io_device	*dist_iodevs;
>>  	struct vgic_io_device	*redist_iodevs;
>> +
>> +	/* Address of LPI configuration table shared by all redistributors */
>> +	u64			propbaser;
> 
> is this an architectural requirement or an implementation decision?
Looks like it is an architectural requirement:
GIC archi spec, 4.8.9 Configuring the Configuration and Pending Tables
for  states:
"Each redistributor shares a table containing the LPI configuration
(group, priority and enables) and has a separate table containing the
pending state for each LPI."

Eric
> 
>> +
>> +	/* Addresses of LPI pending tables per redistributor */
>> +	u64			*pendbaser;
> 
> have we considered having an array of redistributors and putting the
> field in each one of those instead?
> 
>> +
>> +	bool			lpis_enabled;
>>  };
>>  
>>  struct vgic_v2_cpu_if {
>> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
>> index 7d5d630..252b9aff 100644
>> --- a/virt/kvm/arm/vgic/vgic_mmio.c
>> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
>> @@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
>>  				       struct kvm_io_device *this,
>>  				       gpa_t addr, int len, void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +
>> +	write_mask64(dist->propbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> @@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
>>  				        struct kvm_io_device *this,
>>  				        gpa_t addr, int len, const void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +
>> +	/* Storing a value with LPIs already enabled is undefined */
>> +	if (!dist->lpis_enabled)
>> +		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> @@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
>>  				       struct kvm_io_device *this,
>>  				       gpa_t addr, int len, void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
>> +
>> +	write_mask64(pendbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> @@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
>>  				        struct kvm_io_device *this,
>>  				        gpa_t addr, int len, const void *val)
>>  {
>> -	/* TODO: implement */
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
>> +
>> +	/* Storing a value with LPIs already enabled is undefined */
>> +	if (!dist->lpis_enabled)
>> +		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
>>  	return 0;
>>  }
>>  
>> -- 
>> 2.7.3
>>

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-05 15:17     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 15:17 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> In the GICv3 redistributor there are the PENDBASER and PROPBASER
> registers which we did not emulate so far, as they only make sense
> when having an ITS. In preparation for that emulate those MMIO
> accesses by storing the 64-bit data written into it into a variable
> which we later read in the ITS emulation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h       |  8 ++++++++
>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 11344e6..8ea5dd7 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -145,6 +145,14 @@ struct vgic_dist {
>  
>  	struct vgic_io_device	*dist_iodevs;
>  	struct vgic_io_device	*redist_iodevs;
> +
> +	/* Address of LPI configuration table shared by all redistributors */
Doesn't this variable exactly match the PROPBASER register content. If
this statement is correct it does not exactly match the gpa of the
config table but also contains other info (cacheability, shareability,
IDbits). Hence the choice of u64 and not gpa_t. So if correct the
comment is misleading.
> +	u64			propbaser;
> +
> +	/* Addresses of LPI pending tables per redistributor */
> +	u64			*pendbaser;
same comment

Besides Reviewed-by: Eric Auger <eric.auger@linaro.org>

Best Regards

Eric
> +
> +	bool			lpis_enabled;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 7d5d630..252b9aff 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	write_mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	write_mask64(pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-04-05 15:17     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 15:17 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> In the GICv3 redistributor there are the PENDBASER and PROPBASER
> registers which we did not emulate so far, as they only make sense
> when having an ITS. In preparation for that emulate those MMIO
> accesses by storing the 64-bit data written into it into a variable
> which we later read in the ITS emulation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h       |  8 ++++++++
>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 11344e6..8ea5dd7 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -145,6 +145,14 @@ struct vgic_dist {
>  
>  	struct vgic_io_device	*dist_iodevs;
>  	struct vgic_io_device	*redist_iodevs;
> +
> +	/* Address of LPI configuration table shared by all redistributors */
Doesn't this variable exactly match the PROPBASER register content. If
this statement is correct it does not exactly match the gpa of the
config table but also contains other info (cacheability, shareability,
IDbits). Hence the choice of u64 and not gpa_t. So if correct the
comment is misleading.
> +	u64			propbaser;
> +
> +	/* Addresses of LPI pending tables per redistributor */
> +	u64			*pendbaser;
same comment

Besides Reviewed-by: Eric Auger <eric.auger@linaro.org>

Best Regards

Eric
> +
> +	bool			lpis_enabled;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 7d5d630..252b9aff 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -771,7 +771,9 @@ static int vgic_mmio_read_v3r_propbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	write_mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -779,7 +781,11 @@ static int vgic_mmio_write_v3r_propbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		dist->propbaser = mask64(dist->propbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -787,7 +793,12 @@ static int vgic_mmio_read_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				       struct kvm_io_device *this,
>  				       gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 pendbaser = dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	write_mask64(pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> @@ -795,7 +806,14 @@ static int vgic_mmio_write_v3r_pendbase(struct kvm_vcpu *vcpu,
>  				        struct kvm_io_device *this,
>  				        gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement */
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u64 *pendbaser = &dist->pendbaser[iodev->redist_vcpu->vcpu_id];
> +
> +	/* Storing a value with LPIs already enabled is undefined */
> +	if (!dist->lpis_enabled)
> +		*pendbaser = mask64(*pendbaser, addr & 7, len, val);
>  	return 0;
>  }
>  
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-05 16:03     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 16:03 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> The ARM GICv3 ITS emulation code goes into a separate file, but
> needs to be connected to the GICv3 emulation, of which it is an
> option.
> Introduce the skeleton with function stubs to be filled later.
> Introduce the basic ITS data structure and initialize it, but don't
> return any success yet, as we are not yet ready for the show.
This patch also partially implements GICR_CTLR.Enable LPI
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  arch/arm64/kvm/Makefile            |  1 +
>  include/kvm/vgic/vgic.h            |  6 +++
>  include/linux/irqchip/arm-gic-v3.h |  1 +
>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>  7 files changed, 126 insertions(+), 4 deletions(-)
>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
> 
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 2f5d431..3bec10e 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>  else
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 8ea5dd7..c79bed5 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -112,6 +112,11 @@ struct vgic_io_device {
>  	struct kvm_io_device dev;
>  };
>  
> +struct vgic_its {
> +	bool			enabled;
> +	spinlock_t		lock;
> +};
> +
>  struct vgic_dist {
>  	bool			in_kernel;
>  	bool			ready;
> @@ -153,6 +158,7 @@ struct vgic_dist {
>  	u64			*pendbaser;
>  
>  	bool			lpis_enabled;
> +	struct vgic_its		its;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index d5d798b..a813c3e 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -177,6 +177,7 @@
>  #define GITS_CWRITER			0x0088
>  #define GITS_CREADR			0x0090
>  #define GITS_BASER			0x0100
> +#define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
>  
>  #define GITS_TRANSLATER			0x10040
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> new file mode 100644
> index 0000000..49dd5e4
> --- /dev/null
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -0,0 +1,84 @@
> +/*
> + * GICv3 ITS emulation
> + *
> + * Copyright (C) 2015 ARM Ltd.
> + * Author: Andre Przywara <andre.przywara@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/kvm.h>
> +#include <linux/kvm_host.h>
> +#include <linux/interrupt.h>
> +
> +#include <linux/irqchip/arm-gic-v3.h>
> +
> +#include <asm/kvm_emulate.h>
> +#include <asm/kvm_arm.h>
> +#include <asm/kvm_mmu.h>
> +
> +#include "vgic.h"
> +#include "vgic_mmio.h"
> +
> +struct vgic_register_region its_registers[] = {
> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
64bit register with upper 32b reserved. confusing at first sight ;-)
> +	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +};
> +
> +/* This is called on setting the LPI enable bit in the redistributor. */
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +}
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	struct vgic_io_device *regions;
> +	int ret, i;
> +
> +	spin_lock_init(&its->lock);
> +
> +	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> +				sizeof(struct vgic_io_device), GFP_KERNEL);
need to handle ENOMEM case
> +
> +	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
> +		regions[i].base_addr = dist->vgic_its_base;
> +
> +		ret = kvm_vgic_register_mmio_region(kvm, NULL,
> +						    &its_registers[i],
> +						    &regions[i], 0, false);
shouldn't we stop on the first fail and unregister what was previously
registered?
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	its->enabled = false;
> +
> +	return -ENXIO;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index db9dfd7..4e7dcb8 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
>  int vgic_v3_probe(struct device_node *vgic_node);
>  int vgic_v3_map_resources(struct kvm *kvm);
>  int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> +
> +int vits_init(struct kvm *kvm);
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>  {
>  	return -ENODEV;
>  }
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	return 0;
> +}
> +
> +static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> @@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>  
>  int vgic_lazy_init(struct kvm *kvm);
>  int vgic_init(struct kvm *kvm);
> +int vits_init(struct kvm *kvm);
>  void kvm_register_vgic_device(unsigned long type);
>  
>  #endif
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index 2301e03..dcfb93d 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
>  	if (ret)
>  		goto out;
>  
> +	if (vgic_has_its(kvm)) {
> +		ret = vits_init(kvm);
> +		if (ret)
> +			goto out;
> +	}
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 252b9aff..086555e 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
>  				   struct kvm_io_device *this,
>  				   gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +	return 0;
>  }
>  
>  static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
>  				    struct kvm_io_device *this,
>  				    gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
lpis_enabled is a bool
> +
> +	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
> +	    (reg & GICR_CTLR_ENABLE_LPIS)) {
> +		/* Eventually do something */
Looks strange to me to not set/reset dist->lpis_enabled according to the
write val at least, to make things symetrical.

Cheers

Eric
> +	}
> +
> +	return 0;
>  }
>  
>  static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-04-05 16:03     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-05 16:03 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> The ARM GICv3 ITS emulation code goes into a separate file, but
> needs to be connected to the GICv3 emulation, of which it is an
> option.
> Introduce the skeleton with function stubs to be filled later.
> Introduce the basic ITS data structure and initialize it, but don't
> return any success yet, as we are not yet ready for the show.
This patch also partially implements GICR_CTLR.Enable LPI
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  arch/arm64/kvm/Makefile            |  1 +
>  include/kvm/vgic/vgic.h            |  6 +++
>  include/linux/irqchip/arm-gic-v3.h |  1 +
>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>  7 files changed, 126 insertions(+), 4 deletions(-)
>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
> 
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 2f5d431..3bec10e 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>  else
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 8ea5dd7..c79bed5 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -112,6 +112,11 @@ struct vgic_io_device {
>  	struct kvm_io_device dev;
>  };
>  
> +struct vgic_its {
> +	bool			enabled;
> +	spinlock_t		lock;
> +};
> +
>  struct vgic_dist {
>  	bool			in_kernel;
>  	bool			ready;
> @@ -153,6 +158,7 @@ struct vgic_dist {
>  	u64			*pendbaser;
>  
>  	bool			lpis_enabled;
> +	struct vgic_its		its;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index d5d798b..a813c3e 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -177,6 +177,7 @@
>  #define GITS_CWRITER			0x0088
>  #define GITS_CREADR			0x0090
>  #define GITS_BASER			0x0100
> +#define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
>  
>  #define GITS_TRANSLATER			0x10040
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> new file mode 100644
> index 0000000..49dd5e4
> --- /dev/null
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -0,0 +1,84 @@
> +/*
> + * GICv3 ITS emulation
> + *
> + * Copyright (C) 2015 ARM Ltd.
> + * Author: Andre Przywara <andre.przywara@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/kvm.h>
> +#include <linux/kvm_host.h>
> +#include <linux/interrupt.h>
> +
> +#include <linux/irqchip/arm-gic-v3.h>
> +
> +#include <asm/kvm_emulate.h>
> +#include <asm/kvm_arm.h>
> +#include <asm/kvm_mmu.h>
> +
> +#include "vgic.h"
> +#include "vgic_mmio.h"
> +
> +struct vgic_register_region its_registers[] = {
> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
64bit register with upper 32b reserved. confusing at first sight ;-)
> +	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +};
> +
> +/* This is called on setting the LPI enable bit in the redistributor. */
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +}
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	struct vgic_io_device *regions;
> +	int ret, i;
> +
> +	spin_lock_init(&its->lock);
> +
> +	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> +				sizeof(struct vgic_io_device), GFP_KERNEL);
need to handle ENOMEM case
> +
> +	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
> +		regions[i].base_addr = dist->vgic_its_base;
> +
> +		ret = kvm_vgic_register_mmio_region(kvm, NULL,
> +						    &its_registers[i],
> +						    &regions[i], 0, false);
shouldn't we stop on the first fail and unregister what was previously
registered?
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	its->enabled = false;
> +
> +	return -ENXIO;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index db9dfd7..4e7dcb8 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
>  int vgic_v3_probe(struct device_node *vgic_node);
>  int vgic_v3_map_resources(struct kvm *kvm);
>  int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> +
> +int vits_init(struct kvm *kvm);
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>  {
>  	return -ENODEV;
>  }
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	return 0;
> +}
> +
> +static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> @@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>  
>  int vgic_lazy_init(struct kvm *kvm);
>  int vgic_init(struct kvm *kvm);
> +int vits_init(struct kvm *kvm);
>  void kvm_register_vgic_device(unsigned long type);
>  
>  #endif
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index 2301e03..dcfb93d 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
>  	if (ret)
>  		goto out;
>  
> +	if (vgic_has_its(kvm)) {
> +		ret = vits_init(kvm);
> +		if (ret)
> +			goto out;
> +	}
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 252b9aff..086555e 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
>  				   struct kvm_io_device *this,
>  				   gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +	return 0;
>  }
>  
>  static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
>  				    struct kvm_io_device *this,
>  				    gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
lpis_enabled is a bool
> +
> +	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
> +	    (reg & GICR_CTLR_ENABLE_LPIS)) {
> +		/* Eventually do something */
Looks strange to me to not set/reset dist->lpis_enabled according to the
write val at least, to make things symetrical.

Cheers

Eric
> +	}
> +
> +	return 0;
>  }
>  
>  static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-06  9:36     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06  9:36 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> Add emulation for some basic MMIO registers used in the ITS emulation.
> This includes:
> - GITS_{CTLR,TYPER,IIDR}
> - ID registers
> - GITS_{CBASER,CREADR,CWRITER}
>   those implement the ITS command buffer handling
> 
> Most of the handlers are pretty straight forward, but CWRITER goes
> some extra miles to allow fine grained locking. The idea here
> is to let only the first instance iterate through the command ring
> buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
> by that first instance and handled as well. The ITS lock is thus only
> hold for very small periods of time and is dropped before the actual
> command handler is called.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h            |   3 +
>  include/linux/irqchip/arm-gic-v3.h |   8 ++
>  virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h           |   6 +
>  virt/kvm/arm/vgic/vgic_init.c      |   2 +
>  5 files changed, 284 insertions(+), 7 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c79bed5..bafea11 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -115,6 +115,9 @@ struct vgic_io_device {
>  struct vgic_its {
>  	bool			enabled;
>  	spinlock_t		lock;
> +	u64			cbaser;
> +	int			creadr;
> +	int			cwriter;
>  };
>  
>  struct vgic_dist {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index a813c3e..7011b98 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -179,15 +179,23 @@
>  #define GITS_BASER			0x0100
>  #define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
> +#define GITS_PIDR4			0xffd0
> +#define GITS_CIDR0			0xfff0
> +#define GITS_CIDR1			0xfff4
> +#define GITS_CIDR2			0xfff8
> +#define GITS_CIDR3			0xfffc
>  
>  #define GITS_TRANSLATER			0x10040
>  
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
> +#define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
>  #define GITS_TYPER_PTA			(1UL << 19)
> +#define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
>  #define GITS_CBASER_VALID		(1UL << 63)
>  #define GITS_CBASER_nCnB		(0UL << 59)
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 49dd5e4..de8d360 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -31,23 +31,263 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> +
> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u32 reg;
> +
> +	reg = GITS_CTLR_QUIESCENT;
> +	if (its->enabled)
> +		reg |= GITS_CTLR_ENABLE;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +
> +        if (addr - iodev->base_addr == 0)
> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	u64 reg = GITS_TYPER_PLPIS;
> +
> +	/*
> +	 * We use linear CPU numbers for redistributor addressing,
> +	 * so GITS_TYPER.PTA is 0.
> +	 * To avoid memory waste on the guest side, we keep the
> +	 * number of IDBits and DevBits low for the time being.
> +	 * This could later be made configurable by userland.
> +	 * Since we have all collections in linked list, we claim
> +	 * that we can hold all of the collection tables in our
> +	 * own memory and that the ITT entry size is 1 byte (the
> +	 * smallest possible one).
> +	 */
> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u32 reg = 0;
> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
> +
> +	switch (idreg) {
> +	case GITS_PIDR2:
> +		reg = GIC_PIDR2_ARCH_GICv3;
> +		break;
> +	case GITS_PIDR4:
> +		/* This is a 64K software visible page */
> +		reg = 0x40;
> +		break;
> +	/* Those are the ID registers for (any) GIC. */
> +	case GITS_CIDR0:
> +		reg = 0x0d;
> +		break;
> +	case GITS_CIDR1:
> +		reg = 0xf0;
> +		break;
> +	case GITS_CIDR2:
> +		reg = 0x05;
> +		break;
> +	case GITS_CIDR3:
> +		reg = 0xb1;
> +		break;
> +	}
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +/*
> + * This function is called with both the ITS and the distributor lock dropped,
dist lock does not exist anymore ;-)
> + * so the actual command handlers must take the respective locks when needed.
> + */
> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> +{
> +	return -ENODEV;
> +}
> +
> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	write_mask64(its->cbaser, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	if (its->enabled)
> +		return 0;
> +
> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
> +	its->creadr = 0;
shouldn't this be protected by the its lock to guarantee consistency of
cbaser,  creadr?
> +
> +	return 0;
> +}
> +
> +static int its_cmd_buffer_size(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return ((its->cbaser & 0xff) + 1) << 12;
> +}
> +
> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return BASER_BASE_ADDRESS(its->cbaser);
> +}
> +
> +/*
> + * By writing to CWRITER the guest announces new commands to be processed.
> + * Since we cannot read from guest memory inside the ITS spinlock, we
> + * iterate over the command buffer (with the lock dropped) until the read
> + * pointer matches the write pointer. Other VCPUs writing this register in the
> + * meantime will just update the write pointer, leaving the command
> + * processing to the first instance of the function.
> + */
> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
> +				       struct kvm_io_device *this,
> +				       gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
> +	u64 cmd_buf[4];
> +	u32 reg;
> +	bool finished;
> +
> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
> +	reg &= 0xfffe0;
> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
> +		return 0;
> +
> +	spin_lock(&its->lock);
> +
> +	/*
> +	 * If there is still another VCPU handling commands, let this
> +	 * one pick up the new CWRITER and process "our" new commands as well.
> +	 */
> +	finished = (its->cwriter != its->creadr);
empty?
> +	its->cwriter = reg;
> +
> +	spin_unlock(&its->lock);
Assuming we have 2 vcpus attempting a cwriter write at the same moment I
don't understand what does guarantee they are not going to execute the
following loop concurrently and possibly execute vits_handle_command
twice on the same cmd from the same its->creadr start?
> +
> +	while (!finished) {
> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
> +					 cmd_buf, 32);
> +		if (ret) {
> +			/*
> +			 * Gah, we are screwed. Reset CWRITER to that command
> +			 * that we have finished processing and return.
> +			 */
> +			spin_lock(&its->lock);
> +			its->cwriter = its->creadr;
don't get why do you set the queue empty?
> +			spin_unlock(&its->lock);
> +			break;
> +		}
> +		vits_handle_command(vcpu, cmd_buf);
> +
> +		spin_lock(&its->lock);
> +		its->creadr += 32;
> +		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
> +			its->creadr = 0;
> +		finished = (its->creadr == its->cwriter);
> +		spin_unlock(&its->lock);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->cwriter & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->creadr & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
>  struct vgic_register_region its_registers[] = {
>  	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>  		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
>  };
>  
>  /* This is called on setting the LPI enable bit in the redistributor. */
> @@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	int nr_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct vgic_io_device *regions;
>  	int ret, i;
>  
> +	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
> +	if (!dist->pendbaser)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> @@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
>  
>  	return -ENXIO;
>  }
> +
> +void vits_destroy(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +
> +	if (!vgic_has_its(kvm))
> +		return;
> +
> +	kfree(dist->pendbaser);
regions is leaking. By the way its->iodev never is set on init so cannot
be freeed on destructor. Another way to free it is to use the io_dev_ops
destructor.

Cheers

Eric
> +
> +	its->enabled = false;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 4e7dcb8..08f97d1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
>  	return;
>  }
> +
> +static inline void vits_destroy(struct kvm *kvm)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index dcfb93d..e4459e3 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
>  
>  	kvm_vgic_dist_destroy(kvm);
>  
> +	vits_destroy(kvm);
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_destroy(vcpu);
>  }
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-04-06  9:36     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06  9:36 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> Add emulation for some basic MMIO registers used in the ITS emulation.
> This includes:
> - GITS_{CTLR,TYPER,IIDR}
> - ID registers
> - GITS_{CBASER,CREADR,CWRITER}
>   those implement the ITS command buffer handling
> 
> Most of the handlers are pretty straight forward, but CWRITER goes
> some extra miles to allow fine grained locking. The idea here
> is to let only the first instance iterate through the command ring
> buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
> by that first instance and handled as well. The ITS lock is thus only
> hold for very small periods of time and is dropped before the actual
> command handler is called.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h            |   3 +
>  include/linux/irqchip/arm-gic-v3.h |   8 ++
>  virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h           |   6 +
>  virt/kvm/arm/vgic/vgic_init.c      |   2 +
>  5 files changed, 284 insertions(+), 7 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c79bed5..bafea11 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -115,6 +115,9 @@ struct vgic_io_device {
>  struct vgic_its {
>  	bool			enabled;
>  	spinlock_t		lock;
> +	u64			cbaser;
> +	int			creadr;
> +	int			cwriter;
>  };
>  
>  struct vgic_dist {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index a813c3e..7011b98 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -179,15 +179,23 @@
>  #define GITS_BASER			0x0100
>  #define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
> +#define GITS_PIDR4			0xffd0
> +#define GITS_CIDR0			0xfff0
> +#define GITS_CIDR1			0xfff4
> +#define GITS_CIDR2			0xfff8
> +#define GITS_CIDR3			0xfffc
>  
>  #define GITS_TRANSLATER			0x10040
>  
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
> +#define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
>  #define GITS_TYPER_PTA			(1UL << 19)
> +#define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
>  #define GITS_CBASER_VALID		(1UL << 63)
>  #define GITS_CBASER_nCnB		(0UL << 59)
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 49dd5e4..de8d360 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -31,23 +31,263 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> +
> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u32 reg;
> +
> +	reg = GITS_CTLR_QUIESCENT;
> +	if (its->enabled)
> +		reg |= GITS_CTLR_ENABLE;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +
> +        if (addr - iodev->base_addr == 0)
> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	u64 reg = GITS_TYPER_PLPIS;
> +
> +	/*
> +	 * We use linear CPU numbers for redistributor addressing,
> +	 * so GITS_TYPER.PTA is 0.
> +	 * To avoid memory waste on the guest side, we keep the
> +	 * number of IDBits and DevBits low for the time being.
> +	 * This could later be made configurable by userland.
> +	 * Since we have all collections in linked list, we claim
> +	 * that we can hold all of the collection tables in our
> +	 * own memory and that the ITT entry size is 1 byte (the
> +	 * smallest possible one).
> +	 */
> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u32 reg = 0;
> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
> +
> +	switch (idreg) {
> +	case GITS_PIDR2:
> +		reg = GIC_PIDR2_ARCH_GICv3;
> +		break;
> +	case GITS_PIDR4:
> +		/* This is a 64K software visible page */
> +		reg = 0x40;
> +		break;
> +	/* Those are the ID registers for (any) GIC. */
> +	case GITS_CIDR0:
> +		reg = 0x0d;
> +		break;
> +	case GITS_CIDR1:
> +		reg = 0xf0;
> +		break;
> +	case GITS_CIDR2:
> +		reg = 0x05;
> +		break;
> +	case GITS_CIDR3:
> +		reg = 0xb1;
> +		break;
> +	}
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +/*
> + * This function is called with both the ITS and the distributor lock dropped,
dist lock does not exist anymore ;-)
> + * so the actual command handlers must take the respective locks when needed.
> + */
> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> +{
> +	return -ENODEV;
> +}
> +
> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	write_mask64(its->cbaser, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	if (its->enabled)
> +		return 0;
> +
> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
> +	its->creadr = 0;
shouldn't this be protected by the its lock to guarantee consistency of
cbaser,  creadr?
> +
> +	return 0;
> +}
> +
> +static int its_cmd_buffer_size(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return ((its->cbaser & 0xff) + 1) << 12;
> +}
> +
> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return BASER_BASE_ADDRESS(its->cbaser);
> +}
> +
> +/*
> + * By writing to CWRITER the guest announces new commands to be processed.
> + * Since we cannot read from guest memory inside the ITS spinlock, we
> + * iterate over the command buffer (with the lock dropped) until the read
> + * pointer matches the write pointer. Other VCPUs writing this register in the
> + * meantime will just update the write pointer, leaving the command
> + * processing to the first instance of the function.
> + */
> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
> +				       struct kvm_io_device *this,
> +				       gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
> +	u64 cmd_buf[4];
> +	u32 reg;
> +	bool finished;
> +
> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
> +	reg &= 0xfffe0;
> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
> +		return 0;
> +
> +	spin_lock(&its->lock);
> +
> +	/*
> +	 * If there is still another VCPU handling commands, let this
> +	 * one pick up the new CWRITER and process "our" new commands as well.
> +	 */
> +	finished = (its->cwriter != its->creadr);
empty?
> +	its->cwriter = reg;
> +
> +	spin_unlock(&its->lock);
Assuming we have 2 vcpus attempting a cwriter write at the same moment I
don't understand what does guarantee they are not going to execute the
following loop concurrently and possibly execute vits_handle_command
twice on the same cmd from the same its->creadr start?
> +
> +	while (!finished) {
> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
> +					 cmd_buf, 32);
> +		if (ret) {
> +			/*
> +			 * Gah, we are screwed. Reset CWRITER to that command
> +			 * that we have finished processing and return.
> +			 */
> +			spin_lock(&its->lock);
> +			its->cwriter = its->creadr;
don't get why do you set the queue empty?
> +			spin_unlock(&its->lock);
> +			break;
> +		}
> +		vits_handle_command(vcpu, cmd_buf);
> +
> +		spin_lock(&its->lock);
> +		its->creadr += 32;
> +		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
> +			its->creadr = 0;
> +		finished = (its->creadr == its->cwriter);
> +		spin_unlock(&its->lock);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->cwriter & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->creadr & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
>  struct vgic_register_region its_registers[] = {
>  	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>  		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
>  };
>  
>  /* This is called on setting the LPI enable bit in the redistributor. */
> @@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	int nr_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct vgic_io_device *regions;
>  	int ret, i;
>  
> +	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
> +	if (!dist->pendbaser)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> @@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
>  
>  	return -ENXIO;
>  }
> +
> +void vits_destroy(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +
> +	if (!vgic_has_its(kvm))
> +		return;
> +
> +	kfree(dist->pendbaser);
regions is leaking. By the way its->iodev never is set on init so cannot
be freeed on destructor. Another way to free it is to use the io_dev_ops
destructor.

Cheers

Eric
> +
> +	its->enabled = false;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 4e7dcb8..08f97d1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
>  	return;
>  }
> +
> +static inline void vits_destroy(struct kvm *kvm)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index dcfb93d..e4459e3 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
>  
>  	kvm_vgic_dist_destroy(kvm);
>  
> +	vits_destroy(kvm);
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_destroy(vcpu);
>  }
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 07/12] KVM: arm64: add data structures to model ITS interrupt translation
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-06  9:53     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06  9:53 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm

On 03/26/2016 03:14 AM, Andre Przywara wrote:
> The GICv3 Interrupt Translation Service (ITS) uses tables in memory
> to allow a sophisticated interrupt routing. It features device tables,
> an interrupt table per device and a table connecting "collections" to
> actual CPUs (aka. redistributors in the GICv3 lingo).
> Since the interrupt numbers for the LPIs are allocated quite sparsely
> and the range can be quite huge (8192 LPIs being the minimum), using
> bitmaps or arrays for storing information is a waste of memory.
> We use linked lists instead, which we iterate linearily. This works
> very well with the actual number of LPIs/MSIs in the guest being
> quite low. Should the number of LPIs exceed the number where iterating
> through lists seems acceptable, we can later revisit this and use more
> efficient data structures.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h      |  3 ++
>  virt/kvm/arm/vgic/its-emul.c | 66 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 69 insertions(+)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index bafea11..ecf3260 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -22,6 +22,7 @@
>  #include <linux/spinlock.h>
>  #include <linux/types.h>
>  #include <kvm/iodev.h>
> +#include <linux/list.h>
>  
>  #define VGIC_V3_MAX_CPUS	255
>  #define VGIC_V2_MAX_CPUS	8
> @@ -118,6 +119,8 @@ struct vgic_its {
>  	u64			cbaser;
>  	int			creadr;
>  	int			cwriter;
> +	struct list_head	device_list;
> +	struct list_head	collection_list;
>  };
>  
>  struct vgic_dist {
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index de8d360..c0334ff 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -21,6 +21,7 @@
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
>  #include <linux/interrupt.h>
> +#include <linux/list.h>
>  
>  #include <linux/irqchip/arm-gic-v3.h>
>  
> @@ -31,6 +32,34 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +struct its_device {
> +	struct list_head dev_list;
> +
> +	/* the head for the list of ITTEs */
> +	struct list_head itt;
> +	u32 device_id;
> +};
> +
> +#define COLLECTION_NOT_MAPPED ((u32)-1)
> +
> +struct its_collection {
> +	struct list_head coll_list;
> +
> +	u32 collection_id;
> +	u32 target_addr;
> +};
> +
> +#define its_is_collection_mapped(coll) ((coll) && \
> +				((coll)->target_addr != COLLECTION_NOT_MAPPED))
> +
> +struct its_itte {
> +	struct list_head itte_list;
> +
> +	struct its_collection *collection;
> +	u32 lpi;
> +	u32 event_id;
> +};
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>  
>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> @@ -137,6 +166,12 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +static void its_free_itte(struct its_itte *itte)
> +{
> +	list_del(&itte->itte_list);
> +	kfree(itte);
> +}
> +
>  /*
>   * This function is called with both the ITS and the distributor lock dropped,
>   * so the actual command handlers must take the respective locks when needed.
> @@ -309,6 +344,9 @@ int vits_init(struct kvm *kvm)
>  
>  	spin_lock_init(&its->lock);
>  
> +	INIT_LIST_HEAD(&its->device_list);
> +	INIT_LIST_HEAD(&its->collection_list);
> +
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
>  				sizeof(struct vgic_io_device), GFP_KERNEL);
>  
> @@ -332,11 +370,39 @@ void vits_destroy(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	struct its_device *dev;
> +	struct its_itte *itte;
> +	struct list_head *dev_cur, *dev_temp;
> +	struct list_head *cur, *temp;
>  
>  	if (!vgic_has_its(kvm))
>  		return;
>  
> +	/*
> +	 * We may end up here without the lists ever having been initialized.
> +	 * Check this and bail out early to avoid dereferencing a NULL pointer.
> +	 */
> +	if (!its->device_list.next)
> +		return;
don't you check collection_list in a similar way then?
> +
> +	spin_lock(&its->lock);
> +	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
> +		dev = container_of(dev_cur, struct its_device, dev_list);
> +		list_for_each_safe(cur, temp, &dev->itt) {
use list_for_each_safe_entry instead?

Eric
> +			itte = (container_of(cur, struct its_itte, itte_list));
> +			its_free_itte(itte);
> +		}
> +		list_del(dev_cur);
> +		kfree(dev);
> +	}
> +
> +	list_for_each_safe(cur, temp, &its->collection_list) {
> +		list_del(cur);
> +		kfree(container_of(cur, struct its_collection, coll_list));
> +	}
> +
>  	kfree(dist->pendbaser);
>  
>  	its->enabled = false;
> +	spin_unlock(&its->lock);
>  }
> 


^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 07/12] KVM: arm64: add data structures to model ITS interrupt translation
@ 2016-04-06  9:53     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06  9:53 UTC (permalink / raw)
  To: linux-arm-kernel

On 03/26/2016 03:14 AM, Andre Przywara wrote:
> The GICv3 Interrupt Translation Service (ITS) uses tables in memory
> to allow a sophisticated interrupt routing. It features device tables,
> an interrupt table per device and a table connecting "collections" to
> actual CPUs (aka. redistributors in the GICv3 lingo).
> Since the interrupt numbers for the LPIs are allocated quite sparsely
> and the range can be quite huge (8192 LPIs being the minimum), using
> bitmaps or arrays for storing information is a waste of memory.
> We use linked lists instead, which we iterate linearily. This works
> very well with the actual number of LPIs/MSIs in the guest being
> quite low. Should the number of LPIs exceed the number where iterating
> through lists seems acceptable, we can later revisit this and use more
> efficient data structures.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h      |  3 ++
>  virt/kvm/arm/vgic/its-emul.c | 66 ++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 69 insertions(+)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index bafea11..ecf3260 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -22,6 +22,7 @@
>  #include <linux/spinlock.h>
>  #include <linux/types.h>
>  #include <kvm/iodev.h>
> +#include <linux/list.h>
>  
>  #define VGIC_V3_MAX_CPUS	255
>  #define VGIC_V2_MAX_CPUS	8
> @@ -118,6 +119,8 @@ struct vgic_its {
>  	u64			cbaser;
>  	int			creadr;
>  	int			cwriter;
> +	struct list_head	device_list;
> +	struct list_head	collection_list;
>  };
>  
>  struct vgic_dist {
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index de8d360..c0334ff 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -21,6 +21,7 @@
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
>  #include <linux/interrupt.h>
> +#include <linux/list.h>
>  
>  #include <linux/irqchip/arm-gic-v3.h>
>  
> @@ -31,6 +32,34 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +struct its_device {
> +	struct list_head dev_list;
> +
> +	/* the head for the list of ITTEs */
> +	struct list_head itt;
> +	u32 device_id;
> +};
> +
> +#define COLLECTION_NOT_MAPPED ((u32)-1)
> +
> +struct its_collection {
> +	struct list_head coll_list;
> +
> +	u32 collection_id;
> +	u32 target_addr;
> +};
> +
> +#define its_is_collection_mapped(coll) ((coll) && \
> +				((coll)->target_addr != COLLECTION_NOT_MAPPED))
> +
> +struct its_itte {
> +	struct list_head itte_list;
> +
> +	struct its_collection *collection;
> +	u32 lpi;
> +	u32 event_id;
> +};
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>  
>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> @@ -137,6 +166,12 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +static void its_free_itte(struct its_itte *itte)
> +{
> +	list_del(&itte->itte_list);
> +	kfree(itte);
> +}
> +
>  /*
>   * This function is called with both the ITS and the distributor lock dropped,
>   * so the actual command handlers must take the respective locks when needed.
> @@ -309,6 +344,9 @@ int vits_init(struct kvm *kvm)
>  
>  	spin_lock_init(&its->lock);
>  
> +	INIT_LIST_HEAD(&its->device_list);
> +	INIT_LIST_HEAD(&its->collection_list);
> +
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
>  				sizeof(struct vgic_io_device), GFP_KERNEL);
>  
> @@ -332,11 +370,39 @@ void vits_destroy(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	struct its_device *dev;
> +	struct its_itte *itte;
> +	struct list_head *dev_cur, *dev_temp;
> +	struct list_head *cur, *temp;
>  
>  	if (!vgic_has_its(kvm))
>  		return;
>  
> +	/*
> +	 * We may end up here without the lists ever having been initialized.
> +	 * Check this and bail out early to avoid dereferencing a NULL pointer.
> +	 */
> +	if (!its->device_list.next)
> +		return;
don't you check collection_list in a similar way then?
> +
> +	spin_lock(&its->lock);
> +	list_for_each_safe(dev_cur, dev_temp, &its->device_list) {
> +		dev = container_of(dev_cur, struct its_device, dev_list);
> +		list_for_each_safe(cur, temp, &dev->itt) {
use list_for_each_safe_entry instead?

Eric
> +			itte = (container_of(cur, struct its_itte, itte_list));
> +			its_free_itte(itte);
> +		}
> +		list_del(dev_cur);
> +		kfree(dev);
> +	}
> +
> +	list_for_each_safe(cur, temp, &its->collection_list) {
> +		list_del(cur);
> +		kfree(container_of(cur, struct its_collection, coll_list));
> +	}
> +
>  	kfree(dist->pendbaser);
>  
>  	its->enabled = false;
> +	spin_unlock(&its->lock);
>  }
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-06 12:00     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06 12:00 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> LPIs are dynamically created (mapped) at guest runtime and their
> actual numbers can be quite high, but is mostly assigned using a very
> sparse allocation scheme. So arrays are not an ideal data structure
> to hold the information. We use our equivalent of the "Interrupt
> Translation Table Entry" (ITTE) to hold the vgic_irq struct for a
> virtual LPI embedded in in the ITTE.
> Connect the VGIC core code via an accessor function to help it get the
> struct vgic_irq for a certain LPI.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Looks OK to me
Reviewed-by: Eric Auger <eric.auger@linaro.org>

Eric
> ---
>  virt/kvm/arm/vgic/its-emul.c | 29 +++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h     |  8 +++++++-
>  2 files changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index c0334ff..1188e9a 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -55,11 +55,29 @@ struct its_collection {
>  struct its_itte {
>  	struct list_head itte_list;
>  
> +	struct vgic_irq irq;
>  	struct its_collection *collection;
>  	u32 lpi;
>  	u32 event_id;
>  };
>  
> +/* To be used as an iterator this macro misses the enclosing parentheses */
> +#define for_each_lpi(dev, itte, kvm) \
> +	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
> +		list_for_each_entry(itte, &(dev)->itt, itte_list)
> +
> +static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
> +{
> +	struct its_device *device;
> +	struct its_itte *itte;
> +
> +	for_each_lpi(device, itte, kvm) {
> +		if (itte->lpi == lpi)
> +			return itte;
> +	}
> +	return NULL;
> +}
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>  
>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> @@ -166,6 +184,17 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
> +{
> +	struct its_itte *itte;
> +
> +	itte = find_itte_by_lpi(kvm, intid);
> +	if (!itte)
> +		return NULL;
> +
> +	return &itte->irq;
> +}
> +
>  static void its_free_itte(struct its_itte *itte)
>  {
>  	list_del(&itte->itte_list);
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 08f97d1..160c511 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
>  void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
> @@ -129,7 +130,7 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>  	return -ENODEV;
>  }
>  
> -int vits_init(struct kvm *kvm)
> +static inline int vits_init(struct kvm *kvm)
>  {
>  	return 0;
>  }
> @@ -139,6 +140,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  	return;
>  }
>  
> +static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
> +{
> +	return NULL;
> +}
> +
>  static inline void vits_destroy(struct kvm *kvm)
>  {
>  	return;
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation
@ 2016-04-06 12:00     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06 12:00 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> LPIs are dynamically created (mapped) at guest runtime and their
> actual numbers can be quite high, but is mostly assigned using a very
> sparse allocation scheme. So arrays are not an ideal data structure
> to hold the information. We use our equivalent of the "Interrupt
> Translation Table Entry" (ITTE) to hold the vgic_irq struct for a
> virtual LPI embedded in in the ITTE.
> Connect the VGIC core code via an accessor function to help it get the
> struct vgic_irq for a certain LPI.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
Looks OK to me
Reviewed-by: Eric Auger <eric.auger@linaro.org>

Eric
> ---
>  virt/kvm/arm/vgic/its-emul.c | 29 +++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h     |  8 +++++++-
>  2 files changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index c0334ff..1188e9a 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -55,11 +55,29 @@ struct its_collection {
>  struct its_itte {
>  	struct list_head itte_list;
>  
> +	struct vgic_irq irq;
>  	struct its_collection *collection;
>  	u32 lpi;
>  	u32 event_id;
>  };
>  
> +/* To be used as an iterator this macro misses the enclosing parentheses */
> +#define for_each_lpi(dev, itte, kvm) \
> +	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
> +		list_for_each_entry(itte, &(dev)->itt, itte_list)
> +
> +static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
> +{
> +	struct its_device *device;
> +	struct its_itte *itte;
> +
> +	for_each_lpi(device, itte, kvm) {
> +		if (itte->lpi == lpi)
> +			return itte;
> +	}
> +	return NULL;
> +}
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>  
>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> @@ -166,6 +184,17 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>  	return 0;
>  }
>  
> +struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
> +{
> +	struct its_itte *itte;
> +
> +	itte = find_itte_by_lpi(kvm, intid);
> +	if (!itte)
> +		return NULL;
> +
> +	return &itte->irq;
> +}
> +
>  static void its_free_itte(struct its_itte *itte)
>  {
>  	list_del(&itte->itte_list);
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 08f97d1..160c511 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
>  void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
> @@ -129,7 +130,7 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>  	return -ENODEV;
>  }
>  
> -int vits_init(struct kvm *kvm)
> +static inline int vits_init(struct kvm *kvm)
>  {
>  	return 0;
>  }
> @@ -139,6 +140,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  	return;
>  }
>  
> +static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
> +{
> +	return NULL;
> +}
> +
>  static inline void vits_destroy(struct kvm *kvm)
>  {
>  	return;
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-06 13:41     ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06 13:41 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> The LPI configuration and pending tables of the GICv3 LPIs are held
> in tables in (guest) memory. To achieve reasonable performance, we
> cache this data in our own data structures, so we need to sync those
> two views from time to time. This behaviour is well described in the
> GICv3 spec and is also exercised by hardware, so the sync points are
> well known.
> 
> Provide functions that read the guest memory and store the
> information from the configuration and pending tables in the kernel.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h      |   2 +
>  virt/kvm/arm/vgic/its-emul.c | 135 +++++++++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h     |   1 +
>  3 files changed, 138 insertions(+)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index ecf3260..ea98133 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -121,6 +121,8 @@ struct vgic_its {
>  	int			cwriter;
>  	struct list_head	device_list;
>  	struct list_head	collection_list;
> +	/* memory used for buffering guest's memory */
> +	void			*buffer_page;
>  };
>  
>  struct vgic_dist {
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 1188e9a..d82ba9b 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -78,8 +78,130 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>  	return NULL;
>  }
>  
> +#define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
> +#define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
> +
> +/* stores the priority and enable bit for a given LPI */
> +static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
> +{
> +	spin_lock(&itte->irq.irq_lock);
> +	itte->irq.priority = LPI_PROP_PRIORITY(prop);
> +	itte->irq.enabled = LPI_PROP_ENABLE_BIT(prop);
> +
would put a comment saying that vgic_queue_irq is unlocking since it
looks weird.
> +	vgic_queue_irq(kvm, &itte->irq);
> +}
> +
> +#define GIC_LPI_OFFSET 8192
> +
> +/* We scan the table in chunks the size of the smallest page size */
> +#define CHUNK_SIZE 4096U
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>  
> +static int nr_idbits_propbase(u64 propbaser)
> +{
> +	int nr_idbits = (1U << (propbaser & 0x1f)) + 1;
> +
> +	return max(nr_idbits, INTERRUPT_ID_BITS_ITS);
don't get this.
don't you want to take the min of INTERRUPT_ID_BITS_ITS  (16) and
propbaser & 0x1f +1.

Spec says IDbits should not exceed GICD_TYPER.IDbit ==
INTERRUPT_ID_BITS_ITS - 1 = 15.

and in case the purpose of the function is to return the size of the
table, which is not obvious given its name ;-), we should return 1U <<
min(), no?
> +}
> +
> +/*
> + * Scan the whole LPI configuration table and put the LPI configuration
> + * data in our own data structures. This relies on the LPI being
> + * mapped before.
> + */
> +static bool its_update_lpis_configuration(struct kvm *kvm, u64 prop_base_reg)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	u8 *prop = dist->its.buffer_page;
> +	u32 tsize;
> +	gpa_t propbase;
> +	int lpi = GIC_LPI_OFFSET;
> +	struct its_itte *itte;
> +	struct its_device *device;
> +	int ret;
> +
> +	propbase = BASER_BASE_ADDRESS(prop_base_reg);
> +	tsize = nr_idbits_propbase(prop_base_reg);
> +
> +	while (tsize > 0) {
> +		int chunksize = min(tsize, CHUNK_SIZE);
> +
> +		ret = kvm_read_guest(kvm, propbase, prop, chunksize);
> +		if (ret)
> +			return false;
> +
> +		spin_lock(&dist->its.lock);
as we start holding several locks maybe worth to comment somewhere about
order lock: its lock > vcpu lock > irq lock
> +		/*
> +		 * Updating the status for all allocated LPIs. We catch
> +		 * those LPIs that get disabled. We really don't care
> +		 * about unmapped LPIs, as they need to be updated
> +		 * later manually anyway once they get mapped.
> +		 */
> +		for_each_lpi(device, itte, kvm) {
> +			if (itte->lpi < lpi || itte->lpi >= lpi + chunksize)
> +				continue;
> +
> +			update_lpi_config(kvm, itte, prop[itte->lpi - lpi]);
> +		}
> +		spin_unlock(&dist->its.lock);
> +		tsize -= chunksize;
> +		lpi += chunksize;
> +		propbase += chunksize;
> +	}
> +
> +	return true;
> +}
> +
> +/*
> + * Scan the whole LPI pending table and sync the pending bit in there
> + * with our own data structures. This relies on the LPI being
> + * mapped before.
> + */
> +static bool its_sync_lpi_pending_table(struct kvm_vcpu *vcpu, u64 base_addr_reg)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	unsigned long *pendmask = dist->its.buffer_page;
> +	u32 nr_lpis = 1U << INTERRUPT_ID_BITS_ITS;
shouldn't we use value computed as in nr_idbits_propbase instead?
> +	gpa_t pendbase;
> +	int lpi = 0;
> +	struct its_itte *itte;
> +	struct its_device *device;
> +	int ret;
> +	int lpi_bit, nr_bits;
> +
> +	pendbase = BASER_BASE_ADDRESS(base_addr_reg);
> +
I understand we should start reading at pendbase + 1KB. (4.8.5 pending
tables)

Eric
> +	while (nr_lpis > 0) {
> +		nr_bits = min(nr_lpis, CHUNK_SIZE * 8);
> +
> +		ret = kvm_read_guest(vcpu->kvm, pendbase, pendmask,
> +				     nr_bits / 8);
> +		if (ret)
> +			return false;
> +
> +		spin_lock(&dist->its.lock);
> +		for_each_lpi(device, itte, vcpu->kvm) {
> +			lpi_bit = itte->lpi - lpi;
> +			if (lpi_bit < 0 || lpi_bit >= nr_bits)
> +				continue;
> +
> +			if (!test_bit(lpi_bit, pendmask))
> +				continue;
> +
> +			spin_lock(&itte->irq.irq_lock);
> +			itte->irq.pending = true;
> +			vgic_queue_irq(vcpu->kvm, &itte->irq);
> +		}
> +		spin_unlock(&dist->its.lock);
> +		nr_lpis -= nr_bits;
> +		lpi += nr_bits;
> +		pendbase += nr_bits / 8;
> +	}
> +
> +	return true;
> +}
> +
>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>  				   struct kvm_io_device *this,
>  				   gpa_t addr, int len, void *val)
> @@ -357,6 +479,14 @@ struct vgic_register_region its_registers[] = {
>  /* This is called on setting the LPI enable bit in the redistributor. */
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u64 prop_base_reg, pend_base_reg;
> +
> +	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
> +	prop_base_reg = dist->propbaser;
> +
> +	its_update_lpis_configuration(vcpu->kvm, prop_base_reg);
> +	its_sync_lpi_pending_table(vcpu, pend_base_reg);
>  }
>  
>  int vits_init(struct kvm *kvm)
> @@ -371,6 +501,10 @@ int vits_init(struct kvm *kvm)
>  	if (!dist->pendbaser)
>  		return -ENOMEM;
>  
> +	its->buffer_page = kmalloc(CHUNK_SIZE, GFP_KERNEL);
> +	if (!its->buffer_page)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	INIT_LIST_HEAD(&its->device_list);
> @@ -430,6 +564,7 @@ void vits_destroy(struct kvm *kvm)
>  		kfree(container_of(cur, struct its_collection, coll_list));
>  	}
>  
> +	kfree(its->buffer_page);
>  	kfree(dist->pendbaser);
>  
>  	its->enabled = false;
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 160c511..a7218b0 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -23,6 +23,7 @@
>  #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
>  
>  #define INTERRUPT_ID_BITS_SPIS	10
> +#define INTERRUPT_ID_BITS_ITS	16
>  
>  struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
>  			      u32 intid);
> 


^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables
@ 2016-04-06 13:41     ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-06 13:41 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,
On 03/26/2016 03:14 AM, Andre Przywara wrote:
> The LPI configuration and pending tables of the GICv3 LPIs are held
> in tables in (guest) memory. To achieve reasonable performance, we
> cache this data in our own data structures, so we need to sync those
> two views from time to time. This behaviour is well described in the
> GICv3 spec and is also exercised by hardware, so the sync points are
> well known.
> 
> Provide functions that read the guest memory and store the
> information from the configuration and pending tables in the kernel.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h      |   2 +
>  virt/kvm/arm/vgic/its-emul.c | 135 +++++++++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h     |   1 +
>  3 files changed, 138 insertions(+)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index ecf3260..ea98133 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -121,6 +121,8 @@ struct vgic_its {
>  	int			cwriter;
>  	struct list_head	device_list;
>  	struct list_head	collection_list;
> +	/* memory used for buffering guest's memory */
> +	void			*buffer_page;
>  };
>  
>  struct vgic_dist {
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 1188e9a..d82ba9b 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -78,8 +78,130 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>  	return NULL;
>  }
>  
> +#define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
> +#define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
> +
> +/* stores the priority and enable bit for a given LPI */
> +static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
> +{
> +	spin_lock(&itte->irq.irq_lock);
> +	itte->irq.priority = LPI_PROP_PRIORITY(prop);
> +	itte->irq.enabled = LPI_PROP_ENABLE_BIT(prop);
> +
would put a comment saying that vgic_queue_irq is unlocking since it
looks weird.
> +	vgic_queue_irq(kvm, &itte->irq);
> +}
> +
> +#define GIC_LPI_OFFSET 8192
> +
> +/* We scan the table in chunks the size of the smallest page size */
> +#define CHUNK_SIZE 4096U
> +
>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>  
> +static int nr_idbits_propbase(u64 propbaser)
> +{
> +	int nr_idbits = (1U << (propbaser & 0x1f)) + 1;
> +
> +	return max(nr_idbits, INTERRUPT_ID_BITS_ITS);
don't get this.
don't you want to take the min of INTERRUPT_ID_BITS_ITS  (16) and
propbaser & 0x1f +1.

Spec says IDbits should not exceed GICD_TYPER.IDbit ==
INTERRUPT_ID_BITS_ITS - 1 = 15.

and in case the purpose of the function is to return the size of the
table, which is not obvious given its name ;-), we should return 1U <<
min(), no?
> +}
> +
> +/*
> + * Scan the whole LPI configuration table and put the LPI configuration
> + * data in our own data structures. This relies on the LPI being
> + * mapped before.
> + */
> +static bool its_update_lpis_configuration(struct kvm *kvm, u64 prop_base_reg)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	u8 *prop = dist->its.buffer_page;
> +	u32 tsize;
> +	gpa_t propbase;
> +	int lpi = GIC_LPI_OFFSET;
> +	struct its_itte *itte;
> +	struct its_device *device;
> +	int ret;
> +
> +	propbase = BASER_BASE_ADDRESS(prop_base_reg);
> +	tsize = nr_idbits_propbase(prop_base_reg);
> +
> +	while (tsize > 0) {
> +		int chunksize = min(tsize, CHUNK_SIZE);
> +
> +		ret = kvm_read_guest(kvm, propbase, prop, chunksize);
> +		if (ret)
> +			return false;
> +
> +		spin_lock(&dist->its.lock);
as we start holding several locks maybe worth to comment somewhere about
order lock: its lock > vcpu lock > irq lock
> +		/*
> +		 * Updating the status for all allocated LPIs. We catch
> +		 * those LPIs that get disabled. We really don't care
> +		 * about unmapped LPIs, as they need to be updated
> +		 * later manually anyway once they get mapped.
> +		 */
> +		for_each_lpi(device, itte, kvm) {
> +			if (itte->lpi < lpi || itte->lpi >= lpi + chunksize)
> +				continue;
> +
> +			update_lpi_config(kvm, itte, prop[itte->lpi - lpi]);
> +		}
> +		spin_unlock(&dist->its.lock);
> +		tsize -= chunksize;
> +		lpi += chunksize;
> +		propbase += chunksize;
> +	}
> +
> +	return true;
> +}
> +
> +/*
> + * Scan the whole LPI pending table and sync the pending bit in there
> + * with our own data structures. This relies on the LPI being
> + * mapped before.
> + */
> +static bool its_sync_lpi_pending_table(struct kvm_vcpu *vcpu, u64 base_addr_reg)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	unsigned long *pendmask = dist->its.buffer_page;
> +	u32 nr_lpis = 1U << INTERRUPT_ID_BITS_ITS;
shouldn't we use value computed as in nr_idbits_propbase instead?
> +	gpa_t pendbase;
> +	int lpi = 0;
> +	struct its_itte *itte;
> +	struct its_device *device;
> +	int ret;
> +	int lpi_bit, nr_bits;
> +
> +	pendbase = BASER_BASE_ADDRESS(base_addr_reg);
> +
I understand we should start reading at pendbase + 1KB. (4.8.5 pending
tables)

Eric
> +	while (nr_lpis > 0) {
> +		nr_bits = min(nr_lpis, CHUNK_SIZE * 8);
> +
> +		ret = kvm_read_guest(vcpu->kvm, pendbase, pendmask,
> +				     nr_bits / 8);
> +		if (ret)
> +			return false;
> +
> +		spin_lock(&dist->its.lock);
> +		for_each_lpi(device, itte, vcpu->kvm) {
> +			lpi_bit = itte->lpi - lpi;
> +			if (lpi_bit < 0 || lpi_bit >= nr_bits)
> +				continue;
> +
> +			if (!test_bit(lpi_bit, pendmask))
> +				continue;
> +
> +			spin_lock(&itte->irq.irq_lock);
> +			itte->irq.pending = true;
> +			vgic_queue_irq(vcpu->kvm, &itte->irq);
> +		}
> +		spin_unlock(&dist->its.lock);
> +		nr_lpis -= nr_bits;
> +		lpi += nr_bits;
> +		pendbase += nr_bits / 8;
> +	}
> +
> +	return true;
> +}
> +
>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>  				   struct kvm_io_device *this,
>  				   gpa_t addr, int len, void *val)
> @@ -357,6 +479,14 @@ struct vgic_register_region its_registers[] = {
>  /* This is called on setting the LPI enable bit in the redistributor. */
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u64 prop_base_reg, pend_base_reg;
> +
> +	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
> +	prop_base_reg = dist->propbaser;
> +
> +	its_update_lpis_configuration(vcpu->kvm, prop_base_reg);
> +	its_sync_lpi_pending_table(vcpu, pend_base_reg);
>  }
>  
>  int vits_init(struct kvm *kvm)
> @@ -371,6 +501,10 @@ int vits_init(struct kvm *kvm)
>  	if (!dist->pendbaser)
>  		return -ENOMEM;
>  
> +	its->buffer_page = kmalloc(CHUNK_SIZE, GFP_KERNEL);
> +	if (!its->buffer_page)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	INIT_LIST_HEAD(&its->device_list);
> @@ -430,6 +564,7 @@ void vits_destroy(struct kvm *kvm)
>  		kfree(container_of(cur, struct its_collection, coll_list));
>  	}
>  
> +	kfree(its->buffer_page);
>  	kfree(dist->pendbaser);
>  
>  	its->enabled = false;
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 160c511..a7218b0 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -23,6 +23,7 @@
>  #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
>  
>  #define INTERRUPT_ID_BITS_SPIS	10
> +#define INTERRUPT_ID_BITS_ITS	16
>  
>  struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
>  			      u32 intid);
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-07 13:44     ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 13:44 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

On 26/03/16 02:14, Andre Przywara wrote:
> The ARM GICv3 ITS controller requires a separate register frame to
> cover ITS specific registers. Add a new VGIC address type and store
> the address in a field in the vgic_dist structure.
> Provide a function to check whether userland has provided the address,
> so ITS functionality can be guarded by that check.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>  arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>  include/kvm/vgic/vgic.h                        |  5 +++++
>  virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>  virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>  virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>  6 files changed, 34 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
> index 59541d4..087e2d9 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -39,6 +39,15 @@ Groups:
>        Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>        This address needs to be 64K aligned.
>  
> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
> +      Base address in the guest physical address space of the GICv3 ITS
> +      control register frame. The ITS allows MSI(-X) interrupts to be
> +      injected into guests. This extension is optional, if the kernel
> +      does not support the ITS, the call returns -ENODEV.
> +      This memory is solely for the guest to access the ITS control
> +      registers and does not cover the ITS translation register.
> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
> +      This address needs to be 64K aligned and the region covers 64 KByte.
>  
>    KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>    Attributes:
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index f209ea1..c2b257d 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -87,9 +87,11 @@ struct kvm_regs {
>  /* Supported VGICv3 address types  */
>  #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>  #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>  
>  #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>  #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>  
>  #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>  #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 7c1d145..11344e6 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -135,6 +135,9 @@ struct vgic_dist {
>  		gpa_t			vgic_redist_base;
>  	};
>  
> +	/* The base address of the ITS control register frame */
> +	gpa_t			vgic_its_base;
> +

This makes me cringe a bit. The ITS is not part of the distributor, so
having it in vgic_dist is a bit weird. But maybe it is vgic_dist that
has the wrong name, and it is really a bag of vgic-related information.

But my real worry is that this (and the code below) only allows a single
ITS per VM. Is it really what we want? I would have thought that using
the KVM IO bus abstraction would have given us the flexibility of adding
ITSs at will (all sharing the same LPI space).

The reason this is important is that we could want an ITS serving
emulated devices, and one doing direct interrupt injection (once we have
GICv4 support).

Thoughts?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-04-07 13:44     ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 13:44 UTC (permalink / raw)
  To: linux-arm-kernel

On 26/03/16 02:14, Andre Przywara wrote:
> The ARM GICv3 ITS controller requires a separate register frame to
> cover ITS specific registers. Add a new VGIC address type and store
> the address in a field in the vgic_dist structure.
> Provide a function to check whether userland has provided the address,
> so ITS functionality can be guarded by that check.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>  arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>  include/kvm/vgic/vgic.h                        |  5 +++++
>  virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>  virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>  virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>  6 files changed, 34 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
> index 59541d4..087e2d9 100644
> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
> @@ -39,6 +39,15 @@ Groups:
>        Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>        This address needs to be 64K aligned.
>  
> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
> +      Base address in the guest physical address space of the GICv3 ITS
> +      control register frame. The ITS allows MSI(-X) interrupts to be
> +      injected into guests. This extension is optional, if the kernel
> +      does not support the ITS, the call returns -ENODEV.
> +      This memory is solely for the guest to access the ITS control
> +      registers and does not cover the ITS translation register.
> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
> +      This address needs to be 64K aligned and the region covers 64 KByte.
>  
>    KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>    Attributes:
> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
> index f209ea1..c2b257d 100644
> --- a/arch/arm64/include/uapi/asm/kvm.h
> +++ b/arch/arm64/include/uapi/asm/kvm.h
> @@ -87,9 +87,11 @@ struct kvm_regs {
>  /* Supported VGICv3 address types  */
>  #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>  #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>  
>  #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>  #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>  
>  #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>  #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 7c1d145..11344e6 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -135,6 +135,9 @@ struct vgic_dist {
>  		gpa_t			vgic_redist_base;
>  	};
>  
> +	/* The base address of the ITS control register frame */
> +	gpa_t			vgic_its_base;
> +

This makes me cringe a bit. The ITS is not part of the distributor, so
having it in vgic_dist is a bit weird. But maybe it is vgic_dist that
has the wrong name, and it is really a bag of vgic-related information.

But my real worry is that this (and the code below) only allows a single
ITS per VM. Is it really what we want? I would have thought that using
the KVM IO bus abstraction would have given us the flexibility of adding
ITSs at will (all sharing the same LPI space).

The reason this is important is that we could want an ITS serving
emulated devices, and one doing direct interrupt injection (once we have
GICv4 support).

Thoughts?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-07 13:54     ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 13:54 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: linux-arm-kernel, kvmarm, kvm

On 26/03/16 02:14, Andre Przywara wrote:
> In the GICv3 redistributor there are the PENDBASER and PROPBASER
> registers which we did not emulate so far, as they only make sense
> when having an ITS. In preparation for that emulate those MMIO
> accesses by storing the 64-bit data written into it into a variable
> which we later read in the ITS emulation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h       |  8 ++++++++
>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 11344e6..8ea5dd7 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -145,6 +145,14 @@ struct vgic_dist {
>  
>  	struct vgic_io_device	*dist_iodevs;
>  	struct vgic_io_device	*redist_iodevs;
> +
> +	/* Address of LPI configuration table shared by all redistributors */
> +	u64			propbaser;
> +
> +	/* Addresses of LPI pending tables per redistributor */
> +	u64			*pendbaser;

All these should probably either be phys_addr_t or gpa_t.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-04-07 13:54     ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 13:54 UTC (permalink / raw)
  To: linux-arm-kernel

On 26/03/16 02:14, Andre Przywara wrote:
> In the GICv3 redistributor there are the PENDBASER and PROPBASER
> registers which we did not emulate so far, as they only make sense
> when having an ITS. In preparation for that emulate those MMIO
> accesses by storing the 64-bit data written into it into a variable
> which we later read in the ITS emulation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h       |  8 ++++++++
>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 11344e6..8ea5dd7 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -145,6 +145,14 @@ struct vgic_dist {
>  
>  	struct vgic_io_device	*dist_iodevs;
>  	struct vgic_io_device	*redist_iodevs;
> +
> +	/* Address of LPI configuration table shared by all redistributors */
> +	u64			propbaser;
> +
> +	/* Addresses of LPI pending tables per redistributor */
> +	u64			*pendbaser;

All these should probably either be phys_addr_t or gpa_t.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-04-07 13:54     ` Marc Zyngier
@ 2016-04-07 13:58       ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 13:58 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: linux-arm-kernel, kvmarm, kvm

On 07/04/16 14:54, Marc Zyngier wrote:
> On 26/03/16 02:14, Andre Przywara wrote:
>> In the GICv3 redistributor there are the PENDBASER and PROPBASER
>> registers which we did not emulate so far, as they only make sense
>> when having an ITS. In preparation for that emulate those MMIO
>> accesses by storing the 64-bit data written into it into a variable
>> which we later read in the ITS emulation.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  include/kvm/vgic/vgic.h       |  8 ++++++++
>>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 11344e6..8ea5dd7 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -145,6 +145,14 @@ struct vgic_dist {
>>  
>>  	struct vgic_io_device	*dist_iodevs;
>>  	struct vgic_io_device	*redist_iodevs;
>> +
>> +	/* Address of LPI configuration table shared by all redistributors */
>> +	u64			propbaser;
>> +
>> +	/* Addresses of LPI pending tables per redistributor */
>> +	u64			*pendbaser;
> 
> All these should probably either be phys_addr_t or gpa_t.

Also: these registers are per redistributor. They really should appear
as such, and not be shoved in vgic_dist.

As for GICR_PROPBASER being shared: this is an implementation choice.
All the architecture mandates is that the same value is programmed in
all the redistributors.

Yes, you can share the storage, but I'd really want to see a field per
redist.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-04-07 13:58       ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 13:58 UTC (permalink / raw)
  To: linux-arm-kernel

On 07/04/16 14:54, Marc Zyngier wrote:
> On 26/03/16 02:14, Andre Przywara wrote:
>> In the GICv3 redistributor there are the PENDBASER and PROPBASER
>> registers which we did not emulate so far, as they only make sense
>> when having an ITS. In preparation for that emulate those MMIO
>> accesses by storing the 64-bit data written into it into a variable
>> which we later read in the ITS emulation.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  include/kvm/vgic/vgic.h       |  8 ++++++++
>>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 11344e6..8ea5dd7 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -145,6 +145,14 @@ struct vgic_dist {
>>  
>>  	struct vgic_io_device	*dist_iodevs;
>>  	struct vgic_io_device	*redist_iodevs;
>> +
>> +	/* Address of LPI configuration table shared by all redistributors */
>> +	u64			propbaser;
>> +
>> +	/* Addresses of LPI pending tables per redistributor */
>> +	u64			*pendbaser;
> 
> All these should probably either be phys_addr_t or gpa_t.

Also: these registers are per redistributor. They really should appear
as such, and not be shoved in vgic_dist.

As for GICR_PROPBASER being shared: this is an implementation choice.
All the architecture mandates is that the same value is programmed in
all the redistributors.

Yes, you can share the storage, but I'd really want to see a field per
redist.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-07 14:04     ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 14:04 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: linux-arm-kernel, kvmarm, kvm

On 26/03/16 02:14, Andre Przywara wrote:
> The ARM GICv3 ITS emulation code goes into a separate file, but
> needs to be connected to the GICv3 emulation, of which it is an
> option.
> Introduce the skeleton with function stubs to be filled later.
> Introduce the basic ITS data structure and initialize it, but don't
> return any success yet, as we are not yet ready for the show.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  arch/arm64/kvm/Makefile            |  1 +
>  include/kvm/vgic/vgic.h            |  6 +++
>  include/linux/irqchip/arm-gic-v3.h |  1 +
>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>  7 files changed, 126 insertions(+), 4 deletions(-)
>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
> 
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 2f5d431..3bec10e 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>  else
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 8ea5dd7..c79bed5 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -112,6 +112,11 @@ struct vgic_io_device {
>  	struct kvm_io_device dev;
>  };
>  
> +struct vgic_its {
> +	bool			enabled;
> +	spinlock_t		lock;
> +};
> +
>  struct vgic_dist {
>  	bool			in_kernel;
>  	bool			ready;
> @@ -153,6 +158,7 @@ struct vgic_dist {
>  	u64			*pendbaser;
>  
>  	bool			lpis_enabled;
> +	struct vgic_its		its;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index d5d798b..a813c3e 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -177,6 +177,7 @@
>  #define GITS_CWRITER			0x0088
>  #define GITS_CREADR			0x0090
>  #define GITS_BASER			0x0100
> +#define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
>  
>  #define GITS_TRANSLATER			0x10040
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> new file mode 100644
> index 0000000..49dd5e4
> --- /dev/null
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -0,0 +1,84 @@
> +/*
> + * GICv3 ITS emulation
> + *
> + * Copyright (C) 2015 ARM Ltd.
> + * Author: Andre Przywara <andre.przywara@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/kvm.h>
> +#include <linux/kvm_host.h>
> +#include <linux/interrupt.h>
> +
> +#include <linux/irqchip/arm-gic-v3.h>
> +
> +#include <asm/kvm_emulate.h>
> +#include <asm/kvm_arm.h>
> +#include <asm/kvm_mmu.h>
> +
> +#include "vgic.h"
> +#include "vgic_mmio.h"
> +
> +struct vgic_register_region its_registers[] = {
> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),

GITS_TYPER is a 64bit register with plenty of things in the upper bits.

> +	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +};
> +
> +/* This is called on setting the LPI enable bit in the redistributor. */
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +}
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	struct vgic_io_device *regions;
> +	int ret, i;
> +
> +	spin_lock_init(&its->lock);
> +
> +	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> +				sizeof(struct vgic_io_device), GFP_KERNEL);
> +
> +	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
> +		regions[i].base_addr = dist->vgic_its_base;
> +
> +		ret = kvm_vgic_register_mmio_region(kvm, NULL,
> +						    &its_registers[i],
> +						    &regions[i], 0, false);
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	its->enabled = false;
> +
> +	return -ENXIO;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index db9dfd7..4e7dcb8 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
>  int vgic_v3_probe(struct device_node *vgic_node);
>  int vgic_v3_map_resources(struct kvm *kvm);
>  int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> +
> +int vits_init(struct kvm *kvm);
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>  {
>  	return -ENODEV;
>  }
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	return 0;
> +}
> +
> +static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> @@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>  
>  int vgic_lazy_init(struct kvm *kvm);
>  int vgic_init(struct kvm *kvm);
> +int vits_init(struct kvm *kvm);
>  void kvm_register_vgic_device(unsigned long type);
>  
>  #endif
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index 2301e03..dcfb93d 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
>  	if (ret)
>  		goto out;
>  
> +	if (vgic_has_its(kvm)) {
> +		ret = vits_init(kvm);
> +		if (ret)
> +			goto out;
> +	}

You can move the vgic_has_its() test inside vits_init().

> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 252b9aff..086555e 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
>  				   struct kvm_io_device *this,
>  				   gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +	return 0;
>  }
>  
>  static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
>  				    struct kvm_io_device *this,
>  				    gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
> +
> +	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
> +	    (reg & GICR_CTLR_ENABLE_LPIS)) {
> +		/* Eventually do something */
> +	}
> +
> +	return 0;
>  }
>  
>  static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
> 


-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-04-07 14:04     ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 14:04 UTC (permalink / raw)
  To: linux-arm-kernel

On 26/03/16 02:14, Andre Przywara wrote:
> The ARM GICv3 ITS emulation code goes into a separate file, but
> needs to be connected to the GICv3 emulation, of which it is an
> option.
> Introduce the skeleton with function stubs to be filled later.
> Introduce the basic ITS data structure and initialize it, but don't
> return any success yet, as we are not yet ready for the show.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  arch/arm64/kvm/Makefile            |  1 +
>  include/kvm/vgic/vgic.h            |  6 +++
>  include/linux/irqchip/arm-gic-v3.h |  1 +
>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>  7 files changed, 126 insertions(+), 4 deletions(-)
>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
> 
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index 2f5d431..3bec10e 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>  else
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index 8ea5dd7..c79bed5 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -112,6 +112,11 @@ struct vgic_io_device {
>  	struct kvm_io_device dev;
>  };
>  
> +struct vgic_its {
> +	bool			enabled;
> +	spinlock_t		lock;
> +};
> +
>  struct vgic_dist {
>  	bool			in_kernel;
>  	bool			ready;
> @@ -153,6 +158,7 @@ struct vgic_dist {
>  	u64			*pendbaser;
>  
>  	bool			lpis_enabled;
> +	struct vgic_its		its;
>  };
>  
>  struct vgic_v2_cpu_if {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index d5d798b..a813c3e 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -177,6 +177,7 @@
>  #define GITS_CWRITER			0x0088
>  #define GITS_CREADR			0x0090
>  #define GITS_BASER			0x0100
> +#define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
>  
>  #define GITS_TRANSLATER			0x10040
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> new file mode 100644
> index 0000000..49dd5e4
> --- /dev/null
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -0,0 +1,84 @@
> +/*
> + * GICv3 ITS emulation
> + *
> + * Copyright (C) 2015 ARM Ltd.
> + * Author: Andre Przywara <andre.przywara@arm.com>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <linux/cpu.h>
> +#include <linux/kvm.h>
> +#include <linux/kvm_host.h>
> +#include <linux/interrupt.h>
> +
> +#include <linux/irqchip/arm-gic-v3.h>
> +
> +#include <asm/kvm_emulate.h>
> +#include <asm/kvm_arm.h>
> +#include <asm/kvm_mmu.h>
> +
> +#include "vgic.h"
> +#include "vgic_mmio.h"
> +
> +struct vgic_register_region its_registers[] = {
> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),

GITS_TYPER is a 64bit register with plenty of things in the upper bits.

> +	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
> +	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +};
> +
> +/* This is called on setting the LPI enable bit in the redistributor. */
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +}
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	struct vgic_io_device *regions;
> +	int ret, i;
> +
> +	spin_lock_init(&its->lock);
> +
> +	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> +				sizeof(struct vgic_io_device), GFP_KERNEL);
> +
> +	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
> +		regions[i].base_addr = dist->vgic_its_base;
> +
> +		ret = kvm_vgic_register_mmio_region(kvm, NULL,
> +						    &its_registers[i],
> +						    &regions[i], 0, false);
> +	}
> +
> +	if (ret)
> +		return ret;
> +
> +	its->enabled = false;
> +
> +	return -ENXIO;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index db9dfd7..4e7dcb8 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
>  int vgic_v3_probe(struct device_node *vgic_node);
>  int vgic_v3_map_resources(struct kvm *kvm);
>  int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> +
> +int vits_init(struct kvm *kvm);
> +void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>  {
>  	return -ENODEV;
>  }
> +
> +int vits_init(struct kvm *kvm)
> +{
> +	return 0;
> +}
> +
> +static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> @@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>  
>  int vgic_lazy_init(struct kvm *kvm);
>  int vgic_init(struct kvm *kvm);
> +int vits_init(struct kvm *kvm);
>  void kvm_register_vgic_device(unsigned long type);
>  
>  #endif
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index 2301e03..dcfb93d 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
>  	if (ret)
>  		goto out;
>  
> +	if (vgic_has_its(kvm)) {
> +		ret = vits_init(kvm);
> +		if (ret)
> +			goto out;
> +	}

You can move the vgic_has_its() test inside vits_init().

> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
> index 252b9aff..086555e 100644
> --- a/virt/kvm/arm/vgic/vgic_mmio.c
> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
> @@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
>  				   struct kvm_io_device *this,
>  				   gpa_t addr, int len, void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +	return 0;
>  }
>  
>  static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
>  				    struct kvm_io_device *this,
>  				    gpa_t addr, int len, const void *val)
>  {
> -	/* TODO: implement for ITS support */
> -	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
> +
> +	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
> +	    (reg & GICR_CTLR_ENABLE_LPIS)) {
> +		/* Eventually do something */
> +	}
> +
> +	return 0;
>  }
>  
>  static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
> 


-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-04-07 14:04     ` Marc Zyngier
@ 2016-04-07 14:08       ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-07 14:08 UTC (permalink / raw)
  To: Marc Zyngier, Andre Przywara, Christoffer Dall
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Marc,
On 04/07/2016 04:04 PM, Marc Zyngier wrote:
> On 26/03/16 02:14, Andre Przywara wrote:
>> The ARM GICv3 ITS emulation code goes into a separate file, but
>> needs to be connected to the GICv3 emulation, of which it is an
>> option.
>> Introduce the skeleton with function stubs to be filled later.
>> Introduce the basic ITS data structure and initialize it, but don't
>> return any success yet, as we are not yet ready for the show.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  arch/arm64/kvm/Makefile            |  1 +
>>  include/kvm/vgic/vgic.h            |  6 +++
>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>
>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>> index 2f5d431..3bec10e 100644
>> --- a/arch/arm64/kvm/Makefile
>> +++ b/arch/arm64/kvm/Makefile
>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>  else
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 8ea5dd7..c79bed5 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>  	struct kvm_io_device dev;
>>  };
>>  
>> +struct vgic_its {
>> +	bool			enabled;
>> +	spinlock_t		lock;
>> +};
>> +
>>  struct vgic_dist {
>>  	bool			in_kernel;
>>  	bool			ready;
>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>  	u64			*pendbaser;
>>  
>>  	bool			lpis_enabled;
>> +	struct vgic_its		its;
>>  };
>>  
>>  struct vgic_v2_cpu_if {
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index d5d798b..a813c3e 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -177,6 +177,7 @@
>>  #define GITS_CWRITER			0x0088
>>  #define GITS_CREADR			0x0090
>>  #define GITS_BASER			0x0100
>> +#define GITS_IDREGS_BASE		0xffd0
>>  #define GITS_PIDR2			GICR_PIDR2
>>  
>>  #define GITS_TRANSLATER			0x10040
>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>> new file mode 100644
>> index 0000000..49dd5e4
>> --- /dev/null
>> +++ b/virt/kvm/arm/vgic/its-emul.c
>> @@ -0,0 +1,84 @@
>> +/*
>> + * GICv3 ITS emulation
>> + *
>> + * Copyright (C) 2015 ARM Ltd.
>> + * Author: Andre Przywara <andre.przywara@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <linux/cpu.h>
>> +#include <linux/kvm.h>
>> +#include <linux/kvm_host.h>
>> +#include <linux/interrupt.h>
>> +
>> +#include <linux/irqchip/arm-gic-v3.h>
>> +
>> +#include <asm/kvm_emulate.h>
>> +#include <asm/kvm_arm.h>
>> +#include <asm/kvm_mmu.h>
>> +
>> +#include "vgic.h"
>> +#include "vgic_mmio.h"
>> +
>> +struct vgic_register_region its_registers[] = {
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> 
> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
In practice I think the 32 upper bits are reserved.

Eric
> 
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
>> +};
>> +
>> +/* This is called on setting the LPI enable bit in the redistributor. */
>> +void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>> +{
>> +}
>> +
>> +int vits_init(struct kvm *kvm)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	struct vgic_its *its = &dist->its;
>> +	struct vgic_io_device *regions;
>> +	int ret, i;
>> +
>> +	spin_lock_init(&its->lock);
>> +
>> +	regions = kmalloc_array(ARRAY_SIZE(its_registers),
>> +				sizeof(struct vgic_io_device), GFP_KERNEL);
>> +
>> +	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
>> +		regions[i].base_addr = dist->vgic_its_base;
>> +
>> +		ret = kvm_vgic_register_mmio_region(kvm, NULL,
>> +						    &its_registers[i],
>> +						    &regions[i], 0, false);
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	its->enabled = false;
>> +
>> +	return -ENXIO;
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index db9dfd7..4e7dcb8 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
>>  int vgic_v3_probe(struct device_node *vgic_node);
>>  int vgic_v3_map_resources(struct kvm *kvm);
>>  int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>> +
>> +int vits_init(struct kvm *kvm);
>> +void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>>  #else
>>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>>  					       u64 mpidr)
>> @@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>>  {
>>  	return -ENODEV;
>>  }
>> +
>> +int vits_init(struct kvm *kvm)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>> +{
>> +	return;
>> +}
>>  #endif
>>  
>>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>> @@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>>  
>>  int vgic_lazy_init(struct kvm *kvm);
>>  int vgic_init(struct kvm *kvm);
>> +int vits_init(struct kvm *kvm);
>>  void kvm_register_vgic_device(unsigned long type);
>>  
>>  #endif
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index 2301e03..dcfb93d 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
>>  	if (ret)
>>  		goto out;
>>  
>> +	if (vgic_has_its(kvm)) {
>> +		ret = vits_init(kvm);
>> +		if (ret)
>> +			goto out;
>> +	}
> 
> You can move the vgic_has_its() test inside vits_init().
> 
>> +
>>  	kvm_for_each_vcpu(i, vcpu, kvm)
>>  		kvm_vgic_vcpu_init(vcpu);
>>  
>> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
>> index 252b9aff..086555e 100644
>> --- a/virt/kvm/arm/vgic/vgic_mmio.c
>> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
>> @@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
>>  				   struct kvm_io_device *this,
>>  				   gpa_t addr, int len, void *val)
>>  {
>> -	/* TODO: implement for ITS support */
>> -	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +	return 0;
>>  }
>>  
>>  static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
>>  				    struct kvm_io_device *this,
>>  				    gpa_t addr, int len, const void *val)
>>  {
>> -	/* TODO: implement for ITS support */
>> -	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
>> +
>> +	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
>> +	    (reg & GICR_CTLR_ENABLE_LPIS)) {
>> +		/* Eventually do something */
>> +	}
>> +
>> +	return 0;
>>  }
>>  
>>  static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
>>
> 
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-04-07 14:08       ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-07 14:08 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Marc,
On 04/07/2016 04:04 PM, Marc Zyngier wrote:
> On 26/03/16 02:14, Andre Przywara wrote:
>> The ARM GICv3 ITS emulation code goes into a separate file, but
>> needs to be connected to the GICv3 emulation, of which it is an
>> option.
>> Introduce the skeleton with function stubs to be filled later.
>> Introduce the basic ITS data structure and initialize it, but don't
>> return any success yet, as we are not yet ready for the show.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  arch/arm64/kvm/Makefile            |  1 +
>>  include/kvm/vgic/vgic.h            |  6 +++
>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>
>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>> index 2f5d431..3bec10e 100644
>> --- a/arch/arm64/kvm/Makefile
>> +++ b/arch/arm64/kvm/Makefile
>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>  else
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 8ea5dd7..c79bed5 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>  	struct kvm_io_device dev;
>>  };
>>  
>> +struct vgic_its {
>> +	bool			enabled;
>> +	spinlock_t		lock;
>> +};
>> +
>>  struct vgic_dist {
>>  	bool			in_kernel;
>>  	bool			ready;
>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>  	u64			*pendbaser;
>>  
>>  	bool			lpis_enabled;
>> +	struct vgic_its		its;
>>  };
>>  
>>  struct vgic_v2_cpu_if {
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index d5d798b..a813c3e 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -177,6 +177,7 @@
>>  #define GITS_CWRITER			0x0088
>>  #define GITS_CREADR			0x0090
>>  #define GITS_BASER			0x0100
>> +#define GITS_IDREGS_BASE		0xffd0
>>  #define GITS_PIDR2			GICR_PIDR2
>>  
>>  #define GITS_TRANSLATER			0x10040
>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>> new file mode 100644
>> index 0000000..49dd5e4
>> --- /dev/null
>> +++ b/virt/kvm/arm/vgic/its-emul.c
>> @@ -0,0 +1,84 @@
>> +/*
>> + * GICv3 ITS emulation
>> + *
>> + * Copyright (C) 2015 ARM Ltd.
>> + * Author: Andre Przywara <andre.przywara@arm.com>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <linux/cpu.h>
>> +#include <linux/kvm.h>
>> +#include <linux/kvm_host.h>
>> +#include <linux/interrupt.h>
>> +
>> +#include <linux/irqchip/arm-gic-v3.h>
>> +
>> +#include <asm/kvm_emulate.h>
>> +#include <asm/kvm_arm.h>
>> +#include <asm/kvm_mmu.h>
>> +
>> +#include "vgic.h"
>> +#include "vgic_mmio.h"
>> +
>> +struct vgic_register_region its_registers[] = {
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> 
> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
In practice I think the 32 upper bits are reserved.

Eric
> 
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>> +	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
>> +};
>> +
>> +/* This is called on setting the LPI enable bit in the redistributor. */
>> +void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>> +{
>> +}
>> +
>> +int vits_init(struct kvm *kvm)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	struct vgic_its *its = &dist->its;
>> +	struct vgic_io_device *regions;
>> +	int ret, i;
>> +
>> +	spin_lock_init(&its->lock);
>> +
>> +	regions = kmalloc_array(ARRAY_SIZE(its_registers),
>> +				sizeof(struct vgic_io_device), GFP_KERNEL);
>> +
>> +	for (i = 0; i < ARRAY_SIZE(its_registers); i++) {
>> +		regions[i].base_addr = dist->vgic_its_base;
>> +
>> +		ret = kvm_vgic_register_mmio_region(kvm, NULL,
>> +						    &its_registers[i],
>> +						    &regions[i], 0, false);
>> +	}
>> +
>> +	if (ret)
>> +		return ret;
>> +
>> +	its->enabled = false;
>> +
>> +	return -ENXIO;
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index db9dfd7..4e7dcb8 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -60,6 +60,9 @@ void vgic_v3_enable(struct kvm_vcpu *vcpu);
>>  int vgic_v3_probe(struct device_node *vgic_node);
>>  int vgic_v3_map_resources(struct kvm *kvm);
>>  int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>> +
>> +int vits_init(struct kvm *kvm);
>> +void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>>  #else
>>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>>  					       u64 mpidr)
>> @@ -124,6 +127,16 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
>>  {
>>  	return -ENODEV;
>>  }
>> +
>> +int vits_init(struct kvm *kvm)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>> +{
>> +	return;
>> +}
>>  #endif
>>  
>>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>> @@ -131,6 +144,7 @@ void vgic_get_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>>  
>>  int vgic_lazy_init(struct kvm *kvm);
>>  int vgic_init(struct kvm *kvm);
>> +int vits_init(struct kvm *kvm);
>>  void kvm_register_vgic_device(unsigned long type);
>>  
>>  #endif
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index 2301e03..dcfb93d 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -252,6 +252,12 @@ int vgic_init(struct kvm *kvm)
>>  	if (ret)
>>  		goto out;
>>  
>> +	if (vgic_has_its(kvm)) {
>> +		ret = vits_init(kvm);
>> +		if (ret)
>> +			goto out;
>> +	}
> 
> You can move the vgic_has_its() test inside vits_init().
> 
>> +
>>  	kvm_for_each_vcpu(i, vcpu, kvm)
>>  		kvm_vgic_vcpu_init(vcpu);
>>  
>> diff --git a/virt/kvm/arm/vgic/vgic_mmio.c b/virt/kvm/arm/vgic/vgic_mmio.c
>> index 252b9aff..086555e 100644
>> --- a/virt/kvm/arm/vgic/vgic_mmio.c
>> +++ b/virt/kvm/arm/vgic/vgic_mmio.c
>> @@ -726,16 +726,26 @@ static int vgic_mmio_read_v3r_misc(struct kvm_vcpu *vcpu,
>>  				   struct kvm_io_device *this,
>>  				   gpa_t addr, int len, void *val)
>>  {
>> -	/* TODO: implement for ITS support */
>> -	return vgic_mmio_read_raz(vcpu, this, addr, len, val);
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	u32 reg = dist->lpis_enabled ? GICR_CTLR_ENABLE_LPIS : 0;
>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +	return 0;
>>  }
>>  
>>  static int vgic_mmio_write_v3r_misc(struct kvm_vcpu *vcpu,
>>  				    struct kvm_io_device *this,
>>  				    gpa_t addr, int len, const void *val)
>>  {
>> -	/* TODO: implement for ITS support */
>> -	return vgic_mmio_write_wi(vcpu, this, addr, len, val);
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	u32 reg = mask32(!!dist->lpis_enabled, addr & 3, len, val);
>> +
>> +	if (vgic_has_its(vcpu->kvm) && !dist->lpis_enabled &&
>> +	    (reg & GICR_CTLR_ENABLE_LPIS)) {
>> +		/* Eventually do something */
>> +	}
>> +
>> +	return 0;
>>  }
>>  
>>  static int vgic_mmio_read_v3r_iidr(struct kvm_vcpu *vcpu,
>>
> 
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-04-07 14:35     ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 14:35 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

On 26/03/16 02:14, Andre Przywara wrote:
> Add emulation for some basic MMIO registers used in the ITS emulation.
> This includes:
> - GITS_{CTLR,TYPER,IIDR}
> - ID registers
> - GITS_{CBASER,CREADR,CWRITER}
>   those implement the ITS command buffer handling
> 
> Most of the handlers are pretty straight forward, but CWRITER goes
> some extra miles to allow fine grained locking. The idea here
> is to let only the first instance iterate through the command ring
> buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
> by that first instance and handled as well. The ITS lock is thus only
> hold for very small periods of time and is dropped before the actual

s/hold/held/

> command handler is called.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h            |   3 +
>  include/linux/irqchip/arm-gic-v3.h |   8 ++
>  virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h           |   6 +
>  virt/kvm/arm/vgic/vgic_init.c      |   2 +
>  5 files changed, 284 insertions(+), 7 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c79bed5..bafea11 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -115,6 +115,9 @@ struct vgic_io_device {
>  struct vgic_its {
>  	bool			enabled;
>  	spinlock_t		lock;
> +	u64			cbaser;
> +	int			creadr;
> +	int			cwriter;

Irk. Please use explicitly sized types.

>  };
>  
>  struct vgic_dist {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index a813c3e..7011b98 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -179,15 +179,23 @@
>  #define GITS_BASER			0x0100
>  #define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
> +#define GITS_PIDR4			0xffd0
> +#define GITS_CIDR0			0xfff0
> +#define GITS_CIDR1			0xfff4
> +#define GITS_CIDR2			0xfff8
> +#define GITS_CIDR3			0xfffc
>  
>  #define GITS_TRANSLATER			0x10040
>  
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
> +#define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
>  #define GITS_TYPER_PTA			(1UL << 19)
> +#define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
>  #define GITS_CBASER_VALID		(1UL << 63)
>  #define GITS_CBASER_nCnB		(0UL << 59)
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 49dd5e4..de8d360 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -31,23 +31,263 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> +
> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u32 reg;
> +
> +	reg = GITS_CTLR_QUIESCENT;

So your ITS is always in a quiescent state? Even when you're processing
the command queue? You'll have to convince me...

> +	if (its->enabled)
> +		reg |= GITS_CTLR_ENABLE;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +
> +        if (addr - iodev->base_addr == 0)

whitespace issue.

> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	u64 reg = GITS_TYPER_PLPIS;
> +
> +	/*
> +	 * We use linear CPU numbers for redistributor addressing,
> +	 * so GITS_TYPER.PTA is 0.
> +	 * To avoid memory waste on the guest side, we keep the
> +	 * number of IDBits and DevBits low for the time being.
> +	 * This could later be made configurable by userland.
> +	 * Since we have all collections in linked list, we claim
> +	 * that we can hold all of the collection tables in our
> +	 * own memory and that the ITT entry size is 1 byte (the
> +	 * smallest possible one).

All of this is going to bite us when we want to implement migration,
specially the HW collection bit.

> +	 */
> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u32 reg = 0;
> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
> +
> +	switch (idreg) {
> +	case GITS_PIDR2:
> +		reg = GIC_PIDR2_ARCH_GICv3;

Are we leaving the lowest 4 bits to zero?

> +		break;
> +	case GITS_PIDR4:
> +		/* This is a 64K software visible page */
> +		reg = 0x40;

Same question.

Also, how about all the others PIDR registers?

> +		break;
> +	/* Those are the ID registers for (any) GIC. */
> +	case GITS_CIDR0:
> +		reg = 0x0d;
> +		break;
> +	case GITS_CIDR1:
> +		reg = 0xf0;
> +		break;
> +	case GITS_CIDR2:
> +		reg = 0x05;
> +		break;
> +	case GITS_CIDR3:
> +		reg = 0xb1;
> +		break;
> +	}

Given that these values are directly taken from the architecture, and
seem common to the whole GICv3 architecture when implemented by ARM, we
could have a common handler for the whole GICv3 implementatuin. Not a
bit deal though.

> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +/*
> + * This function is called with both the ITS and the distributor lock dropped,
> + * so the actual command handlers must take the respective locks when needed.
> + */
> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> +{
> +	return -ENODEV;
> +}
> +
> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	write_mask64(its->cbaser, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	if (its->enabled)
> +		return 0;
> +
> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
> +	its->creadr = 0;

Don't you need to acquire the command queue lock here?

> +
> +	return 0;
> +}
> +
> +static int its_cmd_buffer_size(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return ((its->cbaser & 0xff) + 1) << 12;
> +}
> +
> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return BASER_BASE_ADDRESS(its->cbaser);
> +}
> +
> +/*
> + * By writing to CWRITER the guest announces new commands to be processed.
> + * Since we cannot read from guest memory inside the ITS spinlock, we
> + * iterate over the command buffer (with the lock dropped) until the read
> + * pointer matches the write pointer. Other VCPUs writing this register in the
> + * meantime will just update the write pointer, leaving the command
> + * processing to the first instance of the function.
> + */
> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
> +				       struct kvm_io_device *this,
> +				       gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
> +	u64 cmd_buf[4];
> +	u32 reg;
> +	bool finished;
> +
> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
> +	reg &= 0xfffe0;
> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
> +		return 0;
> +
> +	spin_lock(&its->lock);
> +
> +	/*
> +	 * If there is still another VCPU handling commands, let this
> +	 * one pick up the new CWRITER and process "our" new commands as well.
> +	 */

How do you detect that condition? All I see is a massive race here, with
two threads processing the queue in parallel, possibly corrupting each
other's data.

Please explain why you think this is safe.

> +	finished = (its->cwriter != its->creadr);
> +	its->cwriter = reg;
> +
> +	spin_unlock(&its->lock);
> +
> +	while (!finished) {
> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
> +					 cmd_buf, 32);
> +		if (ret) {
> +			/*
> +			 * Gah, we are screwed. Reset CWRITER to that command
> +			 * that we have finished processing and return.
> +			 */
> +			spin_lock(&its->lock);
> +			its->cwriter = its->creadr;
> +			spin_unlock(&its->lock);
> +			break;
> +		}
> +		vits_handle_command(vcpu, cmd_buf);
> +
> +		spin_lock(&its->lock);
> +		its->creadr += 32;
> +		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
> +			its->creadr = 0;
> +		finished = (its->creadr == its->cwriter);
> +		spin_unlock(&its->lock);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->cwriter & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->creadr & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
>  struct vgic_register_region its_registers[] = {
>  	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>  		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
>  };
>  
>  /* This is called on setting the LPI enable bit in the redistributor. */
> @@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	int nr_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct vgic_io_device *regions;
>  	int ret, i;
>  
> +	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
> +	if (!dist->pendbaser)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> @@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
>  
>  	return -ENXIO;
>  }
> +
> +void vits_destroy(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +
> +	if (!vgic_has_its(kvm))
> +		return;
> +
> +	kfree(dist->pendbaser);
> +
> +	its->enabled = false;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 4e7dcb8..08f97d1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
>  	return;
>  }
> +
> +static inline void vits_destroy(struct kvm *kvm)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index dcfb93d..e4459e3 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
>  
>  	kvm_vgic_dist_destroy(kvm);
>  
> +	vits_destroy(kvm);
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_destroy(vcpu);
>  }
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-04-07 14:35     ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 14:35 UTC (permalink / raw)
  To: linux-arm-kernel

On 26/03/16 02:14, Andre Przywara wrote:
> Add emulation for some basic MMIO registers used in the ITS emulation.
> This includes:
> - GITS_{CTLR,TYPER,IIDR}
> - ID registers
> - GITS_{CBASER,CREADR,CWRITER}
>   those implement the ITS command buffer handling
> 
> Most of the handlers are pretty straight forward, but CWRITER goes
> some extra miles to allow fine grained locking. The idea here
> is to let only the first instance iterate through the command ring
> buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
> by that first instance and handled as well. The ITS lock is thus only
> hold for very small periods of time and is dropped before the actual

s/hold/held/

> command handler is called.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  include/kvm/vgic/vgic.h            |   3 +
>  include/linux/irqchip/arm-gic-v3.h |   8 ++
>  virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
>  virt/kvm/arm/vgic/vgic.h           |   6 +
>  virt/kvm/arm/vgic/vgic_init.c      |   2 +
>  5 files changed, 284 insertions(+), 7 deletions(-)
> 
> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
> index c79bed5..bafea11 100644
> --- a/include/kvm/vgic/vgic.h
> +++ b/include/kvm/vgic/vgic.h
> @@ -115,6 +115,9 @@ struct vgic_io_device {
>  struct vgic_its {
>  	bool			enabled;
>  	spinlock_t		lock;
> +	u64			cbaser;
> +	int			creadr;
> +	int			cwriter;

Irk. Please use explicitly sized types.

>  };
>  
>  struct vgic_dist {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index a813c3e..7011b98 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -179,15 +179,23 @@
>  #define GITS_BASER			0x0100
>  #define GITS_IDREGS_BASE		0xffd0
>  #define GITS_PIDR2			GICR_PIDR2
> +#define GITS_PIDR4			0xffd0
> +#define GITS_CIDR0			0xfff0
> +#define GITS_CIDR1			0xfff4
> +#define GITS_CIDR2			0xfff8
> +#define GITS_CIDR3			0xfffc
>  
>  #define GITS_TRANSLATER			0x10040
>  
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_QUIESCENT		(1U << 31)
>  
> +#define GITS_TYPER_PLPIS		(1UL << 0)
> +#define GITS_TYPER_IDBITS_SHIFT		8
>  #define GITS_TYPER_DEVBITS_SHIFT	13
>  #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
>  #define GITS_TYPER_PTA			(1UL << 19)
> +#define GITS_TYPER_HWCOLLCNT_SHIFT	24
>  
>  #define GITS_CBASER_VALID		(1UL << 63)
>  #define GITS_CBASER_nCnB		(0UL << 59)
> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
> index 49dd5e4..de8d360 100644
> --- a/virt/kvm/arm/vgic/its-emul.c
> +++ b/virt/kvm/arm/vgic/its-emul.c
> @@ -31,23 +31,263 @@
>  #include "vgic.h"
>  #include "vgic_mmio.h"
>  
> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> +
> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u32 reg;
> +
> +	reg = GITS_CTLR_QUIESCENT;

So your ITS is always in a quiescent state? Even when you're processing
the command queue? You'll have to convince me...

> +	if (its->enabled)
> +		reg |= GITS_CTLR_ENABLE;
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +
> +        if (addr - iodev->base_addr == 0)

whitespace issue.

> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	u64 reg = GITS_TYPER_PLPIS;
> +
> +	/*
> +	 * We use linear CPU numbers for redistributor addressing,
> +	 * so GITS_TYPER.PTA is 0.
> +	 * To avoid memory waste on the guest side, we keep the
> +	 * number of IDBits and DevBits low for the time being.
> +	 * This could later be made configurable by userland.
> +	 * Since we have all collections in linked list, we claim
> +	 * that we can hold all of the collection tables in our
> +	 * own memory and that the ITT entry size is 1 byte (the
> +	 * smallest possible one).

All of this is going to bite us when we want to implement migration,
specially the HW collection bit.

> +	 */
> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
> +				   struct kvm_io_device *this,
> +				   gpa_t addr, int len, void *val)
> +{
> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_io_device *iodev = container_of(this,
> +						    struct vgic_io_device, dev);
> +	u32 reg = 0;
> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
> +
> +	switch (idreg) {
> +	case GITS_PIDR2:
> +		reg = GIC_PIDR2_ARCH_GICv3;

Are we leaving the lowest 4 bits to zero?

> +		break;
> +	case GITS_PIDR4:
> +		/* This is a 64K software visible page */
> +		reg = 0x40;

Same question.

Also, how about all the others PIDR registers?

> +		break;
> +	/* Those are the ID registers for (any) GIC. */
> +	case GITS_CIDR0:
> +		reg = 0x0d;
> +		break;
> +	case GITS_CIDR1:
> +		reg = 0xf0;
> +		break;
> +	case GITS_CIDR2:
> +		reg = 0x05;
> +		break;
> +	case GITS_CIDR3:
> +		reg = 0xb1;
> +		break;
> +	}

Given that these values are directly taken from the architecture, and
seem common to the whole GICv3 architecture when implemented by ARM, we
could have a common handler for the whole GICv3 implementatuin. Not a
bit deal though.

> +
> +	write_mask32(reg, addr & 3, len, val);
> +
> +	return 0;
> +}
> +
> +/*
> + * This function is called with both the ITS and the distributor lock dropped,
> + * so the actual command handlers must take the respective locks when needed.
> + */
> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> +{
> +	return -ENODEV;
> +}
> +
> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
> +				    struct kvm_io_device *this,
> +				    gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	write_mask64(its->cbaser, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +
> +	if (its->enabled)
> +		return 0;
> +
> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
> +	its->creadr = 0;

Don't you need to acquire the command queue lock here?

> +
> +	return 0;
> +}
> +
> +static int its_cmd_buffer_size(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return ((its->cbaser & 0xff) + 1) << 12;
> +}
> +
> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
> +{
> +	struct vgic_its *its = &kvm->arch.vgic.its;
> +
> +	return BASER_BASE_ADDRESS(its->cbaser);
> +}
> +
> +/*
> + * By writing to CWRITER the guest announces new commands to be processed.
> + * Since we cannot read from guest memory inside the ITS spinlock, we
> + * iterate over the command buffer (with the lock dropped) until the read
> + * pointer matches the write pointer. Other VCPUs writing this register in the
> + * meantime will just update the write pointer, leaving the command
> + * processing to the first instance of the function.
> + */
> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
> +				       struct kvm_io_device *this,
> +				       gpa_t addr, int len, const void *val)
> +{
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
> +	u64 cmd_buf[4];
> +	u32 reg;
> +	bool finished;
> +
> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
> +	reg &= 0xfffe0;
> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
> +		return 0;
> +
> +	spin_lock(&its->lock);
> +
> +	/*
> +	 * If there is still another VCPU handling commands, let this
> +	 * one pick up the new CWRITER and process "our" new commands as well.
> +	 */

How do you detect that condition? All I see is a massive race here, with
two threads processing the queue in parallel, possibly corrupting each
other's data.

Please explain why you think this is safe.

> +	finished = (its->cwriter != its->creadr);
> +	its->cwriter = reg;
> +
> +	spin_unlock(&its->lock);
> +
> +	while (!finished) {
> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
> +					 cmd_buf, 32);
> +		if (ret) {
> +			/*
> +			 * Gah, we are screwed. Reset CWRITER to that command
> +			 * that we have finished processing and return.
> +			 */
> +			spin_lock(&its->lock);
> +			its->cwriter = its->creadr;
> +			spin_unlock(&its->lock);
> +			break;
> +		}
> +		vits_handle_command(vcpu, cmd_buf);
> +
> +		spin_lock(&its->lock);
> +		its->creadr += 32;
> +		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
> +			its->creadr = 0;
> +		finished = (its->creadr == its->cwriter);
> +		spin_unlock(&its->lock);
> +	}
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
> +				      struct kvm_io_device *this,
> +				      gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->cwriter & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
> +static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
> +				     struct kvm_io_device *this,
> +				     gpa_t addr, int len, void *val)
> +{
> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
> +	u64 reg = its->creadr & 0xfffe0;
> +
> +	write_mask64(reg, addr & 7, len, val);
> +
> +	return 0;
> +}
> +
>  struct vgic_register_region its_registers[] = {
>  	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
> +		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
> +		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
>  	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
>  		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
>  	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
> -		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
> +		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
>  };
>  
>  /* This is called on setting the LPI enable bit in the redistributor. */
> @@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
>  	struct vgic_its *its = &dist->its;
> +	int nr_vcpus = atomic_read(&kvm->online_vcpus);
>  	struct vgic_io_device *regions;
>  	int ret, i;
>  
> +	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
> +	if (!dist->pendbaser)
> +		return -ENOMEM;
> +
>  	spin_lock_init(&its->lock);
>  
>  	regions = kmalloc_array(ARRAY_SIZE(its_registers),
> @@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
>  
>  	return -ENXIO;
>  }
> +
> +void vits_destroy(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct vgic_its *its = &dist->its;
> +
> +	if (!vgic_has_its(kvm))
> +		return;
> +
> +	kfree(dist->pendbaser);
> +
> +	its->enabled = false;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index 4e7dcb8..08f97d1 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
>  
>  int vits_init(struct kvm *kvm);
>  void vgic_enable_lpis(struct kvm_vcpu *vcpu);
> +void vits_destroy(struct kvm *kvm);
>  #else
>  static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>  					       u64 mpidr)
> @@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>  {
>  	return;
>  }
> +
> +static inline void vits_destroy(struct kvm *kvm)
> +{
> +	return;
> +}
>  #endif
>  
>  void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
> index dcfb93d..e4459e3 100644
> --- a/virt/kvm/arm/vgic/vgic_init.c
> +++ b/virt/kvm/arm/vgic/vgic_init.c
> @@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
>  
>  	kvm_vgic_dist_destroy(kvm);
>  
> +	vits_destroy(kvm);
> +
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_destroy(vcpu);
>  }
> 

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-04-07 14:08       ` Eric Auger
@ 2016-04-07 14:48         ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 14:48 UTC (permalink / raw)
  To: Eric Auger, Andre Przywara, Christoffer Dall
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Eric,

On 07/04/16 15:08, Eric Auger wrote:
> Hi Marc,
> On 04/07/2016 04:04 PM, Marc Zyngier wrote:
>> On 26/03/16 02:14, Andre Przywara wrote:
>>> The ARM GICv3 ITS emulation code goes into a separate file, but
>>> needs to be connected to the GICv3 emulation, of which it is an
>>> option.
>>> Introduce the skeleton with function stubs to be filled later.
>>> Introduce the basic ITS data structure and initialize it, but don't
>>> return any success yet, as we are not yet ready for the show.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>>  arch/arm64/kvm/Makefile            |  1 +
>>>  include/kvm/vgic/vgic.h            |  6 +++
>>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>>
>>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>>> index 2f5d431..3bec10e 100644
>>> --- a/arch/arm64/kvm/Makefile
>>> +++ b/arch/arm64/kvm/Makefile
>>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>>  else
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>> index 8ea5dd7..c79bed5 100644
>>> --- a/include/kvm/vgic/vgic.h
>>> +++ b/include/kvm/vgic/vgic.h
>>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>>  	struct kvm_io_device dev;
>>>  };
>>>  
>>> +struct vgic_its {
>>> +	bool			enabled;
>>> +	spinlock_t		lock;
>>> +};
>>> +
>>>  struct vgic_dist {
>>>  	bool			in_kernel;
>>>  	bool			ready;
>>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>>  	u64			*pendbaser;
>>>  
>>>  	bool			lpis_enabled;
>>> +	struct vgic_its		its;
>>>  };
>>>  
>>>  struct vgic_v2_cpu_if {
>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>> index d5d798b..a813c3e 100644
>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>> @@ -177,6 +177,7 @@
>>>  #define GITS_CWRITER			0x0088
>>>  #define GITS_CREADR			0x0090
>>>  #define GITS_BASER			0x0100
>>> +#define GITS_IDREGS_BASE		0xffd0
>>>  #define GITS_PIDR2			GICR_PIDR2
>>>  
>>>  #define GITS_TRANSLATER			0x10040
>>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>>> new file mode 100644
>>> index 0000000..49dd5e4
>>> --- /dev/null
>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>> @@ -0,0 +1,84 @@
>>> +/*
>>> + * GICv3 ITS emulation
>>> + *
>>> + * Copyright (C) 2015 ARM Ltd.
>>> + * Author: Andre Przywara <andre.przywara@arm.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <linux/cpu.h>
>>> +#include <linux/kvm.h>
>>> +#include <linux/kvm_host.h>
>>> +#include <linux/interrupt.h>
>>> +
>>> +#include <linux/irqchip/arm-gic-v3.h>
>>> +
>>> +#include <asm/kvm_emulate.h>
>>> +#include <asm/kvm_arm.h>
>>> +#include <asm/kvm_mmu.h>
>>> +
>>> +#include "vgic.h"
>>> +#include "vgic_mmio.h"
>>> +
>>> +struct vgic_register_region its_registers[] = {
>>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>
>> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
> In practice I think the 32 upper bits are reserved.

Not quite. Only bits [63:38] are reserved. See 8.19.8 in the
Architecture Specification (IHI 0069B).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-04-07 14:48         ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 14:48 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Eric,

On 07/04/16 15:08, Eric Auger wrote:
> Hi Marc,
> On 04/07/2016 04:04 PM, Marc Zyngier wrote:
>> On 26/03/16 02:14, Andre Przywara wrote:
>>> The ARM GICv3 ITS emulation code goes into a separate file, but
>>> needs to be connected to the GICv3 emulation, of which it is an
>>> option.
>>> Introduce the skeleton with function stubs to be filled later.
>>> Introduce the basic ITS data structure and initialize it, but don't
>>> return any success yet, as we are not yet ready for the show.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>>  arch/arm64/kvm/Makefile            |  1 +
>>>  include/kvm/vgic/vgic.h            |  6 +++
>>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>>
>>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>>> index 2f5d431..3bec10e 100644
>>> --- a/arch/arm64/kvm/Makefile
>>> +++ b/arch/arm64/kvm/Makefile
>>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>>  else
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>> index 8ea5dd7..c79bed5 100644
>>> --- a/include/kvm/vgic/vgic.h
>>> +++ b/include/kvm/vgic/vgic.h
>>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>>  	struct kvm_io_device dev;
>>>  };
>>>  
>>> +struct vgic_its {
>>> +	bool			enabled;
>>> +	spinlock_t		lock;
>>> +};
>>> +
>>>  struct vgic_dist {
>>>  	bool			in_kernel;
>>>  	bool			ready;
>>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>>  	u64			*pendbaser;
>>>  
>>>  	bool			lpis_enabled;
>>> +	struct vgic_its		its;
>>>  };
>>>  
>>>  struct vgic_v2_cpu_if {
>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>> index d5d798b..a813c3e 100644
>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>> @@ -177,6 +177,7 @@
>>>  #define GITS_CWRITER			0x0088
>>>  #define GITS_CREADR			0x0090
>>>  #define GITS_BASER			0x0100
>>> +#define GITS_IDREGS_BASE		0xffd0
>>>  #define GITS_PIDR2			GICR_PIDR2
>>>  
>>>  #define GITS_TRANSLATER			0x10040
>>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>>> new file mode 100644
>>> index 0000000..49dd5e4
>>> --- /dev/null
>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>> @@ -0,0 +1,84 @@
>>> +/*
>>> + * GICv3 ITS emulation
>>> + *
>>> + * Copyright (C) 2015 ARM Ltd.
>>> + * Author: Andre Przywara <andre.przywara@arm.com>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <linux/cpu.h>
>>> +#include <linux/kvm.h>
>>> +#include <linux/kvm_host.h>
>>> +#include <linux/interrupt.h>
>>> +
>>> +#include <linux/irqchip/arm-gic-v3.h>
>>> +
>>> +#include <asm/kvm_emulate.h>
>>> +#include <asm/kvm_arm.h>
>>> +#include <asm/kvm_mmu.h>
>>> +
>>> +#include "vgic.h"
>>> +#include "vgic_mmio.h"
>>> +
>>> +struct vgic_register_region its_registers[] = {
>>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>
>> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
> In practice I think the 32 upper bits are reserved.

Not quite. Only bits [63:38] are reserved. See 8.19.8 in the
Architecture Specification (IHI 0069B).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-04-07 14:48         ` Marc Zyngier
@ 2016-04-07 15:09           ` Eric Auger
  -1 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-07 15:09 UTC (permalink / raw)
  To: Marc Zyngier, Andre Przywara, Christoffer Dall
  Cc: linux-arm-kernel, kvmarm, kvm

On 04/07/2016 04:48 PM, Marc Zyngier wrote:
> Hi Eric,
> 
> On 07/04/16 15:08, Eric Auger wrote:
>> Hi Marc,
>> On 04/07/2016 04:04 PM, Marc Zyngier wrote:
>>> On 26/03/16 02:14, Andre Przywara wrote:
>>>> The ARM GICv3 ITS emulation code goes into a separate file, but
>>>> needs to be connected to the GICv3 emulation, of which it is an
>>>> option.
>>>> Introduce the skeleton with function stubs to be filled later.
>>>> Introduce the basic ITS data structure and initialize it, but don't
>>>> return any success yet, as we are not yet ready for the show.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>> ---
>>>>  arch/arm64/kvm/Makefile            |  1 +
>>>>  include/kvm/vgic/vgic.h            |  6 +++
>>>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>>>
>>>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>>>> index 2f5d431..3bec10e 100644
>>>> --- a/arch/arm64/kvm/Makefile
>>>> +++ b/arch/arm64/kvm/Makefile
>>>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>>>  else
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>>> index 8ea5dd7..c79bed5 100644
>>>> --- a/include/kvm/vgic/vgic.h
>>>> +++ b/include/kvm/vgic/vgic.h
>>>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>>>  	struct kvm_io_device dev;
>>>>  };
>>>>  
>>>> +struct vgic_its {
>>>> +	bool			enabled;
>>>> +	spinlock_t		lock;
>>>> +};
>>>> +
>>>>  struct vgic_dist {
>>>>  	bool			in_kernel;
>>>>  	bool			ready;
>>>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>>>  	u64			*pendbaser;
>>>>  
>>>>  	bool			lpis_enabled;
>>>> +	struct vgic_its		its;
>>>>  };
>>>>  
>>>>  struct vgic_v2_cpu_if {
>>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>>> index d5d798b..a813c3e 100644
>>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>>> @@ -177,6 +177,7 @@
>>>>  #define GITS_CWRITER			0x0088
>>>>  #define GITS_CREADR			0x0090
>>>>  #define GITS_BASER			0x0100
>>>> +#define GITS_IDREGS_BASE		0xffd0
>>>>  #define GITS_PIDR2			GICR_PIDR2
>>>>  
>>>>  #define GITS_TRANSLATER			0x10040
>>>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>>>> new file mode 100644
>>>> index 0000000..49dd5e4
>>>> --- /dev/null
>>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>>> @@ -0,0 +1,84 @@
>>>> +/*
>>>> + * GICv3 ITS emulation
>>>> + *
>>>> + * Copyright (C) 2015 ARM Ltd.
>>>> + * Author: Andre Przywara <andre.przywara@arm.com>
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License version 2 as
>>>> + * published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>> + * GNU General Public License for more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License
>>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <linux/cpu.h>
>>>> +#include <linux/kvm.h>
>>>> +#include <linux/kvm_host.h>
>>>> +#include <linux/interrupt.h>
>>>> +
>>>> +#include <linux/irqchip/arm-gic-v3.h>
>>>> +
>>>> +#include <asm/kvm_emulate.h>
>>>> +#include <asm/kvm_arm.h>
>>>> +#include <asm/kvm_mmu.h>
>>>> +
>>>> +#include "vgic.h"
>>>> +#include "vgic_mmio.h"
>>>> +
>>>> +struct vgic_register_region its_registers[] = {
>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>
>>> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
>> In practice I think the 32 upper bits are reserved.
> 
> Not quite. Only bits [63:38] are reserved. See 8.19.8 in the
> Architecture Specification (IHI 0069B).

Ah OK. Was using an older arch spec(PRD03-GENC-010745 22.0) Now switched
to the indicated one and indeed it's different !;-)

Eric
> 
> Thanks,
> 
> 	M.
> 


^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-04-07 15:09           ` Eric Auger
  0 siblings, 0 replies; 120+ messages in thread
From: Eric Auger @ 2016-04-07 15:09 UTC (permalink / raw)
  To: linux-arm-kernel

On 04/07/2016 04:48 PM, Marc Zyngier wrote:
> Hi Eric,
> 
> On 07/04/16 15:08, Eric Auger wrote:
>> Hi Marc,
>> On 04/07/2016 04:04 PM, Marc Zyngier wrote:
>>> On 26/03/16 02:14, Andre Przywara wrote:
>>>> The ARM GICv3 ITS emulation code goes into a separate file, but
>>>> needs to be connected to the GICv3 emulation, of which it is an
>>>> option.
>>>> Introduce the skeleton with function stubs to be filled later.
>>>> Introduce the basic ITS data structure and initialize it, but don't
>>>> return any success yet, as we are not yet ready for the show.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>> ---
>>>>  arch/arm64/kvm/Makefile            |  1 +
>>>>  include/kvm/vgic/vgic.h            |  6 +++
>>>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>>>
>>>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>>>> index 2f5d431..3bec10e 100644
>>>> --- a/arch/arm64/kvm/Makefile
>>>> +++ b/arch/arm64/kvm/Makefile
>>>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>>>  else
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>>> index 8ea5dd7..c79bed5 100644
>>>> --- a/include/kvm/vgic/vgic.h
>>>> +++ b/include/kvm/vgic/vgic.h
>>>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>>>  	struct kvm_io_device dev;
>>>>  };
>>>>  
>>>> +struct vgic_its {
>>>> +	bool			enabled;
>>>> +	spinlock_t		lock;
>>>> +};
>>>> +
>>>>  struct vgic_dist {
>>>>  	bool			in_kernel;
>>>>  	bool			ready;
>>>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>>>  	u64			*pendbaser;
>>>>  
>>>>  	bool			lpis_enabled;
>>>> +	struct vgic_its		its;
>>>>  };
>>>>  
>>>>  struct vgic_v2_cpu_if {
>>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>>> index d5d798b..a813c3e 100644
>>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>>> @@ -177,6 +177,7 @@
>>>>  #define GITS_CWRITER			0x0088
>>>>  #define GITS_CREADR			0x0090
>>>>  #define GITS_BASER			0x0100
>>>> +#define GITS_IDREGS_BASE		0xffd0
>>>>  #define GITS_PIDR2			GICR_PIDR2
>>>>  
>>>>  #define GITS_TRANSLATER			0x10040
>>>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>>>> new file mode 100644
>>>> index 0000000..49dd5e4
>>>> --- /dev/null
>>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>>> @@ -0,0 +1,84 @@
>>>> +/*
>>>> + * GICv3 ITS emulation
>>>> + *
>>>> + * Copyright (C) 2015 ARM Ltd.
>>>> + * Author: Andre Przywara <andre.przywara@arm.com>
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify
>>>> + * it under the terms of the GNU General Public License version 2 as
>>>> + * published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>> + * GNU General Public License for more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License
>>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <linux/cpu.h>
>>>> +#include <linux/kvm.h>
>>>> +#include <linux/kvm_host.h>
>>>> +#include <linux/interrupt.h>
>>>> +
>>>> +#include <linux/irqchip/arm-gic-v3.h>
>>>> +
>>>> +#include <asm/kvm_emulate.h>
>>>> +#include <asm/kvm_arm.h>
>>>> +#include <asm/kvm_mmu.h>
>>>> +
>>>> +#include "vgic.h"
>>>> +#include "vgic_mmio.h"
>>>> +
>>>> +struct vgic_register_region its_registers[] = {
>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>
>>> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
>> In practice I think the 32 upper bits are reserved.
> 
> Not quite. Only bits [63:38] are reserved. See 8.19.8 in the
> Architecture Specification (IHI 0069B).

Ah OK. Was using an older arch spec(PRD03-GENC-010745 22.0) Now switched
to the indicated one and indeed it's different !;-)

Eric
> 
> Thanks,
> 
> 	M.
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
  2016-04-07 15:09           ` Eric Auger
@ 2016-04-07 15:19             ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 15:19 UTC (permalink / raw)
  To: Eric Auger, Andre Przywara, Christoffer Dall
  Cc: kvmarm, linux-arm-kernel, kvm

On 07/04/16 16:09, Eric Auger wrote:
> On 04/07/2016 04:48 PM, Marc Zyngier wrote:
>> Hi Eric,
>>
>> On 07/04/16 15:08, Eric Auger wrote:
>>> Hi Marc,
>>> On 04/07/2016 04:04 PM, Marc Zyngier wrote:
>>>> On 26/03/16 02:14, Andre Przywara wrote:
>>>>> The ARM GICv3 ITS emulation code goes into a separate file, but
>>>>> needs to be connected to the GICv3 emulation, of which it is an
>>>>> option.
>>>>> Introduce the skeleton with function stubs to be filled later.
>>>>> Introduce the basic ITS data structure and initialize it, but don't
>>>>> return any success yet, as we are not yet ready for the show.
>>>>>
>>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>>> ---
>>>>>  arch/arm64/kvm/Makefile            |  1 +
>>>>>  include/kvm/vgic/vgic.h            |  6 +++
>>>>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>>>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>>>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>>>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>>>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>>>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>>>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>>>>
>>>>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>>>>> index 2f5d431..3bec10e 100644
>>>>> --- a/arch/arm64/kvm/Makefile
>>>>> +++ b/arch/arm64/kvm/Makefile
>>>>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>>>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>>>>  else
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>>>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>>>> index 8ea5dd7..c79bed5 100644
>>>>> --- a/include/kvm/vgic/vgic.h
>>>>> +++ b/include/kvm/vgic/vgic.h
>>>>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>>>>  	struct kvm_io_device dev;
>>>>>  };
>>>>>  
>>>>> +struct vgic_its {
>>>>> +	bool			enabled;
>>>>> +	spinlock_t		lock;
>>>>> +};
>>>>> +
>>>>>  struct vgic_dist {
>>>>>  	bool			in_kernel;
>>>>>  	bool			ready;
>>>>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>>>>  	u64			*pendbaser;
>>>>>  
>>>>>  	bool			lpis_enabled;
>>>>> +	struct vgic_its		its;
>>>>>  };
>>>>>  
>>>>>  struct vgic_v2_cpu_if {
>>>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>>>> index d5d798b..a813c3e 100644
>>>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>>>> @@ -177,6 +177,7 @@
>>>>>  #define GITS_CWRITER			0x0088
>>>>>  #define GITS_CREADR			0x0090
>>>>>  #define GITS_BASER			0x0100
>>>>> +#define GITS_IDREGS_BASE		0xffd0
>>>>>  #define GITS_PIDR2			GICR_PIDR2
>>>>>  
>>>>>  #define GITS_TRANSLATER			0x10040
>>>>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>>>>> new file mode 100644
>>>>> index 0000000..49dd5e4
>>>>> --- /dev/null
>>>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>>>> @@ -0,0 +1,84 @@
>>>>> +/*
>>>>> + * GICv3 ITS emulation
>>>>> + *
>>>>> + * Copyright (C) 2015 ARM Ltd.
>>>>> + * Author: Andre Przywara <andre.przywara@arm.com>
>>>>> + *
>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>> + * it under the terms of the GNU General Public License version 2 as
>>>>> + * published by the Free Software Foundation.
>>>>> + *
>>>>> + * This program is distributed in the hope that it will be useful,
>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>>> + * GNU General Public License for more details.
>>>>> + *
>>>>> + * You should have received a copy of the GNU General Public License
>>>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>>>> + */
>>>>> +
>>>>> +#include <linux/cpu.h>
>>>>> +#include <linux/kvm.h>
>>>>> +#include <linux/kvm_host.h>
>>>>> +#include <linux/interrupt.h>
>>>>> +
>>>>> +#include <linux/irqchip/arm-gic-v3.h>
>>>>> +
>>>>> +#include <asm/kvm_emulate.h>
>>>>> +#include <asm/kvm_arm.h>
>>>>> +#include <asm/kvm_mmu.h>
>>>>> +
>>>>> +#include "vgic.h"
>>>>> +#include "vgic_mmio.h"
>>>>> +
>>>>> +struct vgic_register_region its_registers[] = {
>>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>>
>>>> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
>>> In practice I think the 32 upper bits are reserved.
>>
>> Not quite. Only bits [63:38] are reserved. See 8.19.8 in the
>> Architecture Specification (IHI 0069B).
> 
> Ah OK. Was using an older arch spec(PRD03-GENC-010745 22.0) Now switched
> to the indicated one and indeed it's different !;-)

Ouch! Please burn that one, it is horribly outdated. The internal
revision of that document is now 35.1, but we're phasing it out in
favour of the public Architecture Specification.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions
@ 2016-04-07 15:19             ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-04-07 15:19 UTC (permalink / raw)
  To: linux-arm-kernel

On 07/04/16 16:09, Eric Auger wrote:
> On 04/07/2016 04:48 PM, Marc Zyngier wrote:
>> Hi Eric,
>>
>> On 07/04/16 15:08, Eric Auger wrote:
>>> Hi Marc,
>>> On 04/07/2016 04:04 PM, Marc Zyngier wrote:
>>>> On 26/03/16 02:14, Andre Przywara wrote:
>>>>> The ARM GICv3 ITS emulation code goes into a separate file, but
>>>>> needs to be connected to the GICv3 emulation, of which it is an
>>>>> option.
>>>>> Introduce the skeleton with function stubs to be filled later.
>>>>> Introduce the basic ITS data structure and initialize it, but don't
>>>>> return any success yet, as we are not yet ready for the show.
>>>>>
>>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>>> ---
>>>>>  arch/arm64/kvm/Makefile            |  1 +
>>>>>  include/kvm/vgic/vgic.h            |  6 +++
>>>>>  include/linux/irqchip/arm-gic-v3.h |  1 +
>>>>>  virt/kvm/arm/vgic/its-emul.c       | 84 ++++++++++++++++++++++++++++++++++++++
>>>>>  virt/kvm/arm/vgic/vgic.h           | 14 +++++++
>>>>>  virt/kvm/arm/vgic/vgic_init.c      |  6 +++
>>>>>  virt/kvm/arm/vgic/vgic_mmio.c      | 18 ++++++--
>>>>>  7 files changed, 126 insertions(+), 4 deletions(-)
>>>>>  create mode 100644 virt/kvm/arm/vgic/its-emul.c
>>>>>
>>>>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>>>>> index 2f5d431..3bec10e 100644
>>>>> --- a/arch/arm64/kvm/Makefile
>>>>> +++ b/arch/arm64/kvm/Makefile
>>>>> @@ -28,6 +28,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v2.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-v3.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_mmio.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic_kvm_device.o
>>>>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/its-emul.o
>>>>>  else
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic.o
>>>>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic-v2.o
>>>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>>>> index 8ea5dd7..c79bed5 100644
>>>>> --- a/include/kvm/vgic/vgic.h
>>>>> +++ b/include/kvm/vgic/vgic.h
>>>>> @@ -112,6 +112,11 @@ struct vgic_io_device {
>>>>>  	struct kvm_io_device dev;
>>>>>  };
>>>>>  
>>>>> +struct vgic_its {
>>>>> +	bool			enabled;
>>>>> +	spinlock_t		lock;
>>>>> +};
>>>>> +
>>>>>  struct vgic_dist {
>>>>>  	bool			in_kernel;
>>>>>  	bool			ready;
>>>>> @@ -153,6 +158,7 @@ struct vgic_dist {
>>>>>  	u64			*pendbaser;
>>>>>  
>>>>>  	bool			lpis_enabled;
>>>>> +	struct vgic_its		its;
>>>>>  };
>>>>>  
>>>>>  struct vgic_v2_cpu_if {
>>>>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>>>>> index d5d798b..a813c3e 100644
>>>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>>>> @@ -177,6 +177,7 @@
>>>>>  #define GITS_CWRITER			0x0088
>>>>>  #define GITS_CREADR			0x0090
>>>>>  #define GITS_BASER			0x0100
>>>>> +#define GITS_IDREGS_BASE		0xffd0
>>>>>  #define GITS_PIDR2			GICR_PIDR2
>>>>>  
>>>>>  #define GITS_TRANSLATER			0x10040
>>>>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>>>>> new file mode 100644
>>>>> index 0000000..49dd5e4
>>>>> --- /dev/null
>>>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>>>> @@ -0,0 +1,84 @@
>>>>> +/*
>>>>> + * GICv3 ITS emulation
>>>>> + *
>>>>> + * Copyright (C) 2015 ARM Ltd.
>>>>> + * Author: Andre Przywara <andre.przywara@arm.com>
>>>>> + *
>>>>> + * This program is free software; you can redistribute it and/or modify
>>>>> + * it under the terms of the GNU General Public License version 2 as
>>>>> + * published by the Free Software Foundation.
>>>>> + *
>>>>> + * This program is distributed in the hope that it will be useful,
>>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>>> + * GNU General Public License for more details.
>>>>> + *
>>>>> + * You should have received a copy of the GNU General Public License
>>>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>>>> + */
>>>>> +
>>>>> +#include <linux/cpu.h>
>>>>> +#include <linux/kvm.h>
>>>>> +#include <linux/kvm_host.h>
>>>>> +#include <linux/interrupt.h>
>>>>> +
>>>>> +#include <linux/irqchip/arm-gic-v3.h>
>>>>> +
>>>>> +#include <asm/kvm_emulate.h>
>>>>> +#include <asm/kvm_arm.h>
>>>>> +#include <asm/kvm_mmu.h>
>>>>> +
>>>>> +#include "vgic.h"
>>>>> +#include "vgic_mmio.h"
>>>>> +
>>>>> +struct vgic_register_region its_registers[] = {
>>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>>> +	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>>>>> +		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>>>>
>>>> GITS_TYPER is a 64bit register with plenty of things in the upper bits.
>>> In practice I think the 32 upper bits are reserved.
>>
>> Not quite. Only bits [63:38] are reserved. See 8.19.8 in the
>> Architecture Specification (IHI 0069B).
> 
> Ah OK. Was using an older arch spec(PRD03-GENC-010745 22.0) Now switched
> to the indicated one and indeed it's different !;-)

Ouch! Please burn that one, it is horribly outdated. The internal
revision of that document is now 35.1, but we're phasing it out in
favour of the public Architecture Specification.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
  2016-03-26  2:13   ` Andre Przywara
@ 2016-05-05 17:55     ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 17:55 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 3/25/16, 7:13 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>The ARM GICv3 ITS MSI controller requires a device ID to be able to
>assign the proper interrupt vector. On real hardware, this ID is
>sampled from the bus. To be able to emulate an ITS controller, extend
>the KVM MSI interface to let userspace provide such a device ID. For
>PCI devices, the device ID is simply the 16-bit bus-device-function
>triplet, which should be easily available to the userland tool.

Once VFIO is used it becomes impossible to guess the device ID.
Does the VFIO series take care of this? 
>
>Also there is a new KVM capability which advertises whether the
>current VM requires a device ID to be set along with the MSI data.
>This flag is still reported as not available everywhere, later we will
>enable it when ITS emulation is used.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>Reviewed-by: Eric Auger <eric.auger@linaro.org>
>---
> Documentation/virtual/kvm/api.txt | 12 ++++++++++--
> include/uapi/linux/kvm.h          |  5 ++++-
> 2 files changed, 14 insertions(+), 3 deletions(-)
>
>diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>index cb2ef0b..8f7351d 100644
>--- a/Documentation/virtual/kvm/api.txt
>+++ b/Documentation/virtual/kvm/api.txt
>@@ -2163,10 +2163,18 @@ struct kvm_msi {
> 	__u32 address_hi;
> 	__u32 data;
> 	__u32 flags;
>-	__u8  pad[16];
>+	__u32 devid;
>+	__u8  pad[12];
> };
> 
>-No flags are defined so far. The corresponding field must be 0.
>+flags: KVM_MSI_VALID_DEVID: devid contains a valid value
>+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
>+       for the device that wrote the MSI message.
>+       For PCI, this is usually a BFD identifier in the lower 16 bits.
>+
>+The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
>+the device ID. If this capability is not set, userland cannot rely on
>+the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
> 
> 
> 4.71 KVM_CREATE_PIT2
>diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>index 50f44a2..6a02871 100644
>--- a/include/uapi/linux/kvm.h
>+++ b/include/uapi/linux/kvm.h
>@@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
> #define KVM_CAP_S390_RI 124
> #define KVM_CAP_ARM_PMU_V3 125
> #define KVM_CAP_VCPU_ATTRIBUTES 126
>+#define KVM_CAP_MSI_DEVID 127
> 
> #ifdef KVM_CAP_IRQ_ROUTING
> 
>@@ -1010,12 +1011,14 @@ struct kvm_one_reg {
> 	__u64 addr;
> };
> 
>+#define KVM_MSI_VALID_DEVID	(1U << 0)
> struct kvm_msi {
> 	__u32 address_lo;
> 	__u32 address_hi;
> 	__u32 data;
> 	__u32 flags;
>-	__u8  pad[16];
>+	__u32 devid;
>+	__u8  pad[12];
> };
> 
> struct kvm_arm_device_addr {
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
@ 2016-05-05 17:55     ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 17:55 UTC (permalink / raw)
  To: linux-arm-kernel






On 3/25/16, 7:13 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>The ARM GICv3 ITS MSI controller requires a device ID to be able to
>assign the proper interrupt vector. On real hardware, this ID is
>sampled from the bus. To be able to emulate an ITS controller, extend
>the KVM MSI interface to let userspace provide such a device ID. For
>PCI devices, the device ID is simply the 16-bit bus-device-function
>triplet, which should be easily available to the userland tool.

Once VFIO is used it becomes impossible to guess the device ID.
Does the VFIO series take care of this? 
>
>Also there is a new KVM capability which advertises whether the
>current VM requires a device ID to be set along with the MSI data.
>This flag is still reported as not available everywhere, later we will
>enable it when ITS emulation is used.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>Reviewed-by: Eric Auger <eric.auger@linaro.org>
>---
> Documentation/virtual/kvm/api.txt | 12 ++++++++++--
> include/uapi/linux/kvm.h          |  5 ++++-
> 2 files changed, 14 insertions(+), 3 deletions(-)
>
>diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>index cb2ef0b..8f7351d 100644
>--- a/Documentation/virtual/kvm/api.txt
>+++ b/Documentation/virtual/kvm/api.txt
>@@ -2163,10 +2163,18 @@ struct kvm_msi {
> 	__u32 address_hi;
> 	__u32 data;
> 	__u32 flags;
>-	__u8  pad[16];
>+	__u32 devid;
>+	__u8  pad[12];
> };
> 
>-No flags are defined so far. The corresponding field must be 0.
>+flags: KVM_MSI_VALID_DEVID: devid contains a valid value
>+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
>+       for the device that wrote the MSI message.
>+       For PCI, this is usually a BFD identifier in the lower 16 bits.
>+
>+The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
>+the device ID. If this capability is not set, userland cannot rely on
>+the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
> 
> 
> 4.71 KVM_CREATE_PIT2
>diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>index 50f44a2..6a02871 100644
>--- a/include/uapi/linux/kvm.h
>+++ b/include/uapi/linux/kvm.h
>@@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
> #define KVM_CAP_S390_RI 124
> #define KVM_CAP_ARM_PMU_V3 125
> #define KVM_CAP_VCPU_ATTRIBUTES 126
>+#define KVM_CAP_MSI_DEVID 127
> 
> #ifdef KVM_CAP_IRQ_ROUTING
> 
>@@ -1010,12 +1011,14 @@ struct kvm_one_reg {
> 	__u64 addr;
> };
> 
>+#define KVM_MSI_VALID_DEVID	(1U << 0)
> struct kvm_msi {
> 	__u32 address_lo;
> 	__u32 address_hi;
> 	__u32 data;
> 	__u32 flags;
>-	__u8  pad[16];
>+	__u32 devid;
>+	__u8  pad[12];
> };
> 
> struct kvm_arm_device_addr {
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
  2016-03-26  2:14   ` Andre Przywara
@ 2016-05-05 18:00     ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:00 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>KVM capabilities can be a per-VM property, though ARM/ARM64 currently
>does not pass on the VM pointer to the architecture specific
>capability handlers.
>Add a "struct kvm*" parameter to those function to later allow proper
>per-VM capability reporting.

This looks good
Reviewed-by: Tirumalesh Chalamarla <tchalamarla@caviumnetworks.com> 

>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> arch/arm/include/asm/kvm_host.h   | 2 +-
> arch/arm/kvm/arm.c                | 2 +-
> arch/arm64/include/asm/kvm_host.h | 2 +-
> arch/arm64/kvm/reset.c            | 2 +-
> 4 files changed, 4 insertions(+), 4 deletions(-)
>
>diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
>index 3850701..494b004 100644
>--- a/arch/arm/include/asm/kvm_host.h
>+++ b/arch/arm/include/asm/kvm_host.h
>@@ -265,7 +265,7 @@ static inline void __cpu_init_stage2(void)
> 	kvm_call_hyp(__init_stage2_translation);
> }
> 
>-static inline int kvm_arch_dev_ioctl_check_extension(long ext)
>+static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
> {
> 	return 0;
> }
>diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
>index 5b9d9e3..e2e9be2 100644
>--- a/arch/arm/kvm/arm.c
>+++ b/arch/arm/kvm/arm.c
>@@ -205,7 +205,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> 		r = KVM_MAX_VCPUS;
> 		break;
> 	default:
>-		r = kvm_arch_dev_ioctl_check_extension(ext);
>+		r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
> 		break;
> 	}
> 	return r;
>diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>index 71fa6fe..2cdd7ae 100644
>--- a/arch/arm64/include/asm/kvm_host.h
>+++ b/arch/arm64/include/asm/kvm_host.h
>@@ -46,7 +46,7 @@
> 
> int __attribute_const__ kvm_target_cpu(void);
> int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
>-int kvm_arch_dev_ioctl_check_extension(long ext);
>+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
> 
> struct kvm_arch {
> 	/* The VMID generation used for the virt. memory system */
>diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
>index 9677bf0..2ba7c8d 100644
>--- a/arch/arm64/kvm/reset.c
>+++ b/arch/arm64/kvm/reset.c
>@@ -63,7 +63,7 @@ static bool cpu_has_32bit_el1(void)
>  * We currently assume that the number of HW registers is uniform
>  * across all CPUs (see cpuinfo_sanity_check).
>  */
>-int kvm_arch_dev_ioctl_check_extension(long ext)
>+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
> {
> 	int r;
> 
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
@ 2016-05-05 18:00     ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:00 UTC (permalink / raw)
  To: linux-arm-kernel






On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>KVM capabilities can be a per-VM property, though ARM/ARM64 currently
>does not pass on the VM pointer to the architecture specific
>capability handlers.
>Add a "struct kvm*" parameter to those function to later allow proper
>per-VM capability reporting.

This looks good
Reviewed-by: Tirumalesh Chalamarla <tchalamarla@caviumnetworks.com> 

>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> arch/arm/include/asm/kvm_host.h   | 2 +-
> arch/arm/kvm/arm.c                | 2 +-
> arch/arm64/include/asm/kvm_host.h | 2 +-
> arch/arm64/kvm/reset.c            | 2 +-
> 4 files changed, 4 insertions(+), 4 deletions(-)
>
>diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
>index 3850701..494b004 100644
>--- a/arch/arm/include/asm/kvm_host.h
>+++ b/arch/arm/include/asm/kvm_host.h
>@@ -265,7 +265,7 @@ static inline void __cpu_init_stage2(void)
> 	kvm_call_hyp(__init_stage2_translation);
> }
> 
>-static inline int kvm_arch_dev_ioctl_check_extension(long ext)
>+static inline int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
> {
> 	return 0;
> }
>diff --git a/arch/arm/kvm/arm.c b/arch/arm/kvm/arm.c
>index 5b9d9e3..e2e9be2 100644
>--- a/arch/arm/kvm/arm.c
>+++ b/arch/arm/kvm/arm.c
>@@ -205,7 +205,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> 		r = KVM_MAX_VCPUS;
> 		break;
> 	default:
>-		r = kvm_arch_dev_ioctl_check_extension(ext);
>+		r = kvm_arch_dev_ioctl_check_extension(kvm, ext);
> 		break;
> 	}
> 	return r;
>diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>index 71fa6fe..2cdd7ae 100644
>--- a/arch/arm64/include/asm/kvm_host.h
>+++ b/arch/arm64/include/asm/kvm_host.h
>@@ -46,7 +46,7 @@
> 
> int __attribute_const__ kvm_target_cpu(void);
> int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
>-int kvm_arch_dev_ioctl_check_extension(long ext);
>+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext);
> 
> struct kvm_arch {
> 	/* The VMID generation used for the virt. memory system */
>diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
>index 9677bf0..2ba7c8d 100644
>--- a/arch/arm64/kvm/reset.c
>+++ b/arch/arm64/kvm/reset.c
>@@ -63,7 +63,7 @@ static bool cpu_has_32bit_el1(void)
>  * We currently assume that the number of HW registers is uniform
>  * across all CPUs (see cpuinfo_sanity_check).
>  */
>-int kvm_arch_dev_ioctl_check_extension(long ext)
>+int kvm_arch_dev_ioctl_check_extension(struct kvm *kvm, long ext)
> {
> 	int r;
> 
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
  2016-04-07 13:58       ` Marc Zyngier
@ 2016-05-05 18:06         ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:06 UTC (permalink / raw)
  To: Marc Zyngier, Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 4/7/16, 6:58 AM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Marc Zyngier" <kvmarm-bounces@lists.cs.columbia.edu on behalf of marc.zyngier@arm.com> wrote:

>On 07/04/16 14:54, Marc Zyngier wrote:
>> On 26/03/16 02:14, Andre Przywara wrote:
>>> In the GICv3 redistributor there are the PENDBASER and PROPBASER
>>> registers which we did not emulate so far, as they only make sense
>>> when having an ITS. In preparation for that emulate those MMIO
>>> accesses by storing the 64-bit data written into it into a variable
>>> which we later read in the ITS emulation.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>>  include/kvm/vgic/vgic.h       |  8 ++++++++
>>>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>> index 11344e6..8ea5dd7 100644
>>> --- a/include/kvm/vgic/vgic.h
>>> +++ b/include/kvm/vgic/vgic.h
>>> @@ -145,6 +145,14 @@ struct vgic_dist {
>>>  
>>>  	struct vgic_io_device	*dist_iodevs;
>>>  	struct vgic_io_device	*redist_iodevs;
>>> +
>>> +	/* Address of LPI configuration table shared by all redistributors */
>>> +	u64			propbaser;
>>> +
>>> +	/* Addresses of LPI pending tables per redistributor */
>>> +	u64			*pendbaser;
>> 
>> All these should probably either be phys_addr_t or gpa_t.
>
>Also: these registers are per redistributor. They really should appear
>as such, and not be shoved in vgic_dist.
>
>As for GICR_PROPBASER being shared: this is an implementation choice.
>All the architecture mandates is that the same value is programmed in
>all the redistributors.
>
>Yes, you can share the storage, but I'd really want to see a field per
>redist.
+1


>
>Thanks,
>
>	M.
>-- 
>Jazz is not dead. It just smells funny...
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers
@ 2016-05-05 18:06         ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:06 UTC (permalink / raw)
  To: linux-arm-kernel






On 4/7/16, 6:58 AM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Marc Zyngier" <kvmarm-bounces at lists.cs.columbia.edu on behalf of marc.zyngier@arm.com> wrote:

>On 07/04/16 14:54, Marc Zyngier wrote:
>> On 26/03/16 02:14, Andre Przywara wrote:
>>> In the GICv3 redistributor there are the PENDBASER and PROPBASER
>>> registers which we did not emulate so far, as they only make sense
>>> when having an ITS. In preparation for that emulate those MMIO
>>> accesses by storing the 64-bit data written into it into a variable
>>> which we later read in the ITS emulation.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>>  include/kvm/vgic/vgic.h       |  8 ++++++++
>>>  virt/kvm/arm/vgic/vgic_mmio.c | 26 ++++++++++++++++++++++----
>>>  2 files changed, 30 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>> index 11344e6..8ea5dd7 100644
>>> --- a/include/kvm/vgic/vgic.h
>>> +++ b/include/kvm/vgic/vgic.h
>>> @@ -145,6 +145,14 @@ struct vgic_dist {
>>>  
>>>  	struct vgic_io_device	*dist_iodevs;
>>>  	struct vgic_io_device	*redist_iodevs;
>>> +
>>> +	/* Address of LPI configuration table shared by all redistributors */
>>> +	u64			propbaser;
>>> +
>>> +	/* Addresses of LPI pending tables per redistributor */
>>> +	u64			*pendbaser;
>> 
>> All these should probably either be phys_addr_t or gpa_t.
>
>Also: these registers are per redistributor. They really should appear
>as such, and not be shoved in vgic_dist.
>
>As for GICR_PROPBASER being shared: this is an implementation choice.
>All the architecture mandates is that the same value is programmed in
>all the redistributors.
>
>Yes, you can share the storage, but I'd really want to see a field per
>redist.
+1


>
>Thanks,
>
>	M.
>-- 
>Jazz is not dead. It just smells funny...
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-03-26  2:14   ` Andre Przywara
@ 2016-05-05 18:08     ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:08 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>The ARM GICv3 ITS controller requires a separate register frame to
>cover ITS specific registers. Add a new VGIC address type and store
>the address in a field in the vgic_dist structure.
>Provide a function to check whether userland has provided the address,
>so ITS functionality can be guarded by that check.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
> include/kvm/vgic/vgic.h                        |  5 +++++
> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
> 6 files changed, 34 insertions(+)
>
>diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>index 59541d4..087e2d9 100644
>--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>@@ -39,6 +39,15 @@ Groups:
>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>       This address needs to be 64K aligned.
> 
>+    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>+      Base address in the guest physical address space of the GICv3 ITS
>+      control register frame. The ITS allows MSI(-X) interrupts to be
>+      injected into guests. This extension is optional, if the kernel
>+      does not support the ITS, the call returns -ENODEV.
>+      This memory is solely for the guest to access the ITS control
>+      registers and does not cover the ITS translation register.
>+      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>+      This address needs to be 64K aligned and the region covers 64 KByte.
> 
>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>   Attributes:
>diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>index f209ea1..c2b257d 100644
>--- a/arch/arm64/include/uapi/asm/kvm.h
>+++ b/arch/arm64/include/uapi/asm/kvm.h
>@@ -87,9 +87,11 @@ struct kvm_regs {
> /* Supported VGICv3 address types  */
> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>+#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
> 
> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>+#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
> 
> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>index 7c1d145..11344e6 100644
>--- a/include/kvm/vgic/vgic.h
>+++ b/include/kvm/vgic/vgic.h
>@@ -135,6 +135,9 @@ struct vgic_dist {
> 		gpa_t			vgic_redist_base;
> 	};
> 
>+	/* The base address of the ITS control register frame */
>+	gpa_t			vgic_its_base;
>+
> 	/* distributor enabled */
> 	u32			enabled;
> 
>@@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
> 	return kvm_vgic_global_state.max_gic_vcpus;
> }
> 
>+bool vgic_has_its(struct kvm *kvm);
>+
> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>index 65395af..1de2478 100644
>--- a/virt/kvm/arm/vgic/vgic.c
>+++ b/virt/kvm/arm/vgic/vgic.c
>@@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
> 
> 	return map_is_active;
> }
>+
>+bool vgic_has_its(struct kvm *kvm)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+
>+	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>+		return false;
>+
>+	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>+}
>diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>index ac655b5..2301e03 100644
>--- a/virt/kvm/arm/vgic/vgic_init.c
>+++ b/virt/kvm/arm/vgic/vgic_init.c
>@@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>+	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
> 
> out_unlock:
> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>index 7f78a16..3ec2ac3 100644
>--- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>+++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>@@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
> 		alignment = SZ_64K;
> 		break;
>+	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>+		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>+		addr_ptr = &vgic->vgic_its_base;
>+		block_size = KVM_VGIC_V3_ITS_SIZE;
>+		alignment = SZ_64K;
>+		break;

Should there be a check based on number of vcpus? 
> #endif
> 	default:
> 		r = -ENODEV;
>@@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
> 		switch (attr->attr) {
> 		case KVM_VGIC_V3_ADDR_TYPE_DIST:
> 		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
>+		case KVM_VGIC_V3_ADDR_TYPE_ITS:
> 			return 0;
> 		}
> 		break;
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-05-05 18:08     ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:08 UTC (permalink / raw)
  To: linux-arm-kernel






On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>The ARM GICv3 ITS controller requires a separate register frame to
>cover ITS specific registers. Add a new VGIC address type and store
>the address in a field in the vgic_dist structure.
>Provide a function to check whether userland has provided the address,
>so ITS functionality can be guarded by that check.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
> include/kvm/vgic/vgic.h                        |  5 +++++
> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
> 6 files changed, 34 insertions(+)
>
>diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>index 59541d4..087e2d9 100644
>--- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>+++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>@@ -39,6 +39,15 @@ Groups:
>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>       This address needs to be 64K aligned.
> 
>+    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>+      Base address in the guest physical address space of the GICv3 ITS
>+      control register frame. The ITS allows MSI(-X) interrupts to be
>+      injected into guests. This extension is optional, if the kernel
>+      does not support the ITS, the call returns -ENODEV.
>+      This memory is solely for the guest to access the ITS control
>+      registers and does not cover the ITS translation register.
>+      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>+      This address needs to be 64K aligned and the region covers 64 KByte.
> 
>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>   Attributes:
>diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>index f209ea1..c2b257d 100644
>--- a/arch/arm64/include/uapi/asm/kvm.h
>+++ b/arch/arm64/include/uapi/asm/kvm.h
>@@ -87,9 +87,11 @@ struct kvm_regs {
> /* Supported VGICv3 address types  */
> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>+#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
> 
> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>+#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
> 
> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>index 7c1d145..11344e6 100644
>--- a/include/kvm/vgic/vgic.h
>+++ b/include/kvm/vgic/vgic.h
>@@ -135,6 +135,9 @@ struct vgic_dist {
> 		gpa_t			vgic_redist_base;
> 	};
> 
>+	/* The base address of the ITS control register frame */
>+	gpa_t			vgic_its_base;
>+
> 	/* distributor enabled */
> 	u32			enabled;
> 
>@@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
> 	return kvm_vgic_global_state.max_gic_vcpus;
> }
> 
>+bool vgic_has_its(struct kvm *kvm);
>+
> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>index 65395af..1de2478 100644
>--- a/virt/kvm/arm/vgic/vgic.c
>+++ b/virt/kvm/arm/vgic/vgic.c
>@@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
> 
> 	return map_is_active;
> }
>+
>+bool vgic_has_its(struct kvm *kvm)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+
>+	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>+		return false;
>+
>+	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>+}
>diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>index ac655b5..2301e03 100644
>--- a/virt/kvm/arm/vgic/vgic_init.c
>+++ b/virt/kvm/arm/vgic/vgic_init.c
>@@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>+	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
> 
> out_unlock:
> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>index 7f78a16..3ec2ac3 100644
>--- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>+++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>@@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
> 		alignment = SZ_64K;
> 		break;
>+	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>+		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>+		addr_ptr = &vgic->vgic_its_base;
>+		block_size = KVM_VGIC_V3_ITS_SIZE;
>+		alignment = SZ_64K;
>+		break;

Should there be a check based on number of vcpus? 
> #endif
> 	default:
> 		r = -ENODEV;
>@@ -495,6 +501,7 @@ static int vgic_v3_has_attr(struct kvm_device *dev,
> 		switch (attr->attr) {
> 		case KVM_VGIC_V3_ADDR_TYPE_DIST:
> 		case KVM_VGIC_V3_ADDR_TYPE_REDIST:
>+		case KVM_VGIC_V3_ADDR_TYPE_ITS:
> 			return 0;
> 		}
> 		break;
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-05-05 18:51     ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:51 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>Add emulation for some basic MMIO registers used in the ITS emulation.
>This includes:
>- GITS_{CTLR,TYPER,IIDR}
>- ID registers
>- GITS_{CBASER,CREADR,CWRITER}
>  those implement the ITS command buffer handling
>
>Most of the handlers are pretty straight forward, but CWRITER goes
>some extra miles to allow fine grained locking. The idea here
>is to let only the first instance iterate through the command ring
>buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
>by that first instance and handled as well. The ITS lock is thus only
>hold for very small periods of time and is dropped before the actual
>command handler is called.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> include/kvm/vgic/vgic.h            |   3 +
> include/linux/irqchip/arm-gic-v3.h |   8 ++
> virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
> virt/kvm/arm/vgic/vgic.h           |   6 +
> virt/kvm/arm/vgic/vgic_init.c      |   2 +
> 5 files changed, 284 insertions(+), 7 deletions(-)
>
>diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>index c79bed5..bafea11 100644
>--- a/include/kvm/vgic/vgic.h
>+++ b/include/kvm/vgic/vgic.h
>@@ -115,6 +115,9 @@ struct vgic_io_device {
> struct vgic_its {
> 	bool			enabled;
> 	spinlock_t		lock;
>+	u64			cbaser;
>+	int			creadr;
>+	int			cwriter;
> };
> 
> struct vgic_dist {
>diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>index a813c3e..7011b98 100644
>--- a/include/linux/irqchip/arm-gic-v3.h
>+++ b/include/linux/irqchip/arm-gic-v3.h
>@@ -179,15 +179,23 @@
> #define GITS_BASER			0x0100
> #define GITS_IDREGS_BASE		0xffd0
> #define GITS_PIDR2			GICR_PIDR2
>+#define GITS_PIDR4			0xffd0
>+#define GITS_CIDR0			0xfff0
>+#define GITS_CIDR1			0xfff4
>+#define GITS_CIDR2			0xfff8
>+#define GITS_CIDR3			0xfffc
> 
> #define GITS_TRANSLATER			0x10040
> 
> #define GITS_CTLR_ENABLE		(1U << 0)
> #define GITS_CTLR_QUIESCENT		(1U << 31)
> 
>+#define GITS_TYPER_PLPIS		(1UL << 0)
>+#define GITS_TYPER_IDBITS_SHIFT		8
> #define GITS_TYPER_DEVBITS_SHIFT	13
> #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
> #define GITS_TYPER_PTA			(1UL << 19)
>+#define GITS_TYPER_HWCOLLCNT_SHIFT	24
> 
> #define GITS_CBASER_VALID		(1UL << 63)
> #define GITS_CBASER_nCnB		(0UL << 59)
>diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>index 49dd5e4..de8d360 100644
>--- a/virt/kvm/arm/vgic/its-emul.c
>+++ b/virt/kvm/arm/vgic/its-emul.c
>@@ -31,23 +31,263 @@
> #include "vgic.h"
> #include "vgic_mmio.h"
> 
>+#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>+
>+static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>+				   struct kvm_io_device *this,
>+				   gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	u32 reg;
>+
>+	reg = GITS_CTLR_QUIESCENT;
>+	if (its->enabled)
>+		reg |= GITS_CTLR_ENABLE;
>+
>+	write_mask32(reg, addr & 3, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
>+				    struct kvm_io_device *this,
>+				    gpa_t addr, int len, const void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	struct vgic_io_device *iodev = container_of(this,
>+						    struct vgic_io_device, dev);
>+
>+        if (addr - iodev->base_addr == 0)
>+		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>+				    struct kvm_io_device *this,
>+				    gpa_t addr, int len, void *val)
>+{
>+	u64 reg = GITS_TYPER_PLPIS;
>+
>+	/*
>+	 * We use linear CPU numbers for redistributor addressing,
>+	 * so GITS_TYPER.PTA is 0.
>+	 * To avoid memory waste on the guest side, we keep the
>+	 * number of IDBits and DevBits low for the time being.
>+	 * This could later be made configurable by userland.
>+	 * Since we have all collections in linked list, we claim
>+	 * that we can hold all of the collection tables in our
>+	 * own memory and that the ITT entry size is 1 byte (the
>+	 * smallest possible one).
>+	 */


How are you planning to handle device ids with VFIO?
Will there be a mapping between virtual and physical?
Or will it entirely based on virtual and not planning
to send the commands to corresponding physical ITS.
May be no need. 

>+	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>+	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>+	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>+
>+	write_mask64(reg, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
>+				   struct kvm_io_device *this,
>+				   gpa_t addr, int len, void *val)
>+{
>+	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
>+
Hmm, no other implementor planning to support system emulation. 
>+	write_mask32(reg, addr & 3, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>+				     struct kvm_io_device *this,
>+				     gpa_t addr, int len, void *val)
>+{
>+	struct vgic_io_device *iodev = container_of(this,
>+						    struct vgic_io_device, dev);
>+	u32 reg = 0;
>+	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>+
>+	switch (idreg) {
>+	case GITS_PIDR2:
>+		reg = GIC_PIDR2_ARCH_GICv3;
>+		break;
>+	case GITS_PIDR4:
>+		/* This is a 64K software visible page */
>+		reg = 0x40;
>+		break;
>+	/* Those are the ID registers for (any) GIC. */
>+	case GITS_CIDR0:
>+		reg = 0x0d;
>+		break;
>+	case GITS_CIDR1:
>+		reg = 0xf0;
>+		break;
>+	case GITS_CIDR2:
>+		reg = 0x05;
>+		break;
>+	case GITS_CIDR3:
>+		reg = 0xb1;
>+		break;
>+	}
>+
>+	write_mask32(reg, addr & 3, len, val);
>+
>+	return 0;
>+}
>+
>+/*
>+ * This function is called with both the ITS and the distributor lock dropped,
>+ * so the actual command handlers must take the respective locks when needed.
>+ */
>+static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
>+{
>+	return -ENODEV;
>+}
>+
>+static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>+				    struct kvm_io_device *this,
>+				    gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+
>+	write_mask64(its->cbaser, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
>+				      struct kvm_io_device *this,
>+				      gpa_t addr, int len, const void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+
>+	if (its->enabled)
>+		return 0;
>+
>+	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
>+	its->creadr = 0;
>+
>+	return 0;
>+}
>+
>+static int its_cmd_buffer_size(struct kvm *kvm)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+
>+	return ((its->cbaser & 0xff) + 1) << 12;
>+}
>+
>+static gpa_t its_cmd_buffer_base(struct kvm *kvm)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+
>+	return BASER_BASE_ADDRESS(its->cbaser);
>+}
>+
>+/*
>+ * By writing to CWRITER the guest announces new commands to be processed.
>+ * Since we cannot read from guest memory inside the ITS spinlock, we
>+ * iterate over the command buffer (with the lock dropped) until the read
>+ * pointer matches the write pointer. Other VCPUs writing this register in the
>+ * meantime will just update the write pointer, leaving the command
>+ * processing to the first instance of the function.
>+ */
>+static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>+				       struct kvm_io_device *this,
>+				       gpa_t addr, int len, const void *val)
>+{
>+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>+	struct vgic_its *its = &dist->its;
>+	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>+	u64 cmd_buf[4];
>+	u32 reg;
>+	bool finished;
>+
>+	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>+	reg &= 0xfffe0;
>+	if (reg > its_cmd_buffer_size(vcpu->kvm))
>+		return 0;
>+
>+	spin_lock(&its->lock);
>+
>+	/*
>+	 * If there is still another VCPU handling commands, let this
>+	 * one pick up the new CWRITER and process "our" new commands as well.
>+	 */
>+	finished = (its->cwriter != its->creadr);
>+	its->cwriter = reg;
>+
>+	spin_unlock(&its->lock);
>+
>+	while (!finished) {
>+		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
>+					 cmd_buf, 32);
>+		if (ret) {
>+			/*
>+			 * Gah, we are screwed. Reset CWRITER to that command
>+			 * that we have finished processing and return.
>+			 */
>+			spin_lock(&its->lock);
>+			its->cwriter = its->creadr;

Is this correct?
>+			spin_unlock(&its->lock);
>+			break;
>+		}
>+		vits_handle_command(vcpu, cmd_buf);
>+
>+		spin_lock(&its->lock);
>+		its->creadr += 32;
>+		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
>+			its->creadr = 0;
>+		finished = (its->creadr == its->cwriter);
>+		spin_unlock(&its->lock);
>+	}
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
>+				      struct kvm_io_device *this,
>+				      gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	u64 reg = its->cwriter & 0xfffe0;
>+
>+	write_mask64(reg, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
>+				     struct kvm_io_device *this,
>+				     gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	u64 reg = its->creadr & 0xfffe0;
>+
>+	write_mask64(reg, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
> struct vgic_register_region its_registers[] = {
> 	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>+		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
> 	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>+		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
> 	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>+		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
> 	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>+		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
> 	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>+		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
> 	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>+		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
> 	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
> 		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
> 	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
>+		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
> };
> 
> /* This is called on setting the LPI enable bit in the redistributor. */
>@@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
> {
> 	struct vgic_dist *dist = &kvm->arch.vgic;
> 	struct vgic_its *its = &dist->its;
>+	int nr_vcpus = atomic_read(&kvm->online_vcpus);
> 	struct vgic_io_device *regions;
> 	int ret, i;
> 
>+	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
>+	if (!dist->pendbaser)
>+		return -ENOMEM;
>+
> 	spin_lock_init(&its->lock);
> 
> 	regions = kmalloc_array(ARRAY_SIZE(its_registers),
>@@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
> 
> 	return -ENXIO;
> }
>+
>+void vits_destroy(struct kvm *kvm)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	struct vgic_its *its = &dist->its;
>+
>+	if (!vgic_has_its(kvm))
>+		return;
>+
>+	kfree(dist->pendbaser);
>+
>+	its->enabled = false;
>+}
>diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>index 4e7dcb8..08f97d1 100644
>--- a/virt/kvm/arm/vgic/vgic.h
>+++ b/virt/kvm/arm/vgic/vgic.h
>@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> 
> int vits_init(struct kvm *kvm);
> void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>+void vits_destroy(struct kvm *kvm);
> #else
> static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
> 					       u64 mpidr)
>@@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> {
> 	return;
> }
>+
>+static inline void vits_destroy(struct kvm *kvm)
>+{
>+	return;
>+}
> #endif
> 
> void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>index dcfb93d..e4459e3 100644
>--- a/virt/kvm/arm/vgic/vgic_init.c
>+++ b/virt/kvm/arm/vgic/vgic_init.c
>@@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
> 
> 	kvm_vgic_dist_destroy(kvm);
> 
>+	vits_destroy(kvm);
>+
> 	kvm_for_each_vcpu(i, vcpu, kvm)
> 		kvm_vgic_vcpu_destroy(vcpu);
> }
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-05-05 18:51     ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:51 UTC (permalink / raw)
  To: linux-arm-kernel






On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>Add emulation for some basic MMIO registers used in the ITS emulation.
>This includes:
>- GITS_{CTLR,TYPER,IIDR}
>- ID registers
>- GITS_{CBASER,CREADR,CWRITER}
>  those implement the ITS command buffer handling
>
>Most of the handlers are pretty straight forward, but CWRITER goes
>some extra miles to allow fine grained locking. The idea here
>is to let only the first instance iterate through the command ring
>buffer, CWRITER accesses on other VCPUs meanwhile will be picked up
>by that first instance and handled as well. The ITS lock is thus only
>hold for very small periods of time and is dropped before the actual
>command handler is called.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> include/kvm/vgic/vgic.h            |   3 +
> include/linux/irqchip/arm-gic-v3.h |   8 ++
> virt/kvm/arm/vgic/its-emul.c       | 272 ++++++++++++++++++++++++++++++++++++-
> virt/kvm/arm/vgic/vgic.h           |   6 +
> virt/kvm/arm/vgic/vgic_init.c      |   2 +
> 5 files changed, 284 insertions(+), 7 deletions(-)
>
>diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>index c79bed5..bafea11 100644
>--- a/include/kvm/vgic/vgic.h
>+++ b/include/kvm/vgic/vgic.h
>@@ -115,6 +115,9 @@ struct vgic_io_device {
> struct vgic_its {
> 	bool			enabled;
> 	spinlock_t		lock;
>+	u64			cbaser;
>+	int			creadr;
>+	int			cwriter;
> };
> 
> struct vgic_dist {
>diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>index a813c3e..7011b98 100644
>--- a/include/linux/irqchip/arm-gic-v3.h
>+++ b/include/linux/irqchip/arm-gic-v3.h
>@@ -179,15 +179,23 @@
> #define GITS_BASER			0x0100
> #define GITS_IDREGS_BASE		0xffd0
> #define GITS_PIDR2			GICR_PIDR2
>+#define GITS_PIDR4			0xffd0
>+#define GITS_CIDR0			0xfff0
>+#define GITS_CIDR1			0xfff4
>+#define GITS_CIDR2			0xfff8
>+#define GITS_CIDR3			0xfffc
> 
> #define GITS_TRANSLATER			0x10040
> 
> #define GITS_CTLR_ENABLE		(1U << 0)
> #define GITS_CTLR_QUIESCENT		(1U << 31)
> 
>+#define GITS_TYPER_PLPIS		(1UL << 0)
>+#define GITS_TYPER_IDBITS_SHIFT		8
> #define GITS_TYPER_DEVBITS_SHIFT	13
> #define GITS_TYPER_DEVBITS(r)		((((r) >> GITS_TYPER_DEVBITS_SHIFT) & 0x1f) + 1)
> #define GITS_TYPER_PTA			(1UL << 19)
>+#define GITS_TYPER_HWCOLLCNT_SHIFT	24
> 
> #define GITS_CBASER_VALID		(1UL << 63)
> #define GITS_CBASER_nCnB		(0UL << 59)
>diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>index 49dd5e4..de8d360 100644
>--- a/virt/kvm/arm/vgic/its-emul.c
>+++ b/virt/kvm/arm/vgic/its-emul.c
>@@ -31,23 +31,263 @@
> #include "vgic.h"
> #include "vgic_mmio.h"
> 
>+#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>+
>+static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>+				   struct kvm_io_device *this,
>+				   gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	u32 reg;
>+
>+	reg = GITS_CTLR_QUIESCENT;
>+	if (its->enabled)
>+		reg |= GITS_CTLR_ENABLE;
>+
>+	write_mask32(reg, addr & 3, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
>+				    struct kvm_io_device *this,
>+				    gpa_t addr, int len, const void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	struct vgic_io_device *iodev = container_of(this,
>+						    struct vgic_io_device, dev);
>+
>+        if (addr - iodev->base_addr == 0)
>+		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>+				    struct kvm_io_device *this,
>+				    gpa_t addr, int len, void *val)
>+{
>+	u64 reg = GITS_TYPER_PLPIS;
>+
>+	/*
>+	 * We use linear CPU numbers for redistributor addressing,
>+	 * so GITS_TYPER.PTA is 0.
>+	 * To avoid memory waste on the guest side, we keep the
>+	 * number of IDBits and DevBits low for the time being.
>+	 * This could later be made configurable by userland.
>+	 * Since we have all collections in linked list, we claim
>+	 * that we can hold all of the collection tables in our
>+	 * own memory and that the ITT entry size is 1 byte (the
>+	 * smallest possible one).
>+	 */


How are you planning to handle device ids with VFIO?
Will there be a mapping between virtual and physical?
Or will it entirely based on virtual and not planning
to send the commands to corresponding physical ITS.
May be no need. 

>+	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>+	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>+	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>+
>+	write_mask64(reg, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
>+				   struct kvm_io_device *this,
>+				   gpa_t addr, int len, void *val)
>+{
>+	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
>+
Hmm, no other implementor planning to support system emulation. 
>+	write_mask32(reg, addr & 3, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>+				     struct kvm_io_device *this,
>+				     gpa_t addr, int len, void *val)
>+{
>+	struct vgic_io_device *iodev = container_of(this,
>+						    struct vgic_io_device, dev);
>+	u32 reg = 0;
>+	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>+
>+	switch (idreg) {
>+	case GITS_PIDR2:
>+		reg = GIC_PIDR2_ARCH_GICv3;
>+		break;
>+	case GITS_PIDR4:
>+		/* This is a 64K software visible page */
>+		reg = 0x40;
>+		break;
>+	/* Those are the ID registers for (any) GIC. */
>+	case GITS_CIDR0:
>+		reg = 0x0d;
>+		break;
>+	case GITS_CIDR1:
>+		reg = 0xf0;
>+		break;
>+	case GITS_CIDR2:
>+		reg = 0x05;
>+		break;
>+	case GITS_CIDR3:
>+		reg = 0xb1;
>+		break;
>+	}
>+
>+	write_mask32(reg, addr & 3, len, val);
>+
>+	return 0;
>+}
>+
>+/*
>+ * This function is called with both the ITS and the distributor lock dropped,
>+ * so the actual command handlers must take the respective locks when needed.
>+ */
>+static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
>+{
>+	return -ENODEV;
>+}
>+
>+static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>+				    struct kvm_io_device *this,
>+				    gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+
>+	write_mask64(its->cbaser, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
>+				      struct kvm_io_device *this,
>+				      gpa_t addr, int len, const void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+
>+	if (its->enabled)
>+		return 0;
>+
>+	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
>+	its->creadr = 0;
>+
>+	return 0;
>+}
>+
>+static int its_cmd_buffer_size(struct kvm *kvm)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+
>+	return ((its->cbaser & 0xff) + 1) << 12;
>+}
>+
>+static gpa_t its_cmd_buffer_base(struct kvm *kvm)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+
>+	return BASER_BASE_ADDRESS(its->cbaser);
>+}
>+
>+/*
>+ * By writing to CWRITER the guest announces new commands to be processed.
>+ * Since we cannot read from guest memory inside the ITS spinlock, we
>+ * iterate over the command buffer (with the lock dropped) until the read
>+ * pointer matches the write pointer. Other VCPUs writing this register in the
>+ * meantime will just update the write pointer, leaving the command
>+ * processing to the first instance of the function.
>+ */
>+static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>+				       struct kvm_io_device *this,
>+				       gpa_t addr, int len, const void *val)
>+{
>+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>+	struct vgic_its *its = &dist->its;
>+	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>+	u64 cmd_buf[4];
>+	u32 reg;
>+	bool finished;
>+
>+	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>+	reg &= 0xfffe0;
>+	if (reg > its_cmd_buffer_size(vcpu->kvm))
>+		return 0;
>+
>+	spin_lock(&its->lock);
>+
>+	/*
>+	 * If there is still another VCPU handling commands, let this
>+	 * one pick up the new CWRITER and process "our" new commands as well.
>+	 */
>+	finished = (its->cwriter != its->creadr);
>+	its->cwriter = reg;
>+
>+	spin_unlock(&its->lock);
>+
>+	while (!finished) {
>+		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
>+					 cmd_buf, 32);
>+		if (ret) {
>+			/*
>+			 * Gah, we are screwed. Reset CWRITER to that command
>+			 * that we have finished processing and return.
>+			 */
>+			spin_lock(&its->lock);
>+			its->cwriter = its->creadr;

Is this correct?
>+			spin_unlock(&its->lock);
>+			break;
>+		}
>+		vits_handle_command(vcpu, cmd_buf);
>+
>+		spin_lock(&its->lock);
>+		its->creadr += 32;
>+		if (its->creadr == its_cmd_buffer_size(vcpu->kvm))
>+			its->creadr = 0;
>+		finished = (its->creadr == its->cwriter);
>+		spin_unlock(&its->lock);
>+	}
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_cwriter(struct kvm_vcpu *vcpu,
>+				      struct kvm_io_device *this,
>+				      gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	u64 reg = its->cwriter & 0xfffe0;
>+
>+	write_mask64(reg, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
>+static int vgic_mmio_read_its_creadr(struct kvm_vcpu *vcpu,
>+				     struct kvm_io_device *this,
>+				     gpa_t addr, int len, void *val)
>+{
>+	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>+	u64 reg = its->creadr & 0xfffe0;
>+
>+	write_mask64(reg, addr & 7, len, val);
>+
>+	return 0;
>+}
>+
> struct vgic_register_region its_registers[] = {
> 	REGISTER_DESC_WITH_LENGTH(GITS_CTLR,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>+		vgic_mmio_read_its_ctlr, vgic_mmio_write_its_ctlr, 4),
> 	REGISTER_DESC_WITH_LENGTH(GITS_IIDR,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>+		vgic_mmio_read_its_iidr, vgic_mmio_write_wi, 4),
> 	REGISTER_DESC_WITH_LENGTH(GITS_TYPER,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 4),
>+		vgic_mmio_read_its_typer, vgic_mmio_write_wi, 4),
> 	REGISTER_DESC_WITH_LENGTH(GITS_CBASER,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>+		vgic_mmio_read_its_cbaser, vgic_mmio_write_its_cbaser, 8),
> 	REGISTER_DESC_WITH_LENGTH(GITS_CWRITER,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>+		vgic_mmio_read_its_cwriter, vgic_mmio_write_its_cwriter, 8),
> 	REGISTER_DESC_WITH_LENGTH(GITS_CREADR,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 8),
>+		vgic_mmio_read_its_creadr, vgic_mmio_write_wi, 8),
> 	REGISTER_DESC_WITH_LENGTH(GITS_BASER,
> 		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x40),
> 	REGISTER_DESC_WITH_LENGTH(GITS_IDREGS_BASE,
>-		vgic_mmio_read_raz, vgic_mmio_write_wi, 0x30),
>+		vgic_mmio_read_its_idregs, vgic_mmio_write_wi, 0x30),
> };
> 
> /* This is called on setting the LPI enable bit in the redistributor. */
>@@ -59,9 +299,14 @@ int vits_init(struct kvm *kvm)
> {
> 	struct vgic_dist *dist = &kvm->arch.vgic;
> 	struct vgic_its *its = &dist->its;
>+	int nr_vcpus = atomic_read(&kvm->online_vcpus);
> 	struct vgic_io_device *regions;
> 	int ret, i;
> 
>+	dist->pendbaser = kcalloc(nr_vcpus, sizeof(u64), GFP_KERNEL);
>+	if (!dist->pendbaser)
>+		return -ENOMEM;
>+
> 	spin_lock_init(&its->lock);
> 
> 	regions = kmalloc_array(ARRAY_SIZE(its_registers),
>@@ -82,3 +327,16 @@ int vits_init(struct kvm *kvm)
> 
> 	return -ENXIO;
> }
>+
>+void vits_destroy(struct kvm *kvm)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	struct vgic_its *its = &dist->its;
>+
>+	if (!vgic_has_its(kvm))
>+		return;
>+
>+	kfree(dist->pendbaser);
>+
>+	its->enabled = false;
>+}
>diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>index 4e7dcb8..08f97d1 100644
>--- a/virt/kvm/arm/vgic/vgic.h
>+++ b/virt/kvm/arm/vgic/vgic.h
>@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> 
> int vits_init(struct kvm *kvm);
> void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>+void vits_destroy(struct kvm *kvm);
> #else
> static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
> 					       u64 mpidr)
>@@ -137,6 +138,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> {
> 	return;
> }
>+
>+static inline void vits_destroy(struct kvm *kvm)
>+{
>+	return;
>+}
> #endif
> 
> void vgic_set_vmcr(struct kvm_vcpu *vcpu, struct vgic_vmcr *vmcr);
>diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>index dcfb93d..e4459e3 100644
>--- a/virt/kvm/arm/vgic/vgic_init.c
>+++ b/virt/kvm/arm/vgic/vgic_init.c
>@@ -298,6 +298,8 @@ void kvm_vgic_destroy(struct kvm *kvm)
> 
> 	kvm_vgic_dist_destroy(kvm);
> 
>+	vits_destroy(kvm);
>+
> 	kvm_for_each_vcpu(i, vcpu, kvm)
> 		kvm_vgic_vcpu_destroy(vcpu);
> }
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation
  2016-03-26  2:14   ` Andre Przywara
@ 2016-05-05 18:59     ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:59 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>LPIs are dynamically created (mapped) at guest runtime and their
>actual numbers can be quite high, but is mostly assigned using a very
>sparse allocation scheme. So arrays are not an ideal data structure
>to hold the information. We use our equivalent of the "Interrupt
>Translation Table Entry" (ITTE) to hold the vgic_irq struct for a
>virtual LPI embedded in in the ITTE.
>Connect the VGIC core code via an accessor function to help it get the
>struct vgic_irq for a certain LPI.

Reviewed-by:Tirumalesh Chalamarla<tchalamarla@caviumnetworks.com>

>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> virt/kvm/arm/vgic/its-emul.c | 29 +++++++++++++++++++++++++++++
> virt/kvm/arm/vgic/vgic.h     |  8 +++++++-
> 2 files changed, 36 insertions(+), 1 deletion(-)
>
>diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>index c0334ff..1188e9a 100644
>--- a/virt/kvm/arm/vgic/its-emul.c
>+++ b/virt/kvm/arm/vgic/its-emul.c
>@@ -55,11 +55,29 @@ struct its_collection {
> struct its_itte {
> 	struct list_head itte_list;
> 
>+	struct vgic_irq irq;
> 	struct its_collection *collection;
> 	u32 lpi;
> 	u32 event_id;
> };
> 
>+/* To be used as an iterator this macro misses the enclosing parentheses */
>+#define for_each_lpi(dev, itte, kvm) \
>+	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
>+		list_for_each_entry(itte, &(dev)->itt, itte_list)
>+
>+static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>+{
>+	struct its_device *device;
>+	struct its_itte *itte;
>+
>+	for_each_lpi(device, itte, kvm) {
>+		if (itte->lpi == lpi)
>+			return itte;
>+	}
>+	return NULL;
>+}
>+
> #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> 
> static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>@@ -166,6 +184,17 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> 	return 0;
> }
> 
>+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
>+{
>+	struct its_itte *itte;
>+
>+	itte = find_itte_by_lpi(kvm, intid);
>+	if (!itte)
>+		return NULL;
>+
>+	return &itte->irq;
>+}
>+
> static void its_free_itte(struct its_itte *itte)
> {
> 	list_del(&itte->itte_list);
>diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>index 08f97d1..160c511 100644
>--- a/virt/kvm/arm/vgic/vgic.h
>+++ b/virt/kvm/arm/vgic/vgic.h
>@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> 
> int vits_init(struct kvm *kvm);
> void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
> void vits_destroy(struct kvm *kvm);
> #else
> static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>@@ -129,7 +130,7 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
> 	return -ENODEV;
> }
> 
>-int vits_init(struct kvm *kvm)
>+static inline int vits_init(struct kvm *kvm)
> {
> 	return 0;
> }
>@@ -139,6 +140,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> 	return;
> }
> 
>+static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
>+{
>+	return NULL;
>+}
>+
> static inline void vits_destroy(struct kvm *kvm)
> {
> 	return;
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation
@ 2016-05-05 18:59     ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 18:59 UTC (permalink / raw)
  To: linux-arm-kernel






On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>LPIs are dynamically created (mapped) at guest runtime and their
>actual numbers can be quite high, but is mostly assigned using a very
>sparse allocation scheme. So arrays are not an ideal data structure
>to hold the information. We use our equivalent of the "Interrupt
>Translation Table Entry" (ITTE) to hold the vgic_irq struct for a
>virtual LPI embedded in in the ITTE.
>Connect the VGIC core code via an accessor function to help it get the
>struct vgic_irq for a certain LPI.

Reviewed-by:Tirumalesh Chalamarla<tchalamarla@caviumnetworks.com>

>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> virt/kvm/arm/vgic/its-emul.c | 29 +++++++++++++++++++++++++++++
> virt/kvm/arm/vgic/vgic.h     |  8 +++++++-
> 2 files changed, 36 insertions(+), 1 deletion(-)
>
>diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>index c0334ff..1188e9a 100644
>--- a/virt/kvm/arm/vgic/its-emul.c
>+++ b/virt/kvm/arm/vgic/its-emul.c
>@@ -55,11 +55,29 @@ struct its_collection {
> struct its_itte {
> 	struct list_head itte_list;
> 
>+	struct vgic_irq irq;
> 	struct its_collection *collection;
> 	u32 lpi;
> 	u32 event_id;
> };
> 
>+/* To be used as an iterator this macro misses the enclosing parentheses */
>+#define for_each_lpi(dev, itte, kvm) \
>+	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
>+		list_for_each_entry(itte, &(dev)->itt, itte_list)
>+
>+static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>+{
>+	struct its_device *device;
>+	struct its_itte *itte;
>+
>+	for_each_lpi(device, itte, kvm) {
>+		if (itte->lpi == lpi)
>+			return itte;
>+	}
>+	return NULL;
>+}
>+
> #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
> 
> static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>@@ -166,6 +184,17 @@ static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
> 	return 0;
> }
> 
>+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
>+{
>+	struct its_itte *itte;
>+
>+	itte = find_itte_by_lpi(kvm, intid);
>+	if (!itte)
>+		return NULL;
>+
>+	return &itte->irq;
>+}
>+
> static void its_free_itte(struct its_itte *itte)
> {
> 	list_del(&itte->itte_list);
>diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>index 08f97d1..160c511 100644
>--- a/virt/kvm/arm/vgic/vgic.h
>+++ b/virt/kvm/arm/vgic/vgic.h
>@@ -63,6 +63,7 @@ int vgic_register_redist_regions(struct kvm *kvm, gpa_t dist_base_address);
> 
> int vits_init(struct kvm *kvm);
> void vgic_enable_lpis(struct kvm_vcpu *vcpu);
>+struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid);
> void vits_destroy(struct kvm *kvm);
> #else
> static inline void vgic_v3_irq_change_affinity(struct kvm *kvm, u32 intid,
>@@ -129,7 +130,7 @@ static inline int vgic_register_redist_regions(struct kvm *kvm,
> 	return -ENODEV;
> }
> 
>-int vits_init(struct kvm *kvm)
>+static inline int vits_init(struct kvm *kvm)
> {
> 	return 0;
> }
>@@ -139,6 +140,11 @@ static inline void vgic_enable_lpis(struct kvm_vcpu *vcpu)
> 	return;
> }
> 
>+static inline struct vgic_irq *vgic_its_get_lpi(struct kvm *kvm, u32 intid)
>+{
>+	return NULL;
>+}
>+
> static inline void vits_destroy(struct kvm *kvm)
> {
> 	return;
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers
  2016-03-26  2:14   ` Andre Przywara
@ 2016-05-05 19:12     ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 19:12 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>The connection between a device, an event ID, the LPI number and the
>allocated CPU is stored in in-memory tables in a GICv3, but their
>format is not specified by the spec. Instead software uses a command
>queue in a ring buffer to let the ITS implementation use their own
>format.
>Implement handlers for the various ITS commands and let them store
>the requested relation into our own data structures.
>To avoid kmallocs inside the ITS spinlock, we preallocate possibly
>needed memory outside of the lock and free that if it turns out to
>be not needed (mostly error handling).
>Error handling is very basic at this point, as we don't have a good
>way of communicating errors to the guest (usually a SError).
>The INT command handler is missing at this point, as we gain the
>capability of actually injecting MSIs into the guest only later on.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> include/linux/irqchip/arm-gic-v3.h |  19 +-
> virt/kvm/arm/vgic/its-emul.c       | 514 ++++++++++++++++++++++++++++++++++++-
> 2 files changed, 531 insertions(+), 2 deletions(-)
>
>diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>index 7011b98..c9aa8d8 100644
>--- a/include/linux/irqchip/arm-gic-v3.h
>+++ b/include/linux/irqchip/arm-gic-v3.h
>@@ -254,7 +254,10 @@
>  */
> #define GITS_CMD_MAPD			0x08
> #define GITS_CMD_MAPC			0x09
>-#define GITS_CMD_MAPVI			0x0a
>+#define GITS_CMD_MAPTI			0x0a
>+/* older GIC documentation used MAPVI for this command */
>+#define GITS_CMD_MAPVI			GITS_CMD_MAPTI
>+#define GITS_CMD_MAPI			0x0b
> #define GITS_CMD_MOVI			0x01
> #define GITS_CMD_DISCARD		0x0f
> #define GITS_CMD_INV			0x0c
>@@ -265,6 +268,20 @@
> #define GITS_CMD_SYNC			0x05
> 
> /*
>+ * ITS error numbers
>+ */
>+#define E_ITS_MOVI_UNMAPPED_INTERRUPT           0x010107
>+#define E_ITS_MOVI_UNMAPPED_COLLECTION          0x010109
>+#define E_ITS_CLEAR_UNMAPPED_INTERRUPT          0x010507
>+#define E_ITS_MAPC_PROCNUM_OOR                  0x010902
>+#define E_ITS_MAPTI_UNMAPPED_DEVICE             0x010a04
>+#define E_ITS_MAPTI_PHYSICALID_OOR              0x010a06
>+#define E_ITS_INV_UNMAPPED_INTERRUPT            0x010c07
>+#define E_ITS_INVALL_UNMAPPED_COLLECTION        0x010d09
>+#define E_ITS_MOVALL_PROCNUM_OOR                0x010e01
>+#define E_ITS_DISCARD_UNMAPPED_INTERRUPT        0x010f07
>+
>+/*
>  * CPU interface registers
>  */
> #define ICC_CTLR_EL1_EOImode_drop_dir	(0U << 1)
>diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>index d82ba9b..dcd0cac 100644
>--- a/virt/kvm/arm/vgic/its-emul.c
>+++ b/virt/kvm/arm/vgic/its-emul.c
>@@ -22,6 +22,7 @@
> #include <linux/kvm_host.h>
> #include <linux/interrupt.h>
> #include <linux/list.h>
>+#include <linux/slab.h>
> 
> #include <linux/irqchip/arm-gic-v3.h>
> 
>@@ -61,6 +62,34 @@ struct its_itte {
> 	u32 event_id;
> };
> 
>+static struct its_device *find_its_device(struct kvm *kvm, u32 device_id)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	struct its_device *device;
>+
>+	list_for_each_entry(device, &its->device_list, dev_list)
>+		if (device_id == device->device_id)
>+			return device;
>+
>+	return NULL;
>+}
>+
>+static struct its_itte *find_itte(struct kvm *kvm, u32 device_id, u32 event_id)
>+{
>+	struct its_device *device;
>+	struct its_itte *itte;
>+
>+	device = find_its_device(kvm, device_id);
>+	if (device == NULL)
>+		return NULL;
>+
>+	list_for_each_entry(itte, &device->itt, itte_list)
>+		if (itte->event_id == event_id)
>+			return itte;
>+
>+	return NULL;
>+}
>+
> /* To be used as an iterator this macro misses the enclosing parentheses */
> #define for_each_lpi(dev, itte, kvm) \
> 	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
>@@ -78,6 +107,19 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
> 	return NULL;
> }
> 
>+static struct its_collection *find_collection(struct kvm *kvm, int coll_id)
>+{
>+	struct its_collection *collection;
>+
>+	list_for_each_entry(collection, &kvm->arch.vgic.its.collection_list,
>+			    coll_list) {
>+		if (coll_id == collection->collection_id)
>+			return collection;
>+	}
>+
>+	return NULL;
>+}
>+
> #define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
> #define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
> 
>@@ -91,6 +133,29 @@ static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
> 	vgic_queue_irq(kvm, &itte->irq);
> }
> 
>+/*
>+ * Finds all LPIs which are mapped to this collection and updates the
>+ * struct irq's target_vcpu field accordingly.
>+ * Needs to be called whenever either the collection for a LPIs has
>+ * changed or the collection itself got retargetted.
>+ */
>+static void update_affinity(struct kvm *kvm, struct its_collection *coll)
>+{
>+	struct its_device *device;
>+	struct its_itte *itte;
>+	struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, coll->target_addr);
>+
>+	for_each_lpi(device, itte, kvm) {
>+		if (!itte->collection ||
>+		    coll->collection_id != itte->collection->collection_id)
>+			continue;
>+
>+		spin_lock(&itte->irq.irq_lock);
>+		itte->irq.target_vcpu = vcpu;
>+		spin_unlock(&itte->irq.irq_lock);
>+	}
>+}
>+
> #define GIC_LPI_OFFSET 8192
> 
> /* We scan the table in chunks the size of the smallest page size */
>@@ -323,13 +388,460 @@ static void its_free_itte(struct its_itte *itte)
> 	kfree(itte);
> }
> 
>+static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>+{
>+	return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT_ULL(size) - 1);
>+}
>+
>+#define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
>+#define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
>+#define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>+#define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>+#define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>+#define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
>+#define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
>+
>+/* The DISCARD command frees an Interrupt Translation Table Entry (ITTE). */
>+static int vits_cmd_handle_discard(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 device_id;
>+	u32 event_id;
>+	struct its_itte *itte;
>+	int ret = E_ITS_DISCARD_UNMAPPED_INTERRUPT;
>+
>+	device_id = its_cmd_get_deviceid(its_cmd);
>+	event_id = its_cmd_get_id(its_cmd);
>+
>+	spin_lock(&its->lock);
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (itte && itte->collection) {
>+		/*
>+		 * Though the spec talks about removing the pending state, we
>+		 * don't bother here since we clear the ITTE anyway and the
>+		 * pending state is a property of the ITTE struct.
>+		 */
>+		its_free_itte(itte);
>+		ret = 0;
>+	}
>+
>+	spin_unlock(&its->lock);
>+	return ret;
>+}
>+
>+/* The MOVI command moves an ITTE to a different collection. */
>+static int vits_cmd_handle_movi(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 device_id = its_cmd_get_deviceid(its_cmd);
>+	u32 event_id = its_cmd_get_id(its_cmd);
>+	u32 coll_id = its_cmd_get_collection(its_cmd);
>+	struct its_itte *itte;
>+	struct its_collection *collection;
>+	int ret;
>+
>+	spin_lock(&its->lock);
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (!itte) {
>+		ret = E_ITS_MOVI_UNMAPPED_INTERRUPT;
>+		goto out_unlock;
>+	}
>+	if (!its_is_collection_mapped(itte->collection)) {
>+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>+		goto out_unlock;
>+	}
>+
>+	collection = find_collection(kvm, coll_id);
>+	if (!its_is_collection_mapped(collection)) {
>+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>+		goto out_unlock;
>+	}
>+
>+	itte->collection = collection;
>+	update_affinity(kvm, collection);
>+

Not planning to send this down to Physical ITS in case of VFIO for performance reasons?

>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return ret;
>+}
>+
>+static void vits_init_collection(struct kvm *kvm,
>+				 struct its_collection *collection,
>+				 u32 coll_id)
>+{
>+	collection->collection_id = coll_id;
>+	collection->target_addr = COLLECTION_NOT_MAPPED;
>+
>+	list_add_tail(&collection->coll_list,
>+		&kvm->arch.vgic.its.collection_list);
>+}
>+
>+/* The MAPTI and MAPI commands map LPIs to ITTEs. */
>+static int vits_cmd_handle_mapi(struct kvm *kvm, u64 *its_cmd, u8 cmd)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	u32 device_id = its_cmd_get_deviceid(its_cmd);
>+	u32 event_id = its_cmd_get_id(its_cmd);
>+	u32 coll_id = its_cmd_get_collection(its_cmd);
>+	struct its_itte *itte, *new_itte;
>+	struct its_device *device;
>+	struct its_collection *collection, *new_coll;
>+	int lpi_nr;
>+	int ret = 0;
>+
>+	/* Preallocate possibly needed memory here outside of the lock */
>+	new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
>+	new_itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
>+
>+	spin_lock(&dist->its.lock);
>+
>+	device = find_its_device(kvm, device_id);
>+	if (!device) {
>+		ret = E_ITS_MAPTI_UNMAPPED_DEVICE;
>+		goto out_unlock;
>+	}
>+
>+	collection = find_collection(kvm, coll_id);
>+	if (!collection && !new_coll) {
>+		ret = -ENOMEM;
>+		goto out_unlock;
>+	}
>+
>+	if (cmd == GITS_CMD_MAPTI)
>+		lpi_nr = its_cmd_get_physical_id(its_cmd);
>+	else
>+		lpi_nr = event_id;
>+	if (lpi_nr < GIC_LPI_OFFSET ||
>+	    lpi_nr >= nr_idbits_propbase(dist->propbaser)) {
>+		ret = E_ITS_MAPTI_PHYSICALID_OOR;
>+		goto out_unlock;
>+	}
>+
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (!itte) {
>+		if (!new_itte) {
>+			ret = -ENOMEM;
>+			goto out_unlock;
>+		}
>+		itte = new_itte;
>+
>+		itte->event_id	= event_id;
>+		list_add_tail(&itte->itte_list, &device->itt);
>+	} else {
>+		kfree(new_itte);
>+	}
>+
>+	if (!collection) {
>+		collection = new_coll;
>+		vits_init_collection(kvm, collection, coll_id);
>+	} else {
>+		kfree(new_coll);
>+	}
>+
>+	itte->collection = collection;
>+	itte->lpi = lpi_nr;
>+	itte->irq.intid = lpi_nr;
>+	INIT_LIST_HEAD(&itte->irq.ap_list);
>+	spin_lock_init(&itte->irq.irq_lock);
>+	itte->irq.vcpu = NULL;
>+	update_affinity(kvm, collection);
>+

Same here. 
>+out_unlock:
>+	spin_unlock(&dist->its.lock);
>+	if (ret) {
>+		kfree(new_coll);
>+		kfree(new_itte);
>+	}
>+	return ret;
>+}
>+
>+static void vits_unmap_device(struct kvm *kvm, struct its_device *device)
>+{
>+	struct its_itte *itte, *temp;
>+
>+	/*
>+	 * The spec says that unmapping a device with still valid
>+	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>+	 * since we cannot leave the memory unreferenced.
>+	 */
>+	list_for_each_entry_safe(itte, temp, &device->itt, itte_list)
>+		its_free_itte(itte);
>+
>+	list_del(&device->dev_list);
>+	kfree(device);
>+}
>+
>+/* MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs). */
>+static int vits_cmd_handle_mapd(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	bool valid = its_cmd_get_validbit(its_cmd);
>+	u32 device_id = its_cmd_get_deviceid(its_cmd);
>+	struct its_device *device, *new_device = NULL;
>+
>+	/* We preallocate memory outside of the lock here */
>+	if (valid) {
>+		new_device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
>+		if (!new_device)
>+			return -ENOMEM;
>+	}
>+
>+	spin_lock(&its->lock);
>+
>+	device = find_its_device(kvm, device_id);
>+	if (device)
>+		vits_unmap_device(kvm, device);
>+
>+	/*
>+	 * The spec does not say whether unmapping a not-mapped device
>+	 * is an error, so we are done in any case.
>+	 */
>+	if (!valid)
>+		goto out_unlock;
>+
>+	device = new_device;
>+
>+	device->device_id = device_id;
>+	INIT_LIST_HEAD(&device->itt);
>+
>+	list_add_tail(&device->dev_list,
>+		      &kvm->arch.vgic.its.device_list);
>+
>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return 0;
>+}
>+
>+/* The MAPC command maps collection IDs to redistributors. */
>+static int vits_cmd_handle_mapc(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u16 coll_id;
>+	u32 target_addr;
>+	struct its_collection *collection, *new_coll = NULL;
>+	bool valid;
>+
>+	valid = its_cmd_get_validbit(its_cmd);
>+	coll_id = its_cmd_get_collection(its_cmd);
>+	target_addr = its_cmd_get_target_addr(its_cmd);
>+
>+	if (target_addr >= atomic_read(&kvm->online_vcpus))
>+		return E_ITS_MAPC_PROCNUM_OOR;
>+
>+	/* We preallocate memory outside of the lock here */
>+	if (valid) {
>+		new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
>+		if (!new_coll)
>+			return -ENOMEM;
>+	}
>+
>+	spin_lock(&its->lock);
>+	collection = find_collection(kvm, coll_id);
>+
>+	if (!valid) {
>+		struct its_device *device;
>+		struct its_itte *itte;
>+		/*
>+		 * Clearing the mapping for that collection ID removes the
>+		 * entry from the list. If there wasn't any before, we can
>+		 * go home early.
>+		 */
>+		if (!collection)
>+			goto out_unlock;
>+
>+		for_each_lpi(device, itte, kvm)
>+			if (itte->collection &&
>+			    itte->collection->collection_id == coll_id)
>+				itte->collection = NULL;
>+
>+		list_del(&collection->coll_list);
>+		kfree(collection);
>+	} else {
>+		if (!collection)
>+			collection = new_coll;
>+		else
>+			kfree(new_coll);
>+
>+		vits_init_collection(kvm, collection, coll_id);
>+		collection->target_addr = target_addr;
>+		update_affinity(kvm, collection);
>+	}
>+
>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return 0;
>+}
>+
>+/* The CLEAR command removes the pending state for a particular LPI. */
>+static int vits_cmd_handle_clear(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 device_id;
>+	u32 event_id;
>+	struct its_itte *itte;
>+	int ret = 0;
>+
>+	device_id = its_cmd_get_deviceid(its_cmd);
>+	event_id = its_cmd_get_id(its_cmd);
>+
>+	spin_lock(&its->lock);
>+
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (!itte) {
>+		ret = E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>+		goto out_unlock;
>+	}
>+
>+	itte->irq.pending = false;
>+
>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return ret;
>+}
>+
>+/* The INV command syncs the configuration bits from the memory tables. */
>+static int vits_cmd_handle_inv(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	u32 device_id;
>+	u32 event_id;
>+	struct its_itte *itte, *new_itte;
>+	gpa_t propbase;
>+	int ret;
>+	u8 prop;
>+
>+	device_id = its_cmd_get_deviceid(its_cmd);
>+	event_id = its_cmd_get_id(its_cmd);
>+
>+	spin_lock(&dist->its.lock);
>+	itte = find_itte(kvm, device_id, event_id);
>+	spin_unlock(&dist->its.lock);
>+	if (!itte)
>+		return E_ITS_INV_UNMAPPED_INTERRUPT;
>+
>+	/*
>+	 * We cannot read from guest memory inside the spinlock, so we
>+	 * need to re-read our tables to learn whether the LPI number we are
>+	 * using is still valid.
>+	 */
>+	do {
>+		propbase = BASER_BASE_ADDRESS(dist->propbaser);
>+		ret = kvm_read_guest(kvm, propbase + itte->lpi - GIC_LPI_OFFSET,
>+				     &prop, 1);
>+		if (ret)
>+			return ret;
>+
>+		spin_lock(&dist->its.lock);
>+		new_itte = find_itte(kvm, device_id, event_id);
>+		if (new_itte->lpi != itte->lpi) {
>+			itte = new_itte;
>+			spin_unlock(&dist->its.lock);
>+			continue;
>+		}
>+		update_lpi_config(kvm, itte, prop);
>+		spin_unlock(&dist->its.lock);

Not possible to call the function defined early and update all. How much performance gain will this be?
>+	} while (0);
>+	return 0;
>+}
>+
>+/* The INVALL command requests flushing of all IRQ data in this collection. */
>+static int vits_cmd_handle_invall(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	u64 prop_base_reg, pend_base_reg;
>+	u32 coll_id = its_cmd_get_collection(its_cmd);
>+	struct its_collection *collection;
>+	struct kvm_vcpu *vcpu;
>+
>+	collection = find_collection(kvm, coll_id);
>+	if (!its_is_collection_mapped(collection))
>+		return E_ITS_INVALL_UNMAPPED_COLLECTION;
>+
>+	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>+
>+	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
>+	prop_base_reg = dist->propbaser;
>+
>+	its_update_lpis_configuration(kvm, prop_base_reg);
>+	its_sync_lpi_pending_table(vcpu, pend_base_reg);
>+
>+	return 0;
>+}
>+
>+/* The MOVALL command moves all IRQs from one redistributor to another. */
>+static int vits_cmd_handle_movall(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 target1_addr = its_cmd_get_target_addr(its_cmd);
>+	u32 target2_addr = its_cmd_mask_field(its_cmd, 3, 16, 32);
>+	struct its_collection *collection;
>+
>+	if (target1_addr >= atomic_read(&kvm->online_vcpus) ||
>+	    target2_addr >= atomic_read(&kvm->online_vcpus))
>+		return E_ITS_MOVALL_PROCNUM_OOR;
>+
>+	if (target1_addr == target2_addr)
>+		return 0;
>+
>+	spin_lock(&its->lock);
>+	list_for_each_entry(collection, &its->collection_list,
>+			    coll_list) {
>+		if (collection && collection->target_addr == target1_addr)
>+			collection->target_addr = target2_addr;
>+		update_affinity(kvm, collection);
>+	}

Not planning to send the commands down in case of VFIO? Think about NUMA systems and performance impact. 
>+
>+	spin_unlock(&its->lock);
>+	return 0;
>+}
>+
> /*
>  * This function is called with both the ITS and the distributor lock dropped,
>  * so the actual command handlers must take the respective locks when needed.
>  */
> static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> {
>-	return -ENODEV;
>+	u8 cmd = its_cmd_get_command(its_cmd);
>+	int ret = -ENODEV;
>+
>+	switch (cmd) {
>+	case GITS_CMD_MAPD:
>+		ret = vits_cmd_handle_mapd(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_MAPC:
>+		ret = vits_cmd_handle_mapc(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_MAPI:
>+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
>+		break;
>+	case GITS_CMD_MAPTI:
>+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
>+		break;
>+	case GITS_CMD_MOVI:
>+		ret = vits_cmd_handle_movi(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_DISCARD:
>+		ret = vits_cmd_handle_discard(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_CLEAR:
>+		ret = vits_cmd_handle_clear(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_MOVALL:
>+		ret = vits_cmd_handle_movall(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_INV:
>+		ret = vits_cmd_handle_inv(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_INVALL:
>+		ret = vits_cmd_handle_invall(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_SYNC:
>+		/* we ignore this command: we are in sync all of the time */
>+		ret = 0;
>+		break;
>+	}
>+
>+	return ret;
> }
> 
> static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm@lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers
@ 2016-05-05 19:12     ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-05 19:12 UTC (permalink / raw)
  To: linux-arm-kernel






On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:

>The connection between a device, an event ID, the LPI number and the
>allocated CPU is stored in in-memory tables in a GICv3, but their
>format is not specified by the spec. Instead software uses a command
>queue in a ring buffer to let the ITS implementation use their own
>format.
>Implement handlers for the various ITS commands and let them store
>the requested relation into our own data structures.
>To avoid kmallocs inside the ITS spinlock, we preallocate possibly
>needed memory outside of the lock and free that if it turns out to
>be not needed (mostly error handling).
>Error handling is very basic at this point, as we don't have a good
>way of communicating errors to the guest (usually a SError).
>The INT command handler is missing at this point, as we gain the
>capability of actually injecting MSIs into the guest only later on.
>
>Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>---
> include/linux/irqchip/arm-gic-v3.h |  19 +-
> virt/kvm/arm/vgic/its-emul.c       | 514 ++++++++++++++++++++++++++++++++++++-
> 2 files changed, 531 insertions(+), 2 deletions(-)
>
>diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>index 7011b98..c9aa8d8 100644
>--- a/include/linux/irqchip/arm-gic-v3.h
>+++ b/include/linux/irqchip/arm-gic-v3.h
>@@ -254,7 +254,10 @@
>  */
> #define GITS_CMD_MAPD			0x08
> #define GITS_CMD_MAPC			0x09
>-#define GITS_CMD_MAPVI			0x0a
>+#define GITS_CMD_MAPTI			0x0a
>+/* older GIC documentation used MAPVI for this command */
>+#define GITS_CMD_MAPVI			GITS_CMD_MAPTI
>+#define GITS_CMD_MAPI			0x0b
> #define GITS_CMD_MOVI			0x01
> #define GITS_CMD_DISCARD		0x0f
> #define GITS_CMD_INV			0x0c
>@@ -265,6 +268,20 @@
> #define GITS_CMD_SYNC			0x05
> 
> /*
>+ * ITS error numbers
>+ */
>+#define E_ITS_MOVI_UNMAPPED_INTERRUPT           0x010107
>+#define E_ITS_MOVI_UNMAPPED_COLLECTION          0x010109
>+#define E_ITS_CLEAR_UNMAPPED_INTERRUPT          0x010507
>+#define E_ITS_MAPC_PROCNUM_OOR                  0x010902
>+#define E_ITS_MAPTI_UNMAPPED_DEVICE             0x010a04
>+#define E_ITS_MAPTI_PHYSICALID_OOR              0x010a06
>+#define E_ITS_INV_UNMAPPED_INTERRUPT            0x010c07
>+#define E_ITS_INVALL_UNMAPPED_COLLECTION        0x010d09
>+#define E_ITS_MOVALL_PROCNUM_OOR                0x010e01
>+#define E_ITS_DISCARD_UNMAPPED_INTERRUPT        0x010f07
>+
>+/*
>  * CPU interface registers
>  */
> #define ICC_CTLR_EL1_EOImode_drop_dir	(0U << 1)
>diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>index d82ba9b..dcd0cac 100644
>--- a/virt/kvm/arm/vgic/its-emul.c
>+++ b/virt/kvm/arm/vgic/its-emul.c
>@@ -22,6 +22,7 @@
> #include <linux/kvm_host.h>
> #include <linux/interrupt.h>
> #include <linux/list.h>
>+#include <linux/slab.h>
> 
> #include <linux/irqchip/arm-gic-v3.h>
> 
>@@ -61,6 +62,34 @@ struct its_itte {
> 	u32 event_id;
> };
> 
>+static struct its_device *find_its_device(struct kvm *kvm, u32 device_id)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	struct its_device *device;
>+
>+	list_for_each_entry(device, &its->device_list, dev_list)
>+		if (device_id == device->device_id)
>+			return device;
>+
>+	return NULL;
>+}
>+
>+static struct its_itte *find_itte(struct kvm *kvm, u32 device_id, u32 event_id)
>+{
>+	struct its_device *device;
>+	struct its_itte *itte;
>+
>+	device = find_its_device(kvm, device_id);
>+	if (device == NULL)
>+		return NULL;
>+
>+	list_for_each_entry(itte, &device->itt, itte_list)
>+		if (itte->event_id == event_id)
>+			return itte;
>+
>+	return NULL;
>+}
>+
> /* To be used as an iterator this macro misses the enclosing parentheses */
> #define for_each_lpi(dev, itte, kvm) \
> 	list_for_each_entry(dev, &(kvm)->arch.vgic.its.device_list, dev_list) \
>@@ -78,6 +107,19 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
> 	return NULL;
> }
> 
>+static struct its_collection *find_collection(struct kvm *kvm, int coll_id)
>+{
>+	struct its_collection *collection;
>+
>+	list_for_each_entry(collection, &kvm->arch.vgic.its.collection_list,
>+			    coll_list) {
>+		if (coll_id == collection->collection_id)
>+			return collection;
>+	}
>+
>+	return NULL;
>+}
>+
> #define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
> #define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
> 
>@@ -91,6 +133,29 @@ static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
> 	vgic_queue_irq(kvm, &itte->irq);
> }
> 
>+/*
>+ * Finds all LPIs which are mapped to this collection and updates the
>+ * struct irq's target_vcpu field accordingly.
>+ * Needs to be called whenever either the collection for a LPIs has
>+ * changed or the collection itself got retargetted.
>+ */
>+static void update_affinity(struct kvm *kvm, struct its_collection *coll)
>+{
>+	struct its_device *device;
>+	struct its_itte *itte;
>+	struct kvm_vcpu *vcpu = kvm_get_vcpu(kvm, coll->target_addr);
>+
>+	for_each_lpi(device, itte, kvm) {
>+		if (!itte->collection ||
>+		    coll->collection_id != itte->collection->collection_id)
>+			continue;
>+
>+		spin_lock(&itte->irq.irq_lock);
>+		itte->irq.target_vcpu = vcpu;
>+		spin_unlock(&itte->irq.irq_lock);
>+	}
>+}
>+
> #define GIC_LPI_OFFSET 8192
> 
> /* We scan the table in chunks the size of the smallest page size */
>@@ -323,13 +388,460 @@ static void its_free_itte(struct its_itte *itte)
> 	kfree(itte);
> }
> 
>+static u64 its_cmd_mask_field(u64 *its_cmd, int word, int shift, int size)
>+{
>+	return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT_ULL(size) - 1);
>+}
>+
>+#define its_cmd_get_command(cmd)	its_cmd_mask_field(cmd, 0,  0,  8)
>+#define its_cmd_get_deviceid(cmd)	its_cmd_mask_field(cmd, 0, 32, 32)
>+#define its_cmd_get_id(cmd)		its_cmd_mask_field(cmd, 1,  0, 32)
>+#define its_cmd_get_physical_id(cmd)	its_cmd_mask_field(cmd, 1, 32, 32)
>+#define its_cmd_get_collection(cmd)	its_cmd_mask_field(cmd, 2,  0, 16)
>+#define its_cmd_get_target_addr(cmd)	its_cmd_mask_field(cmd, 2, 16, 32)
>+#define its_cmd_get_validbit(cmd)	its_cmd_mask_field(cmd, 2, 63,  1)
>+
>+/* The DISCARD command frees an Interrupt Translation Table Entry (ITTE). */
>+static int vits_cmd_handle_discard(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 device_id;
>+	u32 event_id;
>+	struct its_itte *itte;
>+	int ret = E_ITS_DISCARD_UNMAPPED_INTERRUPT;
>+
>+	device_id = its_cmd_get_deviceid(its_cmd);
>+	event_id = its_cmd_get_id(its_cmd);
>+
>+	spin_lock(&its->lock);
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (itte && itte->collection) {
>+		/*
>+		 * Though the spec talks about removing the pending state, we
>+		 * don't bother here since we clear the ITTE anyway and the
>+		 * pending state is a property of the ITTE struct.
>+		 */
>+		its_free_itte(itte);
>+		ret = 0;
>+	}
>+
>+	spin_unlock(&its->lock);
>+	return ret;
>+}
>+
>+/* The MOVI command moves an ITTE to a different collection. */
>+static int vits_cmd_handle_movi(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 device_id = its_cmd_get_deviceid(its_cmd);
>+	u32 event_id = its_cmd_get_id(its_cmd);
>+	u32 coll_id = its_cmd_get_collection(its_cmd);
>+	struct its_itte *itte;
>+	struct its_collection *collection;
>+	int ret;
>+
>+	spin_lock(&its->lock);
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (!itte) {
>+		ret = E_ITS_MOVI_UNMAPPED_INTERRUPT;
>+		goto out_unlock;
>+	}
>+	if (!its_is_collection_mapped(itte->collection)) {
>+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>+		goto out_unlock;
>+	}
>+
>+	collection = find_collection(kvm, coll_id);
>+	if (!its_is_collection_mapped(collection)) {
>+		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>+		goto out_unlock;
>+	}
>+
>+	itte->collection = collection;
>+	update_affinity(kvm, collection);
>+

Not planning to send this down to Physical ITS in case of VFIO for performance reasons?

>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return ret;
>+}
>+
>+static void vits_init_collection(struct kvm *kvm,
>+				 struct its_collection *collection,
>+				 u32 coll_id)
>+{
>+	collection->collection_id = coll_id;
>+	collection->target_addr = COLLECTION_NOT_MAPPED;
>+
>+	list_add_tail(&collection->coll_list,
>+		&kvm->arch.vgic.its.collection_list);
>+}
>+
>+/* The MAPTI and MAPI commands map LPIs to ITTEs. */
>+static int vits_cmd_handle_mapi(struct kvm *kvm, u64 *its_cmd, u8 cmd)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	u32 device_id = its_cmd_get_deviceid(its_cmd);
>+	u32 event_id = its_cmd_get_id(its_cmd);
>+	u32 coll_id = its_cmd_get_collection(its_cmd);
>+	struct its_itte *itte, *new_itte;
>+	struct its_device *device;
>+	struct its_collection *collection, *new_coll;
>+	int lpi_nr;
>+	int ret = 0;
>+
>+	/* Preallocate possibly needed memory here outside of the lock */
>+	new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
>+	new_itte = kzalloc(sizeof(struct its_itte), GFP_KERNEL);
>+
>+	spin_lock(&dist->its.lock);
>+
>+	device = find_its_device(kvm, device_id);
>+	if (!device) {
>+		ret = E_ITS_MAPTI_UNMAPPED_DEVICE;
>+		goto out_unlock;
>+	}
>+
>+	collection = find_collection(kvm, coll_id);
>+	if (!collection && !new_coll) {
>+		ret = -ENOMEM;
>+		goto out_unlock;
>+	}
>+
>+	if (cmd == GITS_CMD_MAPTI)
>+		lpi_nr = its_cmd_get_physical_id(its_cmd);
>+	else
>+		lpi_nr = event_id;
>+	if (lpi_nr < GIC_LPI_OFFSET ||
>+	    lpi_nr >= nr_idbits_propbase(dist->propbaser)) {
>+		ret = E_ITS_MAPTI_PHYSICALID_OOR;
>+		goto out_unlock;
>+	}
>+
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (!itte) {
>+		if (!new_itte) {
>+			ret = -ENOMEM;
>+			goto out_unlock;
>+		}
>+		itte = new_itte;
>+
>+		itte->event_id	= event_id;
>+		list_add_tail(&itte->itte_list, &device->itt);
>+	} else {
>+		kfree(new_itte);
>+	}
>+
>+	if (!collection) {
>+		collection = new_coll;
>+		vits_init_collection(kvm, collection, coll_id);
>+	} else {
>+		kfree(new_coll);
>+	}
>+
>+	itte->collection = collection;
>+	itte->lpi = lpi_nr;
>+	itte->irq.intid = lpi_nr;
>+	INIT_LIST_HEAD(&itte->irq.ap_list);
>+	spin_lock_init(&itte->irq.irq_lock);
>+	itte->irq.vcpu = NULL;
>+	update_affinity(kvm, collection);
>+

Same here. 
>+out_unlock:
>+	spin_unlock(&dist->its.lock);
>+	if (ret) {
>+		kfree(new_coll);
>+		kfree(new_itte);
>+	}
>+	return ret;
>+}
>+
>+static void vits_unmap_device(struct kvm *kvm, struct its_device *device)
>+{
>+	struct its_itte *itte, *temp;
>+
>+	/*
>+	 * The spec says that unmapping a device with still valid
>+	 * ITTEs associated is UNPREDICTABLE. We remove all ITTEs,
>+	 * since we cannot leave the memory unreferenced.
>+	 */
>+	list_for_each_entry_safe(itte, temp, &device->itt, itte_list)
>+		its_free_itte(itte);
>+
>+	list_del(&device->dev_list);
>+	kfree(device);
>+}
>+
>+/* MAPD maps or unmaps a device ID to Interrupt Translation Tables (ITTs). */
>+static int vits_cmd_handle_mapd(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	bool valid = its_cmd_get_validbit(its_cmd);
>+	u32 device_id = its_cmd_get_deviceid(its_cmd);
>+	struct its_device *device, *new_device = NULL;
>+
>+	/* We preallocate memory outside of the lock here */
>+	if (valid) {
>+		new_device = kzalloc(sizeof(struct its_device), GFP_KERNEL);
>+		if (!new_device)
>+			return -ENOMEM;
>+	}
>+
>+	spin_lock(&its->lock);
>+
>+	device = find_its_device(kvm, device_id);
>+	if (device)
>+		vits_unmap_device(kvm, device);
>+
>+	/*
>+	 * The spec does not say whether unmapping a not-mapped device
>+	 * is an error, so we are done in any case.
>+	 */
>+	if (!valid)
>+		goto out_unlock;
>+
>+	device = new_device;
>+
>+	device->device_id = device_id;
>+	INIT_LIST_HEAD(&device->itt);
>+
>+	list_add_tail(&device->dev_list,
>+		      &kvm->arch.vgic.its.device_list);
>+
>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return 0;
>+}
>+
>+/* The MAPC command maps collection IDs to redistributors. */
>+static int vits_cmd_handle_mapc(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u16 coll_id;
>+	u32 target_addr;
>+	struct its_collection *collection, *new_coll = NULL;
>+	bool valid;
>+
>+	valid = its_cmd_get_validbit(its_cmd);
>+	coll_id = its_cmd_get_collection(its_cmd);
>+	target_addr = its_cmd_get_target_addr(its_cmd);
>+
>+	if (target_addr >= atomic_read(&kvm->online_vcpus))
>+		return E_ITS_MAPC_PROCNUM_OOR;
>+
>+	/* We preallocate memory outside of the lock here */
>+	if (valid) {
>+		new_coll = kmalloc(sizeof(struct its_collection), GFP_KERNEL);
>+		if (!new_coll)
>+			return -ENOMEM;
>+	}
>+
>+	spin_lock(&its->lock);
>+	collection = find_collection(kvm, coll_id);
>+
>+	if (!valid) {
>+		struct its_device *device;
>+		struct its_itte *itte;
>+		/*
>+		 * Clearing the mapping for that collection ID removes the
>+		 * entry from the list. If there wasn't any before, we can
>+		 * go home early.
>+		 */
>+		if (!collection)
>+			goto out_unlock;
>+
>+		for_each_lpi(device, itte, kvm)
>+			if (itte->collection &&
>+			    itte->collection->collection_id == coll_id)
>+				itte->collection = NULL;
>+
>+		list_del(&collection->coll_list);
>+		kfree(collection);
>+	} else {
>+		if (!collection)
>+			collection = new_coll;
>+		else
>+			kfree(new_coll);
>+
>+		vits_init_collection(kvm, collection, coll_id);
>+		collection->target_addr = target_addr;
>+		update_affinity(kvm, collection);
>+	}
>+
>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return 0;
>+}
>+
>+/* The CLEAR command removes the pending state for a particular LPI. */
>+static int vits_cmd_handle_clear(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 device_id;
>+	u32 event_id;
>+	struct its_itte *itte;
>+	int ret = 0;
>+
>+	device_id = its_cmd_get_deviceid(its_cmd);
>+	event_id = its_cmd_get_id(its_cmd);
>+
>+	spin_lock(&its->lock);
>+
>+	itte = find_itte(kvm, device_id, event_id);
>+	if (!itte) {
>+		ret = E_ITS_CLEAR_UNMAPPED_INTERRUPT;
>+		goto out_unlock;
>+	}
>+
>+	itte->irq.pending = false;
>+
>+out_unlock:
>+	spin_unlock(&its->lock);
>+	return ret;
>+}
>+
>+/* The INV command syncs the configuration bits from the memory tables. */
>+static int vits_cmd_handle_inv(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	u32 device_id;
>+	u32 event_id;
>+	struct its_itte *itte, *new_itte;
>+	gpa_t propbase;
>+	int ret;
>+	u8 prop;
>+
>+	device_id = its_cmd_get_deviceid(its_cmd);
>+	event_id = its_cmd_get_id(its_cmd);
>+
>+	spin_lock(&dist->its.lock);
>+	itte = find_itte(kvm, device_id, event_id);
>+	spin_unlock(&dist->its.lock);
>+	if (!itte)
>+		return E_ITS_INV_UNMAPPED_INTERRUPT;
>+
>+	/*
>+	 * We cannot read from guest memory inside the spinlock, so we
>+	 * need to re-read our tables to learn whether the LPI number we are
>+	 * using is still valid.
>+	 */
>+	do {
>+		propbase = BASER_BASE_ADDRESS(dist->propbaser);
>+		ret = kvm_read_guest(kvm, propbase + itte->lpi - GIC_LPI_OFFSET,
>+				     &prop, 1);
>+		if (ret)
>+			return ret;
>+
>+		spin_lock(&dist->its.lock);
>+		new_itte = find_itte(kvm, device_id, event_id);
>+		if (new_itte->lpi != itte->lpi) {
>+			itte = new_itte;
>+			spin_unlock(&dist->its.lock);
>+			continue;
>+		}
>+		update_lpi_config(kvm, itte, prop);
>+		spin_unlock(&dist->its.lock);

Not possible to call the function defined early and update all. How much performance gain will this be?
>+	} while (0);
>+	return 0;
>+}
>+
>+/* The INVALL command requests flushing of all IRQ data in this collection. */
>+static int vits_cmd_handle_invall(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_dist *dist = &kvm->arch.vgic;
>+	u64 prop_base_reg, pend_base_reg;
>+	u32 coll_id = its_cmd_get_collection(its_cmd);
>+	struct its_collection *collection;
>+	struct kvm_vcpu *vcpu;
>+
>+	collection = find_collection(kvm, coll_id);
>+	if (!its_is_collection_mapped(collection))
>+		return E_ITS_INVALL_UNMAPPED_COLLECTION;
>+
>+	vcpu = kvm_get_vcpu(kvm, collection->target_addr);
>+
>+	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
>+	prop_base_reg = dist->propbaser;
>+
>+	its_update_lpis_configuration(kvm, prop_base_reg);
>+	its_sync_lpi_pending_table(vcpu, pend_base_reg);
>+
>+	return 0;
>+}
>+
>+/* The MOVALL command moves all IRQs from one redistributor to another. */
>+static int vits_cmd_handle_movall(struct kvm *kvm, u64 *its_cmd)
>+{
>+	struct vgic_its *its = &kvm->arch.vgic.its;
>+	u32 target1_addr = its_cmd_get_target_addr(its_cmd);
>+	u32 target2_addr = its_cmd_mask_field(its_cmd, 3, 16, 32);
>+	struct its_collection *collection;
>+
>+	if (target1_addr >= atomic_read(&kvm->online_vcpus) ||
>+	    target2_addr >= atomic_read(&kvm->online_vcpus))
>+		return E_ITS_MOVALL_PROCNUM_OOR;
>+
>+	if (target1_addr == target2_addr)
>+		return 0;
>+
>+	spin_lock(&its->lock);
>+	list_for_each_entry(collection, &its->collection_list,
>+			    coll_list) {
>+		if (collection && collection->target_addr == target1_addr)
>+			collection->target_addr = target2_addr;
>+		update_affinity(kvm, collection);
>+	}

Not planning to send the commands down in case of VFIO? Think about NUMA systems and performance impact. 
>+
>+	spin_unlock(&its->lock);
>+	return 0;
>+}
>+
> /*
>  * This function is called with both the ITS and the distributor lock dropped,
>  * so the actual command handlers must take the respective locks when needed.
>  */
> static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
> {
>-	return -ENODEV;
>+	u8 cmd = its_cmd_get_command(its_cmd);
>+	int ret = -ENODEV;
>+
>+	switch (cmd) {
>+	case GITS_CMD_MAPD:
>+		ret = vits_cmd_handle_mapd(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_MAPC:
>+		ret = vits_cmd_handle_mapc(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_MAPI:
>+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
>+		break;
>+	case GITS_CMD_MAPTI:
>+		ret = vits_cmd_handle_mapi(vcpu->kvm, its_cmd, cmd);
>+		break;
>+	case GITS_CMD_MOVI:
>+		ret = vits_cmd_handle_movi(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_DISCARD:
>+		ret = vits_cmd_handle_discard(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_CLEAR:
>+		ret = vits_cmd_handle_clear(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_MOVALL:
>+		ret = vits_cmd_handle_movall(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_INV:
>+		ret = vits_cmd_handle_inv(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_INVALL:
>+		ret = vits_cmd_handle_invall(vcpu->kvm, its_cmd);
>+		break;
>+	case GITS_CMD_SYNC:
>+		/* we ignore this command: we are in sync all of the time */
>+		ret = 0;
>+		break;
>+	}
>+
>+	return ret;
> }
> 
> static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>-- 
>2.7.3
>
>_______________________________________________
>kvmarm mailing list
>kvmarm at lists.cs.columbia.edu
>https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-05-05 18:08     ` Chalamarla, Tirumalesh
@ 2016-05-09 15:47       ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-09 15:47 UTC (permalink / raw)
  To: Chalamarla, Tirumalesh, Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

On 05/05/16 19:08, Chalamarla, Tirumalesh wrote:
> 
> 
> 
> 
> 
> On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:
> 
>> The ARM GICv3 ITS controller requires a separate register frame to
>> cover ITS specific registers. Add a new VGIC address type and store
>> the address in a field in the vgic_dist structure.
>> Provide a function to check whether userland has provided the address,
>> so ITS functionality can be guarded by that check.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>> include/kvm/vgic/vgic.h                        |  5 +++++
>> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>> 6 files changed, 34 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> index 59541d4..087e2d9 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> @@ -39,6 +39,15 @@ Groups:
>>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>       This address needs to be 64K aligned.
>>
>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>> +      Base address in the guest physical address space of the GICv3 ITS
>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>> +      injected into guests. This extension is optional, if the kernel
>> +      does not support the ITS, the call returns -ENODEV.
>> +      This memory is solely for the guest to access the ITS control
>> +      registers and does not cover the ITS translation register.
>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>> +      This address needs to be 64K aligned and the region covers 64 KByte.
>>
>>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>   Attributes:
>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>> index f209ea1..c2b257d 100644
>> --- a/arch/arm64/include/uapi/asm/kvm.h
>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>> @@ -87,9 +87,11 @@ struct kvm_regs {
>> /* Supported VGICv3 address types  */
>> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>
>> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>
>> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 7c1d145..11344e6 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -135,6 +135,9 @@ struct vgic_dist {
>> 		gpa_t			vgic_redist_base;
>> 	};
>>
>> +	/* The base address of the ITS control register frame */
>> +	gpa_t			vgic_its_base;
>> +
>> 	/* distributor enabled */
>> 	u32			enabled;
>>
>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>> 	return kvm_vgic_global_state.max_gic_vcpus;
>> }
>>
>> +bool vgic_has_its(struct kvm *kvm);
>> +
>> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index 65395af..1de2478 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>
>> 	return map_is_active;
>> }
>> +
>> +bool vgic_has_its(struct kvm *kvm)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +
>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>> +		return false;
>> +
>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index ac655b5..2301e03 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>
>> out_unlock:
>> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> index 7f78a16..3ec2ac3 100644
>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
>> 		alignment = SZ_64K;
>> 		break;
>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>> +		addr_ptr = &vgic->vgic_its_base;
>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>> +		alignment = SZ_64K;
>> +		break;
> 
> Should there be a check based on number of vcpus? 

Why would the number of vcpus influence the size of the ITS?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-05-09 15:47       ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-09 15:47 UTC (permalink / raw)
  To: linux-arm-kernel

On 05/05/16 19:08, Chalamarla, Tirumalesh wrote:
> 
> 
> 
> 
> 
> On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:
> 
>> The ARM GICv3 ITS controller requires a separate register frame to
>> cover ITS specific registers. Add a new VGIC address type and store
>> the address in a field in the vgic_dist structure.
>> Provide a function to check whether userland has provided the address,
>> so ITS functionality can be guarded by that check.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>> include/kvm/vgic/vgic.h                        |  5 +++++
>> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>> 6 files changed, 34 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> index 59541d4..087e2d9 100644
>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>> @@ -39,6 +39,15 @@ Groups:
>>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>       This address needs to be 64K aligned.
>>
>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>> +      Base address in the guest physical address space of the GICv3 ITS
>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>> +      injected into guests. This extension is optional, if the kernel
>> +      does not support the ITS, the call returns -ENODEV.
>> +      This memory is solely for the guest to access the ITS control
>> +      registers and does not cover the ITS translation register.
>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>> +      This address needs to be 64K aligned and the region covers 64 KByte.
>>
>>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>   Attributes:
>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>> index f209ea1..c2b257d 100644
>> --- a/arch/arm64/include/uapi/asm/kvm.h
>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>> @@ -87,9 +87,11 @@ struct kvm_regs {
>> /* Supported VGICv3 address types  */
>> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>
>> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>
>> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index 7c1d145..11344e6 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -135,6 +135,9 @@ struct vgic_dist {
>> 		gpa_t			vgic_redist_base;
>> 	};
>>
>> +	/* The base address of the ITS control register frame */
>> +	gpa_t			vgic_its_base;
>> +
>> 	/* distributor enabled */
>> 	u32			enabled;
>>
>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>> 	return kvm_vgic_global_state.max_gic_vcpus;
>> }
>>
>> +bool vgic_has_its(struct kvm *kvm);
>> +
>> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index 65395af..1de2478 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>
>> 	return map_is_active;
>> }
>> +
>> +bool vgic_has_its(struct kvm *kvm)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +
>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>> +		return false;
>> +
>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>> index ac655b5..2301e03 100644
>> --- a/virt/kvm/arm/vgic/vgic_init.c
>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>
>> out_unlock:
>> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> index 7f78a16..3ec2ac3 100644
>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
>> 		alignment = SZ_64K;
>> 		break;
>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>> +		addr_ptr = &vgic->vgic_its_base;
>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>> +		alignment = SZ_64K;
>> +		break;
> 
> Should there be a check based on number of vcpus? 

Why would the number of vcpus influence the size of the ITS?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-05-09 15:47       ` Marc Zyngier
@ 2016-05-09 16:53         ` Chalamarla, Tirumalesh
  -1 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-09 16:53 UTC (permalink / raw)
  To: Marc Zyngier, Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm






On 5/9/16, 8:47 AM, "Marc Zyngier" <marc.zyngier@arm.com> wrote:

>On 05/05/16 19:08, Chalamarla, Tirumalesh wrote:
>> 
>> 
>> 
>> 
>> 
>> On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:
>> 
>>> The ARM GICv3 ITS controller requires a separate register frame to
>>> cover ITS specific registers. Add a new VGIC address type and store
>>> the address in a field in the vgic_dist structure.
>>> Provide a function to check whether userland has provided the address,
>>> so ITS functionality can be guarded by that check.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>>> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>>> include/kvm/vgic/vgic.h                        |  5 +++++
>>> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>>> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>>> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>>> 6 files changed, 34 insertions(+)
>>>
>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>> index 59541d4..087e2d9 100644
>>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>> @@ -39,6 +39,15 @@ Groups:
>>>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>>       This address needs to be 64K aligned.
>>>
>>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>>> +      Base address in the guest physical address space of the GICv3 ITS
>>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>>> +      injected into guests. This extension is optional, if the kernel
>>> +      does not support the ITS, the call returns -ENODEV.
>>> +      This memory is solely for the guest to access the ITS control
>>> +      registers and does not cover the ITS translation register.
>>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>> +      This address needs to be 64K aligned and the region covers 64 KByte.
>>>
>>>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>>   Attributes:
>>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>>> index f209ea1..c2b257d 100644
>>> --- a/arch/arm64/include/uapi/asm/kvm.h
>>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>>> @@ -87,9 +87,11 @@ struct kvm_regs {
>>> /* Supported VGICv3 address types  */
>>> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>>> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>>
>>> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>>> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>>
>>> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>>> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>> index 7c1d145..11344e6 100644
>>> --- a/include/kvm/vgic/vgic.h
>>> +++ b/include/kvm/vgic/vgic.h
>>> @@ -135,6 +135,9 @@ struct vgic_dist {
>>> 		gpa_t			vgic_redist_base;
>>> 	};
>>>
>>> +	/* The base address of the ITS control register frame */
>>> +	gpa_t			vgic_its_base;
>>> +
>>> 	/* distributor enabled */
>>> 	u32			enabled;
>>>
>>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>> 	return kvm_vgic_global_state.max_gic_vcpus;
>>> }
>>>
>>> +bool vgic_has_its(struct kvm *kvm);
>>> +
>>> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>>> index 65395af..1de2478 100644
>>> --- a/virt/kvm/arm/vgic/vgic.c
>>> +++ b/virt/kvm/arm/vgic/vgic.c
>>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>>
>>> 	return map_is_active;
>>> }
>>> +
>>> +bool vgic_has_its(struct kvm *kvm)
>>> +{
>>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>> +
>>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>>> +		return false;
>>> +
>>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>>> +}
>>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>>> index ac655b5..2301e03 100644
>>> --- a/virt/kvm/arm/vgic/vgic_init.c
>>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>>> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>>> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>>> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>>
>>> out_unlock:
>>> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>> index 7f78a16..3ec2ac3 100644
>>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>>> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
>>> 		alignment = SZ_64K;
>>> 		break;
>>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>>> +		addr_ptr = &vgic->vgic_its_base;
>>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>>> +		alignment = SZ_64K;
>>> +		break;
>> 
>> Should there be a check based on number of vcpus? 
>
>Why would the number of vcpus influence the size of the ITS?

does the re-distributor per VCPU thing? I think its worth to add a check for re distributor size. 
>
>Thanks,
>
>	M.
>-- 
>Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-05-09 16:53         ` Chalamarla, Tirumalesh
  0 siblings, 0 replies; 120+ messages in thread
From: Chalamarla, Tirumalesh @ 2016-05-09 16:53 UTC (permalink / raw)
  To: linux-arm-kernel






On 5/9/16, 8:47 AM, "Marc Zyngier" <marc.zyngier@arm.com> wrote:

>On 05/05/16 19:08, Chalamarla, Tirumalesh wrote:
>> 
>> 
>> 
>> 
>> 
>> On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:
>> 
>>> The ARM GICv3 ITS controller requires a separate register frame to
>>> cover ITS specific registers. Add a new VGIC address type and store
>>> the address in a field in the vgic_dist structure.
>>> Provide a function to check whether userland has provided the address,
>>> so ITS functionality can be guarded by that check.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>>> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>>> include/kvm/vgic/vgic.h                        |  5 +++++
>>> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>>> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>>> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>>> 6 files changed, 34 insertions(+)
>>>
>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>> index 59541d4..087e2d9 100644
>>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>> @@ -39,6 +39,15 @@ Groups:
>>>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>>       This address needs to be 64K aligned.
>>>
>>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>>> +      Base address in the guest physical address space of the GICv3 ITS
>>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>>> +      injected into guests. This extension is optional, if the kernel
>>> +      does not support the ITS, the call returns -ENODEV.
>>> +      This memory is solely for the guest to access the ITS control
>>> +      registers and does not cover the ITS translation register.
>>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>> +      This address needs to be 64K aligned and the region covers 64 KByte.
>>>
>>>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>>   Attributes:
>>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>>> index f209ea1..c2b257d 100644
>>> --- a/arch/arm64/include/uapi/asm/kvm.h
>>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>>> @@ -87,9 +87,11 @@ struct kvm_regs {
>>> /* Supported VGICv3 address types  */
>>> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>>> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>>
>>> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>>> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>>
>>> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>>> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>> index 7c1d145..11344e6 100644
>>> --- a/include/kvm/vgic/vgic.h
>>> +++ b/include/kvm/vgic/vgic.h
>>> @@ -135,6 +135,9 @@ struct vgic_dist {
>>> 		gpa_t			vgic_redist_base;
>>> 	};
>>>
>>> +	/* The base address of the ITS control register frame */
>>> +	gpa_t			vgic_its_base;
>>> +
>>> 	/* distributor enabled */
>>> 	u32			enabled;
>>>
>>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>> 	return kvm_vgic_global_state.max_gic_vcpus;
>>> }
>>>
>>> +bool vgic_has_its(struct kvm *kvm);
>>> +
>>> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>>> index 65395af..1de2478 100644
>>> --- a/virt/kvm/arm/vgic/vgic.c
>>> +++ b/virt/kvm/arm/vgic/vgic.c
>>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>>
>>> 	return map_is_active;
>>> }
>>> +
>>> +bool vgic_has_its(struct kvm *kvm)
>>> +{
>>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>> +
>>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>>> +		return false;
>>> +
>>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>>> +}
>>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>>> index ac655b5..2301e03 100644
>>> --- a/virt/kvm/arm/vgic/vgic_init.c
>>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>>> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>>> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>>> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>>
>>> out_unlock:
>>> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>> index 7f78a16..3ec2ac3 100644
>>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>>> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
>>> 		alignment = SZ_64K;
>>> 		break;
>>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>>> +		addr_ptr = &vgic->vgic_its_base;
>>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>>> +		alignment = SZ_64K;
>>> +		break;
>> 
>> Should there be a check based on number of vcpus? 
>
>Why would the number of vcpus influence the size of the ITS?

does the re-distributor per VCPU thing? I think its worth to add a check for re distributor size. 
>
>Thanks,
>
>	M.
>-- 
>Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
  2016-05-09 16:53         ` Chalamarla, Tirumalesh
@ 2016-05-09 17:09           ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-09 17:09 UTC (permalink / raw)
  To: Chalamarla, Tirumalesh, Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

On 09/05/16 17:53, Chalamarla, Tirumalesh wrote:
> 
> 
> 
> 
> 
> On 5/9/16, 8:47 AM, "Marc Zyngier" <marc.zyngier@arm.com> wrote:
> 
>> On 05/05/16 19:08, Chalamarla, Tirumalesh wrote:
>>>
>>>
>>>
>>>
>>>
>>> On 3/25/16, 7:14 PM, "kvmarm-bounces@lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces@lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:
>>>
>>>> The ARM GICv3 ITS controller requires a separate register frame to
>>>> cover ITS specific registers. Add a new VGIC address type and store
>>>> the address in a field in the vgic_dist structure.
>>>> Provide a function to check whether userland has provided the address,
>>>> so ITS functionality can be guarded by that check.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>> ---
>>>> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>>>> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>>>> include/kvm/vgic/vgic.h                        |  5 +++++
>>>> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>>>> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>>>> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>>>> 6 files changed, 34 insertions(+)
>>>>
>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>>> index 59541d4..087e2d9 100644
>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>>> @@ -39,6 +39,15 @@ Groups:
>>>>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>>>       This address needs to be 64K aligned.
>>>>
>>>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>>>> +      Base address in the guest physical address space of the GICv3 ITS
>>>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>>>> +      injected into guests. This extension is optional, if the kernel
>>>> +      does not support the ITS, the call returns -ENODEV.
>>>> +      This memory is solely for the guest to access the ITS control
>>>> +      registers and does not cover the ITS translation register.
>>>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>>> +      This address needs to be 64K aligned and the region covers 64 KByte.
>>>>
>>>>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>>>   Attributes:
>>>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>>>> index f209ea1..c2b257d 100644
>>>> --- a/arch/arm64/include/uapi/asm/kvm.h
>>>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>>>> @@ -87,9 +87,11 @@ struct kvm_regs {
>>>> /* Supported VGICv3 address types  */
>>>> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>>>> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>>>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>>>
>>>> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>>>> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>>>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>>>
>>>> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>>>> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>>> index 7c1d145..11344e6 100644
>>>> --- a/include/kvm/vgic/vgic.h
>>>> +++ b/include/kvm/vgic/vgic.h
>>>> @@ -135,6 +135,9 @@ struct vgic_dist {
>>>> 		gpa_t			vgic_redist_base;
>>>> 	};
>>>>
>>>> +	/* The base address of the ITS control register frame */
>>>> +	gpa_t			vgic_its_base;
>>>> +
>>>> 	/* distributor enabled */
>>>> 	u32			enabled;
>>>>
>>>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>>> 	return kvm_vgic_global_state.max_gic_vcpus;
>>>> }
>>>>
>>>> +bool vgic_has_its(struct kvm *kvm);
>>>> +
>>>> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>>>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>>>> index 65395af..1de2478 100644
>>>> --- a/virt/kvm/arm/vgic/vgic.c
>>>> +++ b/virt/kvm/arm/vgic/vgic.c
>>>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>>>
>>>> 	return map_is_active;
>>>> }
>>>> +
>>>> +bool vgic_has_its(struct kvm *kvm)
>>>> +{
>>>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>>> +
>>>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>>>> +		return false;
>>>> +
>>>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>>>> +}
>>>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>>>> index ac655b5..2301e03 100644
>>>> --- a/virt/kvm/arm/vgic/vgic_init.c
>>>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>>>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>>>> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>>>> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>>>> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>>>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>>>
>>>> out_unlock:
>>>> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>>>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>>> index 7f78a16..3ec2ac3 100644
>>>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>>>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>>>> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
>>>> 		alignment = SZ_64K;
>>>> 		break;
>>>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>>>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>>>> +		addr_ptr = &vgic->vgic_its_base;
>>>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>>>> +		alignment = SZ_64K;
>>>> +		break;
>>>
>>> Should there be a check based on number of vcpus? 
>>
>> Why would the number of vcpus influence the size of the ITS?
> 
> does the re-distributor per VCPU thing? I think its worth to add a
> check for re distributor size.

This patch is implementing the ITS, not the redistributors.

If you have any comment on the GICv3 implementation (of which the
redistributors are part of), please comment on the relevant patch (see
the 55 patch series that has been posted on Friday).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address
@ 2016-05-09 17:09           ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-09 17:09 UTC (permalink / raw)
  To: linux-arm-kernel

On 09/05/16 17:53, Chalamarla, Tirumalesh wrote:
> 
> 
> 
> 
> 
> On 5/9/16, 8:47 AM, "Marc Zyngier" <marc.zyngier@arm.com> wrote:
> 
>> On 05/05/16 19:08, Chalamarla, Tirumalesh wrote:
>>>
>>>
>>>
>>>
>>>
>>> On 3/25/16, 7:14 PM, "kvmarm-bounces at lists.cs.columbia.edu on behalf of Andre Przywara" <kvmarm-bounces at lists.cs.columbia.edu on behalf of andre.przywara@arm.com> wrote:
>>>
>>>> The ARM GICv3 ITS controller requires a separate register frame to
>>>> cover ITS specific registers. Add a new VGIC address type and store
>>>> the address in a field in the vgic_dist structure.
>>>> Provide a function to check whether userland has provided the address,
>>>> so ITS functionality can be guarded by that check.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>> ---
>>>> Documentation/virtual/kvm/devices/arm-vgic.txt |  9 +++++++++
>>>> arch/arm64/include/uapi/asm/kvm.h              |  2 ++
>>>> include/kvm/vgic/vgic.h                        |  5 +++++
>>>> virt/kvm/arm/vgic/vgic.c                       | 10 ++++++++++
>>>> virt/kvm/arm/vgic/vgic_init.c                  |  1 +
>>>> virt/kvm/arm/vgic/vgic_kvm_device.c            |  7 +++++++
>>>> 6 files changed, 34 insertions(+)
>>>>
>>>> diff --git a/Documentation/virtual/kvm/devices/arm-vgic.txt b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>>> index 59541d4..087e2d9 100644
>>>> --- a/Documentation/virtual/kvm/devices/arm-vgic.txt
>>>> +++ b/Documentation/virtual/kvm/devices/arm-vgic.txt
>>>> @@ -39,6 +39,15 @@ Groups:
>>>>       Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>>>       This address needs to be 64K aligned.
>>>>
>>>> +    KVM_VGIC_V3_ADDR_TYPE_ITS (rw, 64-bit)
>>>> +      Base address in the guest physical address space of the GICv3 ITS
>>>> +      control register frame. The ITS allows MSI(-X) interrupts to be
>>>> +      injected into guests. This extension is optional, if the kernel
>>>> +      does not support the ITS, the call returns -ENODEV.
>>>> +      This memory is solely for the guest to access the ITS control
>>>> +      registers and does not cover the ITS translation register.
>>>> +      Only valid for KVM_DEV_TYPE_ARM_VGIC_V3.
>>>> +      This address needs to be 64K aligned and the region covers 64 KByte.
>>>>
>>>>   KVM_DEV_ARM_VGIC_GRP_DIST_REGS
>>>>   Attributes:
>>>> diff --git a/arch/arm64/include/uapi/asm/kvm.h b/arch/arm64/include/uapi/asm/kvm.h
>>>> index f209ea1..c2b257d 100644
>>>> --- a/arch/arm64/include/uapi/asm/kvm.h
>>>> +++ b/arch/arm64/include/uapi/asm/kvm.h
>>>> @@ -87,9 +87,11 @@ struct kvm_regs {
>>>> /* Supported VGICv3 address types  */
>>>> #define KVM_VGIC_V3_ADDR_TYPE_DIST	2
>>>> #define KVM_VGIC_V3_ADDR_TYPE_REDIST	3
>>>> +#define KVM_VGIC_V3_ADDR_TYPE_ITS	4
>>>>
>>>> #define KVM_VGIC_V3_DIST_SIZE		SZ_64K
>>>> #define KVM_VGIC_V3_REDIST_SIZE		(2 * SZ_64K)
>>>> +#define KVM_VGIC_V3_ITS_SIZE		SZ_64K
>>>>
>>>> #define KVM_ARM_VCPU_POWER_OFF		0 /* CPU is started in OFF state */
>>>> #define KVM_ARM_VCPU_EL1_32BIT		1 /* CPU running a 32bit VM */
>>>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>>>> index 7c1d145..11344e6 100644
>>>> --- a/include/kvm/vgic/vgic.h
>>>> +++ b/include/kvm/vgic/vgic.h
>>>> @@ -135,6 +135,9 @@ struct vgic_dist {
>>>> 		gpa_t			vgic_redist_base;
>>>> 	};
>>>>
>>>> +	/* The base address of the ITS control register frame */
>>>> +	gpa_t			vgic_its_base;
>>>> +
>>>> 	/* distributor enabled */
>>>> 	u32			enabled;
>>>>
>>>> @@ -253,4 +256,6 @@ static inline int kvm_vgic_get_max_vcpus(void)
>>>> 	return kvm_vgic_global_state.max_gic_vcpus;
>>>> }
>>>>
>>>> +bool vgic_has_its(struct kvm *kvm);
>>>> +
>>>> #endif /* __ASM_ARM_KVM_VGIC_VGIC_H */
>>>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>>>> index 65395af..1de2478 100644
>>>> --- a/virt/kvm/arm/vgic/vgic.c
>>>> +++ b/virt/kvm/arm/vgic/vgic.c
>>>> @@ -612,3 +612,13 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, u32 intid)
>>>>
>>>> 	return map_is_active;
>>>> }
>>>> +
>>>> +bool vgic_has_its(struct kvm *kvm)
>>>> +{
>>>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>>> +
>>>> +	if (dist->vgic_model != KVM_DEV_TYPE_ARM_VGIC_V3)
>>>> +		return false;
>>>> +
>>>> +	return !IS_VGIC_ADDR_UNDEF(dist->vgic_its_base);
>>>> +}
>>>> diff --git a/virt/kvm/arm/vgic/vgic_init.c b/virt/kvm/arm/vgic/vgic_init.c
>>>> index ac655b5..2301e03 100644
>>>> --- a/virt/kvm/arm/vgic/vgic_init.c
>>>> +++ b/virt/kvm/arm/vgic/vgic_init.c
>>>> @@ -132,6 +132,7 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
>>>> 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>>>> 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>>>> 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
>>>> +	kvm->arch.vgic.vgic_its_base = VGIC_ADDR_UNDEF;
>>>>
>>>> out_unlock:
>>>> 	for (; vcpu_lock_idx >= 0; vcpu_lock_idx--) {
>>>> diff --git a/virt/kvm/arm/vgic/vgic_kvm_device.c b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>>> index 7f78a16..3ec2ac3 100644
>>>> --- a/virt/kvm/arm/vgic/vgic_kvm_device.c
>>>> +++ b/virt/kvm/arm/vgic/vgic_kvm_device.c
>>>> @@ -109,6 +109,12 @@ int kvm_vgic_addr(struct kvm *kvm, unsigned long type, u64 *addr, bool write)
>>>> 		block_size = KVM_VGIC_V3_REDIST_SIZE;
>>>> 		alignment = SZ_64K;
>>>> 		break;
>>>> +	case KVM_VGIC_V3_ADDR_TYPE_ITS:
>>>> +		type_needed = KVM_DEV_TYPE_ARM_VGIC_V3;
>>>> +		addr_ptr = &vgic->vgic_its_base;
>>>> +		block_size = KVM_VGIC_V3_ITS_SIZE;
>>>> +		alignment = SZ_64K;
>>>> +		break;
>>>
>>> Should there be a check based on number of vcpus? 
>>
>> Why would the number of vcpus influence the size of the ITS?
> 
> does the re-distributor per VCPU thing? I think its worth to add a
> check for re distributor size.

This patch is implementing the ITS, not the redistributors.

If you have any comment on the GICv3 implementation (of which the
redistributors are part of), please comment on the relevant patch (see
the 55 patch series that has been posted on Friday).

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-04-07 14:35     ` Marc Zyngier
@ 2016-05-25 11:37       ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 11:37 UTC (permalink / raw)
  To: Marc Zyngier, Christoffer Dall, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

Hi Marc,

this series is mostly obsolete now, but I wanted to elaborate on the
locking scheme, which hasn't changed.

On 07/04/16 15:35, Marc Zyngier wrote:
> On 26/03/16 02:14, Andre Przywara wrote:

....

>> --- a/virt/kvm/arm/vgic/its-emul.c
>> +++ b/virt/kvm/arm/vgic/its-emul.c
>> @@ -31,23 +31,263 @@
>>  #include "vgic.h"
>>  #include "vgic_mmio.h"
>>  
>> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>> +
>> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>> +				   struct kvm_io_device *this,
>> +				   gpa_t addr, int len, void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +	u32 reg;
>> +
>> +	reg = GITS_CTLR_QUIESCENT;
> 
> So your ITS is always in a quiescent state? Even when you're processing
> the command queue? You'll have to convince me...

Does QUIESCENT actually cover the command queue status? I was under the
impression that this is purely to cope with distributed implementations
and the possible delays caused by propagating commands.
If not so, I can surely take the lock here and respect
(creadr == cwriter) upon setting the bit.

> 
>> +	if (its->enabled)
>> +		reg |= GITS_CTLR_ENABLE;
>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
>> +				    struct kvm_io_device *this,
>> +				    gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +
>> +        if (addr - iodev->base_addr == 0)
> 
> whitespace issue.
> 
>> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>> +				    struct kvm_io_device *this,
>> +				    gpa_t addr, int len, void *val)
>> +{
>> +	u64 reg = GITS_TYPER_PLPIS;
>> +
>> +	/*
>> +	 * We use linear CPU numbers for redistributor addressing,
>> +	 * so GITS_TYPER.PTA is 0.
>> +	 * To avoid memory waste on the guest side, we keep the
>> +	 * number of IDBits and DevBits low for the time being.
>> +	 * This could later be made configurable by userland.
>> +	 * Since we have all collections in linked list, we claim
>> +	 * that we can hold all of the collection tables in our
>> +	 * own memory and that the ITT entry size is 1 byte (the
>> +	 * smallest possible one).
> 
> All of this is going to bite us when we want to implement migration,
> specially the HW collection bit.

How so? Clearly we keep the number of VCPUs constant, so everything
guest visible stays the same even upon migrating to a much different
host? Or are we talking about different things here?

> 
>> +	 */
>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>> +
>> +	write_mask64(reg, addr & 7, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
>> +				   struct kvm_io_device *this,
>> +				   gpa_t addr, int len, void *val)
>> +{
>> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>> +				     struct kvm_io_device *this,
>> +				     gpa_t addr, int len, void *val)
>> +{
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +	u32 reg = 0;
>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>> +
>> +	switch (idreg) {
>> +	case GITS_PIDR2:
>> +		reg = GIC_PIDR2_ARCH_GICv3;
> 
> Are we leaving the lowest 4 bits to zero?

I couldn't stuff "42" in 4 bits, so: yes ;-)

>> +		break;
>> +	case GITS_PIDR4:
>> +		/* This is a 64K software visible page */
>> +		reg = 0x40;
> 
> Same question.
> 
> Also, how about all the others PIDR registers?

I guess I was halfway undecided between going with the pure
architectural requirements (which only mandate bits 7:4 in PIDR2) and
complying with the recommendation later in the spec.
So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.

>> +		break;
>> +	/* Those are the ID registers for (any) GIC. */
>> +	case GITS_CIDR0:
>> +		reg = 0x0d;
>> +		break;
>> +	case GITS_CIDR1:
>> +		reg = 0xf0;
>> +		break;
>> +	case GITS_CIDR2:
>> +		reg = 0x05;
>> +		break;
>> +	case GITS_CIDR3:
>> +		reg = 0xb1;
>> +		break;
>> +	}
> 
> Given that these values are directly taken from the architecture, and
> seem common to the whole GICv3 architecture when implemented by ARM, we
> could have a common handler for the whole GICv3 implementatuin. Not a
> bit deal though.

Agreed.

>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +/*
>> + * This function is called with both the ITS and the distributor lock dropped,
>> + * so the actual command handlers must take the respective locks when needed.
>> + */
>> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
>> +{
>> +	return -ENODEV;
>> +}
>> +
>> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>> +				    struct kvm_io_device *this,
>> +				    gpa_t addr, int len, void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +
>> +	write_mask64(its->cbaser, addr & 7, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
>> +				      struct kvm_io_device *this,
>> +				      gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +
>> +	if (its->enabled)
>> +		return 0;
>> +
>> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
>> +	its->creadr = 0;
> 
> Don't you need to acquire the command queue lock here?

I don't think so, because we "return 0;" above if the ITS is enabled,
which means that nothing is going on at the moment.
But I guess I can just take the lock anyway to be sure.

>> +
>> +	return 0;
>> +}
>> +
>> +static int its_cmd_buffer_size(struct kvm *kvm)
>> +{
>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>> +
>> +	return ((its->cbaser & 0xff) + 1) << 12;
>> +}
>> +
>> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
>> +{
>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>> +
>> +	return BASER_BASE_ADDRESS(its->cbaser);
>> +}
>> +
>> +/*
>> + * By writing to CWRITER the guest announces new commands to be processed.
>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>> + * iterate over the command buffer (with the lock dropped) until the read
>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>> + * meantime will just update the write pointer, leaving the command
>> + * processing to the first instance of the function.
>> + */
>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>> +				       struct kvm_io_device *this,
>> +				       gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_its *its = &dist->its;
>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>> +	u64 cmd_buf[4];
>> +	u32 reg;
>> +	bool finished;
>> +
>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>> +	reg &= 0xfffe0;
>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>> +		return 0;
>> +
>> +	spin_lock(&its->lock);
>> +
>> +	/*
>> +	 * If there is still another VCPU handling commands, let this
>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>> +	 */
> 
> How do you detect that condition? All I see is a massive race here, with
> two threads processing the queue in parallel, possibly corrupting each
> other's data.
> 
> Please explain why you think this is safe.

OK, here you go: Actually we are handling the command queue
synchronously: The first writer to CWRITER will end up here and will
iterate over all commands below. From the guests point of view it will
take some time to do the CWRITER MMIO write, but upon the writel
returning the queue is actually already empty again.
That means that any _sane_ guest will _never_ trigger the case where two
threads/VCPUs are handling the queue, because a concurrent write to
CWRITER without a lock in the guest would be broken by design.

However I am not sure we can rely on this, so I added this precaution
scheme:
The first writer takes our ITS (emulation) lock and examines cwriter and
creadr to see if they are equal. If they are, then the queue is
currently empty, which means we are the first one and need to take care
of the commands. We update our copy of cwriter (now marking the queue as
non-empty) and drop the lock. As we kept the queue-was-empty status in a
local variable, we now can start iterating through the queue. We only
take the lock briefly when really needed in this process (for instance
when one command has been processed and creadr is incremented).

If now a second VCPU writes CWRITER, we also take the lock and compare
creadr and cwriter. Now they are different, because the first thread is
still busy with handling the commands and hasn't finished yet.
So we set our local "finished" to true. Also we update cwriter to the
new value, then drop the lock. The while loop below will not be entered
in this thread and we return to the guest.
Now the first thread has handled another command, takes the lock to
increase creadr and compares it with cwriter. Without the second writer
it may have been finished already, but now cwriter has grown, so it will
just carry on with handling the commands.

A bit like someone washing the dishes while another person adds some
more dirty plates to the pile ;-)

But again: this is just insurance against broken guests and due to guest
locking requirements this would actually never happen.

Does that make sense?

Btw: I have finished rebasing this upon the new VGIC and will post a new
version after -rc1.

Cheers,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-05-25 11:37       ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 11:37 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Marc,

this series is mostly obsolete now, but I wanted to elaborate on the
locking scheme, which hasn't changed.

On 07/04/16 15:35, Marc Zyngier wrote:
> On 26/03/16 02:14, Andre Przywara wrote:

....

>> --- a/virt/kvm/arm/vgic/its-emul.c
>> +++ b/virt/kvm/arm/vgic/its-emul.c
>> @@ -31,23 +31,263 @@
>>  #include "vgic.h"
>>  #include "vgic_mmio.h"
>>  
>> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>> +
>> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>> +				   struct kvm_io_device *this,
>> +				   gpa_t addr, int len, void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +	u32 reg;
>> +
>> +	reg = GITS_CTLR_QUIESCENT;
> 
> So your ITS is always in a quiescent state? Even when you're processing
> the command queue? You'll have to convince me...

Does QUIESCENT actually cover the command queue status? I was under the
impression that this is purely to cope with distributed implementations
and the possible delays caused by propagating commands.
If not so, I can surely take the lock here and respect
(creadr == cwriter) upon setting the bit.

> 
>> +	if (its->enabled)
>> +		reg |= GITS_CTLR_ENABLE;
>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
>> +				    struct kvm_io_device *this,
>> +				    gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +
>> +        if (addr - iodev->base_addr == 0)
> 
> whitespace issue.
> 
>> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>> +				    struct kvm_io_device *this,
>> +				    gpa_t addr, int len, void *val)
>> +{
>> +	u64 reg = GITS_TYPER_PLPIS;
>> +
>> +	/*
>> +	 * We use linear CPU numbers for redistributor addressing,
>> +	 * so GITS_TYPER.PTA is 0.
>> +	 * To avoid memory waste on the guest side, we keep the
>> +	 * number of IDBits and DevBits low for the time being.
>> +	 * This could later be made configurable by userland.
>> +	 * Since we have all collections in linked list, we claim
>> +	 * that we can hold all of the collection tables in our
>> +	 * own memory and that the ITT entry size is 1 byte (the
>> +	 * smallest possible one).
> 
> All of this is going to bite us when we want to implement migration,
> specially the HW collection bit.

How so? Clearly we keep the number of VCPUs constant, so everything
guest visible stays the same even upon migrating to a much different
host? Or are we talking about different things here?

> 
>> +	 */
>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>> +
>> +	write_mask64(reg, addr & 7, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
>> +				   struct kvm_io_device *this,
>> +				   gpa_t addr, int len, void *val)
>> +{
>> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>> +				     struct kvm_io_device *this,
>> +				     gpa_t addr, int len, void *val)
>> +{
>> +	struct vgic_io_device *iodev = container_of(this,
>> +						    struct vgic_io_device, dev);
>> +	u32 reg = 0;
>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>> +
>> +	switch (idreg) {
>> +	case GITS_PIDR2:
>> +		reg = GIC_PIDR2_ARCH_GICv3;
> 
> Are we leaving the lowest 4 bits to zero?

I couldn't stuff "42" in 4 bits, so: yes ;-)

>> +		break;
>> +	case GITS_PIDR4:
>> +		/* This is a 64K software visible page */
>> +		reg = 0x40;
> 
> Same question.
> 
> Also, how about all the others PIDR registers?

I guess I was halfway undecided between going with the pure
architectural requirements (which only mandate bits 7:4 in PIDR2) and
complying with the recommendation later in the spec.
So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.

>> +		break;
>> +	/* Those are the ID registers for (any) GIC. */
>> +	case GITS_CIDR0:
>> +		reg = 0x0d;
>> +		break;
>> +	case GITS_CIDR1:
>> +		reg = 0xf0;
>> +		break;
>> +	case GITS_CIDR2:
>> +		reg = 0x05;
>> +		break;
>> +	case GITS_CIDR3:
>> +		reg = 0xb1;
>> +		break;
>> +	}
> 
> Given that these values are directly taken from the architecture, and
> seem common to the whole GICv3 architecture when implemented by ARM, we
> could have a common handler for the whole GICv3 implementatuin. Not a
> bit deal though.

Agreed.

>> +
>> +	write_mask32(reg, addr & 3, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +/*
>> + * This function is called with both the ITS and the distributor lock dropped,
>> + * so the actual command handlers must take the respective locks when needed.
>> + */
>> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
>> +{
>> +	return -ENODEV;
>> +}
>> +
>> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>> +				    struct kvm_io_device *this,
>> +				    gpa_t addr, int len, void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +
>> +	write_mask64(its->cbaser, addr & 7, len, val);
>> +
>> +	return 0;
>> +}
>> +
>> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
>> +				      struct kvm_io_device *this,
>> +				      gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>> +
>> +	if (its->enabled)
>> +		return 0;
>> +
>> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
>> +	its->creadr = 0;
> 
> Don't you need to acquire the command queue lock here?

I don't think so, because we "return 0;" above if the ITS is enabled,
which means that nothing is going on at the moment.
But I guess I can just take the lock anyway to be sure.

>> +
>> +	return 0;
>> +}
>> +
>> +static int its_cmd_buffer_size(struct kvm *kvm)
>> +{
>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>> +
>> +	return ((its->cbaser & 0xff) + 1) << 12;
>> +}
>> +
>> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
>> +{
>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>> +
>> +	return BASER_BASE_ADDRESS(its->cbaser);
>> +}
>> +
>> +/*
>> + * By writing to CWRITER the guest announces new commands to be processed.
>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>> + * iterate over the command buffer (with the lock dropped) until the read
>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>> + * meantime will just update the write pointer, leaving the command
>> + * processing to the first instance of the function.
>> + */
>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>> +				       struct kvm_io_device *this,
>> +				       gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_its *its = &dist->its;
>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>> +	u64 cmd_buf[4];
>> +	u32 reg;
>> +	bool finished;
>> +
>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>> +	reg &= 0xfffe0;
>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>> +		return 0;
>> +
>> +	spin_lock(&its->lock);
>> +
>> +	/*
>> +	 * If there is still another VCPU handling commands, let this
>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>> +	 */
> 
> How do you detect that condition? All I see is a massive race here, with
> two threads processing the queue in parallel, possibly corrupting each
> other's data.
> 
> Please explain why you think this is safe.

OK, here you go: Actually we are handling the command queue
synchronously: The first writer to CWRITER will end up here and will
iterate over all commands below. From the guests point of view it will
take some time to do the CWRITER MMIO write, but upon the writel
returning the queue is actually already empty again.
That means that any _sane_ guest will _never_ trigger the case where two
threads/VCPUs are handling the queue, because a concurrent write to
CWRITER without a lock in the guest would be broken by design.

However I am not sure we can rely on this, so I added this precaution
scheme:
The first writer takes our ITS (emulation) lock and examines cwriter and
creadr to see if they are equal. If they are, then the queue is
currently empty, which means we are the first one and need to take care
of the commands. We update our copy of cwriter (now marking the queue as
non-empty) and drop the lock. As we kept the queue-was-empty status in a
local variable, we now can start iterating through the queue. We only
take the lock briefly when really needed in this process (for instance
when one command has been processed and creadr is incremented).

If now a second VCPU writes CWRITER, we also take the lock and compare
creadr and cwriter. Now they are different, because the first thread is
still busy with handling the commands and hasn't finished yet.
So we set our local "finished" to true. Also we update cwriter to the
new value, then drop the lock. The while loop below will not be entered
in this thread and we return to the guest.
Now the first thread has handled another command, takes the lock to
increase creadr and compares it with cwriter. Without the second writer
it may have been finished already, but now cwriter has grown, so it will
just carry on with handling the commands.

A bit like someone washing the dishes while another person adds some
more dirty plates to the pile ;-)

But again: this is just insurance against broken guests and due to guest
locking requirements this would actually never happen.

Does that make sense?

Btw: I have finished rebasing this upon the new VGIC and will post a new
version after -rc1.

Cheers,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-04-06  9:36     ` Eric Auger
@ 2016-05-25 13:49       ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 13:49 UTC (permalink / raw)
  To: Eric Auger, Christoffer Dall, Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm

Hi Eric,

On 06/04/16 10:36, Eric Auger wrote:
> Hi Andre,
> On 03/26/2016 03:14 AM, Andre Przywara wrote:
>> Add emulation for some basic MMIO registers used in the ITS emulation.

...

>> +
>> +/*
>> + * By writing to CWRITER the guest announces new commands to be processed.
>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>> + * iterate over the command buffer (with the lock dropped) until the read
>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>> + * meantime will just update the write pointer, leaving the command
>> + * processing to the first instance of the function.
>> + */
>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>> +				       struct kvm_io_device *this,
>> +				       gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_its *its = &dist->its;
>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>> +	u64 cmd_buf[4];
>> +	u32 reg;
>> +	bool finished;
>> +
>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>> +	reg &= 0xfffe0;
>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>> +		return 0;
>> +
>> +	spin_lock(&its->lock);
>> +
>> +	/*
>> +	 * If there is still another VCPU handling commands, let this
>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>> +	 */
>> +	finished = (its->cwriter != its->creadr);
> empty?

Maybe that is indeed less misleading ...

>> +	its->cwriter = reg;
>> +
>> +	spin_unlock(&its->lock);
> Assuming we have 2 vcpus attempting a cwriter write at the same moment I
> don't understand what does guarantee they are not going to execute the
> following loop concurrently and possibly execute vits_handle_command
> twice on the same cmd from the same its->creadr start?

So does my explanation of the algorithm in that other mail today explain
it? Or do you still see a hole in the locking scheme here?

>> +
>> +	while (!finished) {
>> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
>> +					 cmd_buf, 32);
>> +		if (ret) {
>> +			/*
>> +			 * Gah, we are screwed. Reset CWRITER to that command
>> +			 * that we have finished processing and return.
>> +			 */
>> +			spin_lock(&its->lock);
>> +			its->cwriter = its->creadr;
> don't get why do you set the queue empty?

So from a guest's point of view this is something like an ITS internal
error which shouldn't happen. So I just bail out, but leave the queue in
a consistent (read: empty) state to allow possible future commands to be
processed at best effort.
I guess this is the best we can come up with, since there is no feasible
way of communicating errors back(?)

Cheers,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-05-25 13:49       ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 13:49 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Eric,

On 06/04/16 10:36, Eric Auger wrote:
> Hi Andre,
> On 03/26/2016 03:14 AM, Andre Przywara wrote:
>> Add emulation for some basic MMIO registers used in the ITS emulation.

...

>> +
>> +/*
>> + * By writing to CWRITER the guest announces new commands to be processed.
>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>> + * iterate over the command buffer (with the lock dropped) until the read
>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>> + * meantime will just update the write pointer, leaving the command
>> + * processing to the first instance of the function.
>> + */
>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>> +				       struct kvm_io_device *this,
>> +				       gpa_t addr, int len, const void *val)
>> +{
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	struct vgic_its *its = &dist->its;
>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>> +	u64 cmd_buf[4];
>> +	u32 reg;
>> +	bool finished;
>> +
>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>> +	reg &= 0xfffe0;
>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>> +		return 0;
>> +
>> +	spin_lock(&its->lock);
>> +
>> +	/*
>> +	 * If there is still another VCPU handling commands, let this
>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>> +	 */
>> +	finished = (its->cwriter != its->creadr);
> empty?

Maybe that is indeed less misleading ...

>> +	its->cwriter = reg;
>> +
>> +	spin_unlock(&its->lock);
> Assuming we have 2 vcpus attempting a cwriter write at the same moment I
> don't understand what does guarantee they are not going to execute the
> following loop concurrently and possibly execute vits_handle_command
> twice on the same cmd from the same its->creadr start?

So does my explanation of the algorithm in that other mail today explain
it? Or do you still see a hole in the locking scheme here?

>> +
>> +	while (!finished) {
>> +		int ret = kvm_read_guest(vcpu->kvm, cbaser + its->creadr,
>> +					 cmd_buf, 32);
>> +		if (ret) {
>> +			/*
>> +			 * Gah, we are screwed. Reset CWRITER to that command
>> +			 * that we have finished processing and return.
>> +			 */
>> +			spin_lock(&its->lock);
>> +			its->cwriter = its->creadr;
> don't get why do you set the queue empty?

So from a guest's point of view this is something like an ITS internal
error which shouldn't happen. So I just bail out, but leave the queue in
a consistent (read: empty) state to allow possible future commands to be
processed at best effort.
I guess this is the best we can come up with, since there is no feasible
way of communicating errors back(?)

Cheers,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers
  2016-05-05 19:12     ` Chalamarla, Tirumalesh
@ 2016-05-25 14:34       ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 14:34 UTC (permalink / raw)
  To: Chalamarla, Tirumalesh, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Tirumalesh,

....

>> +/* The MOVI command moves an ITTE to a different collection. */
>> +static int vits_cmd_handle_movi(struct kvm *kvm, u64 *its_cmd)
>> +{
>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>> +	u32 device_id = its_cmd_get_deviceid(its_cmd);
>> +	u32 event_id = its_cmd_get_id(its_cmd);
>> +	u32 coll_id = its_cmd_get_collection(its_cmd);
>> +	struct its_itte *itte;
>> +	struct its_collection *collection;
>> +	int ret;
>> +
>> +	spin_lock(&its->lock);
>> +	itte = find_itte(kvm, device_id, event_id);
>> +	if (!itte) {
>> +		ret = E_ITS_MOVI_UNMAPPED_INTERRUPT;
>> +		goto out_unlock;
>> +	}
>> +	if (!its_is_collection_mapped(itte->collection)) {
>> +		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>> +		goto out_unlock;
>> +	}
>> +
>> +	collection = find_collection(kvm, coll_id);
>> +	if (!its_is_collection_mapped(collection)) {
>> +		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>> +		goto out_unlock;
>> +	}
>> +
>> +	itte->collection = collection;
>> +	update_affinity(kvm, collection);
>> +
> 
> Not planning to send this down to Physical ITS in case of VFIO for performance reasons?

Why should we?
So in general: this virtual ITS is a purely emulated one, it does not
depend on any hardware ITS to work. Also at this point we don't focus on
VFIO (in fact this doesn't even work without further patches).
So it would break abstraction to pass on commands. Besides I don't see
the benefit of passing a 32-byte structure around, which needs to be
validated (and thus touched) anyway.
Also in this particular case: Why would a change of the _virtual_
affinity affect a VFIO IRQ on the host at all?
Actually the collection IDs of the guests are _totally_ separate and
unrelated to a host collection ID.

Cheers,
Andre

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers
@ 2016-05-25 14:34       ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 14:34 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Tirumalesh,

....

>> +/* The MOVI command moves an ITTE to a different collection. */
>> +static int vits_cmd_handle_movi(struct kvm *kvm, u64 *its_cmd)
>> +{
>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>> +	u32 device_id = its_cmd_get_deviceid(its_cmd);
>> +	u32 event_id = its_cmd_get_id(its_cmd);
>> +	u32 coll_id = its_cmd_get_collection(its_cmd);
>> +	struct its_itte *itte;
>> +	struct its_collection *collection;
>> +	int ret;
>> +
>> +	spin_lock(&its->lock);
>> +	itte = find_itte(kvm, device_id, event_id);
>> +	if (!itte) {
>> +		ret = E_ITS_MOVI_UNMAPPED_INTERRUPT;
>> +		goto out_unlock;
>> +	}
>> +	if (!its_is_collection_mapped(itte->collection)) {
>> +		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>> +		goto out_unlock;
>> +	}
>> +
>> +	collection = find_collection(kvm, coll_id);
>> +	if (!its_is_collection_mapped(collection)) {
>> +		ret = E_ITS_MOVI_UNMAPPED_COLLECTION;
>> +		goto out_unlock;
>> +	}
>> +
>> +	itte->collection = collection;
>> +	update_affinity(kvm, collection);
>> +
> 
> Not planning to send this down to Physical ITS in case of VFIO for performance reasons?

Why should we?
So in general: this virtual ITS is a purely emulated one, it does not
depend on any hardware ITS to work. Also at this point we don't focus on
VFIO (in fact this doesn't even work without further patches).
So it would break abstraction to pass on commands. Besides I don't see
the benefit of passing a 32-byte structure around, which needs to be
validated (and thus touched) anyway.
Also in this particular case: Why would a change of the _virtual_
affinity affect a VFIO IRQ on the host at all?
Actually the collection IDs of the guests are _totally_ separate and
unrelated to a host collection ID.

Cheers,
Andre

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
  2016-04-03  9:15     ` Christoffer Dall
@ 2016-05-25 15:55       ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 15:55 UTC (permalink / raw)
  To: Christoffer Dall; +Cc: Marc Zyngier, kvm, kvmarm, linux-arm-kernel

Hi Christoffer,

On 03/04/16 10:15, Christoffer Dall wrote:
> On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
>> The ARM GICv3 ITS MSI controller requires a device ID to be able to
>> assign the proper interrupt vector. On real hardware, this ID is
>> sampled from the bus. To be able to emulate an ITS controller, extend
>> the KVM MSI interface to let userspace provide such a device ID. For
>> PCI devices, the device ID is simply the 16-bit bus-device-function
>> triplet, which should be easily available to the userland tool.
>>
>> Also there is a new KVM capability which advertises whether the
>> current VM requires a device ID to be set along with the MSI data.
>> This flag is still reported as not available everywhere, later we will
>> enable it when ITS emulation is used.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> Reviewed-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
>>  include/uapi/linux/kvm.h          |  5 ++++-
>>  2 files changed, 14 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index cb2ef0b..8f7351d 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -2163,10 +2163,18 @@ struct kvm_msi {
>>  	__u32 address_hi;
>>  	__u32 data;
>>  	__u32 flags;
>> -	__u8  pad[16];
>> +	__u32 devid;
> 
> Are we imposing any unfortunate restrictions for other architectures by
> using a u32 over a u64 for the device ID?

Mmmh, good point. I guess not only for other architectures, but also for
the future in general.

Are there any objections against increasing this to a u64?

Cheers,
Andre.

>> +	__u8  pad[12];
>>  };
>>  
>> -No flags are defined so far. The corresponding field must be 0.
>> +flags: KVM_MSI_VALID_DEVID: devid contains a valid value
>> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
>> +       for the device that wrote the MSI message.
>> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
>> +
>> +The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
>> +the device ID. If this capability is not set, userland cannot rely on
>> +the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
>>  
>>  
>>  4.71 KVM_CREATE_PIT2
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 50f44a2..6a02871 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
>>  #define KVM_CAP_S390_RI 124
>>  #define KVM_CAP_ARM_PMU_V3 125
>>  #define KVM_CAP_VCPU_ATTRIBUTES 126
>> +#define KVM_CAP_MSI_DEVID 127
>>  
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>  
>> @@ -1010,12 +1011,14 @@ struct kvm_one_reg {
>>  	__u64 addr;
>>  };
>>  
>> +#define KVM_MSI_VALID_DEVID	(1U << 0)
>>  struct kvm_msi {
>>  	__u32 address_lo;
>>  	__u32 address_hi;
>>  	__u32 data;
>>  	__u32 flags;
>> -	__u8  pad[16];
>> +	__u32 devid;
>> +	__u8  pad[12];
>>  };
>>  
>>  struct kvm_arm_device_addr {
>> -- 
>> 2.7.3
>>
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
@ 2016-05-25 15:55       ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-05-25 15:55 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Christoffer,

On 03/04/16 10:15, Christoffer Dall wrote:
> On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
>> The ARM GICv3 ITS MSI controller requires a device ID to be able to
>> assign the proper interrupt vector. On real hardware, this ID is
>> sampled from the bus. To be able to emulate an ITS controller, extend
>> the KVM MSI interface to let userspace provide such a device ID. For
>> PCI devices, the device ID is simply the 16-bit bus-device-function
>> triplet, which should be easily available to the userland tool.
>>
>> Also there is a new KVM capability which advertises whether the
>> current VM requires a device ID to be set along with the MSI data.
>> This flag is still reported as not available everywhere, later we will
>> enable it when ITS emulation is used.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> Reviewed-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
>>  include/uapi/linux/kvm.h          |  5 ++++-
>>  2 files changed, 14 insertions(+), 3 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index cb2ef0b..8f7351d 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -2163,10 +2163,18 @@ struct kvm_msi {
>>  	__u32 address_hi;
>>  	__u32 data;
>>  	__u32 flags;
>> -	__u8  pad[16];
>> +	__u32 devid;
> 
> Are we imposing any unfortunate restrictions for other architectures by
> using a u32 over a u64 for the device ID?

Mmmh, good point. I guess not only for other architectures, but also for
the future in general.

Are there any objections against increasing this to a u64?

Cheers,
Andre.

>> +	__u8  pad[12];
>>  };
>>  
>> -No flags are defined so far. The corresponding field must be 0.
>> +flags: KVM_MSI_VALID_DEVID: devid contains a valid value
>> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
>> +       for the device that wrote the MSI message.
>> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
>> +
>> +The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
>> +the device ID. If this capability is not set, userland cannot rely on
>> +the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
>>  
>>  
>>  4.71 KVM_CREATE_PIT2
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 50f44a2..6a02871 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -852,6 +852,7 @@ struct kvm_ppc_smmu_info {
>>  #define KVM_CAP_S390_RI 124
>>  #define KVM_CAP_ARM_PMU_V3 125
>>  #define KVM_CAP_VCPU_ATTRIBUTES 126
>> +#define KVM_CAP_MSI_DEVID 127
>>  
>>  #ifdef KVM_CAP_IRQ_ROUTING
>>  
>> @@ -1010,12 +1011,14 @@ struct kvm_one_reg {
>>  	__u64 addr;
>>  };
>>  
>> +#define KVM_MSI_VALID_DEVID	(1U << 0)
>>  struct kvm_msi {
>>  	__u32 address_lo;
>>  	__u32 address_hi;
>>  	__u32 data;
>>  	__u32 flags;
>> -	__u8  pad[16];
>> +	__u32 devid;
>> +	__u8  pad[12];
>>  };
>>  
>>  struct kvm_arm_device_addr {
>> -- 
>> 2.7.3
>>
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
  2016-05-25 15:55       ` Andre Przywara
@ 2016-05-25 16:16         ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-25 16:16 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall
  Cc: Eric Auger, linux-arm-kernel, kvmarm, kvm

On 25/05/16 16:55, Andre Przywara wrote:
> Hi Christoffer,
> 
> On 03/04/16 10:15, Christoffer Dall wrote:
>> On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
>>> The ARM GICv3 ITS MSI controller requires a device ID to be able to
>>> assign the proper interrupt vector. On real hardware, this ID is
>>> sampled from the bus. To be able to emulate an ITS controller, extend
>>> the KVM MSI interface to let userspace provide such a device ID. For
>>> PCI devices, the device ID is simply the 16-bit bus-device-function
>>> triplet, which should be easily available to the userland tool.
>>>
>>> Also there is a new KVM capability which advertises whether the
>>> current VM requires a device ID to be set along with the MSI data.
>>> This flag is still reported as not available everywhere, later we will
>>> enable it when ITS emulation is used.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> Reviewed-by: Eric Auger <eric.auger@linaro.org>
>>> ---
>>>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
>>>  include/uapi/linux/kvm.h          |  5 ++++-
>>>  2 files changed, 14 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>>> index cb2ef0b..8f7351d 100644
>>> --- a/Documentation/virtual/kvm/api.txt
>>> +++ b/Documentation/virtual/kvm/api.txt
>>> @@ -2163,10 +2163,18 @@ struct kvm_msi {
>>>  	__u32 address_hi;
>>>  	__u32 data;
>>>  	__u32 flags;
>>> -	__u8  pad[16];
>>> +	__u32 devid;
>>
>> Are we imposing any unfortunate restrictions for other architectures by
>> using a u32 over a u64 for the device ID?
> 
> Mmmh, good point. I guess not only for other architectures, but also for
> the future in general.
> 
> Are there any objections against increasing this to a u64?

I'm not sure this is really necessary. A PCI RID is 16bit, and expanding
it to 32bit is already making quite a bit of space for further extension.

Also, some architecture might be unhappy of having a 64bit quantity that
is not 64bit aligned...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
@ 2016-05-25 16:16         ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-25 16:16 UTC (permalink / raw)
  To: linux-arm-kernel

On 25/05/16 16:55, Andre Przywara wrote:
> Hi Christoffer,
> 
> On 03/04/16 10:15, Christoffer Dall wrote:
>> On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
>>> The ARM GICv3 ITS MSI controller requires a device ID to be able to
>>> assign the proper interrupt vector. On real hardware, this ID is
>>> sampled from the bus. To be able to emulate an ITS controller, extend
>>> the KVM MSI interface to let userspace provide such a device ID. For
>>> PCI devices, the device ID is simply the 16-bit bus-device-function
>>> triplet, which should be easily available to the userland tool.
>>>
>>> Also there is a new KVM capability which advertises whether the
>>> current VM requires a device ID to be set along with the MSI data.
>>> This flag is still reported as not available everywhere, later we will
>>> enable it when ITS emulation is used.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> Reviewed-by: Eric Auger <eric.auger@linaro.org>
>>> ---
>>>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
>>>  include/uapi/linux/kvm.h          |  5 ++++-
>>>  2 files changed, 14 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>>> index cb2ef0b..8f7351d 100644
>>> --- a/Documentation/virtual/kvm/api.txt
>>> +++ b/Documentation/virtual/kvm/api.txt
>>> @@ -2163,10 +2163,18 @@ struct kvm_msi {
>>>  	__u32 address_hi;
>>>  	__u32 data;
>>>  	__u32 flags;
>>> -	__u8  pad[16];
>>> +	__u32 devid;
>>
>> Are we imposing any unfortunate restrictions for other architectures by
>> using a u32 over a u64 for the device ID?
> 
> Mmmh, good point. I guess not only for other architectures, but also for
> the future in general.
> 
> Are there any objections against increasing this to a u64?

I'm not sure this is really necessary. A PCI RID is 16bit, and expanding
it to 32bit is already making quite a bit of space for further extension.

Also, some architecture might be unhappy of having a 64bit quantity that
is not 64bit aligned...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-05-25 11:37       ` Andre Przywara
@ 2016-05-26  9:10         ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-26  9:10 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

On 25/05/16 12:37, Andre Przywara wrote:
> Hi Marc,
> 
> this series is mostly obsolete now, but I wanted to elaborate on the
> locking scheme, which hasn't changed.
> 
> On 07/04/16 15:35, Marc Zyngier wrote:
>> On 26/03/16 02:14, Andre Przywara wrote:
> 
> ....
> 
>>> --- a/virt/kvm/arm/vgic/its-emul.c
>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>> @@ -31,23 +31,263 @@
>>>  #include "vgic.h"
>>>  #include "vgic_mmio.h"
>>>  
>>> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>>> +
>>> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>>> +				   struct kvm_io_device *this,
>>> +				   gpa_t addr, int len, void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +	u32 reg;
>>> +
>>> +	reg = GITS_CTLR_QUIESCENT;
>>
>> So your ITS is always in a quiescent state? Even when you're processing
>> the command queue? You'll have to convince me...
> 
> Does QUIESCENT actually cover the command queue status? I was under the
> impression that this is purely to cope with distributed implementations
> and the possible delays caused by propagating commands.

Here's what the spec says:

"For the ITS to be quiescent, there must be no transactions in progress.
In addition, all operations required to ensure that mapping data is
consistent with external memory must be complete."

How can you enforce consistency if you have command in progress?

> If not so, I can surely take the lock here and respect
> (creadr == cwriter) upon setting the bit.

Please do.

> 
>>
>>> +	if (its->enabled)
>>> +		reg |= GITS_CTLR_ENABLE;
>>> +
>>> +	write_mask32(reg, addr & 3, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
>>> +				    struct kvm_io_device *this,
>>> +				    gpa_t addr, int len, const void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +	struct vgic_io_device *iodev = container_of(this,
>>> +						    struct vgic_io_device, dev);
>>> +
>>> +        if (addr - iodev->base_addr == 0)
>>
>> whitespace issue.
>>
>>> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>>> +				    struct kvm_io_device *this,
>>> +				    gpa_t addr, int len, void *val)
>>> +{
>>> +	u64 reg = GITS_TYPER_PLPIS;
>>> +
>>> +	/*
>>> +	 * We use linear CPU numbers for redistributor addressing,
>>> +	 * so GITS_TYPER.PTA is 0.
>>> +	 * To avoid memory waste on the guest side, we keep the
>>> +	 * number of IDBits and DevBits low for the time being.
>>> +	 * This could later be made configurable by userland.
>>> +	 * Since we have all collections in linked list, we claim
>>> +	 * that we can hold all of the collection tables in our
>>> +	 * own memory and that the ITT entry size is 1 byte (the
>>> +	 * smallest possible one).
>>
>> All of this is going to bite us when we want to implement migration,
>> specially the HW collection bit.
> 
> How so? Clearly we keep the number of VCPUs constant, so everything
> guest visible stays the same even upon migrating to a much different
> host? Or are we talking about different things here?

Let me rephrase this: How are you going to address HW collections from
userspace in order to dump them? Short of having memory tables, you cannot.

> 
>>
>>> +	 */
>>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>>> +
>>> +	write_mask64(reg, addr & 7, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
>>> +				   struct kvm_io_device *this,
>>> +				   gpa_t addr, int len, void *val)
>>> +{
>>> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
>>> +
>>> +	write_mask32(reg, addr & 3, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>>> +				     struct kvm_io_device *this,
>>> +				     gpa_t addr, int len, void *val)
>>> +{
>>> +	struct vgic_io_device *iodev = container_of(this,
>>> +						    struct vgic_io_device, dev);
>>> +	u32 reg = 0;
>>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>>> +
>>> +	switch (idreg) {
>>> +	case GITS_PIDR2:
>>> +		reg = GIC_PIDR2_ARCH_GICv3;
>>
>> Are we leaving the lowest 4 bits to zero?
> 
> I couldn't stuff "42" in 4 bits, so: yes ;-)

The 4 bottom bits are "Implementation Defined", so you could fit
something in it if you wanted.

> 
>>> +		break;
>>> +	case GITS_PIDR4:
>>> +		/* This is a 64K software visible page */
>>> +		reg = 0x40;
>>
>> Same question.
>>
>> Also, how about all the others PIDR registers?
> 
> I guess I was halfway undecided between going with the pure
> architectural requirements (which only mandate bits 7:4 in PIDR2) and
> complying with the recommendation later in the spec.
> So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.
> 
>>> +		break;
>>> +	/* Those are the ID registers for (any) GIC. */
>>> +	case GITS_CIDR0:
>>> +		reg = 0x0d;
>>> +		break;
>>> +	case GITS_CIDR1:
>>> +		reg = 0xf0;
>>> +		break;
>>> +	case GITS_CIDR2:
>>> +		reg = 0x05;
>>> +		break;
>>> +	case GITS_CIDR3:
>>> +		reg = 0xb1;
>>> +		break;
>>> +	}
>>
>> Given that these values are directly taken from the architecture, and
>> seem common to the whole GICv3 architecture when implemented by ARM, we
>> could have a common handler for the whole GICv3 implementatuin. Not a
>> bit deal though.
> 
> Agreed.
> 
>>> +
>>> +	write_mask32(reg, addr & 3, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +/*
>>> + * This function is called with both the ITS and the distributor lock dropped,
>>> + * so the actual command handlers must take the respective locks when needed.
>>> + */
>>> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
>>> +{
>>> +	return -ENODEV;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>>> +				    struct kvm_io_device *this,
>>> +				    gpa_t addr, int len, void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +
>>> +	write_mask64(its->cbaser, addr & 7, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
>>> +				      struct kvm_io_device *this,
>>> +				      gpa_t addr, int len, const void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +
>>> +	if (its->enabled)
>>> +		return 0;
>>> +
>>> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
>>> +	its->creadr = 0;
>>
>> Don't you need to acquire the command queue lock here?
> 
> I don't think so, because we "return 0;" above if the ITS is enabled,
> which means that nothing is going on at the moment.

What would prevent the ITS from becoming enabled between the two
assignments?

> But I guess I can just take the lock anyway to be sure.

I don't think this is optional.

>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int its_cmd_buffer_size(struct kvm *kvm)
>>> +{
>>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>>> +
>>> +	return ((its->cbaser & 0xff) + 1) << 12;
>>> +}
>>> +
>>> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
>>> +{
>>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>>> +
>>> +	return BASER_BASE_ADDRESS(its->cbaser);
>>> +}
>>> +
>>> +/*
>>> + * By writing to CWRITER the guest announces new commands to be processed.
>>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>>> + * iterate over the command buffer (with the lock dropped) until the read
>>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>>> + * meantime will just update the write pointer, leaving the command
>>> + * processing to the first instance of the function.
>>> + */
>>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>>> +				       struct kvm_io_device *this,
>>> +				       gpa_t addr, int len, const void *val)
>>> +{
>>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>> +	struct vgic_its *its = &dist->its;
>>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>>> +	u64 cmd_buf[4];
>>> +	u32 reg;
>>> +	bool finished;
>>> +
>>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>>> +	reg &= 0xfffe0;
>>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>>> +		return 0;
>>> +
>>> +	spin_lock(&its->lock);
>>> +
>>> +	/*
>>> +	 * If there is still another VCPU handling commands, let this
>>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>>> +	 */
>>
>> How do you detect that condition? All I see is a massive race here, with
>> two threads processing the queue in parallel, possibly corrupting each
>> other's data.
>>
>> Please explain why you think this is safe.
> 
> OK, here you go: Actually we are handling the command queue
> synchronously: The first writer to CWRITER will end up here and will
> iterate over all commands below. From the guests point of view it will
> take some time to do the CWRITER MMIO write, but upon the writel
> returning the queue is actually already empty again.
> That means that any _sane_ guest will _never_ trigger the case where two
> threads/VCPUs are handling the queue, because a concurrent write to
> CWRITER without a lock in the guest would be broken by design.

That feels very fragile, and you're relying on a well behaved guest.

> 
> However I am not sure we can rely on this, so I added this precaution
> scheme:
> The first writer takes our ITS (emulation) lock and examines cwriter and
> creadr to see if they are equal. If they are, then the queue is
> currently empty, which means we are the first one and need to take care
> of the commands. We update our copy of cwriter (now marking the queue as
> non-empty) and drop the lock. As we kept the queue-was-empty status in a
> local variable, we now can start iterating through the queue. We only
> take the lock briefly when really needed in this process (for instance
> when one command has been processed and creadr is incremented).
> 
> If now a second VCPU writes CWRITER, we also take the lock and compare
> creadr and cwriter. Now they are different, because the first thread is
> still busy with handling the commands and hasn't finished yet.
> So we set our local "finished" to true. Also we update cwriter to the
> new value, then drop the lock. The while loop below will not be entered
> in this thread and we return to the guest.
> Now the first thread has handled another command, takes the lock to
> increase creadr and compares it with cwriter. Without the second writer
> it may have been finished already, but now cwriter has grown, so it will
> just carry on with handling the commands.
> 
> A bit like someone washing the dishes while another person adds some
> more dirty plates to the pile ;-)

I'm sorry, this feels incredibly complicated, and will make actual
debugging/tracing a nightmare. It also penalize the first vcpu that gets
to submitting a command, leading to scheduling imbalances.

Why can't you simply turn the ITS lock into a mutex and keep it held,
processing each batch in the context of the vcpu that is writing to
GITC_CWRITER?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-05-26  9:10         ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-05-26  9:10 UTC (permalink / raw)
  To: linux-arm-kernel

On 25/05/16 12:37, Andre Przywara wrote:
> Hi Marc,
> 
> this series is mostly obsolete now, but I wanted to elaborate on the
> locking scheme, which hasn't changed.
> 
> On 07/04/16 15:35, Marc Zyngier wrote:
>> On 26/03/16 02:14, Andre Przywara wrote:
> 
> ....
> 
>>> --- a/virt/kvm/arm/vgic/its-emul.c
>>> +++ b/virt/kvm/arm/vgic/its-emul.c
>>> @@ -31,23 +31,263 @@
>>>  #include "vgic.h"
>>>  #include "vgic_mmio.h"
>>>  
>>> +#define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>>> +
>>> +static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>>> +				   struct kvm_io_device *this,
>>> +				   gpa_t addr, int len, void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +	u32 reg;
>>> +
>>> +	reg = GITS_CTLR_QUIESCENT;
>>
>> So your ITS is always in a quiescent state? Even when you're processing
>> the command queue? You'll have to convince me...
> 
> Does QUIESCENT actually cover the command queue status? I was under the
> impression that this is purely to cope with distributed implementations
> and the possible delays caused by propagating commands.

Here's what the spec says:

"For the ITS to be quiescent, there must be no transactions in progress.
In addition, all operations required to ensure that mapping data is
consistent with external memory must be complete."

How can you enforce consistency if you have command in progress?

> If not so, I can surely take the lock here and respect
> (creadr == cwriter) upon setting the bit.

Please do.

> 
>>
>>> +	if (its->enabled)
>>> +		reg |= GITS_CTLR_ENABLE;
>>> +
>>> +	write_mask32(reg, addr & 3, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_write_its_ctlr(struct kvm_vcpu *vcpu,
>>> +				    struct kvm_io_device *this,
>>> +				    gpa_t addr, int len, const void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +	struct vgic_io_device *iodev = container_of(this,
>>> +						    struct vgic_io_device, dev);
>>> +
>>> +        if (addr - iodev->base_addr == 0)
>>
>> whitespace issue.
>>
>>> +		its->enabled = !!(*(u8*)val & GITS_CTLR_ENABLE);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>>> +				    struct kvm_io_device *this,
>>> +				    gpa_t addr, int len, void *val)
>>> +{
>>> +	u64 reg = GITS_TYPER_PLPIS;
>>> +
>>> +	/*
>>> +	 * We use linear CPU numbers for redistributor addressing,
>>> +	 * so GITS_TYPER.PTA is 0.
>>> +	 * To avoid memory waste on the guest side, we keep the
>>> +	 * number of IDBits and DevBits low for the time being.
>>> +	 * This could later be made configurable by userland.
>>> +	 * Since we have all collections in linked list, we claim
>>> +	 * that we can hold all of the collection tables in our
>>> +	 * own memory and that the ITT entry size is 1 byte (the
>>> +	 * smallest possible one).
>>
>> All of this is going to bite us when we want to implement migration,
>> specially the HW collection bit.
> 
> How so? Clearly we keep the number of VCPUs constant, so everything
> guest visible stays the same even upon migrating to a much different
> host? Or are we talking about different things here?

Let me rephrase this: How are you going to address HW collections from
userspace in order to dump them? Short of having memory tables, you cannot.

> 
>>
>>> +	 */
>>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>>> +
>>> +	write_mask64(reg, addr & 7, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_iidr(struct kvm_vcpu *vcpu,
>>> +				   struct kvm_io_device *this,
>>> +				   gpa_t addr, int len, void *val)
>>> +{
>>> +	u32 reg = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
>>> +
>>> +	write_mask32(reg, addr & 3, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>>> +				     struct kvm_io_device *this,
>>> +				     gpa_t addr, int len, void *val)
>>> +{
>>> +	struct vgic_io_device *iodev = container_of(this,
>>> +						    struct vgic_io_device, dev);
>>> +	u32 reg = 0;
>>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>>> +
>>> +	switch (idreg) {
>>> +	case GITS_PIDR2:
>>> +		reg = GIC_PIDR2_ARCH_GICv3;
>>
>> Are we leaving the lowest 4 bits to zero?
> 
> I couldn't stuff "42" in 4 bits, so: yes ;-)

The 4 bottom bits are "Implementation Defined", so you could fit
something in it if you wanted.

> 
>>> +		break;
>>> +	case GITS_PIDR4:
>>> +		/* This is a 64K software visible page */
>>> +		reg = 0x40;
>>
>> Same question.
>>
>> Also, how about all the others PIDR registers?
> 
> I guess I was halfway undecided between going with the pure
> architectural requirements (which only mandate bits 7:4 in PIDR2) and
> complying with the recommendation later in the spec.
> So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.
> 
>>> +		break;
>>> +	/* Those are the ID registers for (any) GIC. */
>>> +	case GITS_CIDR0:
>>> +		reg = 0x0d;
>>> +		break;
>>> +	case GITS_CIDR1:
>>> +		reg = 0xf0;
>>> +		break;
>>> +	case GITS_CIDR2:
>>> +		reg = 0x05;
>>> +		break;
>>> +	case GITS_CIDR3:
>>> +		reg = 0xb1;
>>> +		break;
>>> +	}
>>
>> Given that these values are directly taken from the architecture, and
>> seem common to the whole GICv3 architecture when implemented by ARM, we
>> could have a common handler for the whole GICv3 implementatuin. Not a
>> bit deal though.
> 
> Agreed.
> 
>>> +
>>> +	write_mask32(reg, addr & 3, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +/*
>>> + * This function is called with both the ITS and the distributor lock dropped,
>>> + * so the actual command handlers must take the respective locks when needed.
>>> + */
>>> +static int vits_handle_command(struct kvm_vcpu *vcpu, u64 *its_cmd)
>>> +{
>>> +	return -ENODEV;
>>> +}
>>> +
>>> +static int vgic_mmio_read_its_cbaser(struct kvm_vcpu *vcpu,
>>> +				    struct kvm_io_device *this,
>>> +				    gpa_t addr, int len, void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +
>>> +	write_mask64(its->cbaser, addr & 7, len, val);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int vgic_mmio_write_its_cbaser(struct kvm_vcpu *vcpu,
>>> +				      struct kvm_io_device *this,
>>> +				      gpa_t addr, int len, const void *val)
>>> +{
>>> +	struct vgic_its *its = &vcpu->kvm->arch.vgic.its;
>>> +
>>> +	if (its->enabled)
>>> +		return 0;
>>> +
>>> +	its->cbaser = mask64(its->cbaser, addr & 7, len, val);
>>> +	its->creadr = 0;
>>
>> Don't you need to acquire the command queue lock here?
> 
> I don't think so, because we "return 0;" above if the ITS is enabled,
> which means that nothing is going on at the moment.

What would prevent the ITS from becoming enabled between the two
assignments?

> But I guess I can just take the lock anyway to be sure.

I don't think this is optional.

>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static int its_cmd_buffer_size(struct kvm *kvm)
>>> +{
>>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>>> +
>>> +	return ((its->cbaser & 0xff) + 1) << 12;
>>> +}
>>> +
>>> +static gpa_t its_cmd_buffer_base(struct kvm *kvm)
>>> +{
>>> +	struct vgic_its *its = &kvm->arch.vgic.its;
>>> +
>>> +	return BASER_BASE_ADDRESS(its->cbaser);
>>> +}
>>> +
>>> +/*
>>> + * By writing to CWRITER the guest announces new commands to be processed.
>>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>>> + * iterate over the command buffer (with the lock dropped) until the read
>>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>>> + * meantime will just update the write pointer, leaving the command
>>> + * processing to the first instance of the function.
>>> + */
>>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>>> +				       struct kvm_io_device *this,
>>> +				       gpa_t addr, int len, const void *val)
>>> +{
>>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>> +	struct vgic_its *its = &dist->its;
>>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>>> +	u64 cmd_buf[4];
>>> +	u32 reg;
>>> +	bool finished;
>>> +
>>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>>> +	reg &= 0xfffe0;
>>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>>> +		return 0;
>>> +
>>> +	spin_lock(&its->lock);
>>> +
>>> +	/*
>>> +	 * If there is still another VCPU handling commands, let this
>>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>>> +	 */
>>
>> How do you detect that condition? All I see is a massive race here, with
>> two threads processing the queue in parallel, possibly corrupting each
>> other's data.
>>
>> Please explain why you think this is safe.
> 
> OK, here you go: Actually we are handling the command queue
> synchronously: The first writer to CWRITER will end up here and will
> iterate over all commands below. From the guests point of view it will
> take some time to do the CWRITER MMIO write, but upon the writel
> returning the queue is actually already empty again.
> That means that any _sane_ guest will _never_ trigger the case where two
> threads/VCPUs are handling the queue, because a concurrent write to
> CWRITER without a lock in the guest would be broken by design.

That feels very fragile, and you're relying on a well behaved guest.

> 
> However I am not sure we can rely on this, so I added this precaution
> scheme:
> The first writer takes our ITS (emulation) lock and examines cwriter and
> creadr to see if they are equal. If they are, then the queue is
> currently empty, which means we are the first one and need to take care
> of the commands. We update our copy of cwriter (now marking the queue as
> non-empty) and drop the lock. As we kept the queue-was-empty status in a
> local variable, we now can start iterating through the queue. We only
> take the lock briefly when really needed in this process (for instance
> when one command has been processed and creadr is incremented).
> 
> If now a second VCPU writes CWRITER, we also take the lock and compare
> creadr and cwriter. Now they are different, because the first thread is
> still busy with handling the commands and hasn't finished yet.
> So we set our local "finished" to true. Also we update cwriter to the
> new value, then drop the lock. The while loop below will not be entered
> in this thread and we return to the guest.
> Now the first thread has handled another command, takes the lock to
> increase creadr and compares it with cwriter. Without the second writer
> it may have been finished already, but now cwriter has grown, so it will
> just carry on with handling the commands.
> 
> A bit like someone washing the dishes while another person adds some
> more dirty plates to the pile ;-)

I'm sorry, this feels incredibly complicated, and will make actual
debugging/tracing a nightmare. It also penalize the first vcpu that gets
to submitting a command, leading to scheduling imbalances.

Why can't you simply turn the ITS lock into a mutex and keep it held,
processing each batch in the context of the vcpu that is writing to
GITC_CWRITER?

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
  2016-05-25 16:16         ` Marc Zyngier
@ 2016-05-31 13:05           ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-05-31 13:05 UTC (permalink / raw)
  To: Marc Zyngier; +Cc: Andre Przywara, Eric Auger, linux-arm-kernel, kvmarm, kvm

On Wed, May 25, 2016 at 05:16:54PM +0100, Marc Zyngier wrote:
> On 25/05/16 16:55, Andre Przywara wrote:
> > Hi Christoffer,
> > 
> > On 03/04/16 10:15, Christoffer Dall wrote:
> >> On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
> >>> The ARM GICv3 ITS MSI controller requires a device ID to be able to
> >>> assign the proper interrupt vector. On real hardware, this ID is
> >>> sampled from the bus. To be able to emulate an ITS controller, extend
> >>> the KVM MSI interface to let userspace provide such a device ID. For
> >>> PCI devices, the device ID is simply the 16-bit bus-device-function
> >>> triplet, which should be easily available to the userland tool.
> >>>
> >>> Also there is a new KVM capability which advertises whether the
> >>> current VM requires a device ID to be set along with the MSI data.
> >>> This flag is still reported as not available everywhere, later we will
> >>> enable it when ITS emulation is used.
> >>>
> >>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> >>> Reviewed-by: Eric Auger <eric.auger@linaro.org>
> >>> ---
> >>>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
> >>>  include/uapi/linux/kvm.h          |  5 ++++-
> >>>  2 files changed, 14 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> >>> index cb2ef0b..8f7351d 100644
> >>> --- a/Documentation/virtual/kvm/api.txt
> >>> +++ b/Documentation/virtual/kvm/api.txt
> >>> @@ -2163,10 +2163,18 @@ struct kvm_msi {
> >>>  	__u32 address_hi;
> >>>  	__u32 data;
> >>>  	__u32 flags;
> >>> -	__u8  pad[16];
> >>> +	__u32 devid;
> >>
> >> Are we imposing any unfortunate restrictions for other architectures by
> >> using a u32 over a u64 for the device ID?
> > 
> > Mmmh, good point. I guess not only for other architectures, but also for
> > the future in general.
> > 
> > Are there any objections against increasing this to a u64?
> 
> I'm not sure this is really necessary. A PCI RID is 16bit, and expanding
> it to 32bit is already making quite a bit of space for further extension.
> 
> Also, some architecture might be unhappy of having a 64bit quantity that
> is not 64bit aligned...
> 
ok, thanks for the explanation.

-Christoffer

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID
@ 2016-05-31 13:05           ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-05-31 13:05 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, May 25, 2016 at 05:16:54PM +0100, Marc Zyngier wrote:
> On 25/05/16 16:55, Andre Przywara wrote:
> > Hi Christoffer,
> > 
> > On 03/04/16 10:15, Christoffer Dall wrote:
> >> On Sat, Mar 26, 2016 at 02:13:59AM +0000, Andre Przywara wrote:
> >>> The ARM GICv3 ITS MSI controller requires a device ID to be able to
> >>> assign the proper interrupt vector. On real hardware, this ID is
> >>> sampled from the bus. To be able to emulate an ITS controller, extend
> >>> the KVM MSI interface to let userspace provide such a device ID. For
> >>> PCI devices, the device ID is simply the 16-bit bus-device-function
> >>> triplet, which should be easily available to the userland tool.
> >>>
> >>> Also there is a new KVM capability which advertises whether the
> >>> current VM requires a device ID to be set along with the MSI data.
> >>> This flag is still reported as not available everywhere, later we will
> >>> enable it when ITS emulation is used.
> >>>
> >>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> >>> Reviewed-by: Eric Auger <eric.auger@linaro.org>
> >>> ---
> >>>  Documentation/virtual/kvm/api.txt | 12 ++++++++++--
> >>>  include/uapi/linux/kvm.h          |  5 ++++-
> >>>  2 files changed, 14 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> >>> index cb2ef0b..8f7351d 100644
> >>> --- a/Documentation/virtual/kvm/api.txt
> >>> +++ b/Documentation/virtual/kvm/api.txt
> >>> @@ -2163,10 +2163,18 @@ struct kvm_msi {
> >>>  	__u32 address_hi;
> >>>  	__u32 data;
> >>>  	__u32 flags;
> >>> -	__u8  pad[16];
> >>> +	__u32 devid;
> >>
> >> Are we imposing any unfortunate restrictions for other architectures by
> >> using a u32 over a u64 for the device ID?
> > 
> > Mmmh, good point. I guess not only for other architectures, but also for
> > the future in general.
> > 
> > Are there any objections against increasing this to a u64?
> 
> I'm not sure this is really necessary. A PCI RID is 16bit, and expanding
> it to 32bit is already making quite a bit of space for further extension.
> 
> Also, some architecture might be unhappy of having a 64bit quantity that
> is not 64bit aligned...
> 
ok, thanks for the explanation.

-Christoffer

^ permalink raw reply	[flat|nested] 120+ messages in thread
* RE: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
  2016-03-26  2:13 ` Andre Przywara
@ 2016-06-03  4:26   ` Bharat Bhushan
  -1 siblings, 0 replies; 120+ messages in thread
From: Bharat Bhushan @ 2016-06-03  4:26 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Marc Zyngier, Eric Auger
  Cc: kvmarm, linux-arm-kernel, kvm

Hi Andre,

We are looking for these patches but seems like these needed to rebased/redesigned based on new GIC emulation changes. Can you give us some sense on when this will be done?
Also do you know if someone is working on QEMU size changes for this, if not then we can help on getting that done.

Thanks
-Bharat

> -----Original Message-----
> From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-
> bounces@lists.cs.columbia.edu] On Behalf Of Andre Przywara
> Sent: Saturday, March 26, 2016 7:44 AM
> To: Christoffer Dall <christoffer.dall@linaro.org>; Marc Zyngier
> <marc.zyngier@arm.com>; Eric Auger <eric.auger@linaro.org>
> Cc: kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org;
> kvm@vger.kernel.org
> Subject: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
> 
> Hi,
> 
> this is a rebased and reworked version of the ITS emulation support.
> It allows KVM guests which use an emulated GICv3 to use LPIs as well,
> though in the moment this is limited to emulated PCI devices.
> It is now based on and requires the new VGIC implementation [1].
> 
> You can find all of this code (and the prerequisites) in the
> its-emul/v4 branch of my repository [2].
> 
> Cheers,
> Andre.
> 
> Changelog v3..v4:
> - adapting to new VGIC (changes in IRQ injection mechanism)
> 
> Changelog v2..v3:
> - adapt to 4.3-rc and Christoffer's timer rework
> - adapt spin locks on handling PROPBASER/PENDBASER registers
> - rework locking in ITS command handling (dropping dist where needed)
> - only clear LPI pending bit if LPI could actually be queued
> - simplify GICR_CTLR handling
> - properly free ITTEs (including our pending bitmap)
> - fix corner cases with unmapped collections
> - keep retire_lr() around
> - rename vgic_handle_base_register to vgic_reg64_access()
> - use kcalloc instead of kmalloc
> - minor fixes, renames and added comments
> 
> Changelog v1..v2
> - fix issues when using non-ITS GICv3 emulation
> - streamline frame address initialization (new patch 05/15)
> - preallocate buffer memory for reading from guest's memory
> - move locking into the actual command handlers
> -   preallocate memory for new structures if needed
> - use non-atomic __set_bit() and __clear_bit() when under the lock
> - add INT command handler to allow LPI injection from the guest
> - rewrite CWRITER handler to align with new locking scheme
> - remove unneeded CONFIG_HAVE_KVM_MSI #ifdefs
> - check memory table size against our LPI limit (65536 interrupts)
> - observe initial gap of 1024 interrupts in pending table
> - use term "configuration table" to be in line with the spec
> - clarify and extend documentation on API extensions
> - introduce new KVM_CAP_MSI_DEVID capability to advertise device ID
> requirement
> - update, fix and add many comments
> - minor style changes as requested by reviewers
> 
> ---------------
> 
> The GICv3 ITS (Interrupt Translation Service) is a part of the ARM GICv3
> interrupt controller [4] used for implementing MSIs.
> It specifies a new kind of interrupts (LPIs), which are mapped to establish a
> connection between a device, its MSI payload value and the target processor
> the IRQ is eventually delivered to.
> In order to allow using MSIs in an ARM64 KVM guest, we emulate this ITS
> widget in the kernel.
> The ITS works by reading commands written by software (from the guest in
> our case) into a (guest allocated) memory region and establishing the
> mapping between a device, the MSI payload and the target CPU.
> We parse these commands and update our internal data structures to reflect
> those changes. On an MSI injection we iterate those structures to learn the
> LPI number we have to inject.
> For the time being we use simple lists to hold the data, this is good enough
> for the small number of entries each of the components currently have.
> Should this become a performance bottleneck in the future, those can be
> extended to arrays or trees if needed.
> 
> Most of the code lives in a separate source file (its-emul.c), though there are
> some changes necessary in the existing VGIC files.
> 
> For the time being this series gives us the ability to use emulated PCI devices
> that can use MSIs in the guest. Those have to be triggered by letting the
> userland device emulation simulate the MSI write with the
> KVM_SIGNAL_MSI ioctl. This will be translated into the proper LPI by the ITS
> emulation and injected into the guest in the usual way (just with a higher IRQ
> number).
> 
> This series is based on 4.5 plus the KVM queue for 4.6 and the new VGIC
> emulation code and can be found at the its-emul/v4 branch of this repository
> [2].
> For this to be used you need a GICv3 host machine (a fast model would do),
> though it does not rely on any host ITS bits (neither in hardware or software).
> 
> To test this you can use the kvmtool patches available in the "its"
> branch here [3].
> Start a guest with: "$ lkvm run --irqchip=gicv3-its --force-pci"
> and see the ITS being used for instance by the virtio devices.
> 
> [1]: https://lists.cs.columbia.edu/pipermail/kvmarm/2016-
> March/019191.html
> [2]: git://linux-arm.org/linux-ap.git
>      http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-
> emul/v4
> [3]: git://linux-arm.org/kvmtool.git
>      http://www.linux-arm.org/git?p=kvmtool.git;a=log;h=refs/heads/its
> [4]:
> http://arminfo.emea.arm.com/help/topic/com.arm.doc.ihi0069a/IHI0069A_g
> ic_architecture_specification.pdf
> 
> Andre Przywara (12):
>   KVM: extend struct kvm_msi to hold a 32-bit device ID
>   KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
>   KVM: arm64: Introduce new MMIO region for the ITS base address
>   KVM: arm64: handle ITS related GICv3 redistributor registers
>   KVM: arm64: introduce ITS emulation file with stub functions
>   KVM: arm64: implement basic ITS register handlers
>   KVM: arm64: add data structures to model ITS interrupt translation
>   KVM: arm64: connect LPIs to the VGIC emulation
>   KVM: arm64: sync LPI configuration and pending tables
>   KVM: arm64: implement ITS command queue command handlers
>   KVM: arm64: implement MSI injection in ITS emulation
>   KVM: arm64: enable ITS emulation as a virtual MSI controller
> 
>  Documentation/virtual/kvm/api.txt              |   14 +-
>  Documentation/virtual/kvm/devices/arm-vgic.txt |    9 +
>  arch/arm/include/asm/kvm_host.h                |    2 +-
>  arch/arm/kvm/arm.c                             |    2 +-
>  arch/arm64/include/asm/kvm_host.h              |    2 +-
>  arch/arm64/include/uapi/asm/kvm.h              |    2 +
>  arch/arm64/kvm/Kconfig                         |    1 +
>  arch/arm64/kvm/Makefile                        |    1 +
>  arch/arm64/kvm/reset.c                         |   12 +-
>  include/kvm/vgic/vgic.h                        |   33 +
>  include/linux/irqchip/arm-gic-v3.h             |   28 +-
>  include/uapi/linux/kvm.h                       |    5 +-
>  virt/kvm/arm/vgic.c                            |    5 +
>  virt/kvm/arm/vgic/its-emul.c                   | 1146 ++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.c                       |   18 +
>  virt/kvm/arm/vgic/vgic.h                       |   33 +
>  virt/kvm/arm/vgic/vgic_init.c                  |    9 +
>  virt/kvm/arm/vgic/vgic_kvm_device.c            |    7 +
>  virt/kvm/arm/vgic/vgic_mmio.c                  |   54 +-
>  19 files changed, 1365 insertions(+), 18 deletions(-)  create mode 100644
> virt/kvm/arm/vgic/its-emul.c
> 
> --
> 2.7.3
> 
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
@ 2016-06-03  4:26   ` Bharat Bhushan
  0 siblings, 0 replies; 120+ messages in thread
From: Bharat Bhushan @ 2016-06-03  4:26 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,

We are looking for these patches but seems like these needed to rebased/redesigned based on new GIC emulation changes. Can you give us some sense on when this will be done?
Also do you know if someone is working on QEMU size changes for this, if not then we can help on getting that done.

Thanks
-Bharat

> -----Original Message-----
> From: kvmarm-bounces at lists.cs.columbia.edu [mailto:kvmarm-
> bounces at lists.cs.columbia.edu] On Behalf Of Andre Przywara
> Sent: Saturday, March 26, 2016 7:44 AM
> To: Christoffer Dall <christoffer.dall@linaro.org>; Marc Zyngier
> <marc.zyngier@arm.com>; Eric Auger <eric.auger@linaro.org>
> Cc: kvmarm at lists.cs.columbia.edu; linux-arm-kernel at lists.infradead.org;
> kvm at vger.kernel.org
> Subject: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
> 
> Hi,
> 
> this is a rebased and reworked version of the ITS emulation support.
> It allows KVM guests which use an emulated GICv3 to use LPIs as well,
> though in the moment this is limited to emulated PCI devices.
> It is now based on and requires the new VGIC implementation [1].
> 
> You can find all of this code (and the prerequisites) in the
> its-emul/v4 branch of my repository [2].
> 
> Cheers,
> Andre.
> 
> Changelog v3..v4:
> - adapting to new VGIC (changes in IRQ injection mechanism)
> 
> Changelog v2..v3:
> - adapt to 4.3-rc and Christoffer's timer rework
> - adapt spin locks on handling PROPBASER/PENDBASER registers
> - rework locking in ITS command handling (dropping dist where needed)
> - only clear LPI pending bit if LPI could actually be queued
> - simplify GICR_CTLR handling
> - properly free ITTEs (including our pending bitmap)
> - fix corner cases with unmapped collections
> - keep retire_lr() around
> - rename vgic_handle_base_register to vgic_reg64_access()
> - use kcalloc instead of kmalloc
> - minor fixes, renames and added comments
> 
> Changelog v1..v2
> - fix issues when using non-ITS GICv3 emulation
> - streamline frame address initialization (new patch 05/15)
> - preallocate buffer memory for reading from guest's memory
> - move locking into the actual command handlers
> -   preallocate memory for new structures if needed
> - use non-atomic __set_bit() and __clear_bit() when under the lock
> - add INT command handler to allow LPI injection from the guest
> - rewrite CWRITER handler to align with new locking scheme
> - remove unneeded CONFIG_HAVE_KVM_MSI #ifdefs
> - check memory table size against our LPI limit (65536 interrupts)
> - observe initial gap of 1024 interrupts in pending table
> - use term "configuration table" to be in line with the spec
> - clarify and extend documentation on API extensions
> - introduce new KVM_CAP_MSI_DEVID capability to advertise device ID
> requirement
> - update, fix and add many comments
> - minor style changes as requested by reviewers
> 
> ---------------
> 
> The GICv3 ITS (Interrupt Translation Service) is a part of the ARM GICv3
> interrupt controller [4] used for implementing MSIs.
> It specifies a new kind of interrupts (LPIs), which are mapped to establish a
> connection between a device, its MSI payload value and the target processor
> the IRQ is eventually delivered to.
> In order to allow using MSIs in an ARM64 KVM guest, we emulate this ITS
> widget in the kernel.
> The ITS works by reading commands written by software (from the guest in
> our case) into a (guest allocated) memory region and establishing the
> mapping between a device, the MSI payload and the target CPU.
> We parse these commands and update our internal data structures to reflect
> those changes. On an MSI injection we iterate those structures to learn the
> LPI number we have to inject.
> For the time being we use simple lists to hold the data, this is good enough
> for the small number of entries each of the components currently have.
> Should this become a performance bottleneck in the future, those can be
> extended to arrays or trees if needed.
> 
> Most of the code lives in a separate source file (its-emul.c), though there are
> some changes necessary in the existing VGIC files.
> 
> For the time being this series gives us the ability to use emulated PCI devices
> that can use MSIs in the guest. Those have to be triggered by letting the
> userland device emulation simulate the MSI write with the
> KVM_SIGNAL_MSI ioctl. This will be translated into the proper LPI by the ITS
> emulation and injected into the guest in the usual way (just with a higher IRQ
> number).
> 
> This series is based on 4.5 plus the KVM queue for 4.6 and the new VGIC
> emulation code and can be found at the its-emul/v4 branch of this repository
> [2].
> For this to be used you need a GICv3 host machine (a fast model would do),
> though it does not rely on any host ITS bits (neither in hardware or software).
> 
> To test this you can use the kvmtool patches available in the "its"
> branch here [3].
> Start a guest with: "$ lkvm run --irqchip=gicv3-its --force-pci"
> and see the ITS being used for instance by the virtio devices.
> 
> [1]: https://lists.cs.columbia.edu/pipermail/kvmarm/2016-
> March/019191.html
> [2]: git://linux-arm.org/linux-ap.git
>      http://www.linux-arm.org/git?p=linux-ap.git;a=log;h=refs/heads/its-
> emul/v4
> [3]: git://linux-arm.org/kvmtool.git
>      http://www.linux-arm.org/git?p=kvmtool.git;a=log;h=refs/heads/its
> [4]:
> http://arminfo.emea.arm.com/help/topic/com.arm.doc.ihi0069a/IHI0069A_g
> ic_architecture_specification.pdf
> 
> Andre Przywara (12):
>   KVM: extend struct kvm_msi to hold a 32-bit device ID
>   KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities
>   KVM: arm64: Introduce new MMIO region for the ITS base address
>   KVM: arm64: handle ITS related GICv3 redistributor registers
>   KVM: arm64: introduce ITS emulation file with stub functions
>   KVM: arm64: implement basic ITS register handlers
>   KVM: arm64: add data structures to model ITS interrupt translation
>   KVM: arm64: connect LPIs to the VGIC emulation
>   KVM: arm64: sync LPI configuration and pending tables
>   KVM: arm64: implement ITS command queue command handlers
>   KVM: arm64: implement MSI injection in ITS emulation
>   KVM: arm64: enable ITS emulation as a virtual MSI controller
> 
>  Documentation/virtual/kvm/api.txt              |   14 +-
>  Documentation/virtual/kvm/devices/arm-vgic.txt |    9 +
>  arch/arm/include/asm/kvm_host.h                |    2 +-
>  arch/arm/kvm/arm.c                             |    2 +-
>  arch/arm64/include/asm/kvm_host.h              |    2 +-
>  arch/arm64/include/uapi/asm/kvm.h              |    2 +
>  arch/arm64/kvm/Kconfig                         |    1 +
>  arch/arm64/kvm/Makefile                        |    1 +
>  arch/arm64/kvm/reset.c                         |   12 +-
>  include/kvm/vgic/vgic.h                        |   33 +
>  include/linux/irqchip/arm-gic-v3.h             |   28 +-
>  include/uapi/linux/kvm.h                       |    5 +-
>  virt/kvm/arm/vgic.c                            |    5 +
>  virt/kvm/arm/vgic/its-emul.c                   | 1146 ++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.c                       |   18 +
>  virt/kvm/arm/vgic/vgic.h                       |   33 +
>  virt/kvm/arm/vgic/vgic_init.c                  |    9 +
>  virt/kvm/arm/vgic/vgic_kvm_device.c            |    7 +
>  virt/kvm/arm/vgic/vgic_mmio.c                  |   54 +-
>  19 files changed, 1365 insertions(+), 18 deletions(-)  create mode 100644
> virt/kvm/arm/vgic/its-emul.c
> 
> --
> 2.7.3
> 
> _______________________________________________
> kvmarm mailing list
> kvmarm at lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables
  2016-04-06 13:41     ` Eric Auger
@ 2016-06-03 14:17       ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-06-03 14:17 UTC (permalink / raw)
  To: Eric Auger, Christoffer Dall, Marc Zyngier; +Cc: kvmarm, linux-arm-kernel, kvm

Salut Eric,

just some rationale below on why I didn't address all of your comments.

On 06/04/16 14:41, Eric Auger wrote:
> Hi Andre,
> On 03/26/2016 03:14 AM, Andre Przywara wrote:
>> The LPI configuration and pending tables of the GICv3 LPIs are held
>> in tables in (guest) memory. To achieve reasonable performance, we
>> cache this data in our own data structures, so we need to sync those
>> two views from time to time. This behaviour is well described in the
>> GICv3 spec and is also exercised by hardware, so the sync points are
>> well known.
>>
>> Provide functions that read the guest memory and store the
>> information from the configuration and pending tables in the kernel.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  include/kvm/vgic/vgic.h      |   2 +
>>  virt/kvm/arm/vgic/its-emul.c | 135 +++++++++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h     |   1 +
>>  3 files changed, 138 insertions(+)
>>
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index ecf3260..ea98133 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -121,6 +121,8 @@ struct vgic_its {
>>  	int			cwriter;
>>  	struct list_head	device_list;
>>  	struct list_head	collection_list;
>> +	/* memory used for buffering guest's memory */
>> +	void			*buffer_page;
>>  };
>>  
>>  struct vgic_dist {
>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>> index 1188e9a..d82ba9b 100644
>> --- a/virt/kvm/arm/vgic/its-emul.c
>> +++ b/virt/kvm/arm/vgic/its-emul.c
>> @@ -78,8 +78,130 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>>  	return NULL;
>>  }
>>  
>> +#define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
>> +#define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
>> +
>> +/* stores the priority and enable bit for a given LPI */
>> +static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
>> +{
>> +	spin_lock(&itte->irq.irq_lock);
>> +	itte->irq.priority = LPI_PROP_PRIORITY(prop);
>> +	itte->irq.enabled = LPI_PROP_ENABLE_BIT(prop);
>> +
> would put a comment saying that vgic_queue_irq is unlocking since it
> looks weird.
>> +	vgic_queue_irq(kvm, &itte->irq);
>> +}
>> +
>> +#define GIC_LPI_OFFSET 8192
>> +
>> +/* We scan the table in chunks the size of the smallest page size */
>> +#define CHUNK_SIZE 4096U
>> +
>>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>>  
>> +static int nr_idbits_propbase(u64 propbaser)
>> +{
>> +	int nr_idbits = (1U << (propbaser & 0x1f)) + 1;
>> +
>> +	return max(nr_idbits, INTERRUPT_ID_BITS_ITS);
> don't get this.
> don't you want to take the min of INTERRUPT_ID_BITS_ITS  (16) and
> propbaser & 0x1f +1.
> 
> Spec says IDbits should not exceed GICD_TYPER.IDbit ==
> INTERRUPT_ID_BITS_ITS - 1 = 15.
> 
> and in case the purpose of the function is to return the size of the
> table, which is not obvious given its name ;-), we should return 1U <<
> min(), no?

Oh dear, indeed that code was totally boll^Wnonsense. Thanks for
pointing this out!

>> +}
>> +
>> +/*
>> + * Scan the whole LPI configuration table and put the LPI configuration
>> + * data in our own data structures. This relies on the LPI being
>> + * mapped before.
>> + */
>> +static bool its_update_lpis_configuration(struct kvm *kvm, u64 prop_base_reg)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	u8 *prop = dist->its.buffer_page;
>> +	u32 tsize;
>> +	gpa_t propbase;
>> +	int lpi = GIC_LPI_OFFSET;
>> +	struct its_itte *itte;
>> +	struct its_device *device;
>> +	int ret;
>> +
>> +	propbase = BASER_BASE_ADDRESS(prop_base_reg);
>> +	tsize = nr_idbits_propbase(prop_base_reg);
>> +
>> +	while (tsize > 0) {
>> +		int chunksize = min(tsize, CHUNK_SIZE);
>> +
>> +		ret = kvm_read_guest(kvm, propbase, prop, chunksize);
>> +		if (ret)
>> +			return false;
>> +
>> +		spin_lock(&dist->its.lock);
> as we start holding several locks maybe worth to comment somewhere about
> order lock: its lock > vcpu lock > irq lock
>> +		/*
>> +		 * Updating the status for all allocated LPIs. We catch
>> +		 * those LPIs that get disabled. We really don't care
>> +		 * about unmapped LPIs, as they need to be updated
>> +		 * later manually anyway once they get mapped.
>> +		 */
>> +		for_each_lpi(device, itte, kvm) {
>> +			if (itte->lpi < lpi || itte->lpi >= lpi + chunksize)
>> +				continue;
>> +
>> +			update_lpi_config(kvm, itte, prop[itte->lpi - lpi]);
>> +		}
>> +		spin_unlock(&dist->its.lock);
>> +		tsize -= chunksize;
>> +		lpi += chunksize;
>> +		propbase += chunksize;
>> +	}
>> +
>> +	return true;
>> +}
>> +
>> +/*
>> + * Scan the whole LPI pending table and sync the pending bit in there
>> + * with our own data structures. This relies on the LPI being
>> + * mapped before.
>> + */
>> +static bool its_sync_lpi_pending_table(struct kvm_vcpu *vcpu, u64 base_addr_reg)
>> +{
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	unsigned long *pendmask = dist->its.buffer_page;
>> +	u32 nr_lpis = 1U << INTERRUPT_ID_BITS_ITS;
> shouldn't we use value computed as in nr_idbits_propbase instead?

In contrast to the PROPBASER there is not size information in PENDBASER,
also we can't use those bits from PROPBASER instead.
So I use the only other information we have about the number of LPIs,
which is what we report in TYPER as well.

>> +	gpa_t pendbase;
>> +	int lpi = 0;
>> +	struct its_itte *itte;
>> +	struct its_device *device;
>> +	int ret;
>> +	int lpi_bit, nr_bits;
>> +
>> +	pendbase = BASER_BASE_ADDRESS(base_addr_reg);
>> +
> I understand we should start reading at pendbase + 1KB. (4.8.5 pending
> tables)

Which stems from the fact that LPI numbers start at 8192, so the first
LPI would naturally start at 1KB. I use non-normalized LPI numbers
below, so that would be covered. Or did I miss something?

Cheers,
Andre.

> Eric
>> +	while (nr_lpis > 0) {
>> +		nr_bits = min(nr_lpis, CHUNK_SIZE * 8);
>> +
>> +		ret = kvm_read_guest(vcpu->kvm, pendbase, pendmask,
>> +				     nr_bits / 8);
>> +		if (ret)
>> +			return false;
>> +
>> +		spin_lock(&dist->its.lock);
>> +		for_each_lpi(device, itte, vcpu->kvm) {
>> +			lpi_bit = itte->lpi - lpi;
>> +			if (lpi_bit < 0 || lpi_bit >= nr_bits)
>> +				continue;
>> +
>> +			if (!test_bit(lpi_bit, pendmask))
>> +				continue;
>> +
>> +			spin_lock(&itte->irq.irq_lock);
>> +			itte->irq.pending = true;
>> +			vgic_queue_irq(vcpu->kvm, &itte->irq);
>> +		}
>> +		spin_unlock(&dist->its.lock);
>> +		nr_lpis -= nr_bits;
>> +		lpi += nr_bits;
>> +		pendbase += nr_bits / 8;
>> +	}
>> +
>> +	return true;
>> +}
>> +
>>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>>  				   struct kvm_io_device *this,
>>  				   gpa_t addr, int len, void *val)
>> @@ -357,6 +479,14 @@ struct vgic_register_region its_registers[] = {
>>  /* This is called on setting the LPI enable bit in the redistributor. */
>>  void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>>  {
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	u64 prop_base_reg, pend_base_reg;
>> +
>> +	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
>> +	prop_base_reg = dist->propbaser;
>> +
>> +	its_update_lpis_configuration(vcpu->kvm, prop_base_reg);
>> +	its_sync_lpi_pending_table(vcpu, pend_base_reg);
>>  }
>>  
>>  int vits_init(struct kvm *kvm)
>> @@ -371,6 +501,10 @@ int vits_init(struct kvm *kvm)
>>  	if (!dist->pendbaser)
>>  		return -ENOMEM;
>>  
>> +	its->buffer_page = kmalloc(CHUNK_SIZE, GFP_KERNEL);
>> +	if (!its->buffer_page)
>> +		return -ENOMEM;
>> +
>>  	spin_lock_init(&its->lock);
>>  
>>  	INIT_LIST_HEAD(&its->device_list);
>> @@ -430,6 +564,7 @@ void vits_destroy(struct kvm *kvm)
>>  		kfree(container_of(cur, struct its_collection, coll_list));
>>  	}
>>  
>> +	kfree(its->buffer_page);
>>  	kfree(dist->pendbaser);
>>  
>>  	its->enabled = false;
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 160c511..a7218b0 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -23,6 +23,7 @@
>>  #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
>>  
>>  #define INTERRUPT_ID_BITS_SPIS	10
>> +#define INTERRUPT_ID_BITS_ITS	16
>>  
>>  struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
>>  			      u32 intid);
>>
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables
@ 2016-06-03 14:17       ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-06-03 14:17 UTC (permalink / raw)
  To: linux-arm-kernel

Salut Eric,

just some rationale below on why I didn't address all of your comments.

On 06/04/16 14:41, Eric Auger wrote:
> Hi Andre,
> On 03/26/2016 03:14 AM, Andre Przywara wrote:
>> The LPI configuration and pending tables of the GICv3 LPIs are held
>> in tables in (guest) memory. To achieve reasonable performance, we
>> cache this data in our own data structures, so we need to sync those
>> two views from time to time. This behaviour is well described in the
>> GICv3 spec and is also exercised by hardware, so the sync points are
>> well known.
>>
>> Provide functions that read the guest memory and store the
>> information from the configuration and pending tables in the kernel.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  include/kvm/vgic/vgic.h      |   2 +
>>  virt/kvm/arm/vgic/its-emul.c | 135 +++++++++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h     |   1 +
>>  3 files changed, 138 insertions(+)
>>
>> diff --git a/include/kvm/vgic/vgic.h b/include/kvm/vgic/vgic.h
>> index ecf3260..ea98133 100644
>> --- a/include/kvm/vgic/vgic.h
>> +++ b/include/kvm/vgic/vgic.h
>> @@ -121,6 +121,8 @@ struct vgic_its {
>>  	int			cwriter;
>>  	struct list_head	device_list;
>>  	struct list_head	collection_list;
>> +	/* memory used for buffering guest's memory */
>> +	void			*buffer_page;
>>  };
>>  
>>  struct vgic_dist {
>> diff --git a/virt/kvm/arm/vgic/its-emul.c b/virt/kvm/arm/vgic/its-emul.c
>> index 1188e9a..d82ba9b 100644
>> --- a/virt/kvm/arm/vgic/its-emul.c
>> +++ b/virt/kvm/arm/vgic/its-emul.c
>> @@ -78,8 +78,130 @@ static struct its_itte *find_itte_by_lpi(struct kvm *kvm, int lpi)
>>  	return NULL;
>>  }
>>  
>> +#define LPI_PROP_ENABLE_BIT(p)	((p) & LPI_PROP_ENABLED)
>> +#define LPI_PROP_PRIORITY(p)	((p) & 0xfc)
>> +
>> +/* stores the priority and enable bit for a given LPI */
>> +static void update_lpi_config(struct kvm *kvm, struct its_itte *itte, u8 prop)
>> +{
>> +	spin_lock(&itte->irq.irq_lock);
>> +	itte->irq.priority = LPI_PROP_PRIORITY(prop);
>> +	itte->irq.enabled = LPI_PROP_ENABLE_BIT(prop);
>> +
> would put a comment saying that vgic_queue_irq is unlocking since it
> looks weird.
>> +	vgic_queue_irq(kvm, &itte->irq);
>> +}
>> +
>> +#define GIC_LPI_OFFSET 8192
>> +
>> +/* We scan the table in chunks the size of the smallest page size */
>> +#define CHUNK_SIZE 4096U
>> +
>>  #define BASER_BASE_ADDRESS(x) ((x) & 0xfffffffff000ULL)
>>  
>> +static int nr_idbits_propbase(u64 propbaser)
>> +{
>> +	int nr_idbits = (1U << (propbaser & 0x1f)) + 1;
>> +
>> +	return max(nr_idbits, INTERRUPT_ID_BITS_ITS);
> don't get this.
> don't you want to take the min of INTERRUPT_ID_BITS_ITS  (16) and
> propbaser & 0x1f +1.
> 
> Spec says IDbits should not exceed GICD_TYPER.IDbit ==
> INTERRUPT_ID_BITS_ITS - 1 = 15.
> 
> and in case the purpose of the function is to return the size of the
> table, which is not obvious given its name ;-), we should return 1U <<
> min(), no?

Oh dear, indeed that code was totally boll^Wnonsense. Thanks for
pointing this out!

>> +}
>> +
>> +/*
>> + * Scan the whole LPI configuration table and put the LPI configuration
>> + * data in our own data structures. This relies on the LPI being
>> + * mapped before.
>> + */
>> +static bool its_update_lpis_configuration(struct kvm *kvm, u64 prop_base_reg)
>> +{
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	u8 *prop = dist->its.buffer_page;
>> +	u32 tsize;
>> +	gpa_t propbase;
>> +	int lpi = GIC_LPI_OFFSET;
>> +	struct its_itte *itte;
>> +	struct its_device *device;
>> +	int ret;
>> +
>> +	propbase = BASER_BASE_ADDRESS(prop_base_reg);
>> +	tsize = nr_idbits_propbase(prop_base_reg);
>> +
>> +	while (tsize > 0) {
>> +		int chunksize = min(tsize, CHUNK_SIZE);
>> +
>> +		ret = kvm_read_guest(kvm, propbase, prop, chunksize);
>> +		if (ret)
>> +			return false;
>> +
>> +		spin_lock(&dist->its.lock);
> as we start holding several locks maybe worth to comment somewhere about
> order lock: its lock > vcpu lock > irq lock
>> +		/*
>> +		 * Updating the status for all allocated LPIs. We catch
>> +		 * those LPIs that get disabled. We really don't care
>> +		 * about unmapped LPIs, as they need to be updated
>> +		 * later manually anyway once they get mapped.
>> +		 */
>> +		for_each_lpi(device, itte, kvm) {
>> +			if (itte->lpi < lpi || itte->lpi >= lpi + chunksize)
>> +				continue;
>> +
>> +			update_lpi_config(kvm, itte, prop[itte->lpi - lpi]);
>> +		}
>> +		spin_unlock(&dist->its.lock);
>> +		tsize -= chunksize;
>> +		lpi += chunksize;
>> +		propbase += chunksize;
>> +	}
>> +
>> +	return true;
>> +}
>> +
>> +/*
>> + * Scan the whole LPI pending table and sync the pending bit in there
>> + * with our own data structures. This relies on the LPI being
>> + * mapped before.
>> + */
>> +static bool its_sync_lpi_pending_table(struct kvm_vcpu *vcpu, u64 base_addr_reg)
>> +{
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	unsigned long *pendmask = dist->its.buffer_page;
>> +	u32 nr_lpis = 1U << INTERRUPT_ID_BITS_ITS;
> shouldn't we use value computed as in nr_idbits_propbase instead?

In contrast to the PROPBASER there is not size information in PENDBASER,
also we can't use those bits from PROPBASER instead.
So I use the only other information we have about the number of LPIs,
which is what we report in TYPER as well.

>> +	gpa_t pendbase;
>> +	int lpi = 0;
>> +	struct its_itte *itte;
>> +	struct its_device *device;
>> +	int ret;
>> +	int lpi_bit, nr_bits;
>> +
>> +	pendbase = BASER_BASE_ADDRESS(base_addr_reg);
>> +
> I understand we should start reading at pendbase + 1KB. (4.8.5 pending
> tables)

Which stems from the fact that LPI numbers start at 8192, so the first
LPI would naturally start at 1KB. I use non-normalized LPI numbers
below, so that would be covered. Or did I miss something?

Cheers,
Andre.

> Eric
>> +	while (nr_lpis > 0) {
>> +		nr_bits = min(nr_lpis, CHUNK_SIZE * 8);
>> +
>> +		ret = kvm_read_guest(vcpu->kvm, pendbase, pendmask,
>> +				     nr_bits / 8);
>> +		if (ret)
>> +			return false;
>> +
>> +		spin_lock(&dist->its.lock);
>> +		for_each_lpi(device, itte, vcpu->kvm) {
>> +			lpi_bit = itte->lpi - lpi;
>> +			if (lpi_bit < 0 || lpi_bit >= nr_bits)
>> +				continue;
>> +
>> +			if (!test_bit(lpi_bit, pendmask))
>> +				continue;
>> +
>> +			spin_lock(&itte->irq.irq_lock);
>> +			itte->irq.pending = true;
>> +			vgic_queue_irq(vcpu->kvm, &itte->irq);
>> +		}
>> +		spin_unlock(&dist->its.lock);
>> +		nr_lpis -= nr_bits;
>> +		lpi += nr_bits;
>> +		pendbase += nr_bits / 8;
>> +	}
>> +
>> +	return true;
>> +}
>> +
>>  static int vgic_mmio_read_its_ctlr(struct kvm_vcpu *vcpu,
>>  				   struct kvm_io_device *this,
>>  				   gpa_t addr, int len, void *val)
>> @@ -357,6 +479,14 @@ struct vgic_register_region its_registers[] = {
>>  /* This is called on setting the LPI enable bit in the redistributor. */
>>  void vgic_enable_lpis(struct kvm_vcpu *vcpu)
>>  {
>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> +	u64 prop_base_reg, pend_base_reg;
>> +
>> +	pend_base_reg = dist->pendbaser[vcpu->vcpu_id];
>> +	prop_base_reg = dist->propbaser;
>> +
>> +	its_update_lpis_configuration(vcpu->kvm, prop_base_reg);
>> +	its_sync_lpi_pending_table(vcpu, pend_base_reg);
>>  }
>>  
>>  int vits_init(struct kvm *kvm)
>> @@ -371,6 +501,10 @@ int vits_init(struct kvm *kvm)
>>  	if (!dist->pendbaser)
>>  		return -ENOMEM;
>>  
>> +	its->buffer_page = kmalloc(CHUNK_SIZE, GFP_KERNEL);
>> +	if (!its->buffer_page)
>> +		return -ENOMEM;
>> +
>>  	spin_lock_init(&its->lock);
>>  
>>  	INIT_LIST_HEAD(&its->device_list);
>> @@ -430,6 +564,7 @@ void vits_destroy(struct kvm *kvm)
>>  		kfree(container_of(cur, struct its_collection, coll_list));
>>  	}
>>  
>> +	kfree(its->buffer_page);
>>  	kfree(dist->pendbaser);
>>  
>>  	its->enabled = false;
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index 160c511..a7218b0 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -23,6 +23,7 @@
>>  #define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
>>  
>>  #define INTERRUPT_ID_BITS_SPIS	10
>> +#define INTERRUPT_ID_BITS_ITS	16
>>  
>>  struct vgic_irq *vgic_get_irq(struct kvm *kvm, struct kvm_vcpu *vcpu,
>>  			      u32 intid);
>>
> 

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
  2016-06-03  4:26   ` Bharat Bhushan
@ 2016-06-03 14:32     ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-06-03 14:32 UTC (permalink / raw)
  To: Bharat Bhushan; +Cc: kvm, Marc Zyngier, kvmarm, linux-arm-kernel

Hi Bharat,

On 03/06/16 05:26, Bharat Bhushan wrote:
> Hi Andre,
> 
> We are looking for these patches but seems like these needed to
> rebased/redesigned based on new GIC emulation changes. Can you give us
> some sense on when this will be done?
> Also do you know if someone is working on QEMU size changes for
> this, if not then we can help on getting that done.

I just pushed a new version of the patches:
https://lists.cs.columbia.edu/pipermail/kvmarm/2016-June/020482.html

I hope that answers your question about the kernel side of things.
For QEMU I have no idea, maybe Christoffer or Peter would know.

Cheers,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
@ 2016-06-03 14:32     ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-06-03 14:32 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Bharat,

On 03/06/16 05:26, Bharat Bhushan wrote:
> Hi Andre,
> 
> We are looking for these patches but seems like these needed to
> rebased/redesigned based on new GIC emulation changes. Can you give us
> some sense on when this will be done?
> Also do you know if someone is working on QEMU size changes for
> this, if not then we can help on getting that done.

I just pushed a new version of the patches:
https://lists.cs.columbia.edu/pipermail/kvmarm/2016-June/020482.html

I hope that answers your question about the kernel side of things.
For QEMU I have no idea, maybe Christoffer or Peter would know.

Cheers,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-05-26  9:10         ` Marc Zyngier
@ 2016-06-03 15:42           ` Andre Przywara
  -1 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-06-03 15:42 UTC (permalink / raw)
  To: Marc Zyngier, Christoffer Dall, Eric Auger; +Cc: kvmarm, linux-arm-kernel, kvm

Hi,

I addressed some comments of yours in v5, just some rationale for the
others (sorry for the late reply on these!):

...

>>>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>>>> +				    struct kvm_io_device *this,
>>>> +				    gpa_t addr, int len, void *val)
>>>> +{
>>>> +	u64 reg = GITS_TYPER_PLPIS;
>>>> +
>>>> +	/*
>>>> +	 * We use linear CPU numbers for redistributor addressing,
>>>> +	 * so GITS_TYPER.PTA is 0.
>>>> +	 * To avoid memory waste on the guest side, we keep the
>>>> +	 * number of IDBits and DevBits low for the time being.
>>>> +	 * This could later be made configurable by userland.
>>>> +	 * Since we have all collections in linked list, we claim
>>>> +	 * that we can hold all of the collection tables in our
>>>> +	 * own memory and that the ITT entry size is 1 byte (the
>>>> +	 * smallest possible one).
>>>
>>> All of this is going to bite us when we want to implement migration,
>>> specially the HW collection bit.
>>
>> How so? Clearly we keep the number of VCPUs constant, so everything
>> guest visible stays the same even upon migrating to a much different
>> host? Or are we talking about different things here?
> 
> Let me rephrase this: How are you going to address HW collections from
> userspace in order to dump them? Short of having memory tables, you cannot.

I was just hoping for addressing first things first. Having the
collection table entirely in the kernel makes things easier for now.

How is that save/restore story covered by a hardware ITS? Is it just
ignored since it's not needed/used there?

That being said I would be happy to change this when we get migration
support for the ITS register state and we have agreed on an API for
userland ITS save/restore.

>>>
>>>> +	 */
>>>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>>>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>>>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>>>> +
>>>> +	write_mask64(reg, addr & 7, len, val);
>>>> +
>>>> +	return 0;
>>>> +}

.....

>>>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>>>> +				     struct kvm_io_device *this,
>>>> +				     gpa_t addr, int len, void *val)
>>>> +{
>>>> +	struct vgic_io_device *iodev = container_of(this,
>>>> +						    struct vgic_io_device, dev);
>>>> +	u32 reg = 0;
>>>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>>>> +
>>>> +	switch (idreg) {
>>>> +	case GITS_PIDR2:
>>>> +		reg = GIC_PIDR2_ARCH_GICv3;
>>>
>>> Are we leaving the lowest 4 bits to zero?
>>
>> I couldn't stuff "42" in 4 bits, so: yes ;-)
> 
> The 4 bottom bits are "Implementation Defined", so you could fit
> something in it if you wanted.
> 
>>
>>>> +		break;
>>>> +	case GITS_PIDR4:
>>>> +		/* This is a 64K software visible page */
>>>> +		reg = 0x40;
>>>
>>> Same question.
>>>
>>> Also, how about all the others PIDR registers?
>>
>> I guess I was halfway undecided between going with the pure
>> architectural requirements (which only mandate bits 7:4 in PIDR2) and
>> complying with the recommendation later in the spec.
>> So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.
>>
>>>> +		break;
>>>> +	/* Those are the ID registers for (any) GIC. */
>>>> +	case GITS_CIDR0:
>>>> +		reg = 0x0d;
>>>> +		break;
>>>> +	case GITS_CIDR1:
>>>> +		reg = 0xf0;
>>>> +		break;
>>>> +	case GITS_CIDR2:
>>>> +		reg = 0x05;
>>>> +		break;
>>>> +	case GITS_CIDR3:
>>>> +		reg = 0xb1;
>>>> +		break;
>>>> +	}
>>>
>>> Given that these values are directly taken from the architecture, and
>>> seem common to the whole GICv3 architecture when implemented by ARM, we
>>> could have a common handler for the whole GICv3 implementatuin. Not a
>>> bit deal though.

I tried it, but it turned out to be more involved than anticipated, so I
shelved it for now. I can make a patch on top later.

>>
>> Agreed.
>>
>>>> +
>>>> +	write_mask32(reg, addr & 3, len, val);
>>>> +
>>>> +	return 0;
>>>> +}

....

>>>> +/*
>>>> + * By writing to CWRITER the guest announces new commands to be processed.
>>>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>>>> + * iterate over the command buffer (with the lock dropped) until the read
>>>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>>>> + * meantime will just update the write pointer, leaving the command
>>>> + * processing to the first instance of the function.
>>>> + */
>>>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>>>> +				       struct kvm_io_device *this,
>>>> +				       gpa_t addr, int len, const void *val)
>>>> +{
>>>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>>> +	struct vgic_its *its = &dist->its;
>>>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>>>> +	u64 cmd_buf[4];
>>>> +	u32 reg;
>>>> +	bool finished;
>>>> +
>>>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>>>> +	reg &= 0xfffe0;
>>>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>>>> +		return 0;
>>>> +
>>>> +	spin_lock(&its->lock);
>>>> +
>>>> +	/*
>>>> +	 * If there is still another VCPU handling commands, let this
>>>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>>>> +	 */
>>>
>>> How do you detect that condition? All I see is a massive race here, with
>>> two threads processing the queue in parallel, possibly corrupting each
>>> other's data.
>>>
>>> Please explain why you think this is safe.
>>
>> OK, here you go: Actually we are handling the command queue
>> synchronously: The first writer to CWRITER will end up here and will
>> iterate over all commands below. From the guests point of view it will
>> take some time to do the CWRITER MMIO write, but upon the writel
>> returning the queue is actually already empty again.
>> That means that any _sane_ guest will _never_ trigger the case where two
>> threads/VCPUs are handling the queue, because a concurrent write to
>> CWRITER without a lock in the guest would be broken by design.
> 
> That feels very fragile, and you're relying on a well behaved guest.

No, I don't rely on it, as mentioned in the next sentence.
I just don't optimize for that case, because in reality it should never
happen - apart from a malicious guest. Let me think about the security
implications, though (whether a misbehaved guest could hog a physical CPU).

>> However I am not sure we can rely on this, so I added this precaution
>> scheme:

>> The first writer takes our ITS (emulation) lock and examines cwriter and
>> creadr to see if they are equal. If they are, then the queue is
>> currently empty, which means we are the first one and need to take care
>> of the commands. We update our copy of cwriter (now marking the queue as
>> non-empty) and drop the lock. As we kept the queue-was-empty status in a
>> local variable, we now can start iterating through the queue. We only
>> take the lock briefly when really needed in this process (for instance
>> when one command has been processed and creadr is incremented).
>>
>> If now a second VCPU writes CWRITER, we also take the lock and compare
>> creadr and cwriter. Now they are different, because the first thread is
>> still busy with handling the commands and hasn't finished yet.
>> So we set our local "finished" to true. Also we update cwriter to the
>> new value, then drop the lock. The while loop below will not be entered
>> in this thread and we return to the guest.
>> Now the first thread has handled another command, takes the lock to
>> increase creadr and compares it with cwriter. Without the second writer
>> it may have been finished already, but now cwriter has grown, so it will
>> just carry on with handling the commands.
>>
>> A bit like someone washing the dishes while another person adds some
>> more dirty plates to the pile ;-)
> 
> I'm sorry, this feels incredibly complicated, and will make actual
> debugging/tracing a nightmare. It also penalize the first vcpu that gets
> to submitting a command, leading to scheduling imbalances.

Well, as mentioned that would never happen apart from that
malicious/misbehaved guest case - so I don't care so much about
scheduling imbalances _for the guest_. As said above I have to think
about how this affects the host cores, though.

And frankly I don't see how this is overly complicated (in the context
of SMP locking schemes in general) - apart from me explaining it badly
above. It's just a bit tricky, but guarantees a strict order of command
processing.
On another approach we could just qualify a concurrent access as illegal
and ignore it - on real hardware you would have no guarantee for this to
work either - issuing two CWRITER MMIO writes at the same time would
just go bonkers.

> Why can't you simply turn the ITS lock into a mutex and keep it held,
> processing each batch in the context of the vcpu that is writing to
> GITC_CWRITER?

Frankly I spent quite some time to make this locking more fine grained
after some comments on earlier revisions - and I think it's better now,
since we don't hog all of the ITS data structures during the command
processing. Pessimistically one core could issue a big number of
commands, which would block the whole ITS.
Instead MSI injections happening in parallel now have a good chance of
iterating our lists and tables in between. This should improve scalability.

Thanks,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-06-03 15:42           ` Andre Przywara
  0 siblings, 0 replies; 120+ messages in thread
From: Andre Przywara @ 2016-06-03 15:42 UTC (permalink / raw)
  To: linux-arm-kernel

Hi,

I addressed some comments of yours in v5, just some rationale for the
others (sorry for the late reply on these!):

...

>>>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>>>> +				    struct kvm_io_device *this,
>>>> +				    gpa_t addr, int len, void *val)
>>>> +{
>>>> +	u64 reg = GITS_TYPER_PLPIS;
>>>> +
>>>> +	/*
>>>> +	 * We use linear CPU numbers for redistributor addressing,
>>>> +	 * so GITS_TYPER.PTA is 0.
>>>> +	 * To avoid memory waste on the guest side, we keep the
>>>> +	 * number of IDBits and DevBits low for the time being.
>>>> +	 * This could later be made configurable by userland.
>>>> +	 * Since we have all collections in linked list, we claim
>>>> +	 * that we can hold all of the collection tables in our
>>>> +	 * own memory and that the ITT entry size is 1 byte (the
>>>> +	 * smallest possible one).
>>>
>>> All of this is going to bite us when we want to implement migration,
>>> specially the HW collection bit.
>>
>> How so? Clearly we keep the number of VCPUs constant, so everything
>> guest visible stays the same even upon migrating to a much different
>> host? Or are we talking about different things here?
> 
> Let me rephrase this: How are you going to address HW collections from
> userspace in order to dump them? Short of having memory tables, you cannot.

I was just hoping for addressing first things first. Having the
collection table entirely in the kernel makes things easier for now.

How is that save/restore story covered by a hardware ITS? Is it just
ignored since it's not needed/used there?

That being said I would be happy to change this when we get migration
support for the ITS register state and we have agreed on an API for
userland ITS save/restore.

>>>
>>>> +	 */
>>>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>>>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>>>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>>>> +
>>>> +	write_mask64(reg, addr & 7, len, val);
>>>> +
>>>> +	return 0;
>>>> +}

.....

>>>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>>>> +				     struct kvm_io_device *this,
>>>> +				     gpa_t addr, int len, void *val)
>>>> +{
>>>> +	struct vgic_io_device *iodev = container_of(this,
>>>> +						    struct vgic_io_device, dev);
>>>> +	u32 reg = 0;
>>>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>>>> +
>>>> +	switch (idreg) {
>>>> +	case GITS_PIDR2:
>>>> +		reg = GIC_PIDR2_ARCH_GICv3;
>>>
>>> Are we leaving the lowest 4 bits to zero?
>>
>> I couldn't stuff "42" in 4 bits, so: yes ;-)
> 
> The 4 bottom bits are "Implementation Defined", so you could fit
> something in it if you wanted.
> 
>>
>>>> +		break;
>>>> +	case GITS_PIDR4:
>>>> +		/* This is a 64K software visible page */
>>>> +		reg = 0x40;
>>>
>>> Same question.
>>>
>>> Also, how about all the others PIDR registers?
>>
>> I guess I was halfway undecided between going with the pure
>> architectural requirements (which only mandate bits 7:4 in PIDR2) and
>> complying with the recommendation later in the spec.
>> So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.
>>
>>>> +		break;
>>>> +	/* Those are the ID registers for (any) GIC. */
>>>> +	case GITS_CIDR0:
>>>> +		reg = 0x0d;
>>>> +		break;
>>>> +	case GITS_CIDR1:
>>>> +		reg = 0xf0;
>>>> +		break;
>>>> +	case GITS_CIDR2:
>>>> +		reg = 0x05;
>>>> +		break;
>>>> +	case GITS_CIDR3:
>>>> +		reg = 0xb1;
>>>> +		break;
>>>> +	}
>>>
>>> Given that these values are directly taken from the architecture, and
>>> seem common to the whole GICv3 architecture when implemented by ARM, we
>>> could have a common handler for the whole GICv3 implementatuin. Not a
>>> bit deal though.

I tried it, but it turned out to be more involved than anticipated, so I
shelved it for now. I can make a patch on top later.

>>
>> Agreed.
>>
>>>> +
>>>> +	write_mask32(reg, addr & 3, len, val);
>>>> +
>>>> +	return 0;
>>>> +}

....

>>>> +/*
>>>> + * By writing to CWRITER the guest announces new commands to be processed.
>>>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>>>> + * iterate over the command buffer (with the lock dropped) until the read
>>>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>>>> + * meantime will just update the write pointer, leaving the command
>>>> + * processing to the first instance of the function.
>>>> + */
>>>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>>>> +				       struct kvm_io_device *this,
>>>> +				       gpa_t addr, int len, const void *val)
>>>> +{
>>>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>>> +	struct vgic_its *its = &dist->its;
>>>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>>>> +	u64 cmd_buf[4];
>>>> +	u32 reg;
>>>> +	bool finished;
>>>> +
>>>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>>>> +	reg &= 0xfffe0;
>>>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>>>> +		return 0;
>>>> +
>>>> +	spin_lock(&its->lock);
>>>> +
>>>> +	/*
>>>> +	 * If there is still another VCPU handling commands, let this
>>>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>>>> +	 */
>>>
>>> How do you detect that condition? All I see is a massive race here, with
>>> two threads processing the queue in parallel, possibly corrupting each
>>> other's data.
>>>
>>> Please explain why you think this is safe.
>>
>> OK, here you go: Actually we are handling the command queue
>> synchronously: The first writer to CWRITER will end up here and will
>> iterate over all commands below. From the guests point of view it will
>> take some time to do the CWRITER MMIO write, but upon the writel
>> returning the queue is actually already empty again.
>> That means that any _sane_ guest will _never_ trigger the case where two
>> threads/VCPUs are handling the queue, because a concurrent write to
>> CWRITER without a lock in the guest would be broken by design.
> 
> That feels very fragile, and you're relying on a well behaved guest.

No, I don't rely on it, as mentioned in the next sentence.
I just don't optimize for that case, because in reality it should never
happen - apart from a malicious guest. Let me think about the security
implications, though (whether a misbehaved guest could hog a physical CPU).

>> However I am not sure we can rely on this, so I added this precaution
>> scheme:

>> The first writer takes our ITS (emulation) lock and examines cwriter and
>> creadr to see if they are equal. If they are, then the queue is
>> currently empty, which means we are the first one and need to take care
>> of the commands. We update our copy of cwriter (now marking the queue as
>> non-empty) and drop the lock. As we kept the queue-was-empty status in a
>> local variable, we now can start iterating through the queue. We only
>> take the lock briefly when really needed in this process (for instance
>> when one command has been processed and creadr is incremented).
>>
>> If now a second VCPU writes CWRITER, we also take the lock and compare
>> creadr and cwriter. Now they are different, because the first thread is
>> still busy with handling the commands and hasn't finished yet.
>> So we set our local "finished" to true. Also we update cwriter to the
>> new value, then drop the lock. The while loop below will not be entered
>> in this thread and we return to the guest.
>> Now the first thread has handled another command, takes the lock to
>> increase creadr and compares it with cwriter. Without the second writer
>> it may have been finished already, but now cwriter has grown, so it will
>> just carry on with handling the commands.
>>
>> A bit like someone washing the dishes while another person adds some
>> more dirty plates to the pile ;-)
> 
> I'm sorry, this feels incredibly complicated, and will make actual
> debugging/tracing a nightmare. It also penalize the first vcpu that gets
> to submitting a command, leading to scheduling imbalances.

Well, as mentioned that would never happen apart from that
malicious/misbehaved guest case - so I don't care so much about
scheduling imbalances _for the guest_. As said above I have to think
about how this affects the host cores, though.

And frankly I don't see how this is overly complicated (in the context
of SMP locking schemes in general) - apart from me explaining it badly
above. It's just a bit tricky, but guarantees a strict order of command
processing.
On another approach we could just qualify a concurrent access as illegal
and ignore it - on real hardware you would have no guarantee for this to
work either - issuing two CWRITER MMIO writes at the same time would
just go bonkers.

> Why can't you simply turn the ITS lock into a mutex and keep it held,
> processing each batch in the context of the vcpu that is writing to
> GITC_CWRITER?

Frankly I spent quite some time to make this locking more fine grained
after some comments on earlier revisions - and I think it's better now,
since we don't hog all of the ITS data structures during the command
processing. Pessimistically one core could issue a big number of
commands, which would block the whole ITS.
Instead MSI injections happening in parallel now have a good chance of
iterating our lists and tables in between. This should improve scalability.

Thanks,
Andre.

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
  2016-06-03 15:42           ` Andre Przywara
@ 2016-06-03 16:54             ` Marc Zyngier
  -1 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-06-03 16:54 UTC (permalink / raw)
  To: Andre Przywara, Christoffer Dall, Eric Auger
  Cc: linux-arm-kernel, kvmarm, kvm

On 03/06/16 16:42, Andre Przywara wrote:
> Hi,
> 
> I addressed some comments of yours in v5, just some rationale for the
> others (sorry for the late reply on these!):
> 
> ...
> 
>>>>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>>>>> +				    struct kvm_io_device *this,
>>>>> +				    gpa_t addr, int len, void *val)
>>>>> +{
>>>>> +	u64 reg = GITS_TYPER_PLPIS;
>>>>> +
>>>>> +	/*
>>>>> +	 * We use linear CPU numbers for redistributor addressing,
>>>>> +	 * so GITS_TYPER.PTA is 0.
>>>>> +	 * To avoid memory waste on the guest side, we keep the
>>>>> +	 * number of IDBits and DevBits low for the time being.
>>>>> +	 * This could later be made configurable by userland.
>>>>> +	 * Since we have all collections in linked list, we claim
>>>>> +	 * that we can hold all of the collection tables in our
>>>>> +	 * own memory and that the ITT entry size is 1 byte (the
>>>>> +	 * smallest possible one).
>>>>
>>>> All of this is going to bite us when we want to implement migration,
>>>> specially the HW collection bit.
>>>
>>> How so? Clearly we keep the number of VCPUs constant, so everything
>>> guest visible stays the same even upon migrating to a much different
>>> host? Or are we talking about different things here?
>>
>> Let me rephrase this: How are you going to address HW collections from
>> userspace in order to dump them? Short of having memory tables, you cannot.
> 
> I was just hoping for addressing first things first. Having the
> collection table entirely in the kernel makes things easier for now.

I can see that. Yet this is not a sustainable approach.

> How is that save/restore story covered by a hardware ITS? Is it just
> ignored since it's not needed/used there?

The save restore story is covered by having tables in memory, and it is
not possible to save/restore a HW collection, full stop. There is no
register to dump it.

> That being said I would be happy to change this when we get migration
> support for the ITS register state and we have agreed on an API for
> userland ITS save/restore.
> 
>>>>
>>>>> +	 */
>>>>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>>>>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>>>>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>>>>> +
>>>>> +	write_mask64(reg, addr & 7, len, val);
>>>>> +
>>>>> +	return 0;
>>>>> +}
> 
> .....
> 
>>>>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>>>>> +				     struct kvm_io_device *this,
>>>>> +				     gpa_t addr, int len, void *val)
>>>>> +{
>>>>> +	struct vgic_io_device *iodev = container_of(this,
>>>>> +						    struct vgic_io_device, dev);
>>>>> +	u32 reg = 0;
>>>>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>>>>> +
>>>>> +	switch (idreg) {
>>>>> +	case GITS_PIDR2:
>>>>> +		reg = GIC_PIDR2_ARCH_GICv3;
>>>>
>>>> Are we leaving the lowest 4 bits to zero?
>>>
>>> I couldn't stuff "42" in 4 bits, so: yes ;-)
>>
>> The 4 bottom bits are "Implementation Defined", so you could fit
>> something in it if you wanted.
>>
>>>
>>>>> +		break;
>>>>> +	case GITS_PIDR4:
>>>>> +		/* This is a 64K software visible page */
>>>>> +		reg = 0x40;
>>>>
>>>> Same question.
>>>>
>>>> Also, how about all the others PIDR registers?
>>>
>>> I guess I was halfway undecided between going with the pure
>>> architectural requirements (which only mandate bits 7:4 in PIDR2) and
>>> complying with the recommendation later in the spec.
>>> So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.
>>>
>>>>> +		break;
>>>>> +	/* Those are the ID registers for (any) GIC. */
>>>>> +	case GITS_CIDR0:
>>>>> +		reg = 0x0d;
>>>>> +		break;
>>>>> +	case GITS_CIDR1:
>>>>> +		reg = 0xf0;
>>>>> +		break;
>>>>> +	case GITS_CIDR2:
>>>>> +		reg = 0x05;
>>>>> +		break;
>>>>> +	case GITS_CIDR3:
>>>>> +		reg = 0xb1;
>>>>> +		break;
>>>>> +	}
>>>>
>>>> Given that these values are directly taken from the architecture, and
>>>> seem common to the whole GICv3 architecture when implemented by ARM, we
>>>> could have a common handler for the whole GICv3 implementatuin. Not a
>>>> bit deal though.
> 
> I tried it, but it turned out to be more involved than anticipated, so I
> shelved it for now. I can make a patch on top later.
> 
>>>
>>> Agreed.
>>>
>>>>> +
>>>>> +	write_mask32(reg, addr & 3, len, val);
>>>>> +
>>>>> +	return 0;
>>>>> +}
> 
> ....
> 
>>>>> +/*
>>>>> + * By writing to CWRITER the guest announces new commands to be processed.
>>>>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>>>>> + * iterate over the command buffer (with the lock dropped) until the read
>>>>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>>>>> + * meantime will just update the write pointer, leaving the command
>>>>> + * processing to the first instance of the function.
>>>>> + */
>>>>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>>>>> +				       struct kvm_io_device *this,
>>>>> +				       gpa_t addr, int len, const void *val)
>>>>> +{
>>>>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>>>> +	struct vgic_its *its = &dist->its;
>>>>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>>>>> +	u64 cmd_buf[4];
>>>>> +	u32 reg;
>>>>> +	bool finished;
>>>>> +
>>>>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>>>>> +	reg &= 0xfffe0;
>>>>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>>>>> +		return 0;
>>>>> +
>>>>> +	spin_lock(&its->lock);
>>>>> +
>>>>> +	/*
>>>>> +	 * If there is still another VCPU handling commands, let this
>>>>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>>>>> +	 */
>>>>
>>>> How do you detect that condition? All I see is a massive race here, with
>>>> two threads processing the queue in parallel, possibly corrupting each
>>>> other's data.
>>>>
>>>> Please explain why you think this is safe.
>>>
>>> OK, here you go: Actually we are handling the command queue
>>> synchronously: The first writer to CWRITER will end up here and will
>>> iterate over all commands below. From the guests point of view it will
>>> take some time to do the CWRITER MMIO write, but upon the writel
>>> returning the queue is actually already empty again.
>>> That means that any _sane_ guest will _never_ trigger the case where two
>>> threads/VCPUs are handling the queue, because a concurrent write to
>>> CWRITER without a lock in the guest would be broken by design.
>>
>> That feels very fragile, and you're relying on a well behaved guest.
> 
> No, I don't rely on it, as mentioned in the next sentence.
> I just don't optimize for that case, because in reality it should never
> happen - apart from a malicious guest. Let me think about the security
> implications, though (whether a misbehaved guest could hog a physical CPU).
> 
>>> However I am not sure we can rely on this, so I added this precaution
>>> scheme:
> 
>>> The first writer takes our ITS (emulation) lock and examines cwriter and
>>> creadr to see if they are equal. If they are, then the queue is
>>> currently empty, which means we are the first one and need to take care
>>> of the commands. We update our copy of cwriter (now marking the queue as
>>> non-empty) and drop the lock. As we kept the queue-was-empty status in a
>>> local variable, we now can start iterating through the queue. We only
>>> take the lock briefly when really needed in this process (for instance
>>> when one command has been processed and creadr is incremented).
>>>
>>> If now a second VCPU writes CWRITER, we also take the lock and compare
>>> creadr and cwriter. Now they are different, because the first thread is
>>> still busy with handling the commands and hasn't finished yet.
>>> So we set our local "finished" to true. Also we update cwriter to the
>>> new value, then drop the lock. The while loop below will not be entered
>>> in this thread and we return to the guest.
>>> Now the first thread has handled another command, takes the lock to
>>> increase creadr and compares it with cwriter. Without the second writer
>>> it may have been finished already, but now cwriter has grown, so it will
>>> just carry on with handling the commands.
>>>
>>> A bit like someone washing the dishes while another person adds some
>>> more dirty plates to the pile ;-)
>>
>> I'm sorry, this feels incredibly complicated, and will make actual
>> debugging/tracing a nightmare. It also penalize the first vcpu that gets
>> to submitting a command, leading to scheduling imbalances.
> 
> Well, as mentioned that would never happen apart from that
> malicious/misbehaved guest case - so I don't care so much about
> scheduling imbalances _for the guest_. As said above I have to think
> about how this affects the host cores, though.
> 
> And frankly I don't see how this is overly complicated (in the context
> of SMP locking schemes in general) - apart from me explaining it badly
> above. It's just a bit tricky, but guarantees a strict order of command
> processing.
> On another approach we could just qualify a concurrent access as illegal
> and ignore it - on real hardware you would have no guarantee for this to
> work either - issuing two CWRITER MMIO writes at the same time would
> just go bonkers.

There is nothing in the spec that indicates that what you're proposing
is legal. And it is likely that the HW would simply serialize the two
write cycles (unless you have a multi-ported register). Bear in mind
that the ITS has been designed for multiprocessor systems (funnily enough).

> 
>> Why can't you simply turn the ITS lock into a mutex and keep it held,
>> processing each batch in the context of the vcpu that is writing to
>> GITC_CWRITER?
> 
> Frankly I spent quite some time to make this locking more fine grained
> after some comments on earlier revisions - and I think it's better now,
> since we don't hog all of the ITS data structures during the command
> processing. Pessimistically one core could issue a big number of
> commands, which would block the whole ITS.
> Instead MSI injections happening in parallel now have a good chance of
> iterating our lists and tables in between. This should improve scalability.

I don't see how having a single "its->lock" makes it "fine-grained".
You're releasing the lock early in order not to cause other issues by
taking this spinlock twice. I call that papering over the issue.

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers
@ 2016-06-03 16:54             ` Marc Zyngier
  0 siblings, 0 replies; 120+ messages in thread
From: Marc Zyngier @ 2016-06-03 16:54 UTC (permalink / raw)
  To: linux-arm-kernel

On 03/06/16 16:42, Andre Przywara wrote:
> Hi,
> 
> I addressed some comments of yours in v5, just some rationale for the
> others (sorry for the late reply on these!):
> 
> ...
> 
>>>>> +static int vgic_mmio_read_its_typer(struct kvm_vcpu *vcpu,
>>>>> +				    struct kvm_io_device *this,
>>>>> +				    gpa_t addr, int len, void *val)
>>>>> +{
>>>>> +	u64 reg = GITS_TYPER_PLPIS;
>>>>> +
>>>>> +	/*
>>>>> +	 * We use linear CPU numbers for redistributor addressing,
>>>>> +	 * so GITS_TYPER.PTA is 0.
>>>>> +	 * To avoid memory waste on the guest side, we keep the
>>>>> +	 * number of IDBits and DevBits low for the time being.
>>>>> +	 * This could later be made configurable by userland.
>>>>> +	 * Since we have all collections in linked list, we claim
>>>>> +	 * that we can hold all of the collection tables in our
>>>>> +	 * own memory and that the ITT entry size is 1 byte (the
>>>>> +	 * smallest possible one).
>>>>
>>>> All of this is going to bite us when we want to implement migration,
>>>> specially the HW collection bit.
>>>
>>> How so? Clearly we keep the number of VCPUs constant, so everything
>>> guest visible stays the same even upon migrating to a much different
>>> host? Or are we talking about different things here?
>>
>> Let me rephrase this: How are you going to address HW collections from
>> userspace in order to dump them? Short of having memory tables, you cannot.
> 
> I was just hoping for addressing first things first. Having the
> collection table entirely in the kernel makes things easier for now.

I can see that. Yet this is not a sustainable approach.

> How is that save/restore story covered by a hardware ITS? Is it just
> ignored since it's not needed/used there?

The save restore story is covered by having tables in memory, and it is
not possible to save/restore a HW collection, full stop. There is no
register to dump it.

> That being said I would be happy to change this when we get migration
> support for the ITS register state and we have agreed on an API for
> userland ITS save/restore.
> 
>>>>
>>>>> +	 */
>>>>> +	reg |= 0xff << GITS_TYPER_HWCOLLCNT_SHIFT;
>>>>> +	reg |= 0x0f << GITS_TYPER_DEVBITS_SHIFT;
>>>>> +	reg |= 0x0f << GITS_TYPER_IDBITS_SHIFT;
>>>>> +
>>>>> +	write_mask64(reg, addr & 7, len, val);
>>>>> +
>>>>> +	return 0;
>>>>> +}
> 
> .....
> 
>>>>> +static int vgic_mmio_read_its_idregs(struct kvm_vcpu *vcpu,
>>>>> +				     struct kvm_io_device *this,
>>>>> +				     gpa_t addr, int len, void *val)
>>>>> +{
>>>>> +	struct vgic_io_device *iodev = container_of(this,
>>>>> +						    struct vgic_io_device, dev);
>>>>> +	u32 reg = 0;
>>>>> +	int idreg = (addr & ~3) - iodev->base_addr + GITS_IDREGS_BASE;
>>>>> +
>>>>> +	switch (idreg) {
>>>>> +	case GITS_PIDR2:
>>>>> +		reg = GIC_PIDR2_ARCH_GICv3;
>>>>
>>>> Are we leaving the lowest 4 bits to zero?
>>>
>>> I couldn't stuff "42" in 4 bits, so: yes ;-)
>>
>> The 4 bottom bits are "Implementation Defined", so you could fit
>> something in it if you wanted.
>>
>>>
>>>>> +		break;
>>>>> +	case GITS_PIDR4:
>>>>> +		/* This is a 64K software visible page */
>>>>> +		reg = 0x40;
>>>>
>>>> Same question.
>>>>
>>>> Also, how about all the others PIDR registers?
>>>
>>> I guess I was halfway undecided between going with the pure
>>> architectural requirements (which only mandate bits 7:4 in PIDR2) and
>>> complying with the recommendation later in the spec.
>>> So I will just fill PIDR0, PIDR2 and the rest of PIDR2 as well.
>>>
>>>>> +		break;
>>>>> +	/* Those are the ID registers for (any) GIC. */
>>>>> +	case GITS_CIDR0:
>>>>> +		reg = 0x0d;
>>>>> +		break;
>>>>> +	case GITS_CIDR1:
>>>>> +		reg = 0xf0;
>>>>> +		break;
>>>>> +	case GITS_CIDR2:
>>>>> +		reg = 0x05;
>>>>> +		break;
>>>>> +	case GITS_CIDR3:
>>>>> +		reg = 0xb1;
>>>>> +		break;
>>>>> +	}
>>>>
>>>> Given that these values are directly taken from the architecture, and
>>>> seem common to the whole GICv3 architecture when implemented by ARM, we
>>>> could have a common handler for the whole GICv3 implementatuin. Not a
>>>> bit deal though.
> 
> I tried it, but it turned out to be more involved than anticipated, so I
> shelved it for now. I can make a patch on top later.
> 
>>>
>>> Agreed.
>>>
>>>>> +
>>>>> +	write_mask32(reg, addr & 3, len, val);
>>>>> +
>>>>> +	return 0;
>>>>> +}
> 
> ....
> 
>>>>> +/*
>>>>> + * By writing to CWRITER the guest announces new commands to be processed.
>>>>> + * Since we cannot read from guest memory inside the ITS spinlock, we
>>>>> + * iterate over the command buffer (with the lock dropped) until the read
>>>>> + * pointer matches the write pointer. Other VCPUs writing this register in the
>>>>> + * meantime will just update the write pointer, leaving the command
>>>>> + * processing to the first instance of the function.
>>>>> + */
>>>>> +static int vgic_mmio_write_its_cwriter(struct kvm_vcpu *vcpu,
>>>>> +				       struct kvm_io_device *this,
>>>>> +				       gpa_t addr, int len, const void *val)
>>>>> +{
>>>>> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>>>> +	struct vgic_its *its = &dist->its;
>>>>> +	gpa_t cbaser = its_cmd_buffer_base(vcpu->kvm);
>>>>> +	u64 cmd_buf[4];
>>>>> +	u32 reg;
>>>>> +	bool finished;
>>>>> +
>>>>> +	reg = mask64(its->cwriter & 0xfffe0, addr & 7, len, val);
>>>>> +	reg &= 0xfffe0;
>>>>> +	if (reg > its_cmd_buffer_size(vcpu->kvm))
>>>>> +		return 0;
>>>>> +
>>>>> +	spin_lock(&its->lock);
>>>>> +
>>>>> +	/*
>>>>> +	 * If there is still another VCPU handling commands, let this
>>>>> +	 * one pick up the new CWRITER and process "our" new commands as well.
>>>>> +	 */
>>>>
>>>> How do you detect that condition? All I see is a massive race here, with
>>>> two threads processing the queue in parallel, possibly corrupting each
>>>> other's data.
>>>>
>>>> Please explain why you think this is safe.
>>>
>>> OK, here you go: Actually we are handling the command queue
>>> synchronously: The first writer to CWRITER will end up here and will
>>> iterate over all commands below. From the guests point of view it will
>>> take some time to do the CWRITER MMIO write, but upon the writel
>>> returning the queue is actually already empty again.
>>> That means that any _sane_ guest will _never_ trigger the case where two
>>> threads/VCPUs are handling the queue, because a concurrent write to
>>> CWRITER without a lock in the guest would be broken by design.
>>
>> That feels very fragile, and you're relying on a well behaved guest.
> 
> No, I don't rely on it, as mentioned in the next sentence.
> I just don't optimize for that case, because in reality it should never
> happen - apart from a malicious guest. Let me think about the security
> implications, though (whether a misbehaved guest could hog a physical CPU).
> 
>>> However I am not sure we can rely on this, so I added this precaution
>>> scheme:
> 
>>> The first writer takes our ITS (emulation) lock and examines cwriter and
>>> creadr to see if they are equal. If they are, then the queue is
>>> currently empty, which means we are the first one and need to take care
>>> of the commands. We update our copy of cwriter (now marking the queue as
>>> non-empty) and drop the lock. As we kept the queue-was-empty status in a
>>> local variable, we now can start iterating through the queue. We only
>>> take the lock briefly when really needed in this process (for instance
>>> when one command has been processed and creadr is incremented).
>>>
>>> If now a second VCPU writes CWRITER, we also take the lock and compare
>>> creadr and cwriter. Now they are different, because the first thread is
>>> still busy with handling the commands and hasn't finished yet.
>>> So we set our local "finished" to true. Also we update cwriter to the
>>> new value, then drop the lock. The while loop below will not be entered
>>> in this thread and we return to the guest.
>>> Now the first thread has handled another command, takes the lock to
>>> increase creadr and compares it with cwriter. Without the second writer
>>> it may have been finished already, but now cwriter has grown, so it will
>>> just carry on with handling the commands.
>>>
>>> A bit like someone washing the dishes while another person adds some
>>> more dirty plates to the pile ;-)
>>
>> I'm sorry, this feels incredibly complicated, and will make actual
>> debugging/tracing a nightmare. It also penalize the first vcpu that gets
>> to submitting a command, leading to scheduling imbalances.
> 
> Well, as mentioned that would never happen apart from that
> malicious/misbehaved guest case - so I don't care so much about
> scheduling imbalances _for the guest_. As said above I have to think
> about how this affects the host cores, though.
> 
> And frankly I don't see how this is overly complicated (in the context
> of SMP locking schemes in general) - apart from me explaining it badly
> above. It's just a bit tricky, but guarantees a strict order of command
> processing.
> On another approach we could just qualify a concurrent access as illegal
> and ignore it - on real hardware you would have no guarantee for this to
> work either - issuing two CWRITER MMIO writes at the same time would
> just go bonkers.

There is nothing in the spec that indicates that what you're proposing
is legal. And it is likely that the HW would simply serialize the two
write cycles (unless you have a multi-ported register). Bear in mind
that the ITS has been designed for multiprocessor systems (funnily enough).

> 
>> Why can't you simply turn the ITS lock into a mutex and keep it held,
>> processing each batch in the context of the vcpu that is writing to
>> GITC_CWRITER?
> 
> Frankly I spent quite some time to make this locking more fine grained
> after some comments on earlier revisions - and I think it's better now,
> since we don't hog all of the ITS data structures during the command
> processing. Pessimistically one core could issue a big number of
> commands, which would block the whole ITS.
> Instead MSI injections happening in parallel now have a good chance of
> iterating our lists and tables in between. This should improve scalability.

I don't see how having a single "its->lock" makes it "fine-grained".
You're releasing the lock early in order not to cause other issues by
taking this spinlock twice. I call that papering over the issue.

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 120+ messages in thread
* RE: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
  2016-06-03 14:32     ` Andre Przywara
@ 2016-06-06  5:29       ` Bharat Bhushan
  -1 siblings, 0 replies; 120+ messages in thread
From: Bharat Bhushan @ 2016-06-06  5:29 UTC (permalink / raw)
  To: Andre Przywara
  Cc: Marc Zyngier, Diana Madalina Craciun, kvmarm, kvm, linux-arm-kernel

Hi Andre,

On clarification, are V5 version of patches are based on new-GIC emulation patches? How are you testing these patches, kvm-tool?

Hi Christoffer/Peter,

Is someone working on QEMU side patches? We need this feature a bit urgently so if no one is working then we can start working on this.

Thanks
-Bharat

> -----Original Message-----
> From: kvmarm-bounces@lists.cs.columbia.edu [mailto:kvmarm-
> bounces@lists.cs.columbia.edu] On Behalf Of Andre Przywara
> Sent: Friday, June 03, 2016 8:02 PM
> To: Bharat Bhushan <bharat.bhushan@nxp.com>
> Cc: kvm@vger.kernel.org; Marc Zyngier <marc.zyngier@arm.com>;
> kvmarm@lists.cs.columbia.edu; linux-arm-kernel@lists.infradead.org
> Subject: Re: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
> 
> Hi Bharat,
> 
> On 03/06/16 05:26, Bharat Bhushan wrote:
> > Hi Andre,
> >
> > We are looking for these patches but seems like these needed to
> > rebased/redesigned based on new GIC emulation changes. Can you give us
> > some sense on when this will be done?
> > Also do you know if someone is working on QEMU size changes for this,
> > if not then we can help on getting that done.
> 
> I just pushed a new version of the patches:
> https://lists.cs.columbia.edu/pipermail/kvmarm/2016-June/020482.html
> 
> I hope that answers your question about the kernel side of things.
> For QEMU I have no idea, maybe Christoffer or Peter would know.
> 
> Cheers,
> Andre.
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
@ 2016-06-06  5:29       ` Bharat Bhushan
  0 siblings, 0 replies; 120+ messages in thread
From: Bharat Bhushan @ 2016-06-06  5:29 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Andre,

On clarification, are V5 version of patches are based on new-GIC emulation patches? How are you testing these patches, kvm-tool?

Hi Christoffer/Peter,

Is someone working on QEMU side patches? We need this feature a bit urgently so if no one is working then we can start working on this.

Thanks
-Bharat

> -----Original Message-----
> From: kvmarm-bounces at lists.cs.columbia.edu [mailto:kvmarm-
> bounces at lists.cs.columbia.edu] On Behalf Of Andre Przywara
> Sent: Friday, June 03, 2016 8:02 PM
> To: Bharat Bhushan <bharat.bhushan@nxp.com>
> Cc: kvm at vger.kernel.org; Marc Zyngier <marc.zyngier@arm.com>;
> kvmarm at lists.cs.columbia.edu; linux-arm-kernel at lists.infradead.org
> Subject: Re: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
> 
> Hi Bharat,
> 
> On 03/06/16 05:26, Bharat Bhushan wrote:
> > Hi Andre,
> >
> > We are looking for these patches but seems like these needed to
> > rebased/redesigned based on new GIC emulation changes. Can you give us
> > some sense on when this will be done?
> > Also do you know if someone is working on QEMU size changes for this,
> > if not then we can help on getting that done.
> 
> I just pushed a new version of the patches:
> https://lists.cs.columbia.edu/pipermail/kvmarm/2016-June/020482.html
> 
> I hope that answers your question about the kernel side of things.
> For QEMU I have no idea, maybe Christoffer or Peter would know.
> 
> Cheers,
> Andre.
> _______________________________________________
> kvmarm mailing list
> kvmarm at lists.cs.columbia.edu
> https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

^ permalink raw reply	[flat|nested] 120+ messages in thread
* Re: [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
  2016-06-06  5:29       ` Bharat Bhushan
@ 2016-06-07  8:02         ` Christoffer Dall
  -1 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-06-07  8:02 UTC (permalink / raw)
  To: Bharat Bhushan
  Cc: kvm, Marc Zyngier, Andre Przywara, Diana Madalina Craciun,
	kvmarm, linux-arm-kernel

Hi Bharat,



On Mon, Jun 6, 2016 at 7:29 AM, Bharat Bhushan <bharat.bhushan@nxp.com> wrote:
> Hi Andre,
>
> On clarification, are V5 version of patches are based on new-GIC emulation patches? How are you testing these patches, kvm-tool?
>
> Hi Christoffer/Peter,
>
> Is someone working on QEMU side patches? We need this feature a bit urgently so if no one is working then we can start working on this.
>
Eric Auger (now with Red Hat) was planning to pick up Pavel's series
and do a respin:

https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg05197.html

If you are ready to commit resources to this *immediately* and if Eric
agrees then it would be good for you to pick up this series, but
should you choose to do so please make sure you can actually commit
resources to the work and to keep up with revisions this time around.

Thanks,
-Christoffer

^ permalink raw reply	[flat|nested] 120+ messages in thread
* [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation
@ 2016-06-07  8:02         ` Christoffer Dall
  0 siblings, 0 replies; 120+ messages in thread
From: Christoffer Dall @ 2016-06-07  8:02 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Bharat,



On Mon, Jun 6, 2016 at 7:29 AM, Bharat Bhushan <bharat.bhushan@nxp.com> wrote:
> Hi Andre,
>
> On clarification, are V5 version of patches are based on new-GIC emulation patches? How are you testing these patches, kvm-tool?
>
> Hi Christoffer/Peter,
>
> Is someone working on QEMU side patches? We need this feature a bit urgently so if no one is working then we can start working on this.
>
Eric Auger (now with Red Hat) was planning to pick up Pavel's series
and do a respin:

https://lists.gnu.org/archive/html/qemu-devel/2015-11/msg05197.html

If you are ready to commit resources to this *immediately* and if Eric
agrees then it would be good for you to pick up this series, but
should you choose to do so please make sure you can actually commit
resources to the work and to keep up with revisions this time around.

Thanks,
-Christoffer

^ permalink raw reply	[flat|nested] 120+ messages in thread
end of thread, other threads:[~2016-06-07  8:02 UTC | newest]

Thread overview: 120+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-03-26  2:13 [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation Andre Przywara
2016-03-26  2:13 ` Andre Przywara
2016-03-26  2:13 ` [PATCH v4 01/12] KVM: extend struct kvm_msi to hold a 32-bit device ID Andre Przywara
2016-03-26  2:13   ` Andre Przywara
2016-04-03  9:15   ` Christoffer Dall
2016-04-03  9:15     ` Christoffer Dall
2016-05-25 15:55     ` Andre Przywara
2016-05-25 15:55       ` Andre Przywara
2016-05-25 16:16       ` Marc Zyngier
2016-05-25 16:16         ` Marc Zyngier
2016-05-31 13:05         ` Christoffer Dall
2016-05-31 13:05           ` Christoffer Dall
2016-05-05 17:55   ` Chalamarla, Tirumalesh
2016-05-05 17:55     ` Chalamarla, Tirumalesh
2016-03-26  2:14 ` [PATCH v4 02/12] KVM: arm/arm64: extend arch CAP checks to allow per-VM capabilities Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-03 10:08   ` Christoffer Dall
2016-04-03 10:08     ` Christoffer Dall
2016-04-05 12:48   ` Eric Auger
2016-04-05 12:48     ` Eric Auger
2016-05-05 18:00   ` Chalamarla, Tirumalesh
2016-05-05 18:00     ` Chalamarla, Tirumalesh
2016-03-26  2:14 ` [PATCH v4 03/12] KVM: arm64: Introduce new MMIO region for the ITS base address Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-03 10:08   ` Christoffer Dall
2016-04-03 10:08     ` Christoffer Dall
2016-04-05 12:47     ` Eric Auger
2016-04-05 12:47       ` Eric Auger
2016-04-07 13:44   ` Marc Zyngier
2016-04-07 13:44     ` Marc Zyngier
2016-05-05 18:08   ` Chalamarla, Tirumalesh
2016-05-05 18:08     ` Chalamarla, Tirumalesh
2016-05-09 15:47     ` Marc Zyngier
2016-05-09 15:47       ` Marc Zyngier
2016-05-09 16:53       ` Chalamarla, Tirumalesh
2016-05-09 16:53         ` Chalamarla, Tirumalesh
2016-05-09 17:09         ` Marc Zyngier
2016-05-09 17:09           ` Marc Zyngier
2016-03-26  2:14 ` [PATCH v4 04/12] KVM: arm64: handle ITS related GICv3 redistributor registers Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-03 10:08   ` Christoffer Dall
2016-04-03 10:08     ` Christoffer Dall
2016-04-05 12:55     ` Eric Auger
2016-04-05 12:55       ` Eric Auger
2016-04-05 15:17   ` Eric Auger
2016-04-05 15:17     ` Eric Auger
2016-04-07 13:54   ` Marc Zyngier
2016-04-07 13:54     ` Marc Zyngier
2016-04-07 13:58     ` Marc Zyngier
2016-04-07 13:58       ` Marc Zyngier
2016-05-05 18:06       ` Chalamarla, Tirumalesh
2016-05-05 18:06         ` Chalamarla, Tirumalesh
2016-03-26  2:14 ` [PATCH v4 05/12] KVM: arm64: introduce ITS emulation file with stub functions Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-05 16:03   ` Eric Auger
2016-04-05 16:03     ` Eric Auger
2016-04-07 14:04   ` Marc Zyngier
2016-04-07 14:04     ` Marc Zyngier
2016-04-07 14:08     ` Eric Auger
2016-04-07 14:08       ` Eric Auger
2016-04-07 14:48       ` Marc Zyngier
2016-04-07 14:48         ` Marc Zyngier
2016-04-07 15:09         ` Eric Auger
2016-04-07 15:09           ` Eric Auger
2016-04-07 15:19           ` Marc Zyngier
2016-04-07 15:19             ` Marc Zyngier
2016-03-26  2:14 ` [PATCH v4 06/12] KVM: arm64: implement basic ITS register handlers Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-03 10:08   ` Christoffer Dall
2016-04-03 10:08     ` Christoffer Dall
2016-04-06  9:36   ` Eric Auger
2016-04-06  9:36     ` Eric Auger
2016-05-25 13:49     ` Andre Przywara
2016-05-25 13:49       ` Andre Przywara
2016-04-07 14:35   ` Marc Zyngier
2016-04-07 14:35     ` Marc Zyngier
2016-05-25 11:37     ` Andre Przywara
2016-05-25 11:37       ` Andre Przywara
2016-05-26  9:10       ` Marc Zyngier
2016-05-26  9:10         ` Marc Zyngier
2016-06-03 15:42         ` Andre Przywara
2016-06-03 15:42           ` Andre Przywara
2016-06-03 16:54           ` Marc Zyngier
2016-06-03 16:54             ` Marc Zyngier
2016-05-05 18:51   ` Chalamarla, Tirumalesh
2016-05-05 18:51     ` Chalamarla, Tirumalesh
2016-03-26  2:14 ` [PATCH v4 07/12] KVM: arm64: add data structures to model ITS interrupt translation Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-06  9:53   ` Eric Auger
2016-04-06  9:53     ` Eric Auger
2016-03-26  2:14 ` [PATCH v4 08/12] KVM: arm64: connect LPIs to the VGIC emulation Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-06 12:00   ` Eric Auger
2016-04-06 12:00     ` Eric Auger
2016-05-05 18:59   ` Chalamarla, Tirumalesh
2016-05-05 18:59     ` Chalamarla, Tirumalesh
2016-03-26  2:14 ` [PATCH v4 09/12] KVM: arm64: sync LPI configuration and pending tables Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-04-06 13:41   ` Eric Auger
2016-04-06 13:41     ` Eric Auger
2016-06-03 14:17     ` Andre Przywara
2016-06-03 14:17       ` Andre Przywara
2016-03-26  2:14 ` [PATCH v4 10/12] KVM: arm64: implement ITS command queue command handlers Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-05-05 19:12   ` Chalamarla, Tirumalesh
2016-05-05 19:12     ` Chalamarla, Tirumalesh
2016-05-25 14:34     ` Andre Przywara
2016-05-25 14:34       ` Andre Przywara
2016-03-26  2:14 ` [PATCH v4 11/12] KVM: arm64: implement MSI injection in ITS emulation Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-03-26  2:14 ` [PATCH v4 12/12] KVM: arm64: enable ITS emulation as a virtual MSI controller Andre Przywara
2016-03-26  2:14   ` Andre Przywara
2016-06-03  4:26 ` [PATCH v4 00/12] KVM: arm64: GICv3 ITS emulation Bharat Bhushan
2016-06-03  4:26   ` Bharat Bhushan
2016-06-03 14:32   ` Andre Przywara
2016-06-03 14:32     ` Andre Przywara
2016-06-06  5:29     ` Bharat Bhushan
2016-06-06  5:29       ` Bharat Bhushan
2016-06-07  8:02       ` Christoffer Dall
2016-06-07  8:02         ` Christoffer Dall

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.