[RFC v9 0/6] KVM: arm/arm64: gsi routing support

[RFC v9 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
becomes mandated for proper VIRTIO-PCI vhost integration.

In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
we previously used direct_mapping trick: this consists in extracting the
IRQ ID found in the MSI message and associating an irqfd to that IRQ ID.
When vhost worker thread gets a new buffer it signals the irqfd and kvm
then injects this IRQ ID on guest. That way although the guest uses MSIs,
no MSI emulation is used.

This worked fine with GICv2m but does not work anymore with GICV3 ITS.
Indeed this latter implements IRQ translation: what is found in the MSI
message no more is the target IRQ ID but is an intermediate event ID used
in the translation process.

Hence true MSI routing is needed so that the vhost back channel irqfd is
associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
injects the MSI through the in-kernel ITS emulation, the MSI is properly
translated and eventually the LPI ID associated to the event ID is injected
on guest.

MSI routing also mandates to integrate irqchip routing. The initial
implementation of irqfd on arm must be upgraded with the integration
of kvm irqchip.c code and the implementation of its standard hooks
in the architecture specific part.

In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
table with flat irqchip routing entries is built enabling to inject gsi
corresponding to the SPI indexes seen by the guest.

As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
default routing table and is responsible for building the whole routing
table.

for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
- only applies to KVM_IRQFD and not to KVM_IRQ_LINE

- irqchip routing was tested on Calxeda midway (VFIO with irqfd)
  with and without explicit routing
- MSI routing was tested on AMD Overdrive and Cavium ThunderX

Code + dependencies can be found at:
https://github.com/eauger/linux/tree/v4.7-rc7-its-emul-v10-gsi-routing-v9

The series depends on

[1]: [PATCH v10 00/17] KVM: arm64: GICv3 ITS emulation
     http://www.spinics.net/lists/kvm/msg135687.html
   + [PATCH] KVM: arm/arm64: fix vGICv2 KVM_DEV_ARM_VGIC_GRP_CPU/DIST_REGS
[2]: [PATCH] KVM: arm/arm64: The GIC is dead, long live the GIC

hence the RFC.

GSI flat routing setup on QEMU can be found at:
https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html

History:
v8 -> v9:
- remove "[RFC v8 7/7] KVM: arm: Enable KVM_SIGNAL_MSI and MSI routing"
- fix spi id check at IRQ injection
- use default in irqchip switch(ue->type)

v7 -> v8:
- take into account Radim's comments:
  x rework kvm_kernel_irq_routing_entry and remove msi_msg now
    replaced by a custom struct
  x check user entry flags depending on user entry type
  x I kept last patch although challenged by Radim. It can be dropped
    according to the consensus.

v6 -> v7:
- take into account Drew's and Andre's comments
- new patch moving declarations of kvm_setup_default_irq_routing and
  kvm_setup_empty_irq_routing outside of kvm_host.h
- vgic_v2m_inject_msi moved in vgic-irqfd
- re-introduce irq.h
- add msi_ prefix to flags/devid kvm_kernel_irq_routing_entry fields
- move kvm_vgic_setup_default_irq_routing declaration in arm_vgic.h
  and definition in vgic-irqfd.c
- remove BUG_ON(!vgic_initialized(kvm) in vgic_irqfd_set_irq
- move KVM_IRQCHIP_NUM_PINS in arm_vgic.h and use VGIC_MAX_SPI instead
  of 1020

v5 -> v6:
- rebase on Andre's v8 + removal of old vgic
- tested on Cavium ThunderX

V4 -> v5:
- rebase on Andre's v7 + final new vgic code
- check msi->data is within SPI range in vgic_v2m_inject_msi
- squashed enable irq routing and default irqchip table patches
- handle default irqchip table allocation failure
- some rephrasing in doc & comment according to Christoffer's feedbacks
- lock issue reported by Pavel seems to have disappear after 4.2 (MSI injection
  fast path)

v3 -> v4:
- rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
  a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
  reflesh/reminder.
- rewrite the cover letter

v2 -> v3:
- don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
  use KVM_MSI_VALID_DEVID flag instead
- propagate user flags downto the kernel to make sure the userspace
  correctly set devid in GICv3 ITS case (still under discussion)

v1 -> v2:
- user API changed:
  x devid id passed in kvm_irq_routing_msi
  x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
- kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
  composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
- Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
  in eventfd.c
- added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
- fix bug reported by Andre: bad setting of msi.flags and msi.devid
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH:
- clearly state limited support on arm/arm64:
  KVM_IRQ_LINE not impacted by GSI routing
- add default routing table feature (new patch file)
- changed uapi to use padding field area
- reword api.txt


Eric Auger (6):
  KVM: api: Pass the devid in the msi routing entry
  KVM: Add devid in kvm_kernel_irq_routing_entry
  KVM: irqchip: Convey devid to kvm_set_msi
  KVM: Move kvm_setup_default/empty_irq_routing declaration in arch
    specific header
  KVM: arm/arm64: Enable irqchip routing
  KVM: arm/arm64: Enable MSI routing

 Documentation/virtual/kvm/api.txt |  34 +++++++++---
 arch/arm/kvm/Kconfig              |   2 +
 arch/arm/kvm/Makefile             |   1 +
 arch/arm/kvm/irq.h                |  19 +++++++
 arch/arm64/kvm/Kconfig            |   2 +
 arch/arm64/kvm/Makefile           |   1 +
 arch/arm64/kvm/irq.h              |  19 +++++++
 arch/x86/kvm/irq.h                |   3 ++
 include/kvm/arm_vgic.h            |   7 +++
 include/linux/kvm_host.h          |  12 +++--
 include/uapi/linux/kvm.h          |   5 +-
 virt/kvm/arm/vgic/vgic-init.c     |   4 ++
 virt/kvm/arm/vgic/vgic-irqfd.c    | 108 +++++++++++++++++++++++++++++++-------
 virt/kvm/arm/vgic/vgic.c          |   7 ---
 virt/kvm/irqchip.c                |  28 ++++++----
 15 files changed, 207 insertions(+), 45 deletions(-)
 create mode 100644 arch/arm/kvm/irq.h
 create mode 100644 arch/arm64/kvm/irq.h

-- 
1.9.1

[RFC v9 1/6] KVM: api: Pass the devid in the msi routing entry

[Eric Auger]

On ARM, the MSI msg (address and data) comes along with
out-of-band device ID information. The device ID encodes the
device that writes the MSI msg. Let's convey the device id in
kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
kvm_irq_routing_entry to indicate the msi devid is populated.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>

---

v6 -> v7:
- Added Andre's R-b

v4 -> v5:
- some rephrasing in api.txt according to Christoffer's comments
v2 -> v3:
- replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
  usage of KVM_MSI_VALID_DEVID flag
- add note about KVM_CAP_MSI_DEVID capability

v1 -> v2:
- devid id passed in kvm_irq_routing_msi instead of in
  kvm_irq_routing_entry

RFC -> PATCH
- remove kvm_irq_routing_extended_msi and use union instead
---
 Documentation/virtual/kvm/api.txt | 19 +++++++++++++++++--
 include/uapi/linux/kvm.h          |  5 ++++-
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index f60b137..0065c8e 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1468,7 +1468,11 @@ struct kvm_irq_routing_entry {
 #define KVM_IRQ_ROUTING_S390_ADAPTER 3
 #define KVM_IRQ_ROUTING_HV_SINT 4
 
-No flags are specified so far, the corresponding field must be set to zero.
+flags:
+- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
+  routing entry type, specifies that the devid field contains
+  a valid value.
+- zero otherwise
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1479,9 +1483,20 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
+       for the device that wrote the MSI message.
+       For PCI, this is usually a BFD identifier in the lower 16 bits.
+
+The per-VM KVM_CAP_MSI_DEVID capability advertises the requirement to
+provide the device ID. If this capability is not set, userland cannot
+rely on the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
+
 struct kvm_irq_routing_s390_adapter {
 	__u64 ind_addr;
 	__u64 summary_addr;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index d8c4c32..eb22208 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -879,7 +879,10 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
 struct kvm_irq_routing_s390_adapter {
-- 
1.9.1

[RFC v9 2/6] KVM: Add devid in kvm_kernel_irq_routing_entry

[Eric Auger]

Enhance kvm_kernel_irq_routing_entry to transport the device id
field, devid. A new flags field makes possible to indicate the
devid is valid. Those additions are used for ARM GICv3 ITS MSI
injection. The original struct msi_msg msi field is replaced by
a new custom structure that embeds the new fields.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Suggested-by: Radim Krčmář <rkrcmar@redhat.com>

---
v7 -> v8:
- replace struct msi_msg by a new custom structure
- removed Christoffer's A-b

v6 -> v7:
- added msi_ prefix to flags and dev_id fields

v4 -> v5:
- add Christoffer's R-b

v2 -> v3:
- add flags

v1 -> v2:
- replace msi_msg field by a struct composed of msi_msg and devid

RFC -> PATCH:
- reword the commit message after change in first patch (uapi)
---
 include/linux/kvm_host.h | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index c87fe6f..e33e85e 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -317,7 +317,13 @@ struct kvm_kernel_irq_routing_entry {
 			unsigned irqchip;
 			unsigned pin;
 		} irqchip;
-		struct msi_msg msi;
+		struct {
+			u32 address_lo;
+			u32 address_hi;
+			u32 data;
+			u32 flags;
+			u32 devid;
+		} msi;
 		struct kvm_s390_adapter_int adapter;
 		struct kvm_hv_sint hv_sint;
 	};
-- 
1.9.1

[RFC v9 3/6] KVM: irqchip: Convey devid to kvm_set_msi

[Eric Auger]

on ARM, a devid field is populated in kvm_msi struct in case the
flag is set to KVM_MSI_VALID_DEVID. Let's propagate both flags and
devid field in kvm_kernel_irq_routing_entry.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

---
v7 -> v8:
- adapt to changes in kvm_kernel_irq_routing_entry struct

v6 -> v7:
- use renamed msi_flags and msi_devid
- added Andre's R-b

v4 -> v5:
- Add Christoffer's R-b

v2 -> v3:
- do not set the type to KVM_IRQ_ROUTING_EXTENDED_MSI anymore as
  suggested by Andre
- correct msi->flags check
- propagate the flags
---
 virt/kvm/irqchip.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 8db197b..0c00054 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -62,12 +62,14 @@ int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
 {
 	struct kvm_kernel_irq_routing_entry route;
 
-	if (!irqchip_in_kernel(kvm) || msi->flags != 0)
+	if (!irqchip_in_kernel(kvm) || (msi->flags & ~KVM_MSI_VALID_DEVID))
 		return -EINVAL;
 
 	route.msi.address_lo = msi->address_lo;
 	route.msi.address_hi = msi->address_hi;
 	route.msi.data = msi->data;
+	route.msi.flags = msi->flags;
+	route.msi.devid = msi->devid;
 
 	return kvm_set_msi(&route, kvm, KVM_USERSPACE_IRQ_SOURCE_ID, 1, false);
 }
-- 
1.9.1

[RFC v9 4/6] KVM: Move kvm_setup_default/empty_irq_routing declaration in arch specific header

[Eric Auger]

kvm_setup_default_irq_routing and kvm_setup_empty_irq_routing are
not used by generic code. So let's move the declarations in x86 irq.h
header instead of kvm_host.h.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Suggested-by: Andre Przywara <andre.przywara@arm.com>

---

- new patch in v7
---
 arch/x86/kvm/irq.h       | 3 +++
 include/linux/kvm_host.h | 2 --
 2 files changed, 3 insertions(+), 2 deletions(-)

diff --git a/arch/x86/kvm/irq.h b/arch/x86/kvm/irq.h
index 61ebdc1..035731e 100644
--- a/arch/x86/kvm/irq.h
+++ b/arch/x86/kvm/irq.h
@@ -120,4 +120,7 @@ void __kvm_migrate_timers(struct kvm_vcpu *vcpu);
 
 int apic_has_pending_timer(struct kvm_vcpu *vcpu);
 
+int kvm_setup_default_irq_routing(struct kvm *kvm);
+int kvm_setup_empty_irq_routing(struct kvm *kvm);
+
 #endif
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e33e85e..49e8152 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1044,8 +1044,6 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
 #define KVM_MAX_IRQ_ROUTES 1024
 #endif
 
-int kvm_setup_default_irq_routing(struct kvm *kvm);
-int kvm_setup_empty_irq_routing(struct kvm *kvm);
 int kvm_set_irq_routing(struct kvm *kvm,
 			const struct kvm_irq_routing_entry *entries,
 			unsigned nr,
-- 
1.9.1

[RFC v9 5/6] KVM: arm/arm64: Enable irqchip routing

[Eric Auger]

This patch adds compilation and link against irqchip.

Main motivation behind using irqchip code is to enable MSI
routing code. In the future irqchip routing may also be useful
when targeting multiple irqchips.

Routing standard callbacks now are implemented in vgic-irqfd:
- kvm_set_routing_entry
- kvm_set_irq
- kvm_set_msi

They only are supported with new_vgic code.

Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.

So from now on IRQCHIP routing is enabled and a routing table entry
must exist for irqfd injection to succeed for a given SPI. This patch
builds a default flat irqchip routing table (gsi=irqchip.pin) covering
all the VGIC SPI indexes. This routing table is overwritten by the
first first user-space call to KVM_SET_GSI_ROUTING ioctl.

MSI routing setup is not yet allowed.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v8 -> v9
- fix bug when checking spi_id before irqfd injection

v7 -> v8:
- adapt to changes in kvm_kernel_irq_routing_entry

v6 -> v7:
- re-introduce irq.h
- use kvm_kernel_irq_routing_entry renamed fields: msi_flags, msi_devid
- moved kvm_vgic_setup_default_irq_routing declaration in arm_vgic.h and
  definition in vgic-irqfd.c
- correct double / in Makefile
- remove BUG_ON(!vgic_initialized(kvm) in vgic_irqfd_set_irq since
  in any case we have a lazy_init in update_irq_pending
- move KVM_IRQCHIP_NUM_PINS in arm_vgic.h
- use VGIC_MAX_SPI

v5 -> v6:
- rebase on top of Andre's v8 + removal of old vgic

v4 -> v5:
- vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
- minor comment changes
- remove trace_kvm_set_irq since it is called in irqchip
- remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
- despite Christoffer's question, in kvm_set_msi, I kept the copy from
  the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
  irqchip callback API into the struct kvm_msi * passed to
  vits_inject_msi. Since vits_inject_msi is directly called by
  kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
  to me to keep the copy.
- squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
  table into that patch
- handle default routing table alloc failure

v3 -> v4:
- provide support only for new-vgic
- code previously in vgic.c now in vgic_irqfd.c

v2 -> v3:
- unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
  by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
- vgic_irqfd_set_irq now is static
- propagate flags
- add comments

v1 -> v2:
- fix bug reported by Andre related to msi.flags and msi.devid setting
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH
- reword api.txt:
  x move MSI routing comments in a subsequent patch,
  x clearly state GSI routing does not apply to KVM_IRQ_LINE
---
 Documentation/virtual/kvm/api.txt |  12 +++--
 arch/arm/kvm/Kconfig              |   2 +
 arch/arm/kvm/Makefile             |   1 +
 arch/arm/kvm/irq.h                |  19 ++++++++
 arch/arm64/kvm/Kconfig            |   2 +
 arch/arm64/kvm/Makefile           |   1 +
 arch/arm64/kvm/irq.h              |  19 ++++++++
 include/kvm/arm_vgic.h            |   7 +++
 virt/kvm/arm/vgic/vgic-init.c     |   4 ++
 virt/kvm/arm/vgic/vgic-irqfd.c    | 100 +++++++++++++++++++++++++++++++-------
 virt/kvm/arm/vgic/vgic.c          |   7 ---
 11 files changed, 145 insertions(+), 29 deletions(-)
 create mode 100644 arch/arm/kvm/irq.h
 create mode 100644 arch/arm64/kvm/irq.h

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 0065c8e..3bb60d3 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
 4.52 KVM_SET_GSI_ROUTING
 
 Capability: KVM_CAP_IRQ_ROUTING
-Architectures: x86 s390
+Architectures: x86 s390 arm arm64
 Type: vm ioctl
 Parameters: struct kvm_irq_routing (in)
 Returns: 0 on success, -1 on error
 
 Sets the GSI routing table entries, overwriting any previously set entries.
 
+On arm/arm64, GSI routing has the following limitation:
+- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
+
 struct kvm_irq_routing {
 	__u32 nr;
 	__u32 flags;
@@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
 irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
 and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
 
-On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
-Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
-given by gsi + 32.
+On arm/arm64, gsi routing being supported, the following can happen:
+- in case no routing entry is associated to this gsi, injection fails
+- in case the gsi is associated to an irqchip routing entry,
+  irqchip.pin + 32 corresponds to the injected SPI ID.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 95a0005..3e1cd04 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -32,6 +32,8 @@ config KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
 	---help---
 	  Support hosting virtualized guest machines.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 5e28df8..10d77a6 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
 obj-y += $(KVM)/arm/vgic/vgic-mmio.o
 obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
 obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
+obj-y += $(KVM)/irqchip.o
 obj-y += $(KVM)/arm/arch_timer.o
diff --git a/arch/arm/kvm/irq.h b/arch/arm/kvm/irq.h
new file mode 100644
index 0000000..b74099b
--- /dev/null
+++ b/arch/arm/kvm/irq.h
@@ -0,0 +1,19 @@
+/*
+ * irq.h: in kernel interrupt controller related definitions
+ * Copyright (c) 2016 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This header is included by irqchip.c. However, on ARM, interrupt
+ * controller declarations are located in include/kvm/arm_vgic.h since
+ * they are mostly shared between arm and arm64.
+ */
+
+#ifndef __IRQ_H
+#define __IRQ_H
+
+#include <kvm/arm_vgic.h>
+
+#endif
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 9d2eff0..9c9edc9 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -37,6 +37,8 @@ config KVM
 	select KVM_ARM_VGIC_V3
 	select KVM_ARM_PMU if HW_PERF_EVENTS
 	select HAVE_KVM_MSI
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	---help---
 	  Support hosting virtualized guest machines.
 	  We don't support KVM with 16K page tables yet, due to the multiple
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index a5b9664..695eb3c 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
 kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
diff --git a/arch/arm64/kvm/irq.h b/arch/arm64/kvm/irq.h
new file mode 100644
index 0000000..b74099b
--- /dev/null
+++ b/arch/arm64/kvm/irq.h
@@ -0,0 +1,19 @@
+/*
+ * irq.h: in kernel interrupt controller related definitions
+ * Copyright (c) 2016 Red Hat, Inc.
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This header is included by irqchip.c. However, on ARM, interrupt
+ * controller declarations are located in include/kvm/arm_vgic.h since
+ * they are mostly shared between arm and arm64.
+ */
+
+#ifndef __IRQ_H
+#define __IRQ_H
+
+#include <kvm/arm_vgic.h>
+
+#endif
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 4e63a07..260c8e9 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -34,6 +34,7 @@
 #define VGIC_MAX_SPI		1019
 #define VGIC_MAX_RESERVED	1023
 #define VGIC_MIN_LPI		8192
+#define KVM_IRQCHIP_NUM_PINS	(1020 - 32)
 
 enum vgic_type {
 	VGIC_V2,		/* Good ol' GICv2 */
@@ -313,4 +314,10 @@ static inline int kvm_vgic_get_max_vcpus(void)
 
 int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
 
+/**
+ * kvm_vgic_setup_default_irq_routing:
+ * Setup a default flat gsi routing table mapping all SPIs
+ */
+int kvm_vgic_setup_default_irq_routing(struct kvm *kvm);
+
 #endif /* __KVM_ARM_VGIC_H */
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index 01a60dc..1aba785 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_init(vcpu);
 
+	ret = kvm_vgic_setup_default_irq_routing(kvm);
+	if (ret)
+		goto out;
+
 	dist->initialized = true;
 out:
 	return ret;
diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
index c675513..6e84d53 100644
--- a/virt/kvm/arm/vgic/vgic-irqfd.c
+++ b/virt/kvm/arm/vgic/vgic-irqfd.c
@@ -17,36 +17,100 @@
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <trace/events/kvm.h>
+#include <kvm/arm_vgic.h>
+#include "vgic.h"
 
-int kvm_irq_map_gsi(struct kvm *kvm,
-		    struct kvm_kernel_irq_routing_entry *entries,
-		    int gsi)
+/**
+ * vgic_irqfd_set_irq: inject the IRQ corresponding to the
+ * irqchip routing entry
+ *
+ * This is the entry point for irqfd IRQ injection
+ */
+static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
+			struct kvm *kvm, int irq_source_id,
+			int level, bool line_status)
 {
-	return 0;
+	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
+
+	if (!vgic_valid_spi(kvm, spi_id))
+		return -EINVAL;
+	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
 }
 
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
-			 unsigned int pin)
+/**
+ * kvm_set_routing_entry: populate a kvm routing entry
+ * from a user routing entry
+ *
+ * @e: kvm kernel routing entry handle
+ * @ue: user api routing entry handle
+ * return 0 on success, -EINVAL on errors.
+ */
+int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
+			  const struct kvm_irq_routing_entry *ue)
 {
-	return pin;
+	int r = -EINVAL;
+
+	switch (ue->type) {
+	case KVM_IRQ_ROUTING_IRQCHIP:
+		e->set = vgic_irqfd_set_irq;
+		e->irqchip.irqchip = ue->u.irqchip.irqchip;
+		e->irqchip.pin = ue->u.irqchip.pin;
+		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
+		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
+			goto out;
+		break;
+	default:
+		goto out;
+	}
+	r = 0;
+out:
+	return r;
 }
 
-int kvm_set_irq(struct kvm *kvm, int irq_source_id,
-		u32 irq, int level, bool line_status)
+/**
+ * kvm_set_msi: inject the MSI corresponding to the
+ * MSI routing entry
+ *
+ * This is the entry point for irqfd MSI injection
+ * and userspace MSI injection.
+ */
+int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
+		struct kvm *kvm, int irq_source_id,
+		int level, bool line_status)
 {
-	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
+	struct kvm_msi msi;
 
-	trace_kvm_set_irq(irq, level, irq_source_id);
+	msi.address_lo = e->msi.address_lo;
+	msi.address_hi = e->msi.address_hi;
+	msi.data = e->msi.data;
+	msi.flags = e->msi.flags;
+	msi.devid = e->msi.devid;
 
-	BUG_ON(!vgic_initialized(kvm));
+	if (!vgic_has_its(kvm))
+		return -ENODEV;
 
-	return kvm_vgic_inject_irq(kvm, 0, spi, level);
+	return vgic_its_inject_msi(kvm, &msi);
 }
 
-/* MSI not implemented yet */
-int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
-		struct kvm *kvm, int irq_source_id,
-		int level, bool line_status)
+int kvm_vgic_setup_default_irq_routing(struct kvm *kvm)
 {
-	return 0;
+	struct kvm_irq_routing_entry *entries;
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 nr = dist->nr_spis;
+	int i, ret;
+
+	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
+			  GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	for (i = 0; i < nr; i++) {
+		entries[i].gsi = i;
+		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
+		entries[i].u.irqchip.irqchip = 0;
+		entries[i].u.irqchip.pin = i;
+	}
+	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
+	kfree(entries);
+	return ret;
 }
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index 424cb9c..1b3e2cd 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -719,10 +719,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
 	return map_is_active;
 }
 
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
-{
-	if (vgic_has_its(kvm))
-		return vgic_its_inject_msi(kvm, msi);
-	else
-		return -ENODEV;
-}
-- 
1.9.1

[RFC v9 6/6] KVM: arm/arm64: Enable MSI routing

[Eric Auger]

Up to now, only irqchip routing entries could be set. This patch
adds the capability to insert MSI routing entries.

For ARM64, let's also increase KVM_MAX_IRQ_ROUTES to 4096: this
include SPI irqchip routes plus MSI routes. In the future this
might be extended.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Reviewed-by: Andre Przywara <andre.przywara@arm.com>

---
v8 -> v9:
- documentation specifies MSI routing only is supported with
  GICv3 ITS use case
- default used when checking ue->type

v7 -> v8:
- adapt to changes in kvm_kernel_irq_routing_entry and check the
  user entry flags according to the user entry type

v6 -> v7:
- added Andre's R-b

v2 -> v3:
- remove any reference to KVM_IRQ_ROUTING_EXTENDED_MSI type
- unconditionnaly uapi flags and devid downto the kernel
  routing entry struct
- handle KVM_MSI_VALID_DEVID flag in kvm_set_irq_routing
- note about KVM_CAP_MSI_DEVID moved in the first patch file
  of the series

v1 -> v2:
- adapt to new routing entry types

RFC -> PATCH:
- move api MSI routing updates into that patch file
- use new devid field of user api struct
---
 Documentation/virtual/kvm/api.txt |  3 +++
 include/linux/kvm_host.h          |  2 ++
 virt/kvm/arm/vgic/vgic-irqfd.c    |  8 ++++++++
 virt/kvm/irqchip.c                | 24 +++++++++++++++---------
 4 files changed, 28 insertions(+), 9 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 3bb60d3..8501e75 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2375,6 +2375,9 @@ On arm/arm64, gsi routing being supported, the following can happen:
 - in case no routing entry is associated to this gsi, injection fails
 - in case the gsi is associated to an irqchip routing entry,
   irqchip.pin + 32 corresponds to the injected SPI ID.
+- in case the gsi is associated to an MSI routing entry, the MSI
+  message and device ID are translated into an LPI (support restricted
+  to GICv3 ITS in-kernel emulation).
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 49e8152..9fa4bc2 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1040,6 +1040,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
 
 #ifdef CONFIG_S390
 #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
+#elif defined(CONFIG_ARM64)
+#define KVM_MAX_IRQ_ROUTES 4096
 #else
 #define KVM_MAX_IRQ_ROUTES 1024
 #endif
diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
index 6e84d53..683a589 100644
--- a/virt/kvm/arm/vgic/vgic-irqfd.c
+++ b/virt/kvm/arm/vgic/vgic-irqfd.c
@@ -59,6 +59,14 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
 		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
 			goto out;
 		break;
+	case KVM_IRQ_ROUTING_MSI:
+		e->set = kvm_set_msi;
+		e->msi.address_lo = ue->u.msi.address_lo;
+		e->msi.address_hi = ue->u.msi.address_hi;
+		e->msi.data = ue->u.msi.data;
+		e->msi.flags = ue->flags;
+		e->msi.devid = ue->u.msi.devid;
+		break;
 	default:
 		goto out;
 	}
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 0c00054..c620219 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -178,6 +178,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
 			unsigned flags)
 {
 	struct kvm_irq_routing_table *new, *old;
+	struct kvm_kernel_irq_routing_entry *e;
 	u32 i, j, nr_rt_entries = 0;
 	int r;
 
@@ -201,23 +202,25 @@ int kvm_set_irq_routing(struct kvm *kvm,
 			new->chip[i][j] = -1;
 
 	for (i = 0; i < nr; ++i) {
-		struct kvm_kernel_irq_routing_entry *e;
-
 		r = -ENOMEM;
 		e = kzalloc(sizeof(*e), GFP_KERNEL);
 		if (!e)
 			goto out;
 
 		r = -EINVAL;
-		if (ue->flags) {
-			kfree(e);
-			goto out;
+		switch (ue->type) {
+		case KVM_IRQ_ROUTING_MSI:
+			if (ue->flags & ~KVM_MSI_VALID_DEVID)
+				goto free_entry;
+			break;
+		default:
+			if (ue->flags)
+				goto free_entry;
+			break;
 		}
 		r = setup_routing_entry(new, e, ue);
-		if (r) {
-			kfree(e);
-			goto out;
-		}
+		if (r)
+			goto free_entry;
 		++ue;
 	}
 
@@ -234,7 +237,10 @@ int kvm_set_irq_routing(struct kvm *kvm,
 
 	new = old;
 	r = 0;
+	goto out;
 
+free_entry:
+	kfree(e);
 out:
 	free_irq_routing_table(new);
 
-- 
1.9.1

Re: [RFC v9 0/6] KVM: arm/arm64: gsi routing support

[Radim Krčmář]

2016-07-22 16:20+0000, Eric Auger:
> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
> becomes mandated for proper VIRTIO-PCI vhost integration.

Changes to interfaces and common code look ok,

Acked-by: Radim Krčmář <rkrcmar@redhat.com>

Good work.

Re: [RFC v9 0/6] KVM: arm/arm64: gsi routing support

[Marc Zyngier]

On 22/07/16 18:13, Radim Krčmář wrote:
> 2016-07-22 16:20+0000, Eric Auger:
>> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
>> becomes mandated for proper VIRTIO-PCI vhost integration.
> 
> Changes to interfaces and common code look ok,
> 
> Acked-by: Radim Krčmář <rkrcmar@redhat.com>

I'll let it sink into -next for a few days, so that I can rerun my
non-reg stuff, and will send another PR if everything turns green.

Thanks you both for the quick turnaround!

	M.
-- 
Jazz is not dead. It just smells funny...
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC v9 0/6] KVM: arm/arm64: gsi routing support

[Auger Eric]

Hi Radim, Marc,
On 22/07/2016 19:33, Marc Zyngier wrote:
> On 22/07/16 18:13, Radim Krčmář wrote:
>> 2016-07-22 16:20+0000, Eric Auger:
>>> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
>>> becomes mandated for proper VIRTIO-PCI vhost integration.
>>
>> Changes to interfaces and common code look ok,
>>
>> Acked-by: Radim Krčmář <rkrcmar@redhat.com>
> 
> I'll let it sink into -next for a few days, so that I can rerun my
> non-reg stuff, and will send another PR if everything turns green.
> 
> Thanks you both for the quick turnaround!
Thank you for your support!

Eric
> 
> 	M.
> 

Re: [RFC v9 0/6] KVM: arm/arm64: gsi routing support

[André Przywara]

Hi,

On 22/07/16 17:20, Eric Auger wrote:
> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
> becomes mandated for proper VIRTIO-PCI vhost integration.
> 
> In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
> we previously used direct_mapping trick: this consists in extracting the
> IRQ ID found in the MSI message and associating an irqfd to that IRQ ID.
> When vhost worker thread gets a new buffer it signals the irqfd and kvm
> then injects this IRQ ID on guest. That way although the guest uses MSIs,
> no MSI emulation is used.
> 
> This worked fine with GICv2m but does not work anymore with GICV3 ITS.
> Indeed this latter implements IRQ translation: what is found in the MSI
> message no more is the target IRQ ID but is an intermediate event ID used
> in the translation process.
> 
> Hence true MSI routing is needed so that the vhost back channel irqfd is
> associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
> injects the MSI through the in-kernel ITS emulation, the MSI is properly
> translated and eventually the LPI ID associated to the event ID is injected
> on guest.
> 
> MSI routing also mandates to integrate irqchip routing. The initial
> implementation of irqfd on arm must be upgraded with the integration
> of kvm irqchip.c code and the implementation of its standard hooks
> in the architecture specific part.
> 
> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
> table with flat irqchip routing entries is built enabling to inject gsi
> corresponding to the SPI indexes seen by the guest.
> 
> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
> default routing table and is responsible for building the whole routing
> table.
> 
> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
> 
> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>   with and without explicit routing
> - MSI routing was tested on AMD Overdrive and Cavium ThunderX
> 
> Code + dependencies can be found at:
> https://github.com/eauger/linux/tree/v4.7-rc7-its-emul-v10-gsi-routing-v9
> 
> The series depends on
> 
> [1]: [PATCH v10 00/17] KVM: arm64: GICv3 ITS emulation
>      http://www.spinics.net/lists/kvm/msg135687.html
>    + [PATCH] KVM: arm/arm64: fix vGICv2 KVM_DEV_ARM_VGIC_GRP_CPU/DIST_REGS
> [2]: [PATCH] KVM: arm/arm64: The GIC is dead, long live the GIC
> 
> hence the RFC.
> 
> GSI flat routing setup on QEMU can be found at:
> https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html

So I gave this series (as part of -next atm) some testing on a Freescale
machine.

I quickly boot tested the following combinations of options:
host kernel: next-20160728 and v4.7.0
guest kernel: next-20160728 and v4.4.0
QEMU: v2.6 and latest HEAD + Eric's ITS & GSI patches (from github)
      gic-version=2 and gic-version=3
      virtio-mmio and virtio-pci
kvmtool: latest HEAD and vits-v7
         --irqchip=gic-{v2,v3,v3-its}
         --force-pci and without (virtio-mmio)

That summed up to 68 combinations for me, all booting fine. If every
part supported ITS and MSIs (host kernel, userland tool, virtio-pci) I
saw the ITS emulation being used.
I also ran three guests (two QEMU and one kvmtool) in parallel with some
interrupt load for some hours.

For the records I could easily trigger our infamous kvmtool virtio hang
with SMP guests on this machine, but as that happened with older kernels
and kvmtools as well, affects only SMP guests and is totally absent when
running with QEMU, it's not a regression and unrelated to this series.

So FWIW:

Tested-by: Andre Przywara <andre.przywara@arm.com>

Cheers,
Andre

> 
> History:
> v8 -> v9:
> - remove "[RFC v8 7/7] KVM: arm: Enable KVM_SIGNAL_MSI and MSI routing"
> - fix spi id check at IRQ injection
> - use default in irqchip switch(ue->type)
> 
> v7 -> v8:
> - take into account Radim's comments:
>   x rework kvm_kernel_irq_routing_entry and remove msi_msg now
>     replaced by a custom struct
>   x check user entry flags depending on user entry type
>   x I kept last patch although challenged by Radim. It can be dropped
>     according to the consensus.
> 
> v6 -> v7:
> - take into account Drew's and Andre's comments
> - new patch moving declarations of kvm_setup_default_irq_routing and
>   kvm_setup_empty_irq_routing outside of kvm_host.h
> - vgic_v2m_inject_msi moved in vgic-irqfd
> - re-introduce irq.h
> - add msi_ prefix to flags/devid kvm_kernel_irq_routing_entry fields
> - move kvm_vgic_setup_default_irq_routing declaration in arm_vgic.h
>   and definition in vgic-irqfd.c
> - remove BUG_ON(!vgic_initialized(kvm) in vgic_irqfd_set_irq
> - move KVM_IRQCHIP_NUM_PINS in arm_vgic.h and use VGIC_MAX_SPI instead
>   of 1020
> 
> v5 -> v6:
> - rebase on Andre's v8 + removal of old vgic
> - tested on Cavium ThunderX
> 
> V4 -> v5:
> - rebase on Andre's v7 + final new vgic code
> - check msi->data is within SPI range in vgic_v2m_inject_msi
> - squashed enable irq routing and default irqchip table patches
> - handle default irqchip table allocation failure
> - some rephrasing in doc & comment according to Christoffer's feedbacks
> - lock issue reported by Pavel seems to have disappear after 4.2 (MSI injection
>   fast path)
> 
> v3 -> v4:
> - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
>   a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
>   reflesh/reminder.
> - rewrite the cover letter
> 
> v2 -> v3:
> - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
>   use KVM_MSI_VALID_DEVID flag instead
> - propagate user flags downto the kernel to make sure the userspace
>   correctly set devid in GICv3 ITS case (still under discussion)
> 
> v1 -> v2:
> - user API changed:
>   x devid id passed in kvm_irq_routing_msi
>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>   in eventfd.c
> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH:
> - clearly state limited support on arm/arm64:
>   KVM_IRQ_LINE not impacted by GSI routing
> - add default routing table feature (new patch file)
> - changed uapi to use padding field area
> - reword api.txt
> 
> 
> Eric Auger (6):
>   KVM: api: Pass the devid in the msi routing entry
>   KVM: Add devid in kvm_kernel_irq_routing_entry
>   KVM: irqchip: Convey devid to kvm_set_msi
>   KVM: Move kvm_setup_default/empty_irq_routing declaration in arch
>     specific header
>   KVM: arm/arm64: Enable irqchip routing
>   KVM: arm/arm64: Enable MSI routing
> 
>  Documentation/virtual/kvm/api.txt |  34 +++++++++---
>  arch/arm/kvm/Kconfig              |   2 +
>  arch/arm/kvm/Makefile             |   1 +
>  arch/arm/kvm/irq.h                |  19 +++++++
>  arch/arm64/kvm/Kconfig            |   2 +
>  arch/arm64/kvm/Makefile           |   1 +
>  arch/arm64/kvm/irq.h              |  19 +++++++
>  arch/x86/kvm/irq.h                |   3 ++
>  include/kvm/arm_vgic.h            |   7 +++
>  include/linux/kvm_host.h          |  12 +++--
>  include/uapi/linux/kvm.h          |   5 +-
>  virt/kvm/arm/vgic/vgic-init.c     |   4 ++
>  virt/kvm/arm/vgic/vgic-irqfd.c    | 108 +++++++++++++++++++++++++++++++-------
>  virt/kvm/arm/vgic/vgic.c          |   7 ---
>  virt/kvm/irqchip.c                |  28 ++++++----
>  15 files changed, 207 insertions(+), 45 deletions(-)
>  create mode 100644 arch/arm/kvm/irq.h
>  create mode 100644 arch/arm64/kvm/irq.h
> 

Re: [RFC v9 0/6] KVM: arm/arm64: gsi routing support

[Auger Eric]

Hi Andre,

On 29/07/2016 00:31, André Przywara wrote:
> Hi,
> 
> On 22/07/16 17:20, Eric Auger wrote:
>> With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
>> becomes mandated for proper VIRTIO-PCI vhost integration.
>>
>> In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
>> we previously used direct_mapping trick: this consists in extracting the
>> IRQ ID found in the MSI message and associating an irqfd to that IRQ ID.
>> When vhost worker thread gets a new buffer it signals the irqfd and kvm
>> then injects this IRQ ID on guest. That way although the guest uses MSIs,
>> no MSI emulation is used.
>>
>> This worked fine with GICv2m but does not work anymore with GICV3 ITS.
>> Indeed this latter implements IRQ translation: what is found in the MSI
>> message no more is the target IRQ ID but is an intermediate event ID used
>> in the translation process.
>>
>> Hence true MSI routing is needed so that the vhost back channel irqfd is
>> associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
>> injects the MSI through the in-kernel ITS emulation, the MSI is properly
>> translated and eventually the LPI ID associated to the event ID is injected
>> on guest.
>>
>> MSI routing also mandates to integrate irqchip routing. The initial
>> implementation of irqfd on arm must be upgraded with the integration
>> of kvm irqchip.c code and the implementation of its standard hooks
>> in the architecture specific part.
>>
>> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
>> table with flat irqchip routing entries is built enabling to inject gsi
>> corresponding to the SPI indexes seen by the guest.
>>
>> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
>> default routing table and is responsible for building the whole routing
>> table.
>>
>> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
>> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
>>
>> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>>   with and without explicit routing
>> - MSI routing was tested on AMD Overdrive and Cavium ThunderX
>>
>> Code + dependencies can be found at:
>> https://github.com/eauger/linux/tree/v4.7-rc7-its-emul-v10-gsi-routing-v9
>>
>> The series depends on
>>
>> [1]: [PATCH v10 00/17] KVM: arm64: GICv3 ITS emulation
>>      http://www.spinics.net/lists/kvm/msg135687.html
>>    + [PATCH] KVM: arm/arm64: fix vGICv2 KVM_DEV_ARM_VGIC_GRP_CPU/DIST_REGS
>> [2]: [PATCH] KVM: arm/arm64: The GIC is dead, long live the GIC
>>
>> hence the RFC.
>>
>> GSI flat routing setup on QEMU can be found at:
>> https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html
> 
> So I gave this series (as part of -next atm) some testing on a Freescale
> machine.
> 
> I quickly boot tested the following combinations of options:
> host kernel: next-20160728 and v4.7.0
> guest kernel: next-20160728 and v4.4.0
> QEMU: v2.6 and latest HEAD + Eric's ITS & GSI patches (from github)
>       gic-version=2 and gic-version=3
>       virtio-mmio and virtio-pci
> kvmtool: latest HEAD and vits-v7
>          --irqchip=gic-{v2,v3,v3-its}
>          --force-pci and without (virtio-mmio)
> 
> That summed up to 68 combinations for me, all booting fine. If every
> part supported ITS and MSIs (host kernel, userland tool, virtio-pci) I
> saw the ITS emulation being used.
> I also ran three guests (two QEMU and one kvmtool) in parallel with some
> interrupt load for some hours.
> 
> For the records I could easily trigger our infamous kvmtool virtio hang
> with SMP guests on this machine, but as that happened with older kernels
> and kvmtools as well, affects only SMP guests and is totally absent when
> running with QEMU, it's not a regression and unrelated to this series.
> 
> So FWIW:
> 
> Tested-by: Andre Przywara <andre.przywara@arm.com>
Thank you very much for this testing time.

Cheers

Eric
> 
> Cheers,
> Andre
> 
>>
>> History:
>> v8 -> v9:
>> - remove "[RFC v8 7/7] KVM: arm: Enable KVM_SIGNAL_MSI and MSI routing"
>> - fix spi id check at IRQ injection
>> - use default in irqchip switch(ue->type)
>>
>> v7 -> v8:
>> - take into account Radim's comments:
>>   x rework kvm_kernel_irq_routing_entry and remove msi_msg now
>>     replaced by a custom struct
>>   x check user entry flags depending on user entry type
>>   x I kept last patch although challenged by Radim. It can be dropped
>>     according to the consensus.
>>
>> v6 -> v7:
>> - take into account Drew's and Andre's comments
>> - new patch moving declarations of kvm_setup_default_irq_routing and
>>   kvm_setup_empty_irq_routing outside of kvm_host.h
>> - vgic_v2m_inject_msi moved in vgic-irqfd
>> - re-introduce irq.h
>> - add msi_ prefix to flags/devid kvm_kernel_irq_routing_entry fields
>> - move kvm_vgic_setup_default_irq_routing declaration in arm_vgic.h
>>   and definition in vgic-irqfd.c
>> - remove BUG_ON(!vgic_initialized(kvm) in vgic_irqfd_set_irq
>> - move KVM_IRQCHIP_NUM_PINS in arm_vgic.h and use VGIC_MAX_SPI instead
>>   of 1020
>>
>> v5 -> v6:
>> - rebase on Andre's v8 + removal of old vgic
>> - tested on Cavium ThunderX
>>
>> V4 -> v5:
>> - rebase on Andre's v7 + final new vgic code
>> - check msi->data is within SPI range in vgic_v2m_inject_msi
>> - squashed enable irq routing and default irqchip table patches
>> - handle default irqchip table allocation failure
>> - some rephrasing in doc & comment according to Christoffer's feedbacks
>> - lock issue reported by Pavel seems to have disappear after 4.2 (MSI injection
>>   fast path)
>>
>> v3 -> v4:
>> - rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
>>   a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
>>   reflesh/reminder.
>> - rewrite the cover letter
>>
>> v2 -> v3:
>> - don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
>>   use KVM_MSI_VALID_DEVID flag instead
>> - propagate user flags downto the kernel to make sure the userspace
>>   correctly set devid in GICv3 ITS case (still under discussion)
>>
>> v1 -> v2:
>> - user API changed:
>>   x devid id passed in kvm_irq_routing_msi
>>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
>> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
>> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>>   in eventfd.c
>> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
>> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH:
>> - clearly state limited support on arm/arm64:
>>   KVM_IRQ_LINE not impacted by GSI routing
>> - add default routing table feature (new patch file)
>> - changed uapi to use padding field area
>> - reword api.txt
>>
>>
>> Eric Auger (6):
>>   KVM: api: Pass the devid in the msi routing entry
>>   KVM: Add devid in kvm_kernel_irq_routing_entry
>>   KVM: irqchip: Convey devid to kvm_set_msi
>>   KVM: Move kvm_setup_default/empty_irq_routing declaration in arch
>>     specific header
>>   KVM: arm/arm64: Enable irqchip routing
>>   KVM: arm/arm64: Enable MSI routing
>>
>>  Documentation/virtual/kvm/api.txt |  34 +++++++++---
>>  arch/arm/kvm/Kconfig              |   2 +
>>  arch/arm/kvm/Makefile             |   1 +
>>  arch/arm/kvm/irq.h                |  19 +++++++
>>  arch/arm64/kvm/Kconfig            |   2 +
>>  arch/arm64/kvm/Makefile           |   1 +
>>  arch/arm64/kvm/irq.h              |  19 +++++++
>>  arch/x86/kvm/irq.h                |   3 ++
>>  include/kvm/arm_vgic.h            |   7 +++
>>  include/linux/kvm_host.h          |  12 +++--
>>  include/uapi/linux/kvm.h          |   5 +-
>>  virt/kvm/arm/vgic/vgic-init.c     |   4 ++
>>  virt/kvm/arm/vgic/vgic-irqfd.c    | 108 +++++++++++++++++++++++++++++++-------
>>  virt/kvm/arm/vgic/vgic.c          |   7 ---
>>  virt/kvm/irqchip.c                |  28 ++++++----
>>  15 files changed, 207 insertions(+), 45 deletions(-)
>>  create mode 100644 arch/arm/kvm/irq.h
>>  create mode 100644 arch/arm64/kvm/irq.h
>>