[PATCH v3 0/7] KVM: arm/arm64: gsi routing support

[PATCH v3 0/7] KVM: arm/arm64: gsi routing support

[Eric Auger]

With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
appears to be requested. More specifically MSI routing is needed.
irqchip routing does not sound to be really useful on arm but usage of
MSI routing also mandates to integrate irqchip routing. The initial
implementation of irqfd on arm must be upgraded with the integration
of kvm irqchip.c code and the implementation of its standard hooks
in the architecture specific part.

In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
table with flat irqchip routing entries is built enabling to inject gsi
corresponding to the SPI indexes seen by the guest.

As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
default routing table and is responsible for building the whole routing
table.

for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
- only applies to KVM_IRQFD and not to KVM_IRQ_LINE

- irqchip routing was tested on Calxeda midway (VFIO with irqfd)
  with and without explicit routing
- MSI routing without GICv3 ITS was tested using APM Xgene-I
  (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
- MSI routing with GICv3 ITS is *NOT* tested.

Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.2-rc4-gsi-routing-v3

It applies on Andre's [PATCH v2 00/15] KVM: arm64: GICv3 ITS emulation
(http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/355727.html)

History:
v2 -> v3:
- eventually got rid of KVM_IRQ_ROUTING_EXTENDED_MSI new type at user
  api level and use KVM_MSI_VALID_DEVID flag instead
- remove usage of KVM_IRQ_ROUTING_EXTENDED_MSI type at kernel level too.
- propagate user flags downto the kernel to make sure the userspace
  correctly set devid in GICv3 ITS case (still under discussion)

v1 -> v2:
- user API changed:
  x devid id passed in kvm_irq_routing_msi
  x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
- kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
  composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
- Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
  in eventfd.c
- added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
- fix bug reported by Andre: bad setting of msi.flags and msi.devid
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH:
- clearly state limited support on arm/arm64:
  KVM_IRQ_LINE not impacted by GSI routing
- add default routing table feature (new patch file)
- changed uapi to use padding field area
- reword api.txt

Eric Auger (7):
  KVM: api: pass the devid in the msi routing entry
  KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
  KVM: irqchip: convey devid to kvm_set_msi
  KVM: arm/arm64: enable irqchip routing
  KVM: arm/arm64: build a default routing table
  KVM: arm/arm64: enable MSI routing
  KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

 Documentation/virtual/kvm/api.txt |  35 ++++++++++---
 arch/arm/include/asm/kvm_host.h   |   2 +
 arch/arm/kvm/Kconfig              |   3 ++
 arch/arm/kvm/Makefile             |   2 +-
 arch/arm64/include/asm/kvm_host.h |   1 +
 arch/arm64/kvm/Kconfig            |   2 +
 arch/arm64/kvm/Makefile           |   2 +-
 include/kvm/arm_vgic.h            |   2 -
 include/linux/kvm_host.h          |   8 ++-
 include/uapi/linux/kvm.h          |   5 +-
 virt/kvm/arm/vgic-v2-emul.c       |  16 ++++++
 virt/kvm/arm/vgic.c               | 107 ++++++++++++++++++++++++++++++--------
 virt/kvm/irqchip.c                |   8 ++-
 13 files changed, 158 insertions(+), 35 deletions(-)

-- 
1.9.1

[PATCH v3 1/7] KVM: api: pass the devid in the msi routing entry

[Eric Auger]

On ARM, the MSI msg (address and data) comes along with
out-of-band device ID information. The device ID encodes the
device that writes the MSI msg. Let's convey the device id in
kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
kvm_irq_routing_entry to indicate the msi devid is populated.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
  usage of KVM_MSI_VALID_DEVID flag
- add note about KVM_CAP_MSI_DEVID capability

v1 -> v2:
- devid id passed in kvm_irq_routing_msi instead of in
  kvm_irq_routing_entry

RFC -> PATCH
- remove kvm_irq_routing_extended_msi and use union instead
---
 Documentation/virtual/kvm/api.txt | 18 ++++++++++++++++--
 include/uapi/linux/kvm.h          |  5 ++++-
 2 files changed, 20 insertions(+), 3 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 1b53155..d4b29df 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1454,7 +1454,10 @@ struct kvm_irq_routing_entry {
 #define KVM_IRQ_ROUTING_MSI 2
 #define KVM_IRQ_ROUTING_S390_ADAPTER 3
 
-No flags are specified so far, the corresponding field must be set to zero.
+flags:
+- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
+  routing entry type, it tells the msi devid contains a valid value.
+- zero otherwise
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1465,9 +1468,20 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
+       for the device that wrote the MSI message.
+       For PCI, this is usually a BFD identifier in the lower 16 bits.
+
+The per-VM KVM_CAP_MSI_DEVID capability advertises the need to provide
+the device ID. If this capability is not set, userland cannot rely on
+the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
+
 struct kvm_irq_routing_s390_adapter {
 	__u64 ind_addr;
 	__u64 summary_addr;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 1c48def..3c865b9 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -830,7 +830,10 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
 struct kvm_irq_routing_s390_adapter {
-- 
1.9.1

[PATCH v3 2/7] KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry

[Eric Auger]

Extend kvm_kernel_irq_routing_entry to transport the device id
field, devid. A new flags field makes possible to indicate the
devid is valid. Those additions are used for ARM GICv3 ITS MSI
injection.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- add flags

v1 -> v2:
- replace msi_msg field by a struct composed of msi_msg and devid

RFC -> PATCH:
- reword the commit message after change in first patch (uapi)
---
 include/linux/kvm_host.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index 05e99b8..b784465 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -321,7 +321,11 @@ struct kvm_kernel_irq_routing_entry {
 			unsigned irqchip;
 			unsigned pin;
 		} irqchip;
-		struct msi_msg msi;
+		struct {
+			struct msi_msg msi;
+			u32 flags;
+			u32 devid;
+		};
 		struct kvm_s390_adapter_int adapter;
 	};
 	struct hlist_node link;
-- 
1.9.1

[PATCH v3 3/7] KVM: irqchip: convey devid to kvm_set_msi

[Eric Auger]

on ARM, a devid field is populated in kvm_msi struct in case the
flag is set to KVM_MSI_VALID_DEVID. Let's propagate both flags and
devid field in kvm_kernel_irq_routing_entry.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- do not set the type to KVM_IRQ_ROUTING_EXTENDED_MSI anymore as
  suggested by Andre
- correct msi->flags check
- propagate the flags
---
 virt/kvm/irqchip.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 21c1424..75c9931 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -72,12 +72,14 @@ int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
 {
 	struct kvm_kernel_irq_routing_entry route;
 
-	if (!irqchip_in_kernel(kvm) || msi->flags != 0)
+	if (!irqchip_in_kernel(kvm) || (msi->flags & ~KVM_MSI_VALID_DEVID))
 		return -EINVAL;
 
 	route.msi.address_lo = msi->address_lo;
 	route.msi.address_hi = msi->address_hi;
 	route.msi.data = msi->data;
+	route.flags = msi->flags;
+	route.devid = msi->devid;
 
 	return kvm_set_msi(&route, kvm, KVM_USERSPACE_IRQ_SOURCE_ID, 1, false);
 }
-- 
1.9.1

[PATCH v3 4/7] KVM: arm/arm64: enable irqchip routing

[Eric Auger]

This patch adds compilation and link against irqchip.

On ARM, irqchip routing is not really useful since there is
a single irqchip. However main motivation behind using irqchip
code is to enable MSI routing code. With the support of in-kernel
GICv3 ITS emulation, it now seems to be a MUST HAVE requirement.

Functions previously implemented in vgic.c and substitute
to more complex irqchip implementation are removed:

- kvm_send_userspace_msi
- kvm_irq_map_chip_pin
- kvm_set_irq
- kvm_irq_map_gsi.

Routing standard hooks are now implemented in vgic:
- kvm_set_routing_entry
- kvm_set_irq
- kvm_set_msi

Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.

MSI routing setup is not yet allowed.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
  by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
- vgic_irqfd_set_irq now is static
- propagate flags
- add comments

v1 -> v2:
- fix bug reported by Andre related to msi.flags and msi.devid setting
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH
- reword api.txt:
  x move MSI routing comments in a subsequent patch,
  x clearly state GSI routing does not apply to KVM_IRQ_LINE

Conflicts:
	include/kvm/arm_vgic.h
	virt/kvm/arm/vgic.c
---
 Documentation/virtual/kvm/api.txt | 12 ++++--
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  2 +
 arch/arm/kvm/Makefile             |  2 +-
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  2 +
 arch/arm64/kvm/Makefile           |  2 +-
 include/kvm/arm_vgic.h            |  2 -
 virt/kvm/arm/vgic.c               | 82 ++++++++++++++++++++++++++++-----------
 virt/kvm/irqchip.c                |  2 +
 10 files changed, 78 insertions(+), 31 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index d4b29df..b6913c9 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1421,13 +1421,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
 4.52 KVM_SET_GSI_ROUTING
 
 Capability: KVM_CAP_IRQ_ROUTING
-Architectures: x86 s390
+Architectures: x86 s390 arm arm64
 Type: vm ioctl
 Parameters: struct kvm_irq_routing (in)
 Returns: 0 on success, -1 on error
 
 Sets the GSI routing table entries, overwriting any previously set entries.
 
+On arm/arm64, GSI routing has the following limitation:
+- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
+
 struct kvm_irq_routing {
 	__u32 nr;
 	__u32 flags;
@@ -2345,9 +2348,10 @@ Note that closing the resamplefd is not sufficient to disable the
 irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
 and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
 
-On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
-Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
-given by gsi + 32.
+On arm/arm64, gsi routing being supported, the following can happen:
+- in case no routing entry is associated to this gsi, injection fails
+- in case the gsi is associated to an irqchip routing entry,
+  irqchip.pin + 32 corresponds to the injected SPI ID.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 56cac05..8a68e9e 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -42,6 +42,8 @@
 
 #define KVM_VCPU_MAX_FEATURES 2
 
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
+
 #include <kvm/arm_vgic.h>
 
 u32 *kvm_vcpu_reg(struct kvm_vcpu *vcpu, u8 reg_num, u32 mode);
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index bfb915d..151e710 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -31,6 +31,8 @@ config KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
 	---help---
 	  Support hosting virtualized guest machines.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index c5eef02c..1a8f48a 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt)
 AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt)
 
 KVM := ../../../virt/kvm
-kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o
+kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o $(KVM)/irqchip.o
 
 obj-y += kvm-arm.o init.o interrupts.o
 obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 8d78a72..38a3291 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -44,6 +44,7 @@
 #include <kvm/arm_arch_timer.h>
 
 #define KVM_VCPU_MAX_FEATURES 3
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPI */
 
 int __attribute_const__ kvm_target_cpu(void);
 int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index ff9722f..1a9900d 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -32,6 +32,8 @@ config KVM
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
 	select HAVE_KVM_MSI
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	---help---
 	  Support hosting virtualized guest machines.
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 9803307..90a08457 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -11,7 +11,7 @@ ARM=../../../arch/arm/kvm
 
 obj-$(CONFIG_KVM_ARM_HOST) += kvm.o
 
-kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o $(KVM)/irqchip.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o
 
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index f50081c..1b370a0 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -375,6 +375,4 @@ static inline int vgic_v3_probe(struct device_node *vgic_node,
 }
 #endif
 
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
-
 #endif
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 081a1ef..8114235 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2215,42 +2215,78 @@ out_free_irq:
 	return ret;
 }
 
-int kvm_irq_map_gsi(struct kvm *kvm,
-		    struct kvm_kernel_irq_routing_entry *entries,
-		    int gsi)
-{
-	return 0;
-}
-
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
-{
-	return pin;
-}
-
-int kvm_set_irq(struct kvm *kvm, int irq_source_id,
-		u32 irq, int level, bool line_status)
+/**
+ * vgic_irqfd_set_irq: inject the IRQ corresponding to the
+ * irqchip routing entry
+ *
+ * This is the entry point for irqfd IRQ injection
+ */
+static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
+			struct kvm *kvm, int irq_source_id,
+			int level, bool line_status)
 {
-	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
+	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
 
-	trace_kvm_set_irq(irq, level, irq_source_id);
+	trace_kvm_set_irq(spi_id, level, irq_source_id);
 
 	BUG_ON(!vgic_initialized(kvm));
 
-	return kvm_vgic_inject_irq(kvm, 0, spi, level);
+	if (spi_id > min(kvm->arch.vgic.nr_irqs, 1020))
+		return -EINVAL;
+	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
+}
+
+/**
+ * kvm_set_routing_entry: populate a kvm routing entry
+ * from a user routing entry
+ *
+ * @e: kvm kernel routing entry handle
+ * @ue: user api routing entry handle
+ * return 0 on success, -EINVAL on errors.
+ */
+int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
+			  const struct kvm_irq_routing_entry *ue)
+{
+	int r = -EINVAL;
+
+	switch (ue->type) {
+	case KVM_IRQ_ROUTING_IRQCHIP:
+		e->set = vgic_irqfd_set_irq;
+		e->irqchip.irqchip = ue->u.irqchip.irqchip;
+		e->irqchip.pin = ue->u.irqchip.pin;
+		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
+		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
+			goto out;
+		break;
+	default:
+		goto out;
+	}
+	r = 0;
+out:
+	return r;
 }
 
-/* MSI not implemented yet */
+/**
+ * kvm_set_msi: inject the MSI corresponding to the
+ * MSI routing entry
+ *
+ * This is the entry point for irqfd MSI injection
+ * and userspace MSI injection.
+ */
 int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 		struct kvm *kvm, int irq_source_id,
 		int level, bool line_status)
 {
-	return 0;
-}
+	struct kvm_msi msi;
+
+	msi.address_lo = e->msi.address_lo;
+	msi.address_hi = e->msi.address_hi;
+	msi.data = e->msi.data;
+	msi.flags = e->flags;
+	msi.devid = e->devid;
 
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
-{
 	if (kvm->arch.vgic.vm_ops.inject_msi)
-		return kvm->arch.vgic.vm_ops.inject_msi(kvm, msi);
+		return kvm->arch.vgic.vm_ops.inject_msi(kvm, &msi);
 	else
 		return -ENODEV;
 }
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 75c9931..e1559d4 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -29,7 +29,9 @@
 #include <linux/srcu.h>
 #include <linux/export.h>
 #include <trace/events/kvm.h>
+#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
 #include "irq.h"
+#endif
 
 struct kvm_irq_routing_table {
 	int chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
-- 
1.9.1

[PATCH v3 5/7] KVM: arm/arm64: build a default routing table

[Eric Auger]

Implement a default routing table made of flat irqchip routing
entries (gsi = irqchip.pin) covering the VGIC SPI indexes.
This routing table is overwritten by the first user-space call
to KVM_SET_GSI_ROUTING ioctl.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

PATCH: creation
---
 virt/kvm/arm/vgic.c | 21 +++++++++++++++++++++
 1 file changed, 21 insertions(+)

diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 8114235..1b092e3 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -1785,6 +1785,8 @@ int vgic_init(struct kvm *kvm)
 	ret |= vgic_init_bitmap(&dist->irq_cfg, nr_cpus, nr_irqs);
 	ret |= vgic_init_bytemap(&dist->irq_priority, nr_cpus, nr_irqs);
 
+	ret |= kvm_setup_default_irq_routing(kvm);
+
 	if (ret)
 		goto out;
 
@@ -2266,6 +2268,25 @@ out:
 	return r;
 }
 
+int kvm_setup_default_irq_routing(struct kvm *kvm)
+{
+	struct kvm_irq_routing_entry *entries;
+	u32 nr = kvm->arch.vgic.nr_irqs - VGIC_NR_PRIVATE_IRQS;
+	int i, ret;
+
+	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
+			  GFP_KERNEL);
+	for (i = 0; i < nr; i++) {
+		entries[i].gsi = i;
+		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
+		entries[i].u.irqchip.irqchip = 0;
+		entries[i].u.irqchip.pin = i;
+	}
+	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
+	kfree(entries);
+	return ret;
+}
+
 /**
  * kvm_set_msi: inject the MSI corresponding to the
  * MSI routing entry
-- 
1.9.1

[PATCH v3 6/7] KVM: arm/arm64: enable MSI routing

[Eric Auger]

Up to now, only irqchip routing entries could be set. This patch
adds the capability to insert MSI routing entries.

For ARM64, let's also increase KVM_MAX_IRQ_ROUTES to 4096: this
include SPI irqchip routes plus MSI routes. In the future this
might be extended.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---
v2 -> v3:
- remove any reference to KVM_IRQ_ROUTING_EXTENDED_MSI type
- unconditionnaly uapi flags and devid downto the kernel
  routing entry struct
- handle KVM_MSI_VALID_DEVID flag in kvm_set_irq_routing
- note about KVM_CAP_MSI_DEVID moved in the first patch file
  of the series

v1 -> v2:
- adapt to new routing entry types

RFC -> PATCH:
- move api MSI routing updates into that patch file
- use new devid field of user api struct
---
 Documentation/virtual/kvm/api.txt | 5 +++++
 include/linux/kvm_host.h          | 2 ++
 virt/kvm/arm/vgic.c               | 8 ++++++++
 virt/kvm/irqchip.c                | 2 +-
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index b6913c9..4f15347 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2352,6 +2352,11 @@ On arm/arm64, gsi routing being supported, the following can happen:
 - in case no routing entry is associated to this gsi, injection fails
 - in case the gsi is associated to an irqchip routing entry,
   irqchip.pin + 32 corresponds to the injected SPI ID.
+- in case the gsi is associated to an MSI routing entry,
+  * without GICv3 ITS in-kernel emulation, MSI data matches the SPI ID
+    of the injected SPI
+  * with GICv3 ITS in-kernel emulation, the MSI message and device ID
+    are translated into an LPI.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index b784465..d00a113 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -995,6 +995,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
 
 #ifdef CONFIG_S390
 #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
+#elif defined(CONFIG_ARM64)
+#define KVM_MAX_IRQ_ROUTES 4096
 #else
 #define KVM_MAX_IRQ_ROUTES 1024
 #endif
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 1b092e3..0165f34 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2260,6 +2260,14 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
 		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
 			goto out;
 		break;
+	case KVM_IRQ_ROUTING_MSI:
+		e->set = kvm_set_msi;
+		e->msi.address_lo = ue->u.msi.address_lo;
+		e->msi.address_hi = ue->u.msi.address_hi;
+		e->msi.data = ue->u.msi.data;
+		e->flags = ue->flags;
+		e->devid = ue->u.msi.devid;
+		break;
 	default:
 		goto out;
 	}
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index e1559d4..48ebc4d 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -217,7 +217,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
 			goto out;
 
 		r = -EINVAL;
-		if (ue->flags)
+		if (ue->flags & ~KVM_MSI_VALID_DEVID)
 			goto out;
 		r = setup_routing_entry(new, e, ue);
 		if (r)
-- 
1.9.1

[PATCH v3 7/7] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Eric Auger]

If the ITS modality is not available, let's simply support MSI
injection by transforming the MSI.data into an SPI ID.

This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
routing for arm too.

Signed-off-by: Eric Auger <eric.auger@linaro.org>

---

v2 -> v3:
- reword the commit message
- add sanity check about devid provision

v1 -> v2:
- introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
  advice
---
 arch/arm/kvm/Kconfig        |  1 +
 virt/kvm/arm/vgic-v2-emul.c | 16 ++++++++++++++++
 2 files changed, 17 insertions(+)

diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 151e710..0f58baf 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -31,6 +31,7 @@ config KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select HAVE_KVM_MSI
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_IRQ_ROUTING
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
diff --git a/virt/kvm/arm/vgic-v2-emul.c b/virt/kvm/arm/vgic-v2-emul.c
index 8faa28c..ba2b393 100644
--- a/virt/kvm/arm/vgic-v2-emul.c
+++ b/virt/kvm/arm/vgic-v2-emul.c
@@ -478,6 +478,21 @@ static bool vgic_v2_queue_sgi(struct kvm_vcpu *vcpu, int irq)
 }
 
 /**
+ * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
+ * the SPI ID matching the msi data
+ *
+ * @kvm: pointer to the kvm struct
+ * @msi: the msi struct handle
+ */
+static int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	if (msi->flags & KVM_MSI_VALID_DEVID)
+		return -EINVAL;
+
+	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);
+}
+
+/**
  * kvm_vgic_map_resources - Configure global VGIC state before running any VCPUs
  * @kvm: pointer to the kvm struct
  *
@@ -566,6 +581,7 @@ void vgic_v2_init_emulation(struct kvm *kvm)
 	dist->vm_ops.add_sgi_source = vgic_v2_add_sgi_source;
 	dist->vm_ops.init_model = vgic_v2_init_model;
 	dist->vm_ops.map_resources = vgic_v2_map_resources;
+	dist->vm_ops.inject_msi = vgic_v2m_inject_msi;
 
 	dist->vgic_cpu_base = VGIC_ADDR_UNDEF;
 	dist->vgic_dist_base = VGIC_ADDR_UNDEF;
-- 
1.9.1

RE: [PATCH v3 0/7] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

Tested-by: Pavel Fedin <p.fedin@samsung.com>

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

> -----Original Message-----
> From: Eric Auger [mailto:eric.auger@linaro.org]
> Sent: Monday, August 03, 2015 6:31 PM
> To: eric.auger@st.com; eric.auger@linaro.org; linux-arm-kernel@lists.infradead.org;
> kvmarm@lists.cs.columbia.edu; kvm@vger.kernel.org; christoffer.dall@linaro.org;
> marc.zyngier@arm.com
> Cc: linux-kernel@vger.kernel.org; patches@linaro.org; pbonzini@redhat.com;
> andre.przywara@arm.com; p.fedin@samsung.com
> Subject: [PATCH v3 0/7] KVM: arm/arm64: gsi routing support
> 
> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
> appears to be requested. More specifically MSI routing is needed.
> irqchip routing does not sound to be really useful on arm but usage of
> MSI routing also mandates to integrate irqchip routing. The initial
> implementation of irqfd on arm must be upgraded with the integration
> of kvm irqchip.c code and the implementation of its standard hooks
> in the architecture specific part.
> 
> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
> table with flat irqchip routing entries is built enabling to inject gsi
> corresponding to the SPI indexes seen by the guest.
> 
> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
> default routing table and is responsible for building the whole routing
> table.
> 
> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
> 
> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>   with and without explicit routing
> - MSI routing without GICv3 ITS was tested using APM Xgene-I
>   (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
> - MSI routing with GICv3 ITS is *NOT* tested.
> 
> Code can be found at
https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.2-rc4-
> gsi-routing-v3
> 
> It applies on Andre's [PATCH v2 00/15] KVM: arm64: GICv3 ITS emulation
> (http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/355727.html)
> 
> History:
> v2 -> v3:
> - eventually got rid of KVM_IRQ_ROUTING_EXTENDED_MSI new type at user
>   api level and use KVM_MSI_VALID_DEVID flag instead
> - remove usage of KVM_IRQ_ROUTING_EXTENDED_MSI type at kernel level too.
> - propagate user flags downto the kernel to make sure the userspace
>   correctly set devid in GICv3 ITS case (still under discussion)
> 
> v1 -> v2:
> - user API changed:
>   x devid id passed in kvm_irq_routing_msi
>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>   in eventfd.c
> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH:
> - clearly state limited support on arm/arm64:
>   KVM_IRQ_LINE not impacted by GSI routing
> - add default routing table feature (new patch file)
> - changed uapi to use padding field area
> - reword api.txt
> 
> Eric Auger (7):
>   KVM: api: pass the devid in the msi routing entry
>   KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
>   KVM: irqchip: convey devid to kvm_set_msi
>   KVM: arm/arm64: enable irqchip routing
>   KVM: arm/arm64: build a default routing table
>   KVM: arm/arm64: enable MSI routing
>   KVM: arm: enable KVM_SIGNAL_MSI and MSI routing
> 
>  Documentation/virtual/kvm/api.txt |  35 ++++++++++---
>  arch/arm/include/asm/kvm_host.h   |   2 +
>  arch/arm/kvm/Kconfig              |   3 ++
>  arch/arm/kvm/Makefile             |   2 +-
>  arch/arm64/include/asm/kvm_host.h |   1 +
>  arch/arm64/kvm/Kconfig            |   2 +
>  arch/arm64/kvm/Makefile           |   2 +-
>  include/kvm/arm_vgic.h            |   2 -
>  include/linux/kvm_host.h          |   8 ++-
>  include/uapi/linux/kvm.h          |   5 +-
>  virt/kvm/arm/vgic-v2-emul.c       |  16 ++++++
>  virt/kvm/arm/vgic.c               | 107 ++++++++++++++++++++++++++++++--------
>  virt/kvm/irqchip.c                |   8 ++-
>  13 files changed, 158 insertions(+), 35 deletions(-)
> 
> --
> 1.9.1

Re: [PATCH v3 0/7] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Pavel,
On 08/06/2015 02:06 PM, Pavel Fedin wrote:
> Tested-by: Pavel Fedin <p.fedin@samsung.com>

Many thanks for testing!

Best Regards

Eric
> 
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
> 
>> -----Original Message-----
>> From: Eric Auger [mailto:eric.auger@linaro.org]
>> Sent: Monday, August 03, 2015 6:31 PM
>> To: eric.auger@st.com; eric.auger@linaro.org; linux-arm-kernel@lists.infradead.org;
>> kvmarm@lists.cs.columbia.edu; kvm@vger.kernel.org; christoffer.dall@linaro.org;
>> marc.zyngier@arm.com
>> Cc: linux-kernel@vger.kernel.org; patches@linaro.org; pbonzini@redhat.com;
>> andre.przywara@arm.com; p.fedin@samsung.com
>> Subject: [PATCH v3 0/7] KVM: arm/arm64: gsi routing support
>>
>> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
>> appears to be requested. More specifically MSI routing is needed.
>> irqchip routing does not sound to be really useful on arm but usage of
>> MSI routing also mandates to integrate irqchip routing. The initial
>> implementation of irqfd on arm must be upgraded with the integration
>> of kvm irqchip.c code and the implementation of its standard hooks
>> in the architecture specific part.
>>
>> In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
>> table with flat irqchip routing entries is built enabling to inject gsi
>> corresponding to the SPI indexes seen by the guest.
>>
>> As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
>> default routing table and is responsible for building the whole routing
>> table.
>>
>> for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
>> - only applies to KVM_IRQFD and not to KVM_IRQ_LINE
>>
>> - irqchip routing was tested on Calxeda midway (VFIO with irqfd)
>>   with and without explicit routing
>> - MSI routing without GICv3 ITS was tested using APM Xgene-I
>>   (qemu VIRTIO-PCI vhost-net without gsi_direct_mapping).
>> - MSI routing with GICv3 ITS is *NOT* tested.
>>
>> Code can be found at
> https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.2-rc4-
>> gsi-routing-v3
>>
>> It applies on Andre's [PATCH v2 00/15] KVM: arm64: GICv3 ITS emulation
>> (http://lists.infradead.org/pipermail/linux-arm-kernel/2015-July/355727.html)
>>
>> History:
>> v2 -> v3:
>> - eventually got rid of KVM_IRQ_ROUTING_EXTENDED_MSI new type at user
>>   api level and use KVM_MSI_VALID_DEVID flag instead
>> - remove usage of KVM_IRQ_ROUTING_EXTENDED_MSI type at kernel level too.
>> - propagate user flags downto the kernel to make sure the userspace
>>   correctly set devid in GICv3 ITS case (still under discussion)
>>
>> v1 -> v2:
>> - user API changed:
>>   x devid id passed in kvm_irq_routing_msi
>>   x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
>> - kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
>>   composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
>> - Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
>>   in eventfd.c
>> - added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
>> - fix bug reported by Andre: bad setting of msi.flags and msi.devid
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH:
>> - clearly state limited support on arm/arm64:
>>   KVM_IRQ_LINE not impacted by GSI routing
>> - add default routing table feature (new patch file)
>> - changed uapi to use padding field area
>> - reword api.txt
>>
>> Eric Auger (7):
>>   KVM: api: pass the devid in the msi routing entry
>>   KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
>>   KVM: irqchip: convey devid to kvm_set_msi
>>   KVM: arm/arm64: enable irqchip routing
>>   KVM: arm/arm64: build a default routing table
>>   KVM: arm/arm64: enable MSI routing
>>   KVM: arm: enable KVM_SIGNAL_MSI and MSI routing
>>
>>  Documentation/virtual/kvm/api.txt |  35 ++++++++++---
>>  arch/arm/include/asm/kvm_host.h   |   2 +
>>  arch/arm/kvm/Kconfig              |   3 ++
>>  arch/arm/kvm/Makefile             |   2 +-
>>  arch/arm64/include/asm/kvm_host.h |   1 +
>>  arch/arm64/kvm/Kconfig            |   2 +
>>  arch/arm64/kvm/Makefile           |   2 +-
>>  include/kvm/arm_vgic.h            |   2 -
>>  include/linux/kvm_host.h          |   8 ++-
>>  include/uapi/linux/kvm.h          |   5 +-
>>  virt/kvm/arm/vgic-v2-emul.c       |  16 ++++++
>>  virt/kvm/arm/vgic.c               | 107 ++++++++++++++++++++++++++++++--------
>>  virt/kvm/irqchip.c                |   8 ++-
>>  13 files changed, 158 insertions(+), 35 deletions(-)
>>
>> --
>> 1.9.1
>