[RFC v6 0/6] KVM: arm/arm64: gsi routing support

[RFC v6 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

With the advent of GICv3 ITS in-kernel emulation, KVM MSI routing
becomes mandated for proper VIRTIO-PCI vhost integration.

In QEMU, when the VIRTIO-PCI device is programmed with the MSI message,
we previously used direct_mapping trick: this consists in extracting the
IRQ ID found in the MSI message and associating an irqfd to that IRQ ID.
When vhost worker thread gets a new buffer it signals the irqfd and kvm
then injects this IRQ ID on guest. That way although the guest uses MSIs,
no MSI emulation is used.

This worked fine with GICv2m but does not work anymore with GICV3 ITS.
Indeed this latter implements IRQ translation: what is found in the MSI
message no more is the target IRQ ID but is an intermediate event ID used
in the translation process.

Hence true MSI routing is needed so that the vhost back channel irqfd is
associated to a dummy gsi ID, routed towards the programmed MSI. When KVM
injects the MSI through the in-kernel ITS emulation, the MSI is properly
translated and eventually the LPI ID associated to the event ID is injected
on guest.

MSI routing also mandates to integrate irqchip routing. The initial
implementation of irqfd on arm must be upgraded with the integration
of kvm irqchip.c code and the implementation of its standard hooks
in the architecture specific part.

In case KVM_SET_GSI_ROUTING ioctl is not called, a default routing
table with flat irqchip routing entries is built enabling to inject gsi
corresponding to the SPI indexes seen by the guest.

As soon as KVM_SET_GSI_ROUTING is called, user-space overwrites this
default routing table and is responsible for building the whole routing
table.

for arm/arm64 KVM_SET_GSI_ROUTING has a limited support:
- only applies to KVM_IRQFD and not to KVM_IRQ_LINE

- irqchip routing was tested on Calxeda midway (VFIO with irqfd)
  with and without explicit routing
- MSI routing was tested on AMD Overdrive and Cavium ThunderX

Code can be found at:
https://github.com/eauger/linux/tree/v4.7-rc6-its-emul-v8-gsi-routing-v6

The series depends on

[1]: [PATCH v8 00/12] KVM: arm64: GICv3 ITS emulation
     http://www.spinics.net/lists/kvm/msg135048.html
[2]: [PATCH] KVM: arm/arm64: The GIC is dead, long live the GIC

hence the RFC.

GSI flat routing setup on QEMU can be found at:
https://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg06262.html

History:
v5 -> v6:
- rebase on Andre's v8 + removal of old vgic
- tested on Cavium ThunderX

V4 -> v5:
- rebase on Andre's v7 + final new vgic code
- check msi->data is within SPI range in vgic_v2m_inject_msi
- squashed enable irq routing and default irqchip table patches
- handle default irqchip table allocation failure
- some rephrasing in doc & comment according to Christoffer's feedbacks
- lock issue reported by Pavel seems to have disappear after 4.2 (MSI injection
  fast path)

v3 -> v4:
- rebase on top of NEW-VGIC RFC and ITS emulation series v4. This is not
  a stable foundation yet. Hence the revert to RFC. This v4 mostly is a
  reflesh/reminder.
- rewrite the cover letter

v2 -> v3:
- don't use KVM_IRQ_ROUTING_EXTENDED_MSI type at uapi and kernel level anymore;
  use KVM_MSI_VALID_DEVID flag instead
- propagate user flags downto the kernel to make sure the userspace
  correctly set devid in GICv3 ITS case (still under discussion)

v1 -> v2:
- user API changed:
  x devid id passed in kvm_irq_routing_msi
  x kept the new routing entry type: KVM_IRQ_ROUTING_EXTENDED_MSI
- kvm_host.h: adopt Andre's proposal to replace the msi_msg by a struct
  composed of the msi_msg and devid in kvm_kernel_irq_routing_entry
- Fix bug reported by Pavel: Added KVM_IRQ_ROUTING_EXTENDED_MSI handling
  in eventfd.c
- added vgic_v2m_inject_msi in vgic-v2-emul.c as suggested by Andre
- fix bug reported by Andre: bad setting of msi.flags and msi.devid
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH:
- clearly state limited support on arm/arm64:
  KVM_IRQ_LINE not impacted by GSI routing
- add default routing table feature (new patch file)
- changed uapi to use padding field area
- reword api.txt


Eric Auger (6):
  KVM: api: pass the devid in the msi routing entry
  KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry
  KVM: irqchip: convey devid to kvm_set_msi
  KVM: arm/arm64: enable irqchip routing
  KVM: arm/arm64: enable MSI routing
  KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

 Documentation/virtual/kvm/api.txt | 36 +++++++++++++---
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  3 ++
 arch/arm/kvm/Makefile             |  1 +
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  2 +
 arch/arm64/kvm/Makefile           |  1 +
 include/linux/kvm_host.h          |  8 +++-
 include/uapi/linux/kvm.h          |  5 ++-
 virt/kvm/arm/vgic/vgic-init.c     | 27 ++++++++++++
 virt/kvm/arm/vgic/vgic-irqfd.c    | 90 ++++++++++++++++++++++++++++++---------
 virt/kvm/arm/vgic/vgic-v2.c       | 17 ++++++++
 virt/kvm/arm/vgic/vgic.c          |  7 ---
 virt/kvm/arm/vgic/vgic.h          |  1 +
 virt/kvm/irqchip.c                |  8 +++-
 15 files changed, 173 insertions(+), 36 deletions(-)

-- 
2.5.5

[RFC v6 1/6] KVM: api: pass the devid in the msi routing entry

[Eric Auger]

On ARM, the MSI msg (address and data) comes along with
out-of-band device ID information. The device ID encodes the
device that writes the MSI msg. Let's convey the device id in
kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
kvm_irq_routing_entry to indicate the msi devid is populated.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v4 -> v5:
- some rephrasing in api.txt according to Christoffer's comments
v2 -> v3:
- replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
  usage of KVM_MSI_VALID_DEVID flag
- add note about KVM_CAP_MSI_DEVID capability

v1 -> v2:
- devid id passed in kvm_irq_routing_msi instead of in
  kvm_irq_routing_entry

RFC -> PATCH
- remove kvm_irq_routing_extended_msi and use union instead
---
 Documentation/virtual/kvm/api.txt | 19 +++++++++++++++++--
 include/uapi/linux/kvm.h          |  5 ++++-
 2 files changed, 21 insertions(+), 3 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index f60b137..0065c8e 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1468,7 +1468,11 @@ struct kvm_irq_routing_entry {
 #define KVM_IRQ_ROUTING_S390_ADAPTER 3
 #define KVM_IRQ_ROUTING_HV_SINT 4
 
-No flags are specified so far, the corresponding field must be set to zero.
+flags:
+- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
+  routing entry type, specifies that the devid field contains
+  a valid value.
+- zero otherwise
 
 struct kvm_irq_routing_irqchip {
 	__u32 irqchip;
@@ -1479,9 +1483,20 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
+devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
+       for the device that wrote the MSI message.
+       For PCI, this is usually a BFD identifier in the lower 16 bits.
+
+The per-VM KVM_CAP_MSI_DEVID capability advertises the requirement to
+provide the device ID. If this capability is not set, userland cannot
+rely on the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
+
 struct kvm_irq_routing_s390_adapter {
 	__u64 ind_addr;
 	__u64 summary_addr;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index d8c4c32..eb22208 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -879,7 +879,10 @@ struct kvm_irq_routing_msi {
 	__u32 address_lo;
 	__u32 address_hi;
 	__u32 data;
-	__u32 pad;
+	union {
+		__u32 pad;
+		__u32 devid;
+	};
 };
 
 struct kvm_irq_routing_s390_adapter {
-- 
2.5.5

[RFC v6 2/6] KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry

[Eric Auger]

Extend kvm_kernel_irq_routing_entry to transport the device id
field, devid. A new flags field makes possible to indicate the
devid is valid. Those additions are used for ARM GICv3 ITS MSI
injection.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

---
v4 -> v5:
- add Christoffer's R-b

v2 -> v3:
- add flags

v1 -> v2:
- replace msi_msg field by a struct composed of msi_msg and devid

RFC -> PATCH:
- reword the commit message after change in first patch (uapi)
---
 include/linux/kvm_host.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index c87fe6f..fffa299 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -317,7 +317,11 @@ struct kvm_kernel_irq_routing_entry {
 			unsigned irqchip;
 			unsigned pin;
 		} irqchip;
-		struct msi_msg msi;
+		struct {
+			struct msi_msg msi;
+			u32 flags;
+			u32 devid;
+		};
 		struct kvm_s390_adapter_int adapter;
 		struct kvm_hv_sint hv_sint;
 	};
-- 
2.5.5

[RFC v6 3/6] KVM: irqchip: convey devid to kvm_set_msi

[Eric Auger]

on ARM, a devid field is populated in kvm_msi struct in case the
flag is set to KVM_MSI_VALID_DEVID. Let's propagate both flags and
devid field in kvm_kernel_irq_routing_entry.

Signed-off-by: Eric Auger <eric.auger@redhat.com>
Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

---
v4 -> v5:
- Add Christoffer's R-b

v2 -> v3:
- do not set the type to KVM_IRQ_ROUTING_EXTENDED_MSI anymore as
  suggested by Andre
- correct msi->flags check
- propagate the flags
---
 virt/kvm/irqchip.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 8db197b..32e5646 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -62,12 +62,14 @@ int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
 {
 	struct kvm_kernel_irq_routing_entry route;
 
-	if (!irqchip_in_kernel(kvm) || msi->flags != 0)
+	if (!irqchip_in_kernel(kvm) || (msi->flags & ~KVM_MSI_VALID_DEVID))
 		return -EINVAL;
 
 	route.msi.address_lo = msi->address_lo;
 	route.msi.address_hi = msi->address_hi;
 	route.msi.data = msi->data;
+	route.flags = msi->flags;
+	route.devid = msi->devid;
 
 	return kvm_set_msi(&route, kvm, KVM_USERSPACE_IRQ_SOURCE_ID, 1, false);
 }
-- 
2.5.5

[RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Eric Auger]

This patch adds compilation and link against irqchip.

Main motivation behind using irqchip code is to enable MSI
routing code. In the future irqchip routing may also be useful
when targeting multiple irqchips.

Routing standard callbacks now are implemented in vgic-irqfd:
- kvm_set_routing_entry
- kvm_set_irq
- kvm_set_msi

They only are supported with new_vgic code.

Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.

So from now on IRQCHIP routing is enabled and a routing table entry
must exist for irqfd injection to succeed for a given SPI. This patch
builds a default flat irqchip routing table (gsi=irqchip.pin) covering
all the VGIC SPI indexes. This routing table is overwritten by the
first first user-space call to KVM_SET_GSI_ROUTING ioctl.

MSI routing setup is not yet allowed.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v5 -> v6:
- rebase on top of Andre's v8 + removal of old vgic

v4 -> v5:
- vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
- minor comment changes
- remove trace_kvm_set_irq since it is called in irqchip
- remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
- despite Christoffer's question, in kvm_set_msi, I kept the copy from
  the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
  irqchip callback API into the struct kvm_msi * passed to
  vits_inject_msi. Since vits_inject_msi is directly called by
  kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
  to me to keep the copy.
- squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
  table into that patch
- handle default routing table alloc failure

v3 -> v4:
- provide support only for new-vgic
- code previously in vgic.c now in vgic_irqfd.c

v2 -> v3:
- unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
  by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
- vgic_irqfd_set_irq now is static
- propagate flags
- add comments

v1 -> v2:
- fix bug reported by Andre related to msi.flags and msi.devid setting
  in kvm_send_userspace_msi
- avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq

RFC -> PATCH
- reword api.txt:
  x move MSI routing comments in a subsequent patch,
  x clearly state GSI routing does not apply to KVM_IRQ_LINE
---
 Documentation/virtual/kvm/api.txt | 12 ++++--
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  2 +
 arch/arm/kvm/Makefile             |  1 +
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  2 +
 arch/arm64/kvm/Makefile           |  1 +
 virt/kvm/arm/vgic/vgic-init.c     | 27 +++++++++++++
 virt/kvm/arm/vgic/vgic-irqfd.c    | 82 ++++++++++++++++++++++++++++++---------
 virt/kvm/arm/vgic/vgic.c          |  7 ----
 virt/kvm/irqchip.c                |  2 +
 11 files changed, 109 insertions(+), 30 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 0065c8e..3bb60d3 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
 4.52 KVM_SET_GSI_ROUTING
 
 Capability: KVM_CAP_IRQ_ROUTING
-Architectures: x86 s390
+Architectures: x86 s390 arm arm64
 Type: vm ioctl
 Parameters: struct kvm_irq_routing (in)
 Returns: 0 on success, -1 on error
 
 Sets the GSI routing table entries, overwriting any previously set entries.
 
+On arm/arm64, GSI routing has the following limitation:
+- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
+
 struct kvm_irq_routing {
 	__u32 nr;
 	__u32 flags;
@@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
 irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
 and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
 
-On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
-Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
-given by gsi + 32.
+On arm/arm64, gsi routing being supported, the following can happen:
+- in case no routing entry is associated to this gsi, injection fails
+- in case the gsi is associated to an irqchip routing entry,
+  irqchip.pin + 32 corresponds to the injected SPI ID.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index 3c40facd..161997e 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -37,6 +37,8 @@
 
 #define KVM_VCPU_MAX_FEATURES 2
 
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
+
 #include <kvm/arm_vgic.h>
 
 #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 95a0005..3e1cd04 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -32,6 +32,8 @@ config KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
 	---help---
 	  Support hosting virtualized guest machines.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index 5e28df8..025d812 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
 obj-y += $(KVM)/arm/vgic/vgic-mmio.o
 obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
 obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
+obj-y += $(KVM)//irqchip.o
 obj-y += $(KVM)/arm/arch_timer.o
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index ebe8904..58f4a60 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -34,6 +34,7 @@
 #define KVM_PRIVATE_MEM_SLOTS 4
 #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
 #define KVM_HALT_POLL_NS_DEFAULT 500000
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
 
 #include <kvm/arm_vgic.h>
 #include <kvm/arm_arch_timer.h>
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 9d2eff0..9c9edc9 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -37,6 +37,8 @@ config KVM
 	select KVM_ARM_VGIC_V3
 	select KVM_ARM_PMU if HW_PERF_EVENTS
 	select HAVE_KVM_MSI
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	---help---
 	  Support hosting virtualized guest machines.
 	  We don't support KVM with 16K page tables yet, due to the multiple
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index a5b9664..695eb3c 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
 kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index 01a60dc..591571e 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
 	kvm_for_each_vcpu(i, vcpu, kvm)
 		kvm_vgic_vcpu_init(vcpu);
 
+	ret = kvm_setup_default_irq_routing(kvm);
+	if (ret)
+		goto out;
+
 	dist->initialized = true;
 out:
 	return ret;
@@ -457,3 +461,26 @@ out_free_irq:
 			kvm_get_running_vcpus());
 	return ret;
 }
+
+int kvm_setup_default_irq_routing(struct kvm *kvm)
+{
+	struct kvm_irq_routing_entry *entries;
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	u32 nr = dist->nr_spis;
+	int i, ret;
+
+	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
+			  GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	for (i = 0; i < nr; i++) {
+		entries[i].gsi = i;
+		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
+		entries[i].u.irqchip.irqchip = 0;
+		entries[i].u.irqchip.pin = i;
+	}
+	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
+	kfree(entries);
+	return ret;
+}
diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
index c675513..b03ab4e 100644
--- a/virt/kvm/arm/vgic/vgic-irqfd.c
+++ b/virt/kvm/arm/vgic/vgic-irqfd.c
@@ -17,36 +17,80 @@
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <trace/events/kvm.h>
+#include <kvm/arm_vgic.h>
+#include "vgic.h"
 
-int kvm_irq_map_gsi(struct kvm *kvm,
-		    struct kvm_kernel_irq_routing_entry *entries,
-		    int gsi)
+/**
+ * vgic_irqfd_set_irq: inject the IRQ corresponding to the
+ * irqchip routing entry
+ *
+ * This is the entry point for irqfd IRQ injection
+ */
+static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
+			struct kvm *kvm, int irq_source_id,
+			int level, bool line_status)
 {
-	return 0;
-}
+	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
+	struct vgic_dist *dist = &kvm->arch.vgic;
 
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
-			 unsigned int pin)
-{
-	return pin;
+	BUG_ON(!vgic_initialized(kvm));
+
+	if (spi_id > min(dist->nr_spis, 1020))
+		return -EINVAL;
+	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
 }
 
-int kvm_set_irq(struct kvm *kvm, int irq_source_id,
-		u32 irq, int level, bool line_status)
+/**
+ * kvm_set_routing_entry: populate a kvm routing entry
+ * from a user routing entry
+ *
+ * @e: kvm kernel routing entry handle
+ * @ue: user api routing entry handle
+ * return 0 on success, -EINVAL on errors.
+ */
+int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
+			  const struct kvm_irq_routing_entry *ue)
 {
-	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
+	int r = -EINVAL;
 
-	trace_kvm_set_irq(irq, level, irq_source_id);
-
-	BUG_ON(!vgic_initialized(kvm));
-
-	return kvm_vgic_inject_irq(kvm, 0, spi, level);
+	switch (ue->type) {
+	case KVM_IRQ_ROUTING_IRQCHIP:
+		e->set = vgic_irqfd_set_irq;
+		e->irqchip.irqchip = ue->u.irqchip.irqchip;
+		e->irqchip.pin = ue->u.irqchip.pin;
+		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
+		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
+			goto out;
+		break;
+	default:
+		goto out;
+	}
+	r = 0;
+out:
+	return r;
 }
 
-/* MSI not implemented yet */
+/**
+ * kvm_set_msi: inject the MSI corresponding to the
+ * MSI routing entry
+ *
+ * This is the entry point for irqfd MSI injection
+ * and userspace MSI injection.
+ */
 int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 		struct kvm *kvm, int irq_source_id,
 		int level, bool line_status)
 {
-	return 0;
+	struct kvm_msi msi;
+
+	msi.address_lo = e->msi.address_lo;
+	msi.address_hi = e->msi.address_hi;
+	msi.data = e->msi.data;
+	msi.flags = e->flags;
+	msi.devid = e->devid;
+
+	if (!vgic_has_its(kvm))
+		return -ENODEV;
+
+	return vgic_its_inject_msi(kvm, &msi);
 }
diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
index c4f3aba..b254833 100644
--- a/virt/kvm/arm/vgic/vgic.c
+++ b/virt/kvm/arm/vgic/vgic.c
@@ -684,10 +684,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
 	return map_is_active;
 }
 
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
-{
-	if (vgic_has_its(kvm))
-		return vgic_its_inject_msi(kvm, msi);
-	else
-		return -ENODEV;
-}
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 32e5646..03632e3 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -29,7 +29,9 @@
 #include <linux/srcu.h>
 #include <linux/export.h>
 #include <trace/events/kvm.h>
+#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
 #include "irq.h"
+#endif
 
 int kvm_irq_map_gsi(struct kvm *kvm,
 		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
-- 
2.5.5

[RFC v6 5/6] KVM: arm/arm64: enable MSI routing

[Eric Auger]

Up to now, only irqchip routing entries could be set. This patch
adds the capability to insert MSI routing entries.

For ARM64, let's also increase KVM_MAX_IRQ_ROUTES to 4096: this
include SPI irqchip routes plus MSI routes. In the future this
might be extended.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---
v2 -> v3:
- remove any reference to KVM_IRQ_ROUTING_EXTENDED_MSI type
- unconditionnaly uapi flags and devid downto the kernel
  routing entry struct
- handle KVM_MSI_VALID_DEVID flag in kvm_set_irq_routing
- note about KVM_CAP_MSI_DEVID moved in the first patch file
  of the series

v1 -> v2:
- adapt to new routing entry types

RFC -> PATCH:
- move api MSI routing updates into that patch file
- use new devid field of user api struct
---
 Documentation/virtual/kvm/api.txt | 5 +++++
 include/linux/kvm_host.h          | 2 ++
 virt/kvm/arm/vgic/vgic-irqfd.c    | 8 ++++++++
 virt/kvm/irqchip.c                | 2 +-
 4 files changed, 16 insertions(+), 1 deletion(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index 3bb60d3..60d4999 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -2375,6 +2375,11 @@ On arm/arm64, gsi routing being supported, the following can happen:
 - in case no routing entry is associated to this gsi, injection fails
 - in case the gsi is associated to an irqchip routing entry,
   irqchip.pin + 32 corresponds to the injected SPI ID.
+- in case the gsi is associated to an MSI routing entry,
+  * without GICv3 ITS in-kernel emulation, MSI data matches the SPI ID
+    of the injected SPI
+  * with GICv3 ITS in-kernel emulation, the MSI message and device ID
+    are translated into an LPI.
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index fffa299..78d62ee 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -1038,6 +1038,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
 
 #ifdef CONFIG_S390
 #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
+#elif defined(CONFIG_ARM64)
+#define KVM_MAX_IRQ_ROUTES 4096
 #else
 #define KVM_MAX_IRQ_ROUTES 1024
 #endif
diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
index b03ab4e..5d9ed39 100644
--- a/virt/kvm/arm/vgic/vgic-irqfd.c
+++ b/virt/kvm/arm/vgic/vgic-irqfd.c
@@ -62,6 +62,14 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
 		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
 			goto out;
 		break;
+	case KVM_IRQ_ROUTING_MSI:
+		e->set = kvm_set_msi;
+		e->msi.address_lo = ue->u.msi.address_lo;
+		e->msi.address_hi = ue->u.msi.address_hi;
+		e->msi.data = ue->u.msi.data;
+		e->flags = ue->flags;
+		e->devid = ue->u.msi.devid;
+		break;
 	default:
 		goto out;
 	}
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 03632e3..2ea5e50 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -211,7 +211,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
 			goto out;
 
 		r = -EINVAL;
-		if (ue->flags) {
+		if (ue->flags & ~KVM_MSI_VALID_DEVID) {
 			kfree(e);
 			goto out;
 		}
-- 
2.5.5

[RFC v6 6/6] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Eric Auger]

If the ITS modality is not available, let's simply support MSI
injection by transforming the MSI.data into an SPI ID.

This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
routing for arm too.

Signed-off-by: Eric Auger <eric.auger@redhat.com>

---

v4 -> v5:
- on vgic_v2m_inject_msi check the msi->data is within the SPI range
- move KVM_HAVE_MSI in the KVM section (to be symetrical with ARM64)

v2 -> v3:
- reword the commit message
- add sanity check about devid provision

v1 -> v2:
- introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
  advice
---
 arch/arm/kvm/Kconfig           |  1 +
 virt/kvm/arm/vgic/vgic-irqfd.c |  2 +-
 virt/kvm/arm/vgic/vgic-v2.c    | 17 +++++++++++++++++
 virt/kvm/arm/vgic/vgic.h       |  1 +
 4 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 3e1cd04..90d0176 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -34,6 +34,7 @@ config KVM
 	select HAVE_KVM_IRQFD
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_IRQ_ROUTING
+	select HAVE_KVM_MSI
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
 	---help---
 	  Support hosting virtualized guest machines.
diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
index 5d9ed39..6612209 100644
--- a/virt/kvm/arm/vgic/vgic-irqfd.c
+++ b/virt/kvm/arm/vgic/vgic-irqfd.c
@@ -98,7 +98,7 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 	msi.devid = e->devid;
 
 	if (!vgic_has_its(kvm))
-		return -ENODEV;
+		return vgic_v2m_inject_msi(kvm, &msi);
 
 	return vgic_its_inject_msi(kvm, &msi);
 }
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index 0bf6709..88e77db 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -241,6 +241,23 @@ static bool vgic_v2_check_base(gpa_t dist_base, gpa_t cpu_base)
 	return false;
 }
 
+/**
+ * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
+ * the SPI ID matching the msi data
+ *
+ * @kvm: pointer to the kvm struct
+ * @msi: the msi struct handle
+ */
+int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
+{
+	if (msi->flags & KVM_MSI_VALID_DEVID)
+		return -EINVAL;
+	if (!vgic_valid_spi(kvm, msi->data))
+		return -EINVAL;
+
+	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);
+}
+
 int vgic_v2_map_resources(struct kvm *kvm)
 {
 	struct vgic_dist *dist = &kvm->arch.vgic;
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index b3e5678..223e946 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -64,6 +64,7 @@ int vgic_v2_probe(const struct gic_kvm_info *info);
 int vgic_v2_map_resources(struct kvm *kvm);
 int vgic_register_dist_iodev(struct kvm *kvm, gpa_t dist_base_address,
 			     enum vgic_type);
+int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
 
 #ifdef CONFIG_KVM_ARM_VGIC_V3
 void vgic_v3_process_maintenance(struct kvm_vcpu *vcpu);
-- 
2.5.5

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Andrew Jones]

On Wed, Jul 06, 2016 at 10:47:53AM +0200, Eric Auger wrote:
> This patch adds compilation and link against irqchip.
> 
> Main motivation behind using irqchip code is to enable MSI
> routing code. In the future irqchip routing may also be useful
> when targeting multiple irqchips.
> 
> Routing standard callbacks now are implemented in vgic-irqfd:
> - kvm_set_routing_entry
> - kvm_set_irq
> - kvm_set_msi
> 
> They only are supported with new_vgic code.
> 
> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
> 
> So from now on IRQCHIP routing is enabled and a routing table entry
> must exist for irqfd injection to succeed for a given SPI. This patch
> builds a default flat irqchip routing table (gsi=irqchip.pin) covering
> all the VGIC SPI indexes. This routing table is overwritten by the
> first first user-space call to KVM_SET_GSI_ROUTING ioctl.
> 
> MSI routing setup is not yet allowed.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v5 -> v6:
> - rebase on top of Andre's v8 + removal of old vgic
> 
> v4 -> v5:
> - vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
> - minor comment changes
> - remove trace_kvm_set_irq since it is called in irqchip
> - remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
> - despite Christoffer's question, in kvm_set_msi, I kept the copy from
>   the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
>   irqchip callback API into the struct kvm_msi * passed to
>   vits_inject_msi. Since vits_inject_msi is directly called by
>   kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
>   to me to keep the copy.
> - squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
>   table into that patch
> - handle default routing table alloc failure
> 
> v3 -> v4:
> - provide support only for new-vgic
> - code previously in vgic.c now in vgic_irqfd.c
> 
> v2 -> v3:
> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
> - vgic_irqfd_set_irq now is static
> - propagate flags
> - add comments
> 
> v1 -> v2:
> - fix bug reported by Andre related to msi.flags and msi.devid setting
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH
> - reword api.txt:
>   x move MSI routing comments in a subsequent patch,
>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
> ---
>  Documentation/virtual/kvm/api.txt | 12 ++++--
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  2 +
>  arch/arm/kvm/Makefile             |  1 +
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  2 +
>  arch/arm64/kvm/Makefile           |  1 +
>  virt/kvm/arm/vgic/vgic-init.c     | 27 +++++++++++++
>  virt/kvm/arm/vgic/vgic-irqfd.c    | 82 ++++++++++++++++++++++++++++++---------
>  virt/kvm/arm/vgic/vgic.c          |  7 ----
>  virt/kvm/irqchip.c                |  2 +
>  11 files changed, 109 insertions(+), 30 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 0065c8e..3bb60d3 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>  4.52 KVM_SET_GSI_ROUTING
>  
>  Capability: KVM_CAP_IRQ_ROUTING
> -Architectures: x86 s390
> +Architectures: x86 s390 arm arm64
>  Type: vm ioctl
>  Parameters: struct kvm_irq_routing (in)
>  Returns: 0 on success, -1 on error
>  
>  Sets the GSI routing table entries, overwriting any previously set entries.
>  
> +On arm/arm64, GSI routing has the following limitation:
> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
> +
>  struct kvm_irq_routing {
>  	__u32 nr;
>  	__u32 flags;
> @@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>  
> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
> -given by gsi + 32.
> +On arm/arm64, gsi routing being supported, the following can happen:
> +- in case no routing entry is associated to this gsi, injection fails
> +- in case the gsi is associated to an irqchip routing entry,
> +  irqchip.pin + 32 corresponds to the injected SPI ID.
>  
>  4.76 KVM_PPC_ALLOCATE_HTAB
>  
> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> index 3c40facd..161997e 100644
> --- a/arch/arm/include/asm/kvm_host.h
> +++ b/arch/arm/include/asm/kvm_host.h
> @@ -37,6 +37,8 @@
>  
>  #define KVM_VCPU_MAX_FEATURES 2
>  
> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */

I wonder if it's time for include/linux/irqchip/arm-gic-common.h to
gain some defines like include/kvm/vgic/vgic.h has, in order to
replace all the scatterings of 1020s and 32s throughout irq-gic*.c
code. In any case, just a nite, but I'd write this define as

 #define KVM_IRQCHIP_NUM_PINS (1020 - 32) /* number of SPIs */

> +
>  #include <kvm/arm_vgic.h>
>  
>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 95a0005..3e1cd04 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -32,6 +32,8 @@ config KVM
>  	select KVM_VFIO
>  	select HAVE_KVM_EVENTFD
>  	select HAVE_KVM_IRQFD
> +	select HAVE_KVM_IRQCHIP
> +	select HAVE_KVM_IRQ_ROUTING
>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
>  	---help---
>  	  Support hosting virtualized guest machines.
> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> index 5e28df8..025d812 100644
> --- a/arch/arm/kvm/Makefile
> +++ b/arch/arm/kvm/Makefile
> @@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
>  obj-y += $(KVM)/arm/vgic/vgic-mmio.o
>  obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
>  obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
> +obj-y += $(KVM)//irqchip.o

extra '/'

>  obj-y += $(KVM)/arm/arch_timer.o
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index ebe8904..58f4a60 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -34,6 +34,7 @@
>  #define KVM_PRIVATE_MEM_SLOTS 4
>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
>  #define KVM_HALT_POLL_NS_DEFAULT 500000
> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */

same comment as above

>  
>  #include <kvm/arm_vgic.h>
>  #include <kvm/arm_arch_timer.h>
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 9d2eff0..9c9edc9 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -37,6 +37,8 @@ config KVM
>  	select KVM_ARM_VGIC_V3
>  	select KVM_ARM_PMU if HW_PERF_EVENTS
>  	select HAVE_KVM_MSI
> +	select HAVE_KVM_IRQCHIP
> +	select HAVE_KVM_IRQ_ROUTING
>  	---help---
>  	  Support hosting virtualized guest machines.
>  	  We don't support KVM with 16K page tables yet, due to the multiple
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index a5b9664..695eb3c 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
>  kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
> diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
> index 01a60dc..591571e 100644
> --- a/virt/kvm/arm/vgic/vgic-init.c
> +++ b/virt/kvm/arm/vgic/vgic-init.c
> @@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> +	ret = kvm_setup_default_irq_routing(kvm);
> +	if (ret)
> +		goto out;
> +
>  	dist->initialized = true;
>  out:
>  	return ret;
> @@ -457,3 +461,26 @@ out_free_irq:
>  			kvm_get_running_vcpus());
>  	return ret;
>  }
> +
> +int kvm_setup_default_irq_routing(struct kvm *kvm)
> +{
> +	struct kvm_irq_routing_entry *entries;
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	u32 nr = dist->nr_spis;
> +	int i, ret;
> +
> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
> +			  GFP_KERNEL);
> +	if (!entries)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < nr; i++) {
> +		entries[i].gsi = i;
> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
> +		entries[i].u.irqchip.irqchip = 0;
> +		entries[i].u.irqchip.pin = i;
> +	}
> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
> +	kfree(entries);
> +	return ret;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
> index c675513..b03ab4e 100644
> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
> @@ -17,36 +17,80 @@
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
>  #include <trace/events/kvm.h>
> +#include <kvm/arm_vgic.h>
> +#include "vgic.h"
>  
> -int kvm_irq_map_gsi(struct kvm *kvm,
> -		    struct kvm_kernel_irq_routing_entry *entries,
> -		    int gsi)
> +/**
> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
> + * irqchip routing entry
> + *
> + * This is the entry point for irqfd IRQ injection
> + */
> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
> +			struct kvm *kvm, int irq_source_id,
> +			int level, bool line_status)
>  {
> -	return 0;
> -}
> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
> +	struct vgic_dist *dist = &kvm->arch.vgic;
>  
> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
> -			 unsigned int pin)
> -{
> -	return pin;
> +	BUG_ON(!vgic_initialized(kvm));

Is it possible for userspace to trigger this by [intentionally]
doing setup out of order? If so, then we should only error here.

> +
> +	if (spi_id > min(dist->nr_spis, 1020))

Another 1020.

> +		return -EINVAL;
> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
>  }
>  
> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
> -		u32 irq, int level, bool line_status)
> +/**
> + * kvm_set_routing_entry: populate a kvm routing entry
> + * from a user routing entry
> + *
> + * @e: kvm kernel routing entry handle
> + * @ue: user api routing entry handle
> + * return 0 on success, -EINVAL on errors.
> + */
> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
> +			  const struct kvm_irq_routing_entry *ue)
>  {
> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
> +	int r = -EINVAL;
>  
> -	trace_kvm_set_irq(irq, level, irq_source_id);
> -
> -	BUG_ON(!vgic_initialized(kvm));
> -
> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
> +	switch (ue->type) {
> +	case KVM_IRQ_ROUTING_IRQCHIP:
> +		e->set = vgic_irqfd_set_irq;
> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
> +		e->irqchip.pin = ue->u.irqchip.pin;
> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
> +			goto out;
> +		break;
> +	default:
> +		goto out;
> +	}
> +	r = 0;
> +out:
> +	return r;
>  }
>  
> -/* MSI not implemented yet */
> +/**
> + * kvm_set_msi: inject the MSI corresponding to the
> + * MSI routing entry
> + *
> + * This is the entry point for irqfd MSI injection
> + * and userspace MSI injection.
> + */
>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>  		struct kvm *kvm, int irq_source_id,
>  		int level, bool line_status)
>  {
> -	return 0;
> +	struct kvm_msi msi;
> +
> +	msi.address_lo = e->msi.address_lo;
> +	msi.address_hi = e->msi.address_hi;
> +	msi.data = e->msi.data;
> +	msi.flags = e->flags;
> +	msi.devid = e->devid;
> +
> +	if (!vgic_has_its(kvm))
> +		return -ENODEV;
> +
> +	return vgic_its_inject_msi(kvm, &msi);
>  }
> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
> index c4f3aba..b254833 100644
> --- a/virt/kvm/arm/vgic/vgic.c
> +++ b/virt/kvm/arm/vgic/vgic.c
> @@ -684,10 +684,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
>  	return map_is_active;
>  }
>  
> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
> -{
> -	if (vgic_has_its(kvm))
> -		return vgic_its_inject_msi(kvm, msi);
> -	else
> -		return -ENODEV;
> -}
> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> index 32e5646..03632e3 100644
> --- a/virt/kvm/irqchip.c
> +++ b/virt/kvm/irqchip.c
> @@ -29,7 +29,9 @@
>  #include <linux/srcu.h>
>  #include <linux/export.h>
>  #include <trace/events/kvm.h>
> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>  #include "irq.h"
> +#endif

Instead of doing this, shouldn't we add arch/arm[64]/kvm/irq.h files.
Probably a simple one like ./arch/s390/kvm/irq.h ?

Thanks,
drew

>  
>  int kvm_irq_map_gsi(struct kvm *kvm,
>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
> -- 
> 2.5.5
> 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Auger Eric]

Hi Drew,

On 07/07/2016 19:20, Andrew Jones wrote:
> On Wed, Jul 06, 2016 at 10:47:53AM +0200, Eric Auger wrote:
>> This patch adds compilation and link against irqchip.
>>
>> Main motivation behind using irqchip code is to enable MSI
>> routing code. In the future irqchip routing may also be useful
>> when targeting multiple irqchips.
>>
>> Routing standard callbacks now are implemented in vgic-irqfd:
>> - kvm_set_routing_entry
>> - kvm_set_irq
>> - kvm_set_msi
>>
>> They only are supported with new_vgic code.
>>
>> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
>> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
>>
>> So from now on IRQCHIP routing is enabled and a routing table entry
>> must exist for irqfd injection to succeed for a given SPI. This patch
>> builds a default flat irqchip routing table (gsi=irqchip.pin) covering
>> all the VGIC SPI indexes. This routing table is overwritten by the
>> first first user-space call to KVM_SET_GSI_ROUTING ioctl.
>>
>> MSI routing setup is not yet allowed.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v5 -> v6:
>> - rebase on top of Andre's v8 + removal of old vgic
>>
>> v4 -> v5:
>> - vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
>> - minor comment changes
>> - remove trace_kvm_set_irq since it is called in irqchip
>> - remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
>> - despite Christoffer's question, in kvm_set_msi, I kept the copy from
>>   the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
>>   irqchip callback API into the struct kvm_msi * passed to
>>   vits_inject_msi. Since vits_inject_msi is directly called by
>>   kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
>>   to me to keep the copy.
>> - squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
>>   table into that patch
>> - handle default routing table alloc failure
>>
>> v3 -> v4:
>> - provide support only for new-vgic
>> - code previously in vgic.c now in vgic_irqfd.c
>>
>> v2 -> v3:
>> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
>>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
>> - vgic_irqfd_set_irq now is static
>> - propagate flags
>> - add comments
>>
>> v1 -> v2:
>> - fix bug reported by Andre related to msi.flags and msi.devid setting
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH
>> - reword api.txt:
>>   x move MSI routing comments in a subsequent patch,
>>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
>> ---
>>  Documentation/virtual/kvm/api.txt | 12 ++++--
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  2 +
>>  arch/arm/kvm/Makefile             |  1 +
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  2 +
>>  arch/arm64/kvm/Makefile           |  1 +
>>  virt/kvm/arm/vgic/vgic-init.c     | 27 +++++++++++++
>>  virt/kvm/arm/vgic/vgic-irqfd.c    | 82 ++++++++++++++++++++++++++++++---------
>>  virt/kvm/arm/vgic/vgic.c          |  7 ----
>>  virt/kvm/irqchip.c                |  2 +
>>  11 files changed, 109 insertions(+), 30 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 0065c8e..3bb60d3 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>>  4.52 KVM_SET_GSI_ROUTING
>>  
>>  Capability: KVM_CAP_IRQ_ROUTING
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm arm64
>>  Type: vm ioctl
>>  Parameters: struct kvm_irq_routing (in)
>>  Returns: 0 on success, -1 on error
>>  
>>  Sets the GSI routing table entries, overwriting any previously set entries.
>>  
>> +On arm/arm64, GSI routing has the following limitation:
>> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
>> +
>>  struct kvm_irq_routing {
>>  	__u32 nr;
>>  	__u32 flags;
>> @@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
>>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>  
>> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
>> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
>> -given by gsi + 32.
>> +On arm/arm64, gsi routing being supported, the following can happen:
>> +- in case no routing entry is associated to this gsi, injection fails
>> +- in case the gsi is associated to an irqchip routing entry,
>> +  irqchip.pin + 32 corresponds to the injected SPI ID.
>>  
>>  4.76 KVM_PPC_ALLOCATE_HTAB
>>  
>> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
>> index 3c40facd..161997e 100644
>> --- a/arch/arm/include/asm/kvm_host.h
>> +++ b/arch/arm/include/asm/kvm_host.h
>> @@ -37,6 +37,8 @@
>>  
>>  #define KVM_VCPU_MAX_FEATURES 2
>>  
>> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
> 
> I wonder if it's time for include/linux/irqchip/arm-gic-common.h to
> gain some defines like include/kvm/vgic/vgic.h has, in order to
> replace all the scatterings of 1020s and 32s throughout irq-gic*.c
> code. In any case, just a nite, but I'd write this define as

Marc, any opinion on this?
> 
>  #define KVM_IRQCHIP_NUM_PINS (1020 - 32) /* number of SPIs */

sure
> 
>> +
>>  #include <kvm/arm_vgic.h>
>>  
>>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 95a0005..3e1cd04 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -32,6 +32,8 @@ config KVM
>>  	select KVM_VFIO
>>  	select HAVE_KVM_EVENTFD
>>  	select HAVE_KVM_IRQFD
>> +	select HAVE_KVM_IRQCHIP
>> +	select HAVE_KVM_IRQ_ROUTING
>>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
>>  	---help---
>>  	  Support hosting virtualized guest machines.
>> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
>> index 5e28df8..025d812 100644
>> --- a/arch/arm/kvm/Makefile
>> +++ b/arch/arm/kvm/Makefile
>> @@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
>>  obj-y += $(KVM)/arm/vgic/vgic-mmio.o
>>  obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
>>  obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
>> +obj-y += $(KVM)//irqchip.o
> 
> extra '/'
ok thanks
> 
>>  obj-y += $(KVM)/arm/arch_timer.o
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>> index ebe8904..58f4a60 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
>> @@ -34,6 +34,7 @@
>>  #define KVM_PRIVATE_MEM_SLOTS 4
>>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
>>  #define KVM_HALT_POLL_NS_DEFAULT 500000
>> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
> 
> same comment as above
yep
> 
>>  
>>  #include <kvm/arm_vgic.h>
>>  #include <kvm/arm_arch_timer.h>
>> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
>> index 9d2eff0..9c9edc9 100644
>> --- a/arch/arm64/kvm/Kconfig
>> +++ b/arch/arm64/kvm/Kconfig
>> @@ -37,6 +37,8 @@ config KVM
>>  	select KVM_ARM_VGIC_V3
>>  	select KVM_ARM_PMU if HW_PERF_EVENTS
>>  	select HAVE_KVM_MSI
>> +	select HAVE_KVM_IRQCHIP
>> +	select HAVE_KVM_IRQ_ROUTING
>>  	---help---
>>  	  Support hosting virtualized guest machines.
>>  	  We don't support KVM with 16K page tables yet, due to the multiple
>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>> index a5b9664..695eb3c 100644
>> --- a/arch/arm64/kvm/Makefile
>> +++ b/arch/arm64/kvm/Makefile
>> @@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
>>  kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
>> diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
>> index 01a60dc..591571e 100644
>> --- a/virt/kvm/arm/vgic/vgic-init.c
>> +++ b/virt/kvm/arm/vgic/vgic-init.c
>> @@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
>>  	kvm_for_each_vcpu(i, vcpu, kvm)
>>  		kvm_vgic_vcpu_init(vcpu);
>>  
>> +	ret = kvm_setup_default_irq_routing(kvm);
>> +	if (ret)
>> +		goto out;
>> +
>>  	dist->initialized = true;
>>  out:
>>  	return ret;
>> @@ -457,3 +461,26 @@ out_free_irq:
>>  			kvm_get_running_vcpus());
>>  	return ret;
>>  }
>> +
>> +int kvm_setup_default_irq_routing(struct kvm *kvm)
>> +{
>> +	struct kvm_irq_routing_entry *entries;
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	u32 nr = dist->nr_spis;
>> +	int i, ret;
>> +
>> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
>> +			  GFP_KERNEL);
>> +	if (!entries)
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < nr; i++) {
>> +		entries[i].gsi = i;
>> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
>> +		entries[i].u.irqchip.irqchip = 0;
>> +		entries[i].u.irqchip.pin = i;
>> +	}
>> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
>> +	kfree(entries);
>> +	return ret;
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
>> index c675513..b03ab4e 100644
>> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
>> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
>> @@ -17,36 +17,80 @@
>>  #include <linux/kvm.h>
>>  #include <linux/kvm_host.h>
>>  #include <trace/events/kvm.h>
>> +#include <kvm/arm_vgic.h>
>> +#include "vgic.h"
>>  
>> -int kvm_irq_map_gsi(struct kvm *kvm,
>> -		    struct kvm_kernel_irq_routing_entry *entries,
>> -		    int gsi)
>> +/**
>> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
>> + * irqchip routing entry
>> + *
>> + * This is the entry point for irqfd IRQ injection
>> + */
>> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
>> +			struct kvm *kvm, int irq_source_id,
>> +			int level, bool line_status)
>>  {
>> -	return 0;
>> -}
>> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>  
>> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
>> -			 unsigned int pin)
>> -{
>> -	return pin;
>> +	BUG_ON(!vgic_initialized(kvm));
> 
> Is it possible for userspace to trigger this by [intentionally]
> doing setup out of order? If so, then we should only error here.
kvm_irq_map_chip_pin is called from kvm_irq_has_notifier and
kvm_notify_acked_irq. On ARM we only use the latter. This is basically
used to retrieved the gsi associated with a physical (irqchip/pin) and
eventually signal the resamplefd associated to this gsi, if any.
kvm_notify_acked_irq is called from *process_maintenance meaning a
level-sensitive vIRQ was deactivated. So at that point the vGIC is
initialized since a virtual IRQ was already injected. So I think it is
even safe to remove the check.

> 
>> +
>> +	if (spi_id > min(dist->nr_spis, 1020))
> 
> Another 1020.
ok
> 
>> +		return -EINVAL;
>> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
>>  }
>>  
>> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
>> -		u32 irq, int level, bool line_status)
>> +/**
>> + * kvm_set_routing_entry: populate a kvm routing entry
>> + * from a user routing entry
>> + *
>> + * @e: kvm kernel routing entry handle
>> + * @ue: user api routing entry handle
>> + * return 0 on success, -EINVAL on errors.
>> + */
>> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
>> +			  const struct kvm_irq_routing_entry *ue)
>>  {
>> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
>> +	int r = -EINVAL;
>>  
>> -	trace_kvm_set_irq(irq, level, irq_source_id);
>> -
>> -	BUG_ON(!vgic_initialized(kvm));
>> -
>> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
>> +	switch (ue->type) {
>> +	case KVM_IRQ_ROUTING_IRQCHIP:
>> +		e->set = vgic_irqfd_set_irq;
>> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
>> +		e->irqchip.pin = ue->u.irqchip.pin;
>> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
>> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
>> +			goto out;
>> +		break;
>> +	default:
>> +		goto out;
>> +	}
>> +	r = 0;
>> +out:
>> +	return r;
>>  }
>>  
>> -/* MSI not implemented yet */
>> +/**
>> + * kvm_set_msi: inject the MSI corresponding to the
>> + * MSI routing entry
>> + *
>> + * This is the entry point for irqfd MSI injection
>> + * and userspace MSI injection.
>> + */
>>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>>  		struct kvm *kvm, int irq_source_id,
>>  		int level, bool line_status)
>>  {
>> -	return 0;
>> +	struct kvm_msi msi;
>> +
>> +	msi.address_lo = e->msi.address_lo;
>> +	msi.address_hi = e->msi.address_hi;
>> +	msi.data = e->msi.data;
>> +	msi.flags = e->flags;
>> +	msi.devid = e->devid;
>> +
>> +	if (!vgic_has_its(kvm))
>> +		return -ENODEV;
>> +
>> +	return vgic_its_inject_msi(kvm, &msi);
>>  }
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index c4f3aba..b254833 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -684,10 +684,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
>>  	return map_is_active;
>>  }
>>  
>> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
>> -{
>> -	if (vgic_has_its(kvm))
>> -		return vgic_its_inject_msi(kvm, msi);
>> -	else
>> -		return -ENODEV;
>> -}
>> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
>> index 32e5646..03632e3 100644
>> --- a/virt/kvm/irqchip.c
>> +++ b/virt/kvm/irqchip.c
>> @@ -29,7 +29,9 @@
>>  #include <linux/srcu.h>
>>  #include <linux/export.h>
>>  #include <trace/events/kvm.h>
>> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>>  #include "irq.h"
>> +#endif
> 
> Instead of doing this, shouldn't we add arch/arm[64]/kvm/irq.h files.
> Probably a simple one like ./arch/s390/kvm/irq.h ?

Well I considered this solution in the past but I did not find much to
put there (it was even void). typically irqchip_in_kernel is in
include/kvm/arm_vgic.h since the macro can be shared between arm/arm64.

Thank you for your time!

Eric
> 
> Thanks,
> drew
> 
>>  
>>  int kvm_irq_map_gsi(struct kvm *kvm,
>>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
>> -- 
>> 2.5.5
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe kvm" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Andrew Jones]

On Fri, Jul 08, 2016 at 10:16:53AM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 07/07/2016 19:20, Andrew Jones wrote:
> > On Wed, Jul 06, 2016 at 10:47:53AM +0200, Eric Auger wrote:
> >> This patch adds compilation and link against irqchip.
> >>
> >> Main motivation behind using irqchip code is to enable MSI
> >> routing code. In the future irqchip routing may also be useful
> >> when targeting multiple irqchips.
> >>
> >> Routing standard callbacks now are implemented in vgic-irqfd:
> >> - kvm_set_routing_entry
> >> - kvm_set_irq
> >> - kvm_set_msi
> >>
> >> They only are supported with new_vgic code.
> >>
> >> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
> >> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
> >>
> >> So from now on IRQCHIP routing is enabled and a routing table entry
> >> must exist for irqfd injection to succeed for a given SPI. This patch
> >> builds a default flat irqchip routing table (gsi=irqchip.pin) covering
> >> all the VGIC SPI indexes. This routing table is overwritten by the
> >> first first user-space call to KVM_SET_GSI_ROUTING ioctl.
> >>
> >> MSI routing setup is not yet allowed.
> >>
> >> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> >>
> >> ---
> >> v5 -> v6:
> >> - rebase on top of Andre's v8 + removal of old vgic
> >>
> >> v4 -> v5:
> >> - vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
> >> - minor comment changes
> >> - remove trace_kvm_set_irq since it is called in irqchip
> >> - remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
> >> - despite Christoffer's question, in kvm_set_msi, I kept the copy from
> >>   the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
> >>   irqchip callback API into the struct kvm_msi * passed to
> >>   vits_inject_msi. Since vits_inject_msi is directly called by
> >>   kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
> >>   to me to keep the copy.
> >> - squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
> >>   table into that patch
> >> - handle default routing table alloc failure
> >>
> >> v3 -> v4:
> >> - provide support only for new-vgic
> >> - code previously in vgic.c now in vgic_irqfd.c
> >>
> >> v2 -> v3:
> >> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
> >>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
> >> - vgic_irqfd_set_irq now is static
> >> - propagate flags
> >> - add comments
> >>
> >> v1 -> v2:
> >> - fix bug reported by Andre related to msi.flags and msi.devid setting
> >>   in kvm_send_userspace_msi
> >> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> >>
> >> RFC -> PATCH
> >> - reword api.txt:
> >>   x move MSI routing comments in a subsequent patch,
> >>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
> >> ---
> >>  Documentation/virtual/kvm/api.txt | 12 ++++--
> >>  arch/arm/include/asm/kvm_host.h   |  2 +
> >>  arch/arm/kvm/Kconfig              |  2 +
> >>  arch/arm/kvm/Makefile             |  1 +
> >>  arch/arm64/include/asm/kvm_host.h |  1 +
> >>  arch/arm64/kvm/Kconfig            |  2 +
> >>  arch/arm64/kvm/Makefile           |  1 +
> >>  virt/kvm/arm/vgic/vgic-init.c     | 27 +++++++++++++
> >>  virt/kvm/arm/vgic/vgic-irqfd.c    | 82 ++++++++++++++++++++++++++++++---------
> >>  virt/kvm/arm/vgic/vgic.c          |  7 ----
> >>  virt/kvm/irqchip.c                |  2 +
> >>  11 files changed, 109 insertions(+), 30 deletions(-)
> >>
> >> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> >> index 0065c8e..3bb60d3 100644
> >> --- a/Documentation/virtual/kvm/api.txt
> >> +++ b/Documentation/virtual/kvm/api.txt
> >> @@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
> >>  4.52 KVM_SET_GSI_ROUTING
> >>  
> >>  Capability: KVM_CAP_IRQ_ROUTING
> >> -Architectures: x86 s390
> >> +Architectures: x86 s390 arm arm64
> >>  Type: vm ioctl
> >>  Parameters: struct kvm_irq_routing (in)
> >>  Returns: 0 on success, -1 on error
> >>  
> >>  Sets the GSI routing table entries, overwriting any previously set entries.
> >>  
> >> +On arm/arm64, GSI routing has the following limitation:
> >> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
> >> +
> >>  struct kvm_irq_routing {
> >>  	__u32 nr;
> >>  	__u32 flags;
> >> @@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
> >>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
> >>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
> >>  
> >> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
> >> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
> >> -given by gsi + 32.
> >> +On arm/arm64, gsi routing being supported, the following can happen:
> >> +- in case no routing entry is associated to this gsi, injection fails
> >> +- in case the gsi is associated to an irqchip routing entry,
> >> +  irqchip.pin + 32 corresponds to the injected SPI ID.
> >>  
> >>  4.76 KVM_PPC_ALLOCATE_HTAB
> >>  
> >> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> >> index 3c40facd..161997e 100644
> >> --- a/arch/arm/include/asm/kvm_host.h
> >> +++ b/arch/arm/include/asm/kvm_host.h
> >> @@ -37,6 +37,8 @@
> >>  
> >>  #define KVM_VCPU_MAX_FEATURES 2
> >>  
> >> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
> > 
> > I wonder if it's time for include/linux/irqchip/arm-gic-common.h to
> > gain some defines like include/kvm/vgic/vgic.h has, in order to
> > replace all the scatterings of 1020s and 32s throughout irq-gic*.c
> > code. In any case, just a nite, but I'd write this define as
> 
> Marc, any opinion on this?
> > 
> >  #define KVM_IRQCHIP_NUM_PINS (1020 - 32) /* number of SPIs */
> 
> sure
> > 
> >> +
> >>  #include <kvm/arm_vgic.h>
> >>  
> >>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
> >> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> >> index 95a0005..3e1cd04 100644
> >> --- a/arch/arm/kvm/Kconfig
> >> +++ b/arch/arm/kvm/Kconfig
> >> @@ -32,6 +32,8 @@ config KVM
> >>  	select KVM_VFIO
> >>  	select HAVE_KVM_EVENTFD
> >>  	select HAVE_KVM_IRQFD
> >> +	select HAVE_KVM_IRQCHIP
> >> +	select HAVE_KVM_IRQ_ROUTING
> >>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
> >>  	---help---
> >>  	  Support hosting virtualized guest machines.
> >> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> >> index 5e28df8..025d812 100644
> >> --- a/arch/arm/kvm/Makefile
> >> +++ b/arch/arm/kvm/Makefile
> >> @@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
> >>  obj-y += $(KVM)/arm/vgic/vgic-mmio.o
> >>  obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
> >>  obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
> >> +obj-y += $(KVM)//irqchip.o
> > 
> > extra '/'
> ok thanks
> > 
> >>  obj-y += $(KVM)/arm/arch_timer.o
> >> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> >> index ebe8904..58f4a60 100644
> >> --- a/arch/arm64/include/asm/kvm_host.h
> >> +++ b/arch/arm64/include/asm/kvm_host.h
> >> @@ -34,6 +34,7 @@
> >>  #define KVM_PRIVATE_MEM_SLOTS 4
> >>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
> >>  #define KVM_HALT_POLL_NS_DEFAULT 500000
> >> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
> > 
> > same comment as above
> yep
> > 
> >>  
> >>  #include <kvm/arm_vgic.h>
> >>  #include <kvm/arm_arch_timer.h>
> >> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> >> index 9d2eff0..9c9edc9 100644
> >> --- a/arch/arm64/kvm/Kconfig
> >> +++ b/arch/arm64/kvm/Kconfig
> >> @@ -37,6 +37,8 @@ config KVM
> >>  	select KVM_ARM_VGIC_V3
> >>  	select KVM_ARM_PMU if HW_PERF_EVENTS
> >>  	select HAVE_KVM_MSI
> >> +	select HAVE_KVM_IRQCHIP
> >> +	select HAVE_KVM_IRQ_ROUTING
> >>  	---help---
> >>  	  Support hosting virtualized guest machines.
> >>  	  We don't support KVM with 16K page tables yet, due to the multiple
> >> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> >> index a5b9664..695eb3c 100644
> >> --- a/arch/arm64/kvm/Makefile
> >> +++ b/arch/arm64/kvm/Makefile
> >> @@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
> >>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
> >>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
> >>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
> >> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
> >>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
> >>  kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
> >> diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
> >> index 01a60dc..591571e 100644
> >> --- a/virt/kvm/arm/vgic/vgic-init.c
> >> +++ b/virt/kvm/arm/vgic/vgic-init.c
> >> @@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
> >>  	kvm_for_each_vcpu(i, vcpu, kvm)
> >>  		kvm_vgic_vcpu_init(vcpu);
> >>  
> >> +	ret = kvm_setup_default_irq_routing(kvm);
> >> +	if (ret)
> >> +		goto out;
> >> +
> >>  	dist->initialized = true;
> >>  out:
> >>  	return ret;
> >> @@ -457,3 +461,26 @@ out_free_irq:
> >>  			kvm_get_running_vcpus());
> >>  	return ret;
> >>  }
> >> +
> >> +int kvm_setup_default_irq_routing(struct kvm *kvm)
> >> +{
> >> +	struct kvm_irq_routing_entry *entries;
> >> +	struct vgic_dist *dist = &kvm->arch.vgic;
> >> +	u32 nr = dist->nr_spis;
> >> +	int i, ret;
> >> +
> >> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
> >> +			  GFP_KERNEL);
> >> +	if (!entries)
> >> +		return -ENOMEM;
> >> +
> >> +	for (i = 0; i < nr; i++) {
> >> +		entries[i].gsi = i;
> >> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
> >> +		entries[i].u.irqchip.irqchip = 0;
> >> +		entries[i].u.irqchip.pin = i;
> >> +	}
> >> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
> >> +	kfree(entries);
> >> +	return ret;
> >> +}
> >> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
> >> index c675513..b03ab4e 100644
> >> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
> >> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
> >> @@ -17,36 +17,80 @@
> >>  #include <linux/kvm.h>
> >>  #include <linux/kvm_host.h>
> >>  #include <trace/events/kvm.h>
> >> +#include <kvm/arm_vgic.h>
> >> +#include "vgic.h"
> >>  
> >> -int kvm_irq_map_gsi(struct kvm *kvm,
> >> -		    struct kvm_kernel_irq_routing_entry *entries,
> >> -		    int gsi)
> >> +/**
> >> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
> >> + * irqchip routing entry
> >> + *
> >> + * This is the entry point for irqfd IRQ injection
> >> + */
> >> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
> >> +			struct kvm *kvm, int irq_source_id,
> >> +			int level, bool line_status)
> >>  {
> >> -	return 0;
> >> -}
> >> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
> >> +	struct vgic_dist *dist = &kvm->arch.vgic;
> >>  
> >> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
> >> -			 unsigned int pin)
> >> -{
> >> -	return pin;
> >> +	BUG_ON(!vgic_initialized(kvm));
> > 
> > Is it possible for userspace to trigger this by [intentionally]
> > doing setup out of order? If so, then we should only error here.
> kvm_irq_map_chip_pin is called from kvm_irq_has_notifier and
> kvm_notify_acked_irq. On ARM we only use the latter. This is basically
> used to retrieved the gsi associated with a physical (irqchip/pin) and
> eventually signal the resamplefd associated to this gsi, if any.
> kvm_notify_acked_irq is called from *process_maintenance meaning a
> level-sensitive vIRQ was deactivated. So at that point the vGIC is
> initialized since a virtual IRQ was already injected. So I think it is
> even safe to remove the check.

I vote we remove it then.

> 
> > 
> >> +
> >> +	if (spi_id > min(dist->nr_spis, 1020))
> > 
> > Another 1020.
> ok
> > 
> >> +		return -EINVAL;
> >> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
> >>  }
> >>  
> >> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
> >> -		u32 irq, int level, bool line_status)
> >> +/**
> >> + * kvm_set_routing_entry: populate a kvm routing entry
> >> + * from a user routing entry
> >> + *
> >> + * @e: kvm kernel routing entry handle
> >> + * @ue: user api routing entry handle
> >> + * return 0 on success, -EINVAL on errors.
> >> + */
> >> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
> >> +			  const struct kvm_irq_routing_entry *ue)
> >>  {
> >> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
> >> +	int r = -EINVAL;
> >>  
> >> -	trace_kvm_set_irq(irq, level, irq_source_id);
> >> -
> >> -	BUG_ON(!vgic_initialized(kvm));
> >> -
> >> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
> >> +	switch (ue->type) {
> >> +	case KVM_IRQ_ROUTING_IRQCHIP:
> >> +		e->set = vgic_irqfd_set_irq;
> >> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
> >> +		e->irqchip.pin = ue->u.irqchip.pin;
> >> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
> >> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
> >> +			goto out;
> >> +		break;
> >> +	default:
> >> +		goto out;
> >> +	}
> >> +	r = 0;
> >> +out:
> >> +	return r;
> >>  }
> >>  
> >> -/* MSI not implemented yet */
> >> +/**
> >> + * kvm_set_msi: inject the MSI corresponding to the
> >> + * MSI routing entry
> >> + *
> >> + * This is the entry point for irqfd MSI injection
> >> + * and userspace MSI injection.
> >> + */
> >>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
> >>  		struct kvm *kvm, int irq_source_id,
> >>  		int level, bool line_status)
> >>  {
> >> -	return 0;
> >> +	struct kvm_msi msi;
> >> +
> >> +	msi.address_lo = e->msi.address_lo;
> >> +	msi.address_hi = e->msi.address_hi;
> >> +	msi.data = e->msi.data;
> >> +	msi.flags = e->flags;
> >> +	msi.devid = e->devid;
> >> +
> >> +	if (!vgic_has_its(kvm))
> >> +		return -ENODEV;
> >> +
> >> +	return vgic_its_inject_msi(kvm, &msi);
> >>  }
> >> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
> >> index c4f3aba..b254833 100644
> >> --- a/virt/kvm/arm/vgic/vgic.c
> >> +++ b/virt/kvm/arm/vgic/vgic.c
> >> @@ -684,10 +684,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
> >>  	return map_is_active;
> >>  }
> >>  
> >> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
> >> -{
> >> -	if (vgic_has_its(kvm))
> >> -		return vgic_its_inject_msi(kvm, msi);
> >> -	else
> >> -		return -ENODEV;
> >> -}
> >> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> >> index 32e5646..03632e3 100644
> >> --- a/virt/kvm/irqchip.c
> >> +++ b/virt/kvm/irqchip.c
> >> @@ -29,7 +29,9 @@
> >>  #include <linux/srcu.h>
> >>  #include <linux/export.h>
> >>  #include <trace/events/kvm.h>
> >> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
> >>  #include "irq.h"
> >> +#endif
> > 
> > Instead of doing this, shouldn't we add arch/arm[64]/kvm/irq.h files.
> > Probably a simple one like ./arch/s390/kvm/irq.h ?
> 
> Well I considered this solution in the past but I did not find much to
> put there (it was even void). typically irqchip_in_kernel is in
> include/kvm/arm_vgic.h since the macro can be shared between arm/arm64.

I think I'd prefer a nearly empty file to the #ifdef's, but Paolo and
Radim should chime in.

Thanks,
drew

> 
> Thank you for your time!
> 
> Eric
> > 
> > Thanks,
> > drew
> > 
> >>  
> >>  int kvm_irq_map_gsi(struct kvm *kvm,
> >>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
> >> -- 
> >> 2.5.5
> >>
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe kvm" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > --
> > To unsubscribe from this list: send the line "unsubscribe kvm" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Radim Krčmář]

2016-07-08 10:52+0200, Andrew Jones:
> On Fri, Jul 08, 2016 at 10:16:53AM +0200, Auger Eric wrote:
>> On 07/07/2016 19:20, Andrew Jones wrote:
>> > On Wed, Jul 06, 2016 at 10:47:53AM +0200, Eric Auger wrote:
>> >> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
>> >> @@ -29,7 +29,9 @@
>> >>  #include <linux/srcu.h>
>> >>  #include <linux/export.h>
>> >>  #include <trace/events/kvm.h>
>> >> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>> >>  #include "irq.h"
>> >> +#endif
>> > 
>> > Instead of doing this, shouldn't we add arch/arm[64]/kvm/irq.h files.
>> > Probably a simple one like ./arch/s390/kvm/irq.h ?
>> 
>> Well I considered this solution in the past but I did not find much to
>> put there (it was even void). typically irqchip_in_kernel is in
>> include/kvm/arm_vgic.h since the macro can be shared between arm/arm64.
> 
> I think I'd prefer a nearly empty file to the #ifdef's, but Paolo and
> Radim should chime in.

I concur, hiding ugliness in header files is what we strive for.

The files could #include <include/kvm/arm_vgic.h>, which might make
their existence easier to understand.

Re: [RFC v6 1/6] KVM: api: pass the devid in the msi routing entry

[Andre Przywara]

Hi Eric,

On 06/07/16 09:47, Eric Auger wrote:
> On ARM, the MSI msg (address and data) comes along with
> out-of-band device ID information. The device ID encodes the
> device that writes the MSI msg. Let's convey the device id in
> kvm_irq_routing_msi and use KVM_MSI_VALID_DEVID flag value in
> kvm_irq_routing_entry to indicate the msi devid is populated.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Reviewed-by: Andre Przywara <andre.przywara@arm.com>

Cheers,
Andre.

> 
> ---
> v4 -> v5:
> - some rephrasing in api.txt according to Christoffer's comments
> v2 -> v3:
> - replace usage of KVM_IRQ_ROUTING_EXTENDED_MSI type by
>   usage of KVM_MSI_VALID_DEVID flag
> - add note about KVM_CAP_MSI_DEVID capability
> 
> v1 -> v2:
> - devid id passed in kvm_irq_routing_msi instead of in
>   kvm_irq_routing_entry
> 
> RFC -> PATCH
> - remove kvm_irq_routing_extended_msi and use union instead
> ---
>  Documentation/virtual/kvm/api.txt | 19 +++++++++++++++++--
>  include/uapi/linux/kvm.h          |  5 ++++-
>  2 files changed, 21 insertions(+), 3 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index f60b137..0065c8e 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1468,7 +1468,11 @@ struct kvm_irq_routing_entry {
>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
>  #define KVM_IRQ_ROUTING_HV_SINT 4
>  
> -No flags are specified so far, the corresponding field must be set to zero.
> +flags:
> +- KVM_MSI_VALID_DEVID: used along with KVM_IRQ_ROUTING_MSI
> +  routing entry type, specifies that the devid field contains
> +  a valid value.
> +- zero otherwise
>  
>  struct kvm_irq_routing_irqchip {
>  	__u32 irqchip;
> @@ -1479,9 +1483,20 @@ struct kvm_irq_routing_msi {
>  	__u32 address_lo;
>  	__u32 address_hi;
>  	__u32 data;
> -	__u32 pad;
> +	union {
> +		__u32 pad;
> +		__u32 devid;
> +	};
>  };
>  
> +devid: If KVM_MSI_VALID_DEVID is set, contains a unique device identifier
> +       for the device that wrote the MSI message.
> +       For PCI, this is usually a BFD identifier in the lower 16 bits.
> +
> +The per-VM KVM_CAP_MSI_DEVID capability advertises the requirement to
> +provide the device ID. If this capability is not set, userland cannot
> +rely on the kernel to allow the KVM_MSI_VALID_DEVID flag being set.
> +
>  struct kvm_irq_routing_s390_adapter {
>  	__u64 ind_addr;
>  	__u64 summary_addr;
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index d8c4c32..eb22208 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -879,7 +879,10 @@ struct kvm_irq_routing_msi {
>  	__u32 address_lo;
>  	__u32 address_hi;
>  	__u32 data;
> -	__u32 pad;
> +	union {
> +		__u32 pad;
> +		__u32 devid;
> +	};
>  };
>  
>  struct kvm_irq_routing_s390_adapter {
> 

Re: [RFC v6 2/6] KVM: kvm_host: add devid in kvm_kernel_irq_routing_entry

[Andre Przywara]

Hi,

On 06/07/16 09:47, Eric Auger wrote:
> Extend kvm_kernel_irq_routing_entry to transport the device id
> field, devid. A new flags field makes possible to indicate the
> devid is valid. Those additions are used for ARM GICv3 ITS MSI
> injection.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
> 
> ---
> v4 -> v5:
> - add Christoffer's R-b
> 
> v2 -> v3:
> - add flags
> 
> v1 -> v2:
> - replace msi_msg field by a struct composed of msi_msg and devid
> 
> RFC -> PATCH:
> - reword the commit message after change in first patch (uapi)
> ---
>  include/linux/kvm_host.h | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index c87fe6f..fffa299 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -317,7 +317,11 @@ struct kvm_kernel_irq_routing_entry {
>  			unsigned irqchip;
>  			unsigned pin;
>  		} irqchip;
> -		struct msi_msg msi;
> +		struct {
> +			struct msi_msg msi;
> +			u32 flags;
> +			u32 devid;

I wonder if we should rename those two members to have an "msi_" prefix.
Otherwise they turn into generic "route.flags" and "route.devid" (as
seen in the next patch), which is a bit misleading.

Cheers,
Andre.

> +		};
>  		struct kvm_s390_adapter_int adapter;
>  		struct kvm_hv_sint hv_sint;
>  	};
> 

Re: [RFC v6 3/6] KVM: irqchip: convey devid to kvm_set_msi

[Andre Przywara]

Hi,

On 06/07/16 09:47, Eric Auger wrote:
> on ARM, a devid field is populated in kvm_msi struct in case the
> flag is set to KVM_MSI_VALID_DEVID. Let's propagate both flags and
> devid field in kvm_kernel_irq_routing_entry.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> Acked-by: Christoffer Dall <christoffer.dall@linaro.org>

So apart from the renaming part I mentioned in the previous patch:
Reviewed-by: Andre Przywara <andre.przywara@arm.com>

> 
> ---
> v4 -> v5:
> - Add Christoffer's R-b
> 
> v2 -> v3:
> - do not set the type to KVM_IRQ_ROUTING_EXTENDED_MSI anymore as
>   suggested by Andre
> - correct msi->flags check
> - propagate the flags
> ---
>  virt/kvm/irqchip.c | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> index 8db197b..32e5646 100644
> --- a/virt/kvm/irqchip.c
> +++ b/virt/kvm/irqchip.c
> @@ -62,12 +62,14 @@ int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
>  {
>  	struct kvm_kernel_irq_routing_entry route;
>  
> -	if (!irqchip_in_kernel(kvm) || msi->flags != 0)
> +	if (!irqchip_in_kernel(kvm) || (msi->flags & ~KVM_MSI_VALID_DEVID))
>  		return -EINVAL;
>  
>  	route.msi.address_lo = msi->address_lo;
>  	route.msi.address_hi = msi->address_hi;
>  	route.msi.data = msi->data;
> +	route.flags = msi->flags;
> +	route.devid = msi->devid;
>  
>  	return kvm_set_msi(&route, kvm, KVM_USERSPACE_IRQ_SOURCE_ID, 1, false);
>  }
> 

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Andre Przywara]

Hi Eric,

On 06/07/16 09:47, Eric Auger wrote:
> This patch adds compilation and link against irqchip.
> 
> Main motivation behind using irqchip code is to enable MSI
> routing code. In the future irqchip routing may also be useful
> when targeting multiple irqchips.
> 
> Routing standard callbacks now are implemented in vgic-irqfd:
> - kvm_set_routing_entry
> - kvm_set_irq
> - kvm_set_msi
> 
> They only are supported with new_vgic code.
> 
> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
> 
> So from now on IRQCHIP routing is enabled and a routing table entry
> must exist for irqfd injection to succeed for a given SPI. This patch
> builds a default flat irqchip routing table (gsi=irqchip.pin) covering
> all the VGIC SPI indexes. This routing table is overwritten by the
> first first user-space call to KVM_SET_GSI_ROUTING ioctl.
> 
> MSI routing setup is not yet allowed.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> v5 -> v6:
> - rebase on top of Andre's v8 + removal of old vgic
> 
> v4 -> v5:
> - vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
> - minor comment changes
> - remove trace_kvm_set_irq since it is called in irqchip
> - remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
> - despite Christoffer's question, in kvm_set_msi, I kept the copy from
>   the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
>   irqchip callback API into the struct kvm_msi * passed to
>   vits_inject_msi. Since vits_inject_msi is directly called by
>   kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
>   to me to keep the copy.
> - squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
>   table into that patch
> - handle default routing table alloc failure
> 
> v3 -> v4:
> - provide support only for new-vgic
> - code previously in vgic.c now in vgic_irqfd.c
> 
> v2 -> v3:
> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
> - vgic_irqfd_set_irq now is static
> - propagate flags
> - add comments
> 
> v1 -> v2:
> - fix bug reported by Andre related to msi.flags and msi.devid setting
>   in kvm_send_userspace_msi
> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
> 
> RFC -> PATCH
> - reword api.txt:
>   x move MSI routing comments in a subsequent patch,
>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
> ---
>  Documentation/virtual/kvm/api.txt | 12 ++++--
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  2 +
>  arch/arm/kvm/Makefile             |  1 +
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  2 +
>  arch/arm64/kvm/Makefile           |  1 +
>  virt/kvm/arm/vgic/vgic-init.c     | 27 +++++++++++++
>  virt/kvm/arm/vgic/vgic-irqfd.c    | 82 ++++++++++++++++++++++++++++++---------
>  virt/kvm/arm/vgic/vgic.c          |  7 ----
>  virt/kvm/irqchip.c                |  2 +
>  11 files changed, 109 insertions(+), 30 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 0065c8e..3bb60d3 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>  4.52 KVM_SET_GSI_ROUTING
>  
>  Capability: KVM_CAP_IRQ_ROUTING
> -Architectures: x86 s390
> +Architectures: x86 s390 arm arm64
>  Type: vm ioctl
>  Parameters: struct kvm_irq_routing (in)
>  Returns: 0 on success, -1 on error
>  
>  Sets the GSI routing table entries, overwriting any previously set entries.
>  
> +On arm/arm64, GSI routing has the following limitation:
> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.

Do you have an idea whether this is a limitation? Or something that
requires arch specific code in userland?
Or is there another issue with extending this to KVM_IRQ_LINE (ignoring
the fact that we don't need it)?

> +
>  struct kvm_irq_routing {
>  	__u32 nr;
>  	__u32 flags;
> @@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>  
> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
> -given by gsi + 32.
> +On arm/arm64, gsi routing being supported, the following can happen:
> +- in case no routing entry is associated to this gsi, injection fails
> +- in case the gsi is associated to an irqchip routing entry,
> +  irqchip.pin + 32 corresponds to the injected SPI ID.
>  
>  4.76 KVM_PPC_ALLOCATE_HTAB
>  
> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
> index 3c40facd..161997e 100644
> --- a/arch/arm/include/asm/kvm_host.h
> +++ b/arch/arm/include/asm/kvm_host.h
> @@ -37,6 +37,8 @@
>  
>  #define KVM_VCPU_MAX_FEATURES 2
>  
> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
> +
>  #include <kvm/arm_vgic.h>
>  
>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 95a0005..3e1cd04 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -32,6 +32,8 @@ config KVM
>  	select KVM_VFIO
>  	select HAVE_KVM_EVENTFD
>  	select HAVE_KVM_IRQFD
> +	select HAVE_KVM_IRQCHIP
> +	select HAVE_KVM_IRQ_ROUTING
>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
>  	---help---
>  	  Support hosting virtualized guest machines.
> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
> index 5e28df8..025d812 100644
> --- a/arch/arm/kvm/Makefile
> +++ b/arch/arm/kvm/Makefile
> @@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
>  obj-y += $(KVM)/arm/vgic/vgic-mmio.o
>  obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
>  obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
> +obj-y += $(KVM)//irqchip.o
>  obj-y += $(KVM)/arm/arch_timer.o
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index ebe8904..58f4a60 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -34,6 +34,7 @@
>  #define KVM_PRIVATE_MEM_SLOTS 4
>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
>  #define KVM_HALT_POLL_NS_DEFAULT 500000
> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
>  
>  #include <kvm/arm_vgic.h>
>  #include <kvm/arm_arch_timer.h>
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 9d2eff0..9c9edc9 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -37,6 +37,8 @@ config KVM
>  	select KVM_ARM_VGIC_V3
>  	select KVM_ARM_PMU if HW_PERF_EVENTS
>  	select HAVE_KVM_MSI
> +	select HAVE_KVM_IRQCHIP
> +	select HAVE_KVM_IRQ_ROUTING
>  	---help---
>  	  Support hosting virtualized guest machines.
>  	  We don't support KVM with 16K page tables yet, due to the multiple
> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
> index a5b9664..695eb3c 100644
> --- a/arch/arm64/kvm/Makefile
> +++ b/arch/arm64/kvm/Makefile
> @@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
>  kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
> diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
> index 01a60dc..591571e 100644
> --- a/virt/kvm/arm/vgic/vgic-init.c
> +++ b/virt/kvm/arm/vgic/vgic-init.c
> @@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
>  	kvm_for_each_vcpu(i, vcpu, kvm)
>  		kvm_vgic_vcpu_init(vcpu);
>  
> +	ret = kvm_setup_default_irq_routing(kvm);
> +	if (ret)
> +		goto out;
> +
>  	dist->initialized = true;
>  out:
>  	return ret;
> @@ -457,3 +461,26 @@ out_free_irq:
>  			kvm_get_running_vcpus());
>  	return ret;
>  }
> +
> +int kvm_setup_default_irq_routing(struct kvm *kvm)

Can you make this static (and rename it to avoid the clash)? I see that
x86 also has it public, but there is no architecture agnostic code using
that, so I consider this just a leftover on their side.

Cheers,
Andre.

> +{
> +	struct kvm_irq_routing_entry *entries;
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	u32 nr = dist->nr_spis;
> +	int i, ret;
> +
> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
> +			  GFP_KERNEL);
> +	if (!entries)
> +		return -ENOMEM;
> +
> +	for (i = 0; i < nr; i++) {
> +		entries[i].gsi = i;
> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
> +		entries[i].u.irqchip.irqchip = 0;
> +		entries[i].u.irqchip.pin = i;
> +	}
> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
> +	kfree(entries);
> +	return ret;
> +}
> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
> index c675513..b03ab4e 100644
> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
> @@ -17,36 +17,80 @@
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
>  #include <trace/events/kvm.h>
> +#include <kvm/arm_vgic.h>
> +#include "vgic.h"
>  
> -int kvm_irq_map_gsi(struct kvm *kvm,
> -		    struct kvm_kernel_irq_routing_entry *entries,
> -		    int gsi)
> +/**
> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
> + * irqchip routing entry
> + *
> + * This is the entry point for irqfd IRQ injection
> + */
> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
> +			struct kvm *kvm, int irq_source_id,
> +			int level, bool line_status)
>  {
> -	return 0;
> -}
> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
> +	struct vgic_dist *dist = &kvm->arch.vgic;
>  
> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
> -			 unsigned int pin)
> -{
> -	return pin;
> +	BUG_ON(!vgic_initialized(kvm));
> +
> +	if (spi_id > min(dist->nr_spis, 1020))
> +		return -EINVAL;
> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
>  }
>  
> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
> -		u32 irq, int level, bool line_status)
> +/**
> + * kvm_set_routing_entry: populate a kvm routing entry
> + * from a user routing entry
> + *
> + * @e: kvm kernel routing entry handle
> + * @ue: user api routing entry handle
> + * return 0 on success, -EINVAL on errors.
> + */
> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
> +			  const struct kvm_irq_routing_entry *ue)
>  {
> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
> +	int r = -EINVAL;
>  
> -	trace_kvm_set_irq(irq, level, irq_source_id);
> -
> -	BUG_ON(!vgic_initialized(kvm));
> -
> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
> +	switch (ue->type) {
> +	case KVM_IRQ_ROUTING_IRQCHIP:
> +		e->set = vgic_irqfd_set_irq;
> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
> +		e->irqchip.pin = ue->u.irqchip.pin;
> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
> +			goto out;
> +		break;
> +	default:
> +		goto out;
> +	}
> +	r = 0;
> +out:
> +	return r;
>  }
>  
> -/* MSI not implemented yet */
> +/**
> + * kvm_set_msi: inject the MSI corresponding to the
> + * MSI routing entry
> + *
> + * This is the entry point for irqfd MSI injection
> + * and userspace MSI injection.
> + */
>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>  		struct kvm *kvm, int irq_source_id,
>  		int level, bool line_status)
>  {
> -	return 0;
> +	struct kvm_msi msi;
> +
> +	msi.address_lo = e->msi.address_lo;
> +	msi.address_hi = e->msi.address_hi;
> +	msi.data = e->msi.data;
> +	msi.flags = e->flags;
> +	msi.devid = e->devid;
> +
> +	if (!vgic_has_its(kvm))
> +		return -ENODEV;
> +
> +	return vgic_its_inject_msi(kvm, &msi);
>  }
> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
> index c4f3aba..b254833 100644
> --- a/virt/kvm/arm/vgic/vgic.c
> +++ b/virt/kvm/arm/vgic/vgic.c
> @@ -684,10 +684,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
>  	return map_is_active;
>  }
>  
> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
> -{
> -	if (vgic_has_its(kvm))
> -		return vgic_its_inject_msi(kvm, msi);
> -	else
> -		return -ENODEV;
> -}
> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> index 32e5646..03632e3 100644
> --- a/virt/kvm/irqchip.c
> +++ b/virt/kvm/irqchip.c
> @@ -29,7 +29,9 @@
>  #include <linux/srcu.h>
>  #include <linux/export.h>
>  #include <trace/events/kvm.h>
> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>  #include "irq.h"
> +#endif
>  
>  int kvm_irq_map_gsi(struct kvm *kvm,
>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
> 

Re: [RFC v6 5/6] KVM: arm/arm64: enable MSI routing

[Andre Przywara]

Hi,

On 06/07/16 09:47, Eric Auger wrote:
> Up to now, only irqchip routing entries could be set. This patch
> adds the capability to insert MSI routing entries.
> 
> For ARM64, let's also increase KVM_MAX_IRQ_ROUTES to 4096: this
> include SPI irqchip routes plus MSI routes. In the future this
> might be extended.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>

Reviewed-by: Andre Przywara <andre.przywara@arm.com>

Cheers,
Andre.

> ---
> v2 -> v3:
> - remove any reference to KVM_IRQ_ROUTING_EXTENDED_MSI type
> - unconditionnaly uapi flags and devid downto the kernel
>   routing entry struct
> - handle KVM_MSI_VALID_DEVID flag in kvm_set_irq_routing
> - note about KVM_CAP_MSI_DEVID moved in the first patch file
>   of the series
> 
> v1 -> v2:
> - adapt to new routing entry types
> 
> RFC -> PATCH:
> - move api MSI routing updates into that patch file
> - use new devid field of user api struct
> ---
>  Documentation/virtual/kvm/api.txt | 5 +++++
>  include/linux/kvm_host.h          | 2 ++
>  virt/kvm/arm/vgic/vgic-irqfd.c    | 8 ++++++++
>  virt/kvm/irqchip.c                | 2 +-
>  4 files changed, 16 insertions(+), 1 deletion(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index 3bb60d3..60d4999 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -2375,6 +2375,11 @@ On arm/arm64, gsi routing being supported, the following can happen:
>  - in case no routing entry is associated to this gsi, injection fails
>  - in case the gsi is associated to an irqchip routing entry,
>    irqchip.pin + 32 corresponds to the injected SPI ID.
> +- in case the gsi is associated to an MSI routing entry,
> +  * without GICv3 ITS in-kernel emulation, MSI data matches the SPI ID
> +    of the injected SPI
> +  * with GICv3 ITS in-kernel emulation, the MSI message and device ID
> +    are translated into an LPI.
>  
>  4.76 KVM_PPC_ALLOCATE_HTAB
>  
> diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
> index fffa299..78d62ee 100644
> --- a/include/linux/kvm_host.h
> +++ b/include/linux/kvm_host.h
> @@ -1038,6 +1038,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
>  
>  #ifdef CONFIG_S390
>  #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
> +#elif defined(CONFIG_ARM64)
> +#define KVM_MAX_IRQ_ROUTES 4096
>  #else
>  #define KVM_MAX_IRQ_ROUTES 1024
>  #endif
> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
> index b03ab4e..5d9ed39 100644
> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
> @@ -62,6 +62,14 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
>  		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
>  			goto out;
>  		break;
> +	case KVM_IRQ_ROUTING_MSI:
> +		e->set = kvm_set_msi;
> +		e->msi.address_lo = ue->u.msi.address_lo;
> +		e->msi.address_hi = ue->u.msi.address_hi;
> +		e->msi.data = ue->u.msi.data;
> +		e->flags = ue->flags;
> +		e->devid = ue->u.msi.devid;
> +		break;
>  	default:
>  		goto out;
>  	}
> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
> index 03632e3..2ea5e50 100644
> --- a/virt/kvm/irqchip.c
> +++ b/virt/kvm/irqchip.c
> @@ -211,7 +211,7 @@ int kvm_set_irq_routing(struct kvm *kvm,
>  			goto out;
>  
>  		r = -EINVAL;
> -		if (ue->flags) {
> +		if (ue->flags & ~KVM_MSI_VALID_DEVID) {
>  			kfree(e);
>  			goto out;
>  		}
> 

Re: [RFC v6 6/6] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Andre Przywara]

Hi,

On 06/07/16 09:47, Eric Auger wrote:
> If the ITS modality is not available, let's simply support MSI
> injection by transforming the MSI.data into an SPI ID.
> 
> This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
> routing for arm too.
> 
> Signed-off-by: Eric Auger <eric.auger@redhat.com>
> 
> ---
> 
> v4 -> v5:
> - on vgic_v2m_inject_msi check the msi->data is within the SPI range
> - move KVM_HAVE_MSI in the KVM section (to be symetrical with ARM64)
> 
> v2 -> v3:
> - reword the commit message
> - add sanity check about devid provision
> 
> v1 -> v2:
> - introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
>   advice
> ---
>  arch/arm/kvm/Kconfig           |  1 +
>  virt/kvm/arm/vgic/vgic-irqfd.c |  2 +-
>  virt/kvm/arm/vgic/vgic-v2.c    | 17 +++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h       |  1 +
>  4 files changed, 20 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
> index 3e1cd04..90d0176 100644
> --- a/arch/arm/kvm/Kconfig
> +++ b/arch/arm/kvm/Kconfig
> @@ -34,6 +34,7 @@ config KVM
>  	select HAVE_KVM_IRQFD
>  	select HAVE_KVM_IRQCHIP
>  	select HAVE_KVM_IRQ_ROUTING
> +	select HAVE_KVM_MSI
>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
>  	---help---
>  	  Support hosting virtualized guest machines.
> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
> index 5d9ed39..6612209 100644
> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
> @@ -98,7 +98,7 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>  	msi.devid = e->devid;
>  
>  	if (!vgic_has_its(kvm))
> -		return -ENODEV;
> +		return vgic_v2m_inject_msi(kvm, &msi);
>  
>  	return vgic_its_inject_msi(kvm, &msi);
>  }
> diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
> index 0bf6709..88e77db 100644
> --- a/virt/kvm/arm/vgic/vgic-v2.c
> +++ b/virt/kvm/arm/vgic/vgic-v2.c
> @@ -241,6 +241,23 @@ static bool vgic_v2_check_base(gpa_t dist_base, gpa_t cpu_base)
>  	return false;
>  }
>  
> +/**
> + * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
> + * the SPI ID matching the msi data
> + *
> + * @kvm: pointer to the kvm struct
> + * @msi: the msi struct handle
> + */
> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
> +{
> +	if (msi->flags & KVM_MSI_VALID_DEVID)
> +		return -EINVAL;
> +	if (!vgic_valid_spi(kvm, msi->data))
> +		return -EINVAL;
> +
> +	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);
> +}
> +

Is that the right file to live in?
If I got this right, then vgic-v2.c contains functions specific to a
GICv2 host, whereas this function is connected to the GICv2 _emulation_,
which would work as well on top of a GICv3 host.
So can you move this to either vgic.c or vgic-irqfd.c?

>  int vgic_v2_map_resources(struct kvm *kvm)
>  {
>  	struct vgic_dist *dist = &kvm->arch.vgic;
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index b3e5678..223e946 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -64,6 +64,7 @@ int vgic_v2_probe(const struct gic_kvm_info *info);
>  int vgic_v2_map_resources(struct kvm *kvm);
>  int vgic_register_dist_iodev(struct kvm *kvm, gpa_t dist_base_address,
>  			     enum vgic_type);
> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi);

If you would move this to vgic-irqfd.c, you could avoid this prototype here.

Cheers,
Andre.

>  
>  #ifdef CONFIG_KVM_ARM_VGIC_V3
>  void vgic_v3_process_maintenance(struct kvm_vcpu *vcpu);
> 

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Auger Eric]

Hi Andre,

On 11/07/2016 18:25, Andre Przywara wrote:
> Hi Eric,
> 
> On 06/07/16 09:47, Eric Auger wrote:
>> This patch adds compilation and link against irqchip.
>>
>> Main motivation behind using irqchip code is to enable MSI
>> routing code. In the future irqchip routing may also be useful
>> when targeting multiple irqchips.
>>
>> Routing standard callbacks now are implemented in vgic-irqfd:
>> - kvm_set_routing_entry
>> - kvm_set_irq
>> - kvm_set_msi
>>
>> They only are supported with new_vgic code.
>>
>> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
>> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
>>
>> So from now on IRQCHIP routing is enabled and a routing table entry
>> must exist for irqfd injection to succeed for a given SPI. This patch
>> builds a default flat irqchip routing table (gsi=irqchip.pin) covering
>> all the VGIC SPI indexes. This routing table is overwritten by the
>> first first user-space call to KVM_SET_GSI_ROUTING ioctl.
>>
>> MSI routing setup is not yet allowed.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>> v5 -> v6:
>> - rebase on top of Andre's v8 + removal of old vgic
>>
>> v4 -> v5:
>> - vgic_irqfd.c was renamed into vgic-irqfd.c by Andre
>> - minor comment changes
>> - remove trace_kvm_set_irq since it is called in irqchip
>> - remove CONFIG_HAVE_KVM_MSI setting (done in KVM section)
>> - despite Christoffer's question, in kvm_set_msi, I kept the copy from
>>   the input "struct kvm_kernel_irq_routing_entry *e" imposed by the
>>   irqchip callback API into the struct kvm_msi * passed to
>>   vits_inject_msi. Since vits_inject_msi is directly called by
>>   kvm_send_userspace_msi which takes a struct kvm_msi*, makes sense
>>   to me to keep the copy.
>> - squash former [PATCH v4 5/7] KVM: arm/arm64: build a default routing
>>   table into that patch
>> - handle default routing table alloc failure
>>
>> v3 -> v4:
>> - provide support only for new-vgic
>> - code previously in vgic.c now in vgic_irqfd.c
>>
>> v2 -> v3:
>> - unconditionally set devid and KVM_MSI_VALID_DEVID flag as suggested
>>   by Andre (KVM_IRQ_ROUTING_EXTENDED_MSI type not used anymore)
>> - vgic_irqfd_set_irq now is static
>> - propagate flags
>> - add comments
>>
>> v1 -> v2:
>> - fix bug reported by Andre related to msi.flags and msi.devid setting
>>   in kvm_send_userspace_msi
>> - avoid injecting reserved IRQ numbers in vgic_irqfd_set_irq
>>
>> RFC -> PATCH
>> - reword api.txt:
>>   x move MSI routing comments in a subsequent patch,
>>   x clearly state GSI routing does not apply to KVM_IRQ_LINE
>> ---
>>  Documentation/virtual/kvm/api.txt | 12 ++++--
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  2 +
>>  arch/arm/kvm/Makefile             |  1 +
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  2 +
>>  arch/arm64/kvm/Makefile           |  1 +
>>  virt/kvm/arm/vgic/vgic-init.c     | 27 +++++++++++++
>>  virt/kvm/arm/vgic/vgic-irqfd.c    | 82 ++++++++++++++++++++++++++++++---------
>>  virt/kvm/arm/vgic/vgic.c          |  7 ----
>>  virt/kvm/irqchip.c                |  2 +
>>  11 files changed, 109 insertions(+), 30 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index 0065c8e..3bb60d3 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -1433,13 +1433,16 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>>  4.52 KVM_SET_GSI_ROUTING
>>  
>>  Capability: KVM_CAP_IRQ_ROUTING
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm arm64
>>  Type: vm ioctl
>>  Parameters: struct kvm_irq_routing (in)
>>  Returns: 0 on success, -1 on error
>>  
>>  Sets the GSI routing table entries, overwriting any previously set entries.
>>  
>> +On arm/arm64, GSI routing has the following limitation:
>> +- GSI routing does not apply to KVM_IRQ_LINE but only to KVM_IRQFD.
> 
> Do you have an idea whether this is a limitation? Or something that
> requires arch specific code in userland?
No, currently I have not fully identified the consequences. I guess they
will come when we will start really using GSI routing with several irqchips.

> Or is there another issue with extending this to KVM_IRQ_LINE (ignoring
> the fact that we don't need it)?
The problem comes from the GSI semantic used originally for ARM which is
really hardcoded:


bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
field: | irq_type  | vcpu_index |     irq_id     |

userspace or kernel (default) would need to create routing entries for
all the GSIs likely to be injected from userspace.

> 
>> +
>>  struct kvm_irq_routing {
>>  	__u32 nr;
>>  	__u32 flags;
>> @@ -2368,9 +2371,10 @@ Note that closing the resamplefd is not sufficient to disable the
>>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>  
>> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
>> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
>> -given by gsi + 32.
>> +On arm/arm64, gsi routing being supported, the following can happen:
>> +- in case no routing entry is associated to this gsi, injection fails
>> +- in case the gsi is associated to an irqchip routing entry,
>> +  irqchip.pin + 32 corresponds to the injected SPI ID.
>>  
>>  4.76 KVM_PPC_ALLOCATE_HTAB
>>  
>> diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
>> index 3c40facd..161997e 100644
>> --- a/arch/arm/include/asm/kvm_host.h
>> +++ b/arch/arm/include/asm/kvm_host.h
>> @@ -37,6 +37,8 @@
>>  
>>  #define KVM_VCPU_MAX_FEATURES 2
>>  
>> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
>> +
>>  #include <kvm/arm_vgic.h>
>>  
>>  #define KVM_MAX_VCPUS VGIC_V2_MAX_CPUS
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 95a0005..3e1cd04 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -32,6 +32,8 @@ config KVM
>>  	select KVM_VFIO
>>  	select HAVE_KVM_EVENTFD
>>  	select HAVE_KVM_IRQFD
>> +	select HAVE_KVM_IRQCHIP
>> +	select HAVE_KVM_IRQ_ROUTING
>>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
>>  	---help---
>>  	  Support hosting virtualized guest machines.
>> diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
>> index 5e28df8..025d812 100644
>> --- a/arch/arm/kvm/Makefile
>> +++ b/arch/arm/kvm/Makefile
>> @@ -29,4 +29,5 @@ obj-y += $(KVM)/arm/vgic/vgic-v2.o
>>  obj-y += $(KVM)/arm/vgic/vgic-mmio.o
>>  obj-y += $(KVM)/arm/vgic/vgic-mmio-v2.o
>>  obj-y += $(KVM)/arm/vgic/vgic-kvm-device.o
>> +obj-y += $(KVM)//irqchip.o
>>  obj-y += $(KVM)/arm/arch_timer.o
>> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
>> index ebe8904..58f4a60 100644
>> --- a/arch/arm64/include/asm/kvm_host.h
>> +++ b/arch/arm64/include/asm/kvm_host.h
>> @@ -34,6 +34,7 @@
>>  #define KVM_PRIVATE_MEM_SLOTS 4
>>  #define KVM_COALESCED_MMIO_PAGE_OFFSET 1
>>  #define KVM_HALT_POLL_NS_DEFAULT 500000
>> +#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 - 32 is the number of SPIs */
>>  
>>  #include <kvm/arm_vgic.h>
>>  #include <kvm/arm_arch_timer.h>
>> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
>> index 9d2eff0..9c9edc9 100644
>> --- a/arch/arm64/kvm/Kconfig
>> +++ b/arch/arm64/kvm/Kconfig
>> @@ -37,6 +37,8 @@ config KVM
>>  	select KVM_ARM_VGIC_V3
>>  	select KVM_ARM_PMU if HW_PERF_EVENTS
>>  	select HAVE_KVM_MSI
>> +	select HAVE_KVM_IRQCHIP
>> +	select HAVE_KVM_IRQ_ROUTING
>>  	---help---
>>  	  Support hosting virtualized guest machines.
>>  	  We don't support KVM with 16K page tables yet, due to the multiple
>> diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
>> index a5b9664..695eb3c 100644
>> --- a/arch/arm64/kvm/Makefile
>> +++ b/arch/arm64/kvm/Makefile
>> @@ -30,5 +30,6 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v2.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-mmio-v3.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-kvm-device.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/vgic/vgic-its.o
>> +kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/irqchip.o
>>  kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arch_timer.o
>>  kvm-$(CONFIG_KVM_ARM_PMU) += $(KVM)/arm/pmu.o
>> diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
>> index 01a60dc..591571e 100644
>> --- a/virt/kvm/arm/vgic/vgic-init.c
>> +++ b/virt/kvm/arm/vgic/vgic-init.c
>> @@ -264,6 +264,10 @@ int vgic_init(struct kvm *kvm)
>>  	kvm_for_each_vcpu(i, vcpu, kvm)
>>  		kvm_vgic_vcpu_init(vcpu);
>>  
>> +	ret = kvm_setup_default_irq_routing(kvm);
>> +	if (ret)
>> +		goto out;
>> +
>>  	dist->initialized = true;
>>  out:
>>  	return ret;
>> @@ -457,3 +461,26 @@ out_free_irq:
>>  			kvm_get_running_vcpus());
>>  	return ret;
>>  }
>> +
>> +int kvm_setup_default_irq_routing(struct kvm *kvm)
> 
> Can you make this static (and rename it to avoid the clash)? I see that
> x86 also has it public, but there is no architecture agnostic code using
> that, so I consider this just a leftover on their side.

OK. I think I will move the kvm_setup_default_irq_routing definition
into vgic-irqfd and indeed I can propose to retire the declaration from
kvm_host.h and move it to architecture folders.

Thanks

Eric
> 
> Cheers,
> Andre.
> 
>> +{
>> +	struct kvm_irq_routing_entry *entries;
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>> +	u32 nr = dist->nr_spis;
>> +	int i, ret;
>> +
>> +	entries = kcalloc(nr, sizeof(struct kvm_kernel_irq_routing_entry),
>> +			  GFP_KERNEL);
>> +	if (!entries)
>> +		return -ENOMEM;
>> +
>> +	for (i = 0; i < nr; i++) {
>> +		entries[i].gsi = i;
>> +		entries[i].type = KVM_IRQ_ROUTING_IRQCHIP;
>> +		entries[i].u.irqchip.irqchip = 0;
>> +		entries[i].u.irqchip.pin = i;
>> +	}
>> +	ret = kvm_set_irq_routing(kvm, entries, nr, 0);
>> +	kfree(entries);
>> +	return ret;
>> +}
>> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
>> index c675513..b03ab4e 100644
>> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
>> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
>> @@ -17,36 +17,80 @@
>>  #include <linux/kvm.h>
>>  #include <linux/kvm_host.h>
>>  #include <trace/events/kvm.h>
>> +#include <kvm/arm_vgic.h>
>> +#include "vgic.h"
>>  
>> -int kvm_irq_map_gsi(struct kvm *kvm,
>> -		    struct kvm_kernel_irq_routing_entry *entries,
>> -		    int gsi)
>> +/**
>> + * vgic_irqfd_set_irq: inject the IRQ corresponding to the
>> + * irqchip routing entry
>> + *
>> + * This is the entry point for irqfd IRQ injection
>> + */
>> +static int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
>> +			struct kvm *kvm, int irq_source_id,
>> +			int level, bool line_status)
>>  {
>> -	return 0;
>> -}
>> +	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
>> +	struct vgic_dist *dist = &kvm->arch.vgic;
>>  
>> -int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned int irqchip,
>> -			 unsigned int pin)
>> -{
>> -	return pin;
>> +	BUG_ON(!vgic_initialized(kvm));
>> +
>> +	if (spi_id > min(dist->nr_spis, 1020))
>> +		return -EINVAL;
>> +	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
>>  }
>>  
>> -int kvm_set_irq(struct kvm *kvm, int irq_source_id,
>> -		u32 irq, int level, bool line_status)
>> +/**
>> + * kvm_set_routing_entry: populate a kvm routing entry
>> + * from a user routing entry
>> + *
>> + * @e: kvm kernel routing entry handle
>> + * @ue: user api routing entry handle
>> + * return 0 on success, -EINVAL on errors.
>> + */
>> +int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
>> +			  const struct kvm_irq_routing_entry *ue)
>>  {
>> -	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
>> +	int r = -EINVAL;
>>  
>> -	trace_kvm_set_irq(irq, level, irq_source_id);
>> -
>> -	BUG_ON(!vgic_initialized(kvm));
>> -
>> -	return kvm_vgic_inject_irq(kvm, 0, spi, level);
>> +	switch (ue->type) {
>> +	case KVM_IRQ_ROUTING_IRQCHIP:
>> +		e->set = vgic_irqfd_set_irq;
>> +		e->irqchip.irqchip = ue->u.irqchip.irqchip;
>> +		e->irqchip.pin = ue->u.irqchip.pin;
>> +		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
>> +		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
>> +			goto out;
>> +		break;
>> +	default:
>> +		goto out;
>> +	}
>> +	r = 0;
>> +out:
>> +	return r;
>>  }
>>  
>> -/* MSI not implemented yet */
>> +/**
>> + * kvm_set_msi: inject the MSI corresponding to the
>> + * MSI routing entry
>> + *
>> + * This is the entry point for irqfd MSI injection
>> + * and userspace MSI injection.
>> + */
>>  int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>>  		struct kvm *kvm, int irq_source_id,
>>  		int level, bool line_status)
>>  {
>> -	return 0;
>> +	struct kvm_msi msi;
>> +
>> +	msi.address_lo = e->msi.address_lo;
>> +	msi.address_hi = e->msi.address_hi;
>> +	msi.data = e->msi.data;
>> +	msi.flags = e->flags;
>> +	msi.devid = e->devid;
>> +
>> +	if (!vgic_has_its(kvm))
>> +		return -ENODEV;
>> +
>> +	return vgic_its_inject_msi(kvm, &msi);
>>  }
>> diff --git a/virt/kvm/arm/vgic/vgic.c b/virt/kvm/arm/vgic/vgic.c
>> index c4f3aba..b254833 100644
>> --- a/virt/kvm/arm/vgic/vgic.c
>> +++ b/virt/kvm/arm/vgic/vgic.c
>> @@ -684,10 +684,3 @@ bool kvm_vgic_map_is_active(struct kvm_vcpu *vcpu, unsigned int virt_irq)
>>  	return map_is_active;
>>  }
>>  
>> -int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
>> -{
>> -	if (vgic_has_its(kvm))
>> -		return vgic_its_inject_msi(kvm, msi);
>> -	else
>> -		return -ENODEV;
>> -}
>> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
>> index 32e5646..03632e3 100644
>> --- a/virt/kvm/irqchip.c
>> +++ b/virt/kvm/irqchip.c
>> @@ -29,7 +29,9 @@
>>  #include <linux/srcu.h>
>>  #include <linux/export.h>
>>  #include <trace/events/kvm.h>
>> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>>  #include "irq.h"
>> +#endif
>>  
>>  int kvm_irq_map_gsi(struct kvm *kvm,
>>  		    struct kvm_kernel_irq_routing_entry *entries, int gsi)
>>

Re: [RFC v6 4/6] KVM: arm/arm64: enable irqchip routing

[Auger Eric]

Hi Drew, Radim,

On 08/07/2016 22:55, Radim Krčmář wrote:
> 2016-07-08 10:52+0200, Andrew Jones:
>> On Fri, Jul 08, 2016 at 10:16:53AM +0200, Auger Eric wrote:
>>> On 07/07/2016 19:20, Andrew Jones wrote:
>>>> On Wed, Jul 06, 2016 at 10:47:53AM +0200, Eric Auger wrote:
>>>>> diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
>>>>> @@ -29,7 +29,9 @@
>>>>>  #include <linux/srcu.h>
>>>>>  #include <linux/export.h>
>>>>>  #include <trace/events/kvm.h>
>>>>> +#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
>>>>>  #include "irq.h"
>>>>> +#endif
>>>>
>>>> Instead of doing this, shouldn't we add arch/arm[64]/kvm/irq.h files.
>>>> Probably a simple one like ./arch/s390/kvm/irq.h ?
>>>
>>> Well I considered this solution in the past but I did not find much to
>>> put there (it was even void). typically irqchip_in_kernel is in
>>> include/kvm/arm_vgic.h since the macro can be shared between arm/arm64.
>>
>> I think I'd prefer a nearly empty file to the #ifdef's, but Paolo and
>> Radim should chime in.
> 
> I concur, hiding ugliness in header files is what we strive for.
> 
> The files could #include <include/kvm/arm_vgic.h>, which might make
> their existence easier to understand.
> 
Ok I will introduce irq.h which such inclusion on ARM/ARM64.

Thanks

Eric
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm

Re: [RFC v6 6/6] KVM: arm: enable KVM_SIGNAL_MSI and MSI routing

[Auger Eric]

Hi Andre,

On 11/07/2016 18:26, Andre Przywara wrote:
> Hi,
> 
> On 06/07/16 09:47, Eric Auger wrote:
>> If the ITS modality is not available, let's simply support MSI
>> injection by transforming the MSI.data into an SPI ID.
>>
>> This becomes possible to use KVM_SIGNAL_MSI ioctl and MSI
>> routing for arm too.
>>
>> Signed-off-by: Eric Auger <eric.auger@redhat.com>
>>
>> ---
>>
>> v4 -> v5:
>> - on vgic_v2m_inject_msi check the msi->data is within the SPI range
>> - move KVM_HAVE_MSI in the KVM section (to be symetrical with ARM64)
>>
>> v2 -> v3:
>> - reword the commit message
>> - add sanity check about devid provision
>>
>> v1 -> v2:
>> - introduce vgic_v2m_inject_msi in vgic-v2-emul.c following Andre's
>>   advice
>> ---
>>  arch/arm/kvm/Kconfig           |  1 +
>>  virt/kvm/arm/vgic/vgic-irqfd.c |  2 +-
>>  virt/kvm/arm/vgic/vgic-v2.c    | 17 +++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h       |  1 +
>>  4 files changed, 20 insertions(+), 1 deletion(-)
>>
>> diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
>> index 3e1cd04..90d0176 100644
>> --- a/arch/arm/kvm/Kconfig
>> +++ b/arch/arm/kvm/Kconfig
>> @@ -34,6 +34,7 @@ config KVM
>>  	select HAVE_KVM_IRQFD
>>  	select HAVE_KVM_IRQCHIP
>>  	select HAVE_KVM_IRQ_ROUTING
>> +	select HAVE_KVM_MSI
>>  	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
>>  	---help---
>>  	  Support hosting virtualized guest machines.
>> diff --git a/virt/kvm/arm/vgic/vgic-irqfd.c b/virt/kvm/arm/vgic/vgic-irqfd.c
>> index 5d9ed39..6612209 100644
>> --- a/virt/kvm/arm/vgic/vgic-irqfd.c
>> +++ b/virt/kvm/arm/vgic/vgic-irqfd.c
>> @@ -98,7 +98,7 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
>>  	msi.devid = e->devid;
>>  
>>  	if (!vgic_has_its(kvm))
>> -		return -ENODEV;
>> +		return vgic_v2m_inject_msi(kvm, &msi);
>>  
>>  	return vgic_its_inject_msi(kvm, &msi);
>>  }
>> diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
>> index 0bf6709..88e77db 100644
>> --- a/virt/kvm/arm/vgic/vgic-v2.c
>> +++ b/virt/kvm/arm/vgic/vgic-v2.c
>> @@ -241,6 +241,23 @@ static bool vgic_v2_check_base(gpa_t dist_base, gpa_t cpu_base)
>>  	return false;
>>  }
>>  
>> +/**
>> + * vgic_v2m_inject_msi: emulates GICv2M MSI injection by injecting
>> + * the SPI ID matching the msi data
>> + *
>> + * @kvm: pointer to the kvm struct
>> + * @msi: the msi struct handle
>> + */
>> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi)
>> +{
>> +	if (msi->flags & KVM_MSI_VALID_DEVID)
>> +		return -EINVAL;
>> +	if (!vgic_valid_spi(kvm, msi->data))
>> +		return -EINVAL;
>> +
>> +	return kvm_vgic_inject_irq(kvm, 0, msi->data, 1);
>> +}
>> +
> 
> Is that the right file to live in?
> If I got this right, then vgic-v2.c contains functions specific to a
> GICv2 host, whereas this function is connected to the GICv2 _emulation_,
> which would work as well on top of a GICv3 host.
> So can you move this to either vgic.c or vgic-irqfd.c?
Yes makes sense to me too. I will move the function to vgic-irqfd then

Thanks

Eric
> 
>>  int vgic_v2_map_resources(struct kvm *kvm)
>>  {
>>  	struct vgic_dist *dist = &kvm->arch.vgic;
>> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
>> index b3e5678..223e946 100644
>> --- a/virt/kvm/arm/vgic/vgic.h
>> +++ b/virt/kvm/arm/vgic/vgic.h
>> @@ -64,6 +64,7 @@ int vgic_v2_probe(const struct gic_kvm_info *info);
>>  int vgic_v2_map_resources(struct kvm *kvm);
>>  int vgic_register_dist_iodev(struct kvm *kvm, gpa_t dist_base_address,
>>  			     enum vgic_type);
>> +int vgic_v2m_inject_msi(struct kvm *kvm, struct kvm_msi *msi);
> 
> If you would move this to vgic-irqfd.c, you could avoid this prototype here.
> 
> Cheers,
> Andre.
> 
>>  
>>  #ifdef CONFIG_KVM_ARM_VGIC_V3
>>  void vgic_v3_process_maintenance(struct kvm_vcpu *vcpu);
>>