[RFC 0/6] KVM: arm/arm64: gsi routing support

[RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
appears to be requested. More specifically MSI routing is needed.
irqchip routing does not sound to be really useful on arm but usage of
MSI routing also mandates to integrate irqchip routing. The initial
implementation of irqfd on arm must be upgraded with the integration
of kvm irqchip.c code and the implementation of its standard hooks
in the architecture specific part.

The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
ioctl along with KVM_IRQFD. If the userspace does not define any routing
table, no irqfd injection can happen. The user-space can use
KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.

for irqchip routing, the convention is, only SPI can be injected and the
SPI ID corresponds to irqchip.pin + 32. For MSI routing the interrupt ID
matches the MSI msg data. API evolve to support associating a device ID
to a routine entry.

Known Issues of this RFC:

- One of the biggest is the API inconsistencies on ARM. Blame me.
  Routing should apply to KVM_IRQ_LINE ioctl which is not the case yet
  in this series. It only applies to irqfd.
  on x86 typically this KVM_IRQ_LINE is plugged onto irqchip.c kvm_set_irq
  whereas on ARM we inject directly through kvm_vgic_inject_irq
  x on arm/arm64 gsi has a specific structure:
    bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
    field: | irq_type  | vcpu_index |     irq_id     |
    where irq_id matches the Interrupt ID
- for KVM_IRQFD without routing (current implementation) the gsi field
  corresponds to an SPI index = irq_id (above) -32.
- as far as understand qemu integration, gsi is supposed to be within
  [0, KVM_MAX_IRQ_ROUTES]. Difficult to use KVM_IRQ_LINE gsi.
- to be defined what we choose as a convention with irqchip routing is
  applied: gsi -> irqchip input pin.
- Or shouldn't we simply rule out any userspace irqchip routing and stick
  to MSI routing? we could define a fixed identity in-kernel irqchip mapping
  and only offer MSI routing.
- static allocation of chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
  arbitrary put KVM_IRQCHIP_NUM_PINS = 1020 - 32 (SPI count). On s390
  this is even bigger.

Currently tested on irqchip routing only (Calxeda midway only),
ie NOT TESTED on MSI routing yet.

This is a very preliminary RFC to ease the discussion.

Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc8-gsi-routing-rfc

It applies on Andre's [PATCH 00/13] arm64: KVM: GICv3 ITS emulation
(http://www.spinics.net/lists/kvm/msg117402.html)

Eric Auger (6):
  KVM: api: add kvm_irq_routing_extended_msi
  KVM: kvm_host: add kvm_extended_msi
  KVM: irqchip: convey devid to kvm_set_msi
  KVM: arm/arm64: enable irqchip routing
  KVM: arm/arm64: enable MSI routing
  KVM: arm: implement kvm_set_msi by gsi direct mapping

 Documentation/virtual/kvm/api.txt | 20 ++++++--
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  3 ++
 arch/arm/kvm/Makefile             |  2 +-
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  2 +
 arch/arm64/kvm/Makefile           |  2 +-
 include/kvm/arm_vgic.h            |  9 ----
 include/linux/kvm_host.h          | 10 ++++
 include/uapi/linux/kvm.h          |  9 ++++
 virt/kvm/arm/vgic.c               | 96 +++++++++++++++++++++++++++------------
 virt/kvm/irqchip.c                | 20 ++++++--
 12 files changed, 128 insertions(+), 48 deletions(-)

-- 
1.9.1

[RFC 1/6] KVM: api: add kvm_irq_routing_extended_msi

[Eric Auger]

On ARM, the MSI msg (address and data) comes along with
out-of-band device ID information. The device ID encodes the device
that composes the MSI msg. Let's create a new routing entry structure
that enables to encode that information on top of standard MSI
message

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 Documentation/virtual/kvm/api.txt | 9 +++++++++
 include/uapi/linux/kvm.h          | 9 +++++++++
 2 files changed, 18 insertions(+)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index d20fd94..bcec91e 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1419,6 +1419,7 @@ struct kvm_irq_routing_entry {
 		struct kvm_irq_routing_irqchip irqchip;
 		struct kvm_irq_routing_msi msi;
 		struct kvm_irq_routing_s390_adapter adapter;
+		struct kvm_irq_routing_extended_msi ext_msi;
 		__u32 pad[8];
 	} u;
 };
@@ -1427,6 +1428,7 @@ struct kvm_irq_routing_entry {
 #define KVM_IRQ_ROUTING_IRQCHIP 1
 #define KVM_IRQ_ROUTING_MSI 2
 #define KVM_IRQ_ROUTING_S390_ADAPTER 3
+#define KVM_IRQ_ROUTING_EXTENDED_MSI 4
 
 No flags are specified so far, the corresponding field must be set to zero.
 
@@ -1442,6 +1444,13 @@ struct kvm_irq_routing_msi {
 	__u32 pad;
 };
 
+struct kvm_irq_routing_extended_msi {
+	__u32 address_lo;
+	__u32 address_hi;
+	__u32 data;
+	__u32 devid;
+};
+
 struct kvm_irq_routing_s390_adapter {
 	__u64 ind_addr;
 	__u64 summary_addr;
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 2a23705..e3f65a0 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -829,6 +829,13 @@ struct kvm_irq_routing_msi {
 	__u32 pad;
 };
 
+struct kvm_irq_routing_extended_msi {
+	__u32 address_lo;
+	__u32 address_hi;
+	__u32 data;
+	__u32 devid;
+};
+
 struct kvm_irq_routing_s390_adapter {
 	__u64 ind_addr;
 	__u64 summary_addr;
@@ -841,6 +848,7 @@ struct kvm_irq_routing_s390_adapter {
 #define KVM_IRQ_ROUTING_IRQCHIP 1
 #define KVM_IRQ_ROUTING_MSI 2
 #define KVM_IRQ_ROUTING_S390_ADAPTER 3
+#define KVM_IRQ_ROUTING_EXTENDED_MSI 4
 
 struct kvm_irq_routing_entry {
 	__u32 gsi;
@@ -851,6 +859,7 @@ struct kvm_irq_routing_entry {
 		struct kvm_irq_routing_irqchip irqchip;
 		struct kvm_irq_routing_msi msi;
 		struct kvm_irq_routing_s390_adapter adapter;
+		struct kvm_irq_routing_extended_msi ext_msi;
 		__u32 pad[8];
 	} u;
 };
-- 
1.9.1

[RFC 2/6] KVM: kvm_host: add kvm_extended_msi

[Eric Auger]

As a follow-up of user API extension let's create a corresponding
kernel side structure

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 include/linux/kvm_host.h | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index ad45054..e1c1c0d 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -304,6 +304,13 @@ struct kvm_s390_adapter_int {
 	u32 adapter_id;
 };
 
+struct kvm_extended_msi {
+	u32     address_lo;     /* low 32 bits of msi message address */
+	u32     address_hi;     /* high 32 bits of msi message address */
+	u32     data;           /* 16 bits of msi message data */
+	u32	devid;		/* out-of-band device ID */
+};
+
 struct kvm_kernel_irq_routing_entry {
 	u32 gsi;
 	u32 type;
@@ -317,6 +324,7 @@ struct kvm_kernel_irq_routing_entry {
 		} irqchip;
 		struct msi_msg msi;
 		struct kvm_s390_adapter_int adapter;
+		struct kvm_extended_msi ext_msi;
 	};
 	struct hlist_node link;
 };
-- 
1.9.1

[RFC 3/6] KVM: irqchip: convey devid to kvm_set_msi

[Eric Auger]

on ARM, a devid field is conveyed in kvm_msi struct. Let's choose the
rooting type and struct according to its availability and fill the
corresponding struct. Also remove the flag check now this latter can
be non null.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 virt/kvm/irqchip.c | 18 ++++++++++++++----
 1 file changed, 14 insertions(+), 4 deletions(-)

diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index 1d56a90..e76c7d2 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -73,12 +73,22 @@ int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
 {
 	struct kvm_kernel_irq_routing_entry route;
 
-	if (!irqchip_in_kernel(kvm) || msi->flags != 0)
+	if (!irqchip_in_kernel(kvm))
 		return -EINVAL;
 
-	route.msi.address_lo = msi->address_lo;
-	route.msi.address_hi = msi->address_hi;
-	route.msi.data = msi->data;
+	if (msi->flags & KVM_MSI_VALID_DEVID) {
+		route.type = KVM_IRQ_ROUTING_EXTENDED_MSI;
+		route.ext_msi.address_lo = msi->address_lo;
+		route.ext_msi.address_hi = msi->address_hi;
+		route.ext_msi.data = msi->data;
+		route.ext_msi.devid= msi->devid;
+	}
+	else {
+		route.type = KVM_IRQ_ROUTING_MSI;
+		route.msi.address_lo = msi->address_lo;
+		route.msi.address_hi = msi->address_hi;
+		route.msi.data = msi->data;
+	}
 
 	return kvm_set_msi(&route, kvm, KVM_USERSPACE_IRQ_SOURCE_ID, 1, false);
 }
-- 
1.9.1

[RFC 4/6] KVM: arm/arm64: enable irqchip routing

[Eric Auger]

This patch adds compilation and link against irqchip.

On ARM, irqchip routing is not really useful since there is
a single irqchip. However main motivation behind using irqchip
code is to enable MSI routing code. With the support of in-kernel
GICv3 ITS emulation, it now seems to be a MUST HAVE requirement.

Functions previously implemented in vgic.c and substitute
to more complex irqchip implementation are removed:

- kvm_send_userspace_msi
- kvm_irq_map_chip_pin
- kvm_set_irq
- kvm_irq_map_gsi.

They implemented a kernel default identity GSI routing. This is now
replaced by user-side provided routing.

Routing standard hooks are now implemented in vgic:
- kvm_set_routing_entry
- kvm_set_irq
- kvm_set_msi

Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.

MSI routing is not yet allowed.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 Documentation/virtual/kvm/api.txt | 11 ++++--
 arch/arm/include/asm/kvm_host.h   |  2 +
 arch/arm/kvm/Kconfig              |  2 +
 arch/arm/kvm/Makefile             |  2 +-
 arch/arm64/include/asm/kvm_host.h |  1 +
 arch/arm64/kvm/Kconfig            |  2 +
 arch/arm64/kvm/Makefile           |  2 +-
 include/kvm/arm_vgic.h            |  9 -----
 virt/kvm/arm/vgic.c               | 78 ++++++++++++++++++++++++---------------
 virt/kvm/irqchip.c                |  2 +
 10 files changed, 67 insertions(+), 44 deletions(-)

diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
index bcec91e..2bc96e1 100644
--- a/Documentation/virtual/kvm/api.txt
+++ b/Documentation/virtual/kvm/api.txt
@@ -1395,7 +1395,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
 4.52 KVM_SET_GSI_ROUTING
 
 Capability: KVM_CAP_IRQ_ROUTING
-Architectures: x86 s390
+Architectures: x86 s390 arm arm64
 Type: vm ioctl
 Parameters: struct kvm_irq_routing (in)
 Returns: 0 on success, -1 on error
@@ -2310,9 +2310,12 @@ Note that closing the resamplefd is not sufficient to disable the
 irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
 and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
 
-On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
-Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
-given by gsi + 32.
+On ARM/ARM64, when GSI routing is not used, the gsi field in the
+kvm_irqfd struct specifies the Shared Peripheral Interrupt (SPI) index,
+such that the GIC interrupt ID is given by gsi + 32. When GSI routing is
+setup:
+- if irqchip routing: irqchip.pin + 32 is the SPI ID that is injected
+- if MSI routing: the MSI data is used as interrupt ID (SPI or LPI).
 
 4.76 KVM_PPC_ALLOCATE_HTAB
 
diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index d71607c..452697e 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -42,6 +42,8 @@
 
 #define KVM_VCPU_MAX_FEATURES 2
 
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 -32 is the number of SPI */
+
 #include <kvm/arm_vgic.h>
 
 u32 *kvm_vcpu_reg(struct kvm_vcpu *vcpu, u8 reg_num, u32 mode);
diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index bfb915d..151e710 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -31,6 +31,8 @@ config KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
 	---help---
 	  Support hosting virtualized guest machines.
diff --git a/arch/arm/kvm/Makefile b/arch/arm/kvm/Makefile
index c5eef02c..1a8f48a 100644
--- a/arch/arm/kvm/Makefile
+++ b/arch/arm/kvm/Makefile
@@ -15,7 +15,7 @@ AFLAGS_init.o := -Wa,-march=armv7-a$(plus_virt)
 AFLAGS_interrupts.o := -Wa,-march=armv7-a$(plus_virt)
 
 KVM := ../../../virt/kvm
-kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o
+kvm-arm-y = $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o $(KVM)/irqchip.o
 
 obj-y += kvm-arm.o init.o interrupts.o
 obj-y += arm.o handle_exit.o guest.o mmu.o emulate.o reset.o
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index f0f58c9..751210a 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -44,6 +44,7 @@
 #include <kvm/arm_arch_timer.h>
 
 #define KVM_VCPU_MAX_FEATURES 3
+#define KVM_IRQCHIP_NUM_PINS 988 /* 1020 -32 is the number of SPI */
 
 int __attribute_const__ kvm_target_cpu(void);
 int kvm_reset_vcpu(struct kvm_vcpu *vcpu);
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index ff9722f..1a9900d 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -32,6 +32,8 @@ config KVM
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
 	select HAVE_KVM_MSI
+	select HAVE_KVM_IRQCHIP
+	select HAVE_KVM_IRQ_ROUTING
 	---help---
 	  Support hosting virtualized guest machines.
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 9803307..90a08457 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -11,7 +11,7 @@ ARM=../../../arch/arm/kvm
 
 obj-$(CONFIG_KVM_ARM_HOST) += kvm.o
 
-kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o
+kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/eventfd.o $(KVM)/vfio.o $(KVM)/irqchip.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/arm.o $(ARM)/mmu.o $(ARM)/mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(ARM)/psci.o $(ARM)/perf.o
 
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 8f1be6a..10dc596 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -366,13 +366,4 @@ static inline int vgic_v3_probe(struct device_node *vgic_node,
 }
 #endif
 
-#ifdef CONFIG_HAVE_KVM_MSI
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi);
-#else
-static inline int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
-{
-	return -ENODEV;
-}
-#endif
-
 #endif
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 09b1f46..212a5ff 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2220,47 +2220,67 @@ out_free_irq:
 	return ret;
 }
 
-int kvm_irq_map_gsi(struct kvm *kvm,
-		    struct kvm_kernel_irq_routing_entry *entries,
-		    int gsi)
+int vgic_irqfd_set_irq(struct kvm_kernel_irq_routing_entry *e,
+			struct kvm *kvm, int irq_source_id,
+			int level, bool line_status)
 {
-	return 0;
-}
-
-int kvm_irq_map_chip_pin(struct kvm *kvm, unsigned irqchip, unsigned pin)
-{
-	return pin;
-}
-
-int kvm_set_irq(struct kvm *kvm, int irq_source_id,
-		u32 irq, int level, bool line_status)
-{
-	unsigned int spi = irq + VGIC_NR_PRIVATE_IRQS;
+	unsigned int spi_id = e->irqchip.pin + VGIC_NR_PRIVATE_IRQS;
 
-	trace_kvm_set_irq(irq, level, irq_source_id);
+	trace_kvm_set_irq(spi_id, level, irq_source_id);
 
 	BUG_ON(!vgic_initialized(kvm));
 
-	if (spi > kvm->arch.vgic.nr_irqs)
+	if (spi_id > kvm->arch.vgic.nr_irqs)
 		return -EINVAL;
-	return kvm_vgic_inject_irq(kvm, 0, spi, level);
+	return kvm_vgic_inject_irq(kvm, 0, spi_id, level);
 
 }
 
-/* MSI not implemented yet */
+/**
+ * Populates a kvm routing entry from a user routing entry
+ * @e: kvm internal formatted entry
+ * @ue: user api formatted entry
+ * return 0 on success, -EINVAL on errors.
+ */
+int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
+			  const struct kvm_irq_routing_entry *ue)
+{
+	int r = -EINVAL;
+
+	switch (ue->type) {
+	case KVM_IRQ_ROUTING_IRQCHIP:
+		e->set = vgic_irqfd_set_irq;
+		e->irqchip.irqchip = ue->u.irqchip.irqchip;
+		e->irqchip.pin = ue->u.irqchip.pin;
+		if ((e->irqchip.pin >= KVM_IRQCHIP_NUM_PINS) ||
+		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
+			goto out;
+		break;
+	default:
+		goto out;
+	}
+	r = 0;
+out:
+	return r;
+}
+
 int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 		struct kvm *kvm, int irq_source_id,
 		int level, bool line_status)
 {
-	return 0;
-}
+	struct kvm_msi msi;
 
-#ifdef CONFIG_HAVE_KVM_MSI
-int kvm_send_userspace_msi(struct kvm *kvm, struct kvm_msi *msi)
-{
-	if (kvm->arch.vgic.vm_ops.inject_msi)
-		return kvm->arch.vgic.vm_ops.inject_msi(kvm, msi);
-	else
-		return -ENODEV;
+	switch (e->type) {
+	case KVM_IRQ_ROUTING_EXTENDED_MSI:
+		msi.address_lo = e->ext_msi.address_lo;
+		msi.address_hi = e->ext_msi.address_hi;
+		msi.data = e->ext_msi.data;
+		msi.flags = e->ext_msi.devid;
+		if (kvm->arch.vgic.vm_ops.inject_msi)
+			return kvm->arch.vgic.vm_ops.inject_msi(kvm, &msi);
+		else
+			return -ENODEV;
+	default:
+		return -EINVAL;
+	}
 }
-#endif
diff --git a/virt/kvm/irqchip.c b/virt/kvm/irqchip.c
index e76c7d2..924a29d 100644
--- a/virt/kvm/irqchip.c
+++ b/virt/kvm/irqchip.c
@@ -29,7 +29,9 @@
 #include <linux/srcu.h>
 #include <linux/export.h>
 #include <trace/events/kvm.h>
+#if !defined(CONFIG_ARM) && !defined(CONFIG_ARM64)
 #include "irq.h"
+#endif
 
 struct kvm_irq_routing_table {
 	int chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
-- 
1.9.1

[RFC 5/6] KVM: arm/arm64: enable MSI routing

[Eric Auger]

Up to now, only irqchip routing entries could be set. This patch
adds the capability to insert MSI routing entries, extended or
standard ones. Although standard MSI entries can be set, their
injection still is not supported. For ARM64, let's also increase
KVM_MAX_IRQ_ROUTES to 4096.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 include/linux/kvm_host.h |  2 ++
 virt/kvm/arm/vgic.c      | 13 +++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h
index e1c1c0d..6cacf11 100644
--- a/include/linux/kvm_host.h
+++ b/include/linux/kvm_host.h
@@ -927,6 +927,8 @@ static inline int mmu_notifier_retry(struct kvm *kvm, unsigned long mmu_seq)
 
 #ifdef CONFIG_S390
 #define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that...
+#elif defined(CONFIG_ARM64)
+#define KVM_MAX_IRQ_ROUTES 4096 //FIXME: we can have more than that too...
 #else
 #define KVM_MAX_IRQ_ROUTES 1024
 #endif
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 212a5ff..16d232f 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2256,6 +2256,19 @@ int kvm_set_routing_entry(struct kvm_kernel_irq_routing_entry *e,
 		    (e->irqchip.irqchip >= KVM_NR_IRQCHIPS))
 			goto out;
 		break;
+	case KVM_IRQ_ROUTING_MSI:
+		e->set = kvm_set_msi;
+		e->msi.address_lo = ue->u.msi.address_lo;
+		e->msi.address_hi = ue->u.msi.address_hi;
+		e->msi.data = ue->u.msi.data;
+		break;
+	case KVM_IRQ_ROUTING_EXTENDED_MSI:
+		e->set = kvm_set_msi;
+		e->ext_msi.address_lo = ue->u.ext_msi.address_lo;
+		e->ext_msi.address_hi = ue->u.ext_msi.address_hi;
+		e->ext_msi.data = ue->u.ext_msi.data;
+		e->ext_msi.devid = ue->u.ext_msi.devid;
+		break;
 	default:
 		goto out;
 	}
-- 
1.9.1

[RFC 6/6] KVM: arm: implement kvm_set_msi by gsi direct mapping

[Eric Auger]

If the ITS modality is not available, let's simply support MSI
injection by transforming the MSI.data into an SPI ID.

This becomes possible to use KVM_SIGNAL_MSI ioctl for arm too.

Signed-off-by: Eric Auger <eric.auger@linaro.org>
---
 arch/arm/kvm/Kconfig | 1 +
 virt/kvm/arm/vgic.c  | 5 +++++
 2 files changed, 6 insertions(+)

diff --git a/arch/arm/kvm/Kconfig b/arch/arm/kvm/Kconfig
index 151e710..0f58baf 100644
--- a/arch/arm/kvm/Kconfig
+++ b/arch/arm/kvm/Kconfig
@@ -31,6 +31,7 @@ config KVM
 	select KVM_VFIO
 	select HAVE_KVM_EVENTFD
 	select HAVE_KVM_IRQFD
+	select HAVE_KVM_MSI
 	select HAVE_KVM_IRQCHIP
 	select HAVE_KVM_IRQ_ROUTING
 	depends on ARM_VIRT_EXT && ARM_LPAE && ARM_ARCH_TIMER
diff --git a/virt/kvm/arm/vgic.c b/virt/kvm/arm/vgic.c
index 16d232f..40e96f9 100644
--- a/virt/kvm/arm/vgic.c
+++ b/virt/kvm/arm/vgic.c
@@ -2293,6 +2293,11 @@ int kvm_set_msi(struct kvm_kernel_irq_routing_entry *e,
 			return kvm->arch.vgic.vm_ops.inject_msi(kvm, &msi);
 		else
 			return -ENODEV;
+	case KVM_IRQ_ROUTING_MSI:
+		if (kvm->arch.vgic.vm_ops.inject_msi)
+			return -EINVAL;
+		else
+			return kvm_vgic_inject_irq(kvm, 0, e->msi.data, level);
 	default:
 		return -EINVAL;
 	}
-- 
1.9.1

Re: [RFC 4/6] KVM: arm/arm64: enable irqchip routing

[Marc Zyngier]

Hi Eric,

On 18/06/15 18:40, Eric Auger wrote:
> This patch adds compilation and link against irqchip.
> 
> On ARM, irqchip routing is not really useful since there is
> a single irqchip. However main motivation behind using irqchip
> code is to enable MSI routing code. With the support of in-kernel
> GICv3 ITS emulation, it now seems to be a MUST HAVE requirement.
> 
> Functions previously implemented in vgic.c and substitute
> to more complex irqchip implementation are removed:
> 
> - kvm_send_userspace_msi
> - kvm_irq_map_chip_pin
> - kvm_set_irq
> - kvm_irq_map_gsi.
> 
> They implemented a kernel default identity GSI routing. This is now
> replaced by user-side provided routing.
> 
> Routing standard hooks are now implemented in vgic:
> - kvm_set_routing_entry
> - kvm_set_irq
> - kvm_set_msi
> 
> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
> 
> MSI routing is not yet allowed.
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  Documentation/virtual/kvm/api.txt | 11 ++++--
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  2 +
>  arch/arm/kvm/Makefile             |  2 +-
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  2 +
>  arch/arm64/kvm/Makefile           |  2 +-
>  include/kvm/arm_vgic.h            |  9 -----
>  virt/kvm/arm/vgic.c               | 78 ++++++++++++++++++++++++---------------
>  virt/kvm/irqchip.c                |  2 +
>  10 files changed, 67 insertions(+), 44 deletions(-)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index bcec91e..2bc96e1 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1395,7 +1395,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>  4.52 KVM_SET_GSI_ROUTING
>  
>  Capability: KVM_CAP_IRQ_ROUTING
> -Architectures: x86 s390
> +Architectures: x86 s390 arm arm64
>  Type: vm ioctl
>  Parameters: struct kvm_irq_routing (in)
>  Returns: 0 on success, -1 on error
> @@ -2310,9 +2310,12 @@ Note that closing the resamplefd is not sufficient to disable the
>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>  
> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
> -given by gsi + 32.
> +On ARM/ARM64, when GSI routing is not used, the gsi field in the
> +kvm_irqfd struct specifies the Shared Peripheral Interrupt (SPI) index,
> +such that the GIC interrupt ID is given by gsi + 32. When GSI routing is
> +setup:
> +- if irqchip routing: irqchip.pin + 32 is the SPI ID that is injected
> +- if MSI routing: the MSI data is used as interrupt ID (SPI or LPI).

This feels just wrong. With GICv3, the MSI data is not the LPI at all.
It is an opaque value that gets translated into an LPI when combined
with the DeviceID.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

Re: [RFC 4/6] KVM: arm/arm64: enable irqchip routing

[Eric Auger]

Hi Marc,
On 06/18/2015 07:53 PM, Marc Zyngier wrote:
> Hi Eric,
> 
> On 18/06/15 18:40, Eric Auger wrote:
>> This patch adds compilation and link against irqchip.
>>
>> On ARM, irqchip routing is not really useful since there is
>> a single irqchip. However main motivation behind using irqchip
>> code is to enable MSI routing code. With the support of in-kernel
>> GICv3 ITS emulation, it now seems to be a MUST HAVE requirement.
>>
>> Functions previously implemented in vgic.c and substitute
>> to more complex irqchip implementation are removed:
>>
>> - kvm_send_userspace_msi
>> - kvm_irq_map_chip_pin
>> - kvm_set_irq
>> - kvm_irq_map_gsi.
>>
>> They implemented a kernel default identity GSI routing. This is now
>> replaced by user-side provided routing.
>>
>> Routing standard hooks are now implemented in vgic:
>> - kvm_set_routing_entry
>> - kvm_set_irq
>> - kvm_set_msi
>>
>> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
>> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
>>
>> MSI routing is not yet allowed.
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  Documentation/virtual/kvm/api.txt | 11 ++++--
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  2 +
>>  arch/arm/kvm/Makefile             |  2 +-
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  2 +
>>  arch/arm64/kvm/Makefile           |  2 +-
>>  include/kvm/arm_vgic.h            |  9 -----
>>  virt/kvm/arm/vgic.c               | 78 ++++++++++++++++++++++++---------------
>>  virt/kvm/irqchip.c                |  2 +
>>  10 files changed, 67 insertions(+), 44 deletions(-)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index bcec91e..2bc96e1 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -1395,7 +1395,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>>  4.52 KVM_SET_GSI_ROUTING
>>  
>>  Capability: KVM_CAP_IRQ_ROUTING
>> -Architectures: x86 s390
>> +Architectures: x86 s390 arm arm64
>>  Type: vm ioctl
>>  Parameters: struct kvm_irq_routing (in)
>>  Returns: 0 on success, -1 on error
>> @@ -2310,9 +2310,12 @@ Note that closing the resamplefd is not sufficient to disable the
>>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>  
>> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
>> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
>> -given by gsi + 32.
>> +On ARM/ARM64, when GSI routing is not used, the gsi field in the
>> +kvm_irqfd struct specifies the Shared Peripheral Interrupt (SPI) index,
>> +such that the GIC interrupt ID is given by gsi + 32. When GSI routing is
>> +setup:
>> +- if irqchip routing: irqchip.pin + 32 is the SPI ID that is injected
>> +- if MSI routing: the MSI data is used as interrupt ID (SPI or LPI).
> 
> This feels just wrong. With GICv3, the MSI data is not the LPI at all.
> It is an opaque value that gets translated into an LPI when combined
> with the DeviceID.
I agree with you. I need to rephrase that. In practice this is what
should happen in the code since I use Andre's MSI injection routine
which does the translation; except for GICv2 where last patch attempts
to do direct gsi mapping from msi msg data!

Thanks

Eric
> 
> Thanks,
> 
> 	M.
> 

Re: [RFC 4/6] KVM: arm/arm64: enable irqchip routing

[Marc Zyngier]

On 18/06/15 19:00, Eric Auger wrote:
> Hi Marc,
> On 06/18/2015 07:53 PM, Marc Zyngier wrote:
>> Hi Eric,
>>
>> On 18/06/15 18:40, Eric Auger wrote:
>>> This patch adds compilation and link against irqchip.
>>>
>>> On ARM, irqchip routing is not really useful since there is
>>> a single irqchip. However main motivation behind using irqchip
>>> code is to enable MSI routing code. With the support of in-kernel
>>> GICv3 ITS emulation, it now seems to be a MUST HAVE requirement.
>>>
>>> Functions previously implemented in vgic.c and substitute
>>> to more complex irqchip implementation are removed:
>>>
>>> - kvm_send_userspace_msi
>>> - kvm_irq_map_chip_pin
>>> - kvm_set_irq
>>> - kvm_irq_map_gsi.
>>>
>>> They implemented a kernel default identity GSI routing. This is now
>>> replaced by user-side provided routing.
>>>
>>> Routing standard hooks are now implemented in vgic:
>>> - kvm_set_routing_entry
>>> - kvm_set_irq
>>> - kvm_set_msi
>>>
>>> Both HAVE_KVM_IRQCHIP and HAVE_KVM_IRQ_ROUTING are defined.
>>> KVM_CAP_IRQ_ROUTING is advertised and KVM_SET_GSI_ROUTING is allowed.
>>>
>>> MSI routing is not yet allowed.
>>>
>>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>>> ---
>>>  Documentation/virtual/kvm/api.txt | 11 ++++--
>>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>>  arch/arm/kvm/Kconfig              |  2 +
>>>  arch/arm/kvm/Makefile             |  2 +-
>>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>>  arch/arm64/kvm/Kconfig            |  2 +
>>>  arch/arm64/kvm/Makefile           |  2 +-
>>>  include/kvm/arm_vgic.h            |  9 -----
>>>  virt/kvm/arm/vgic.c               | 78 ++++++++++++++++++++++++---------------
>>>  virt/kvm/irqchip.c                |  2 +
>>>  10 files changed, 67 insertions(+), 44 deletions(-)
>>>
>>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>>> index bcec91e..2bc96e1 100644
>>> --- a/Documentation/virtual/kvm/api.txt
>>> +++ b/Documentation/virtual/kvm/api.txt
>>> @@ -1395,7 +1395,7 @@ KVM_ASSIGN_DEV_IRQ. Partial deassignment of host or guest IRQ is allowed.
>>>  4.52 KVM_SET_GSI_ROUTING
>>>  
>>>  Capability: KVM_CAP_IRQ_ROUTING
>>> -Architectures: x86 s390
>>> +Architectures: x86 s390 arm arm64
>>>  Type: vm ioctl
>>>  Parameters: struct kvm_irq_routing (in)
>>>  Returns: 0 on success, -1 on error
>>> @@ -2310,9 +2310,12 @@ Note that closing the resamplefd is not sufficient to disable the
>>>  irqfd.  The KVM_IRQFD_FLAG_RESAMPLE is only necessary on assignment
>>>  and need not be specified with KVM_IRQFD_FLAG_DEASSIGN.
>>>  
>>> -On ARM/ARM64, the gsi field in the kvm_irqfd struct specifies the Shared
>>> -Peripheral Interrupt (SPI) index, such that the GIC interrupt ID is
>>> -given by gsi + 32.
>>> +On ARM/ARM64, when GSI routing is not used, the gsi field in the
>>> +kvm_irqfd struct specifies the Shared Peripheral Interrupt (SPI) index,
>>> +such that the GIC interrupt ID is given by gsi + 32. When GSI routing is
>>> +setup:
>>> +- if irqchip routing: irqchip.pin + 32 is the SPI ID that is injected
>>> +- if MSI routing: the MSI data is used as interrupt ID (SPI or LPI).
>>
>> This feels just wrong. With GICv3, the MSI data is not the LPI at all.
>> It is an opaque value that gets translated into an LPI when combined
>> with the DeviceID.
> I agree with you. I need to rephrase that. In practice this is what
> should happen in the code since I use Andre's MSI injection routine
> which does the translation; except for GICv2 where last patch attempts
> to do direct gsi mapping from msi msg data!

Agreed. The code seems to do the right thing, only the documentation is
misleading.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

RE: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

 Hello!

> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
> ioctl along with KVM_IRQFD. If the userspace does not define any routing
> table, no irqfd injection can happen. The user-space can use
> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.

 Yesterday, half-sleeping in the train back home, i've got a simple idea how to resolve
conflicts with existing static GSI->SPI routing without bringing in any more
inconsistencies.
 So far, in current implementation GSI is an SPI index (let alone KVM_IRQ_LINE, because
it's already another story on ARM). In order to maintain this convention we could simply
implement default routing which sets all GSIs to corresponding SPI pins. So, if the
userland never cares about KVM_SET_GSI_ROUTING, everything works as before. But it will be
possible to re-route GSIs to MSI. It will perfectly work because SPI signaling is used
with GICv2m, and MSI with GICv3(+), which cannot be used at the same time.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Pavel,
On 06/19/2015 08:37 AM, Pavel Fedin wrote:
>  Hello!
> 
>> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
>> ioctl along with KVM_IRQFD. If the userspace does not define any routing
>> table, no irqfd injection can happen. The user-space can use
>> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.
> 
>  Yesterday, half-sleeping in the train back home, i've got a simple idea how to resolve
> conflicts with existing static GSI->SPI routing without bringing in any more
> inconsistencies.
>  So far, in current implementation GSI is an SPI index (let alone KVM_IRQ_LINE, because
> it's already another story on ARM). In order to maintain this convention we could simply
> implement default routing which sets all GSIs to corresponding SPI pins. So, if the
> userland never cares about KVM_SET_GSI_ROUTING, everything works as before. But it will be
> possible to re-route GSIs to MSI. It will perfectly work because SPI signaling is used
> with GICv2m, and MSI with GICv3(+), which cannot be used at the same time.
I agree with you and I suggested the same approach in my cover letter.
Since applying GSI routing to KVM_IRQ_LINE is quite problematic, I would
be also in favour to forbid userspace GSI routing setting and implement
it kernel-side. Userspace would only be allowed to define MSI routing
entries.

I will respin accordingly and validate it further with qemu.

Best Regards

Eric
> 
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
> 
> 

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Andre Przywara]

Hi Eric,

I briefly looked over the series, the patches itself look good overall.
I have one or two comments on the actual code, but want to discuss the
general approach first (more a dump of some first thoughts):

On 18/06/15 18:40, Eric Auger wrote:
> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
> appears to be requested. More specifically MSI routing is needed.
> irqchip routing does not sound to be really useful on arm but usage of
> MSI routing also mandates to integrate irqchip routing. The initial
> implementation of irqfd on arm must be upgraded with the integration
> of kvm irqchip.c code and the implementation of its standard hooks
> in the architecture specific part.
> 
> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
> ioctl along with KVM_IRQFD. If the userspace does not define any routing
> table, no irqfd injection can happen. The user-space can use
> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.
> 
> for irqchip routing, the convention is, only SPI can be injected and the
> SPI ID corresponds to irqchip.pin + 32. For MSI routing the interrupt ID
> matches the MSI msg data. API evolve to support associating a device ID
> to a routine entry.

So if I get this right, in a guest ITS case we have now three different
IRQ name spaces:
a) the LPI number, which is guest internal. The ITS driver in the guest
maintains it. We can track assignments and changes when handling the
MAPVI command in the host kernel, but this would stay in the kernel, as
I don't see an efficient way of propagating this to userland.
b) the GSI number, which is used in communication between userland and
the host kernel. The guest kernel does not know about this at all. Also
the ioctl requires us to set the routing for _all_ GSIs, and I read it
that it assumes starting at GSI 0. So we cannot even pretend to have
LPIs here, because we would need at least 8192 empty entries then, not
to speak of the possibly sparse allocation above. So we have a
completely distinct name space here.
c) The DevID:EvID pair, which actually identifies an IRQ in all the
three regimes and is the only authoritative ID.

So that means we need to maintain the connection between all the three,
somehow duplicating the whole ITS mapping again to map GSIs to DevID:EvID.

So I wonder if we could use DevID:EvID directly.
The KVM_IRQFD ioctl struct has some space, so we could put the DevID
into the pad area.
Also (more forward-looking) KVM_CAP_ASSIGN_DEV_IRQ identifies guest IRQs
by an u32, but again there is quite some padding area available.

In general I am a bit reluctant to introduce just another level of
complexity to the already quite convoluted way of doing IRQs and MSIs on
ARM(64), that's why I will investigate if we can use DevID:EvID to refer
to an interrupt.

So far,
Andre.

> 
> Known Issues of this RFC:
> 
> - One of the biggest is the API inconsistencies on ARM. Blame me.
>   Routing should apply to KVM_IRQ_LINE ioctl which is not the case yet
>   in this series. It only applies to irqfd.
>   on x86 typically this KVM_IRQ_LINE is plugged onto irqchip.c kvm_set_irq
>   whereas on ARM we inject directly through kvm_vgic_inject_irq
>   x on arm/arm64 gsi has a specific structure:
>     bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
>     field: | irq_type  | vcpu_index |     irq_id     |
>     where irq_id matches the Interrupt ID
> - for KVM_IRQFD without routing (current implementation) the gsi field
>   corresponds to an SPI index = irq_id (above) -32.
> - as far as understand qemu integration, gsi is supposed to be within
>   [0, KVM_MAX_IRQ_ROUTES]. Difficult to use KVM_IRQ_LINE gsi.
> - to be defined what we choose as a convention with irqchip routing is
>   applied: gsi -> irqchip input pin.
> - Or shouldn't we simply rule out any userspace irqchip routing and stick
>   to MSI routing? we could define a fixed identity in-kernel irqchip mapping
>   and only offer MSI routing.
> - static allocation of chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
>   arbitrary put KVM_IRQCHIP_NUM_PINS = 1020 - 32 (SPI count). On s390
>   this is even bigger.
> 
> Currently tested on irqchip routing only (Calxeda midway only),
> ie NOT TESTED on MSI routing yet.
> 
> This is a very preliminary RFC to ease the discussion.
> 
> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc8-gsi-routing-rfc
> 
> It applies on Andre's [PATCH 00/13] arm64: KVM: GICv3 ITS emulation
> (http://www.spinics.net/lists/kvm/msg117402.html)
> 
> Eric Auger (6):
>   KVM: api: add kvm_irq_routing_extended_msi
>   KVM: kvm_host: add kvm_extended_msi
>   KVM: irqchip: convey devid to kvm_set_msi
>   KVM: arm/arm64: enable irqchip routing
>   KVM: arm/arm64: enable MSI routing
>   KVM: arm: implement kvm_set_msi by gsi direct mapping
> 
>  Documentation/virtual/kvm/api.txt | 20 ++++++--
>  arch/arm/include/asm/kvm_host.h   |  2 +
>  arch/arm/kvm/Kconfig              |  3 ++
>  arch/arm/kvm/Makefile             |  2 +-
>  arch/arm64/include/asm/kvm_host.h |  1 +
>  arch/arm64/kvm/Kconfig            |  2 +
>  arch/arm64/kvm/Makefile           |  2 +-
>  include/kvm/arm_vgic.h            |  9 ----
>  include/linux/kvm_host.h          | 10 ++++
>  include/uapi/linux/kvm.h          |  9 ++++
>  virt/kvm/arm/vgic.c               | 96 +++++++++++++++++++++++++++------------
>  virt/kvm/irqchip.c                | 20 ++++++--
>  12 files changed, 128 insertions(+), 48 deletions(-)
> 

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

On 06/22/2015 10:40 AM, Andre Przywara wrote:
> Hi Eric,
> 
> I briefly looked over the series, the patches itself look good overall.
> I have one or two comments on the actual code, but want to discuss the
> general approach first (more a dump of some first thoughts):
> 
> On 18/06/15 18:40, Eric Auger wrote:
>> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
>> appears to be requested. More specifically MSI routing is needed.
>> irqchip routing does not sound to be really useful on arm but usage of
>> MSI routing also mandates to integrate irqchip routing. The initial
>> implementation of irqfd on arm must be upgraded with the integration
>> of kvm irqchip.c code and the implementation of its standard hooks
>> in the architecture specific part.
>>
>> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
>> ioctl along with KVM_IRQFD. If the userspace does not define any routing
>> table, no irqfd injection can happen. The user-space can use
>> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.
>>
>> for irqchip routing, the convention is, only SPI can be injected and the
>> SPI ID corresponds to irqchip.pin + 32. For MSI routing the interrupt ID
>> matches the MSI msg data. API evolve to support associating a device ID
>> to a routine entry.
> 
> So if I get this right, in a guest ITS case we have now three different
> IRQ name spaces:
> a) the LPI number, which is guest internal. The ITS driver in the guest
> maintains it. We can track assignments and changes when handling the
> MAPVI command in the host kernel, but this would stay in the kernel, as
> I don't see an efficient way of propagating this to userland.
> b) the GSI number, which is used in communication between userland and
> the host kernel. The guest kernel does not know about this at all. Also
> the ioctl requires us to set the routing for _all_ GSIs, and I read it
> that it assumes starting at GSI 0.
all injected GSI must effectively have a routing entry in KVM. Starting
at 0 that's not requested. At qemu level there's just the constaint gsi
fits between [0, max route number].
 So we cannot even pretend to have
> LPIs here, because we would need at least 8192 empty entries then, not
> to speak of the possibly sparse allocation above. So we have a
> completely distinct name space here.
What is done currently at qemu level for other archs -  if I understand
it correctly -  is there is static GSI routing for standard IRQ. For MSI
irqfd setup they use spare gsi number not yet used for GSI routing < max
route number. So this is sparse for MSI but not for standard IRQs.
Effectively we do not plan to have GSI routing for LPIs but only MSI
routing.
> c) The DevID:EvID pair, which actually identifies an IRQ in all the
> three regimes and is the only authoritative ID.
> 
> So that means we need to maintain the connection between all the three,
> somehow duplicating the whole ITS mapping again to map GSIs to DevID:EvID.
Currently the KVM routing table indeed stores GSI/DevID:EvID mapping.
> 
> So I wonder if we could use DevID:EvID directly.
> The KVM_IRQFD ioctl struct has some space, so we could put the DevID
> into the pad area.
> Also (more forward-looking) KVM_CAP_ASSIGN_DEV_IRQ identifies guest IRQs
> by an u32, but again there is quite some padding area available.
ASSIGN_DEV_IRQ is a deprecated feature. We should not use that API I think.

Eric
> 
> In general I am a bit reluctant to introduce just another level of
> complexity to the already quite convoluted way of doing IRQs and MSIs on
> ARM(64), that's why I will investigate if we can use DevID:EvID to refer
> to an interrupt.
> 
> So far,
> Andre.
> 
>>
>> Known Issues of this RFC:
>>
>> - One of the biggest is the API inconsistencies on ARM. Blame me.
>>   Routing should apply to KVM_IRQ_LINE ioctl which is not the case yet
>>   in this series. It only applies to irqfd.
>>   on x86 typically this KVM_IRQ_LINE is plugged onto irqchip.c kvm_set_irq
>>   whereas on ARM we inject directly through kvm_vgic_inject_irq
>>   x on arm/arm64 gsi has a specific structure:
>>     bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
>>     field: | irq_type  | vcpu_index |     irq_id     |
>>     where irq_id matches the Interrupt ID
>> - for KVM_IRQFD without routing (current implementation) the gsi field
>>   corresponds to an SPI index = irq_id (above) -32.
>> - as far as understand qemu integration, gsi is supposed to be within
>>   [0, KVM_MAX_IRQ_ROUTES]. Difficult to use KVM_IRQ_LINE gsi.
>> - to be defined what we choose as a convention with irqchip routing is
>>   applied: gsi -> irqchip input pin.
>> - Or shouldn't we simply rule out any userspace irqchip routing and stick
>>   to MSI routing? we could define a fixed identity in-kernel irqchip mapping
>>   and only offer MSI routing.
>> - static allocation of chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
>>   arbitrary put KVM_IRQCHIP_NUM_PINS = 1020 - 32 (SPI count). On s390
>>   this is even bigger.
>>
>> Currently tested on irqchip routing only (Calxeda midway only),
>> ie NOT TESTED on MSI routing yet.
>>
>> This is a very preliminary RFC to ease the discussion.
>>
>> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc8-gsi-routing-rfc
>>
>> It applies on Andre's [PATCH 00/13] arm64: KVM: GICv3 ITS emulation
>> (http://www.spinics.net/lists/kvm/msg117402.html)
>>
>> Eric Auger (6):
>>   KVM: api: add kvm_irq_routing_extended_msi
>>   KVM: kvm_host: add kvm_extended_msi
>>   KVM: irqchip: convey devid to kvm_set_msi
>>   KVM: arm/arm64: enable irqchip routing
>>   KVM: arm/arm64: enable MSI routing
>>   KVM: arm: implement kvm_set_msi by gsi direct mapping
>>
>>  Documentation/virtual/kvm/api.txt | 20 ++++++--
>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>  arch/arm/kvm/Kconfig              |  3 ++
>>  arch/arm/kvm/Makefile             |  2 +-
>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>  arch/arm64/kvm/Kconfig            |  2 +
>>  arch/arm64/kvm/Makefile           |  2 +-
>>  include/kvm/arm_vgic.h            |  9 ----
>>  include/linux/kvm_host.h          | 10 ++++
>>  include/uapi/linux/kvm.h          |  9 ++++
>>  virt/kvm/arm/vgic.c               | 96 +++++++++++++++++++++++++++------------
>>  virt/kvm/irqchip.c                | 20 ++++++--
>>  12 files changed, 128 insertions(+), 48 deletions(-)
>>

--
To unsubscribe from this list: send the line "unsubscribe kvm" in

Re: [RFC 1/6] KVM: api: add kvm_irq_routing_extended_msi

[Andre Przywara]

Hi Eric,

On 18/06/15 18:40, Eric Auger wrote:
> On ARM, the MSI msg (address and data) comes along with
> out-of-band device ID information. The device ID encodes the device
> that composes the MSI msg. Let's create a new routing entry structure
> that enables to encode that information on top of standard MSI
> message
> 
> Signed-off-by: Eric Auger <eric.auger@linaro.org>
> ---
>  Documentation/virtual/kvm/api.txt | 9 +++++++++
>  include/uapi/linux/kvm.h          | 9 +++++++++
>  2 files changed, 18 insertions(+)
> 
> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
> index d20fd94..bcec91e 100644
> --- a/Documentation/virtual/kvm/api.txt
> +++ b/Documentation/virtual/kvm/api.txt
> @@ -1419,6 +1419,7 @@ struct kvm_irq_routing_entry {
>  		struct kvm_irq_routing_irqchip irqchip;
>  		struct kvm_irq_routing_msi msi;
>  		struct kvm_irq_routing_s390_adapter adapter;
> +		struct kvm_irq_routing_extended_msi ext_msi;
>  		__u32 pad[8];
>  	} u;
>  };
> @@ -1427,6 +1428,7 @@ struct kvm_irq_routing_entry {
>  #define KVM_IRQ_ROUTING_IRQCHIP 1
>  #define KVM_IRQ_ROUTING_MSI 2
>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
> +#define KVM_IRQ_ROUTING_EXTENDED_MSI 4
>  
>  No flags are specified so far, the corresponding field must be set to zero.
>  
> @@ -1442,6 +1444,13 @@ struct kvm_irq_routing_msi {
>  	__u32 pad;
>  };
>  
> +struct kvm_irq_routing_extended_msi {
> +	__u32 address_lo;
> +	__u32 address_hi;
> +	__u32 data;
> +	__u32 devid;
> +};
> +

I wonder if we could re-use the existing struct kvm_irq_routing_msi,
which has an u32 pad field already. Since we use a different type
number, this should not break. Admittedly not the nicest thing, but
reduces interface bloat:

struct kvm_irq_routing_msi {
	__u32 address_lo;
	__u32 address_hi;
	__u32 data;
	union {
		__u32 pad;
		__u32 devid;
	};
};

to maintain backward compatibility on the userland source level.

Cheers,
Andre.

>  struct kvm_irq_routing_s390_adapter {
>  	__u64 ind_addr;
>  	__u64 summary_addr;
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 2a23705..e3f65a0 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -829,6 +829,13 @@ struct kvm_irq_routing_msi {
>  	__u32 pad;
>  };
>  
> +struct kvm_irq_routing_extended_msi {
> +	__u32 address_lo;
> +	__u32 address_hi;
> +	__u32 data;
> +	__u32 devid;
> +};
> +
>  struct kvm_irq_routing_s390_adapter {
>  	__u64 ind_addr;
>  	__u64 summary_addr;
> @@ -841,6 +848,7 @@ struct kvm_irq_routing_s390_adapter {
>  #define KVM_IRQ_ROUTING_IRQCHIP 1
>  #define KVM_IRQ_ROUTING_MSI 2
>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
> +#define KVM_IRQ_ROUTING_EXTENDED_MSI 4
>  
>  struct kvm_irq_routing_entry {
>  	__u32 gsi;
> @@ -851,6 +859,7 @@ struct kvm_irq_routing_entry {
>  		struct kvm_irq_routing_irqchip irqchip;
>  		struct kvm_irq_routing_msi msi;
>  		struct kvm_irq_routing_s390_adapter adapter;
> +		struct kvm_irq_routing_extended_msi ext_msi;
>  		__u32 pad[8];
>  	} u;
>  };
> 
--
To unsubscribe from this list: send the line "unsubscribe kvm" in

Re: [RFC 1/6] KVM: api: add kvm_irq_routing_extended_msi

[Eric Auger]

Hi André,
On 06/22/2015 06:32 PM, Andre Przywara wrote:
> Hi Eric,
> 
> On 18/06/15 18:40, Eric Auger wrote:
>> On ARM, the MSI msg (address and data) comes along with
>> out-of-band device ID information. The device ID encodes the device
>> that composes the MSI msg. Let's create a new routing entry structure
>> that enables to encode that information on top of standard MSI
>> message
>>
>> Signed-off-by: Eric Auger <eric.auger@linaro.org>
>> ---
>>  Documentation/virtual/kvm/api.txt | 9 +++++++++
>>  include/uapi/linux/kvm.h          | 9 +++++++++
>>  2 files changed, 18 insertions(+)
>>
>> diff --git a/Documentation/virtual/kvm/api.txt b/Documentation/virtual/kvm/api.txt
>> index d20fd94..bcec91e 100644
>> --- a/Documentation/virtual/kvm/api.txt
>> +++ b/Documentation/virtual/kvm/api.txt
>> @@ -1419,6 +1419,7 @@ struct kvm_irq_routing_entry {
>>  		struct kvm_irq_routing_irqchip irqchip;
>>  		struct kvm_irq_routing_msi msi;
>>  		struct kvm_irq_routing_s390_adapter adapter;
>> +		struct kvm_irq_routing_extended_msi ext_msi;
>>  		__u32 pad[8];
>>  	} u;
>>  };
>> @@ -1427,6 +1428,7 @@ struct kvm_irq_routing_entry {
>>  #define KVM_IRQ_ROUTING_IRQCHIP 1
>>  #define KVM_IRQ_ROUTING_MSI 2
>>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
>> +#define KVM_IRQ_ROUTING_EXTENDED_MSI 4
>>  
>>  No flags are specified so far, the corresponding field must be set to zero.
>>  
>> @@ -1442,6 +1444,13 @@ struct kvm_irq_routing_msi {
>>  	__u32 pad;
>>  };
>>  
>> +struct kvm_irq_routing_extended_msi {
>> +	__u32 address_lo;
>> +	__u32 address_hi;
>> +	__u32 data;
>> +	__u32 devid;
>> +};
>> +
> 
> I wonder if we could re-use the existing struct kvm_irq_routing_msi,
> which has an u32 pad field already. Since we use a different type
> number, this should not break. Admittedly not the nicest thing, but
> reduces interface bloat:
> 
> struct kvm_irq_routing_msi {
> 	__u32 address_lo;
> 	__u32 address_hi;
> 	__u32 data;
> 	union {
> 		__u32 pad;
> 		__u32 devid;
> 	};
> };
> 
> to maintain backward compatibility on the userland source level.
I also envisioned that solution. I eventually chose adding a new struct,
for sake of homogeneity since the fellow kvm struct does not have such a
padding field: struct kvm_kernel_irq_routing_entry (kvm_host.h).

But I don't have a strong preference so if it is a general preference I
will follow, of course.

Thanks

Eric


> 
> Cheers,
> Andre.
> 
>>  struct kvm_irq_routing_s390_adapter {
>>  	__u64 ind_addr;
>>  	__u64 summary_addr;
>> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
>> index 2a23705..e3f65a0 100644
>> --- a/include/uapi/linux/kvm.h
>> +++ b/include/uapi/linux/kvm.h
>> @@ -829,6 +829,13 @@ struct kvm_irq_routing_msi {
>>  	__u32 pad;
>>  };
>>  
>> +struct kvm_irq_routing_extended_msi {
>> +	__u32 address_lo;
>> +	__u32 address_hi;
>> +	__u32 data;
>> +	__u32 devid;
>> +};
>> +
>>  struct kvm_irq_routing_s390_adapter {
>>  	__u64 ind_addr;
>>  	__u64 summary_addr;
>> @@ -841,6 +848,7 @@ struct kvm_irq_routing_s390_adapter {
>>  #define KVM_IRQ_ROUTING_IRQCHIP 1
>>  #define KVM_IRQ_ROUTING_MSI 2
>>  #define KVM_IRQ_ROUTING_S390_ADAPTER 3
>> +#define KVM_IRQ_ROUTING_EXTENDED_MSI 4
>>  
>>  struct kvm_irq_routing_entry {
>>  	__u32 gsi;
>> @@ -851,6 +859,7 @@ struct kvm_irq_routing_entry {
>>  		struct kvm_irq_routing_irqchip irqchip;
>>  		struct kvm_irq_routing_msi msi;
>>  		struct kvm_irq_routing_s390_adapter adapter;
>> +		struct kvm_irq_routing_extended_msi ext_msi;
>>  		__u32 pad[8];
>>  	} u;
>>  };
>>

--
To unsubscribe from this list: send the line "unsubscribe kvm" in

RE: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

 Hello!

> What is done currently at qemu level for other archs -  if I understand
> it correctly -  is there is static GSI routing for standard IRQ.

 Yes, this is correct. GSI number = SPI number for ARM. Also something similar for MIPS (haven't
study that).

> For MSI
> irqfd setup they use spare gsi number not yet used for GSI routing < max
> route number.

 No.
 On ARM with vGICv2m MSI data is effectively IRQ offset, with base being 32. So, original convention
works without any change, they simply retrieve SPI number from MSI data and set up irqfd to that
SPI. No allocation is done.
 Allocation is done only on x86, with routing. They just allocate GSI numbers starting from zero and
route them to something (either irqchip or MSI).

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia


--
To unsubscribe from this list: send the line "unsubscribe kvm" in

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Pavel,
On 06/23/2015 09:38 AM, Pavel Fedin wrote:
>  Hello!
> 
>> What is done currently at qemu level for other archs -  if I understand
>> it correctly -  is there is static GSI routing for standard IRQ.
> 
>  Yes, this is correct. GSI number = SPI number for ARM. Also something similar for MIPS (haven't
> study that).
> 
>> For MSI
>> irqfd setup they use spare gsi number not yet used for GSI routing < max
>> route number.
> 
>  No.
>  On ARM with vGICv2m MSI data is effectively IRQ offset, with base being 32. So, original convention
> works without any change, they simply retrieve SPI number from MSI data and set up irqfd to that
> SPI. No allocation is done. 
Yes on ARM we use gsi direct mapping feature. I was talking about what
is done without this "gsi direct mapping" feature. In that case the low
GSI numbers are used by irqchip routing entries. So kvm_irqchip_get_virq
returns a gsi beyond the irqchip routed gsi's.

Eric
>  Allocation is done only on x86, with routing. They just allocate GSI numbers starting from zero and
> route them to something (either irqchip or MSI).
> 
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
> 
> 

RE: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

 Hello!

> I was talking about what
> is done without this "gsi direct mapping" feature. In that case the low
> GSI numbers are used by irqchip routing entries. So kvm_irqchip_get_virq
> returns a gsi beyond the irqchip routed gsi's.

 But aren't these low numbers just routed by the qemu too? I don't see qemu getting any predefined
routing from the kernel. kvm_init_irq_routing() allocates used_gsi_bitmap as all zeros, and
kvm_irqchip_get_virq() just starts to give those free numbers, starting from zero. I believe during
machine init we allocate virq's and route them to irqchip explicitly, don't we ?
 Sorry for being perhaps ignorant, i haven't studied x86 qemu code too much.

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Andre Przywara]

Hi Eric,

I went back reading the code and looked at how the x86 APIC works more
closely to understand the GSI routing better.
See below for more ...

On 22/06/15 10:21, Eric Auger wrote:
> On 06/22/2015 10:40 AM, Andre Przywara wrote:
>> Hi Eric,
>>
>> I briefly looked over the series, the patches itself look good overall.
>> I have one or two comments on the actual code, but want to discuss the
>> general approach first (more a dump of some first thoughts):
>>
>> On 18/06/15 18:40, Eric Auger wrote:
>>> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
>>> appears to be requested. More specifically MSI routing is needed.
>>> irqchip routing does not sound to be really useful on arm but usage of
>>> MSI routing also mandates to integrate irqchip routing. The initial
>>> implementation of irqfd on arm must be upgraded with the integration
>>> of kvm irqchip.c code and the implementation of its standard hooks
>>> in the architecture specific part.
>>>
>>> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
>>> ioctl along with KVM_IRQFD. If the userspace does not define any routing
>>> table, no irqfd injection can happen. The user-space can use
>>> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.
>>>
>>> for irqchip routing, the convention is, only SPI can be injected and the
>>> SPI ID corresponds to irqchip.pin + 32. For MSI routing the interrupt ID
>>> matches the MSI msg data. API evolve to support associating a device ID
>>> to a routine entry.
>>
>> So if I get this right, in a guest ITS case we have now three different
>> IRQ name spaces:
>> a) the LPI number, which is guest internal. The ITS driver in the guest
>> maintains it. We can track assignments and changes when handling the
>> MAPVI command in the host kernel, but this would stay in the kernel, as
>> I don't see an efficient way of propagating this to userland.
>> b) the GSI number, which is used in communication between userland and
>> the host kernel. The guest kernel does not know about this at all. Also
>> the ioctl requires us to set the routing for _all_ GSIs, and I read it
>> that it assumes starting at GSI 0.
> all injected GSI must effectively have a routing entry in KVM. Starting
> at 0 that's not requested. At qemu level there's just the constaint gsi
> fits between [0, max route number].

Yeah, you are right, I somehow missed that each routing entry has a gsi
field in it. So we have to allocate all of them at once with one ioctl,
but they can be sparse.

>  So we cannot even pretend to have
>> LPIs here, because we would need at least 8192 empty entries then, not
>> to speak of the possibly sparse allocation above. So we have a
>> completely distinct name space here.
> What is done currently at qemu level for other archs -  if I understand
> it correctly -  is there is static GSI routing for standard IRQ. For MSI
> irqfd setup they use spare gsi number not yet used for GSI routing < max
> route number. So this is sparse for MSI but not for standard IRQs.
> Effectively we do not plan to have GSI routing for LPIs but only MSI
> routing.

That seems to make sense to me. Since we already limit the number of
SPIs to something sensible with our KVM_DEV_ARM_VGIC_GRP_NR_IRQS, we
could infer an implicit direct routing for those SPIs. KVM could check
the IRQ number against vgic.nr_irqs to see whether an IRQ is routed or not.
Any GSI beyond that number would be an MSI with your enhanced DevID:EvID
pair in it, which gets injected via the ITS emulation code (or the
respective GICv2m code).

That would be the idea, but if it turns out that not routing SPIs but
only MSIs requires too many changes to the (core) KVM code (haven't
looked yet), we could require routing entries for SPIs as well.
After all that's what for instance kvmtool sets up for x86, creating
default 1:1 mappings for ISA and low APIC IRQs and allocating MSIs on
demand after that.

>> c) The DevID:EvID pair, which actually identifies an IRQ in all the
>> three regimes and is the only authoritative ID.
>>
>> So that means we need to maintain the connection between all the three,
>> somehow duplicating the whole ITS mapping again to map GSIs to DevID:EvID.
> Currently the KVM routing table indeed stores GSI/DevID:EvID mapping.
>>
>> So I wonder if we could use DevID:EvID directly.
>> The KVM_IRQFD ioctl struct has some space, so we could put the DevID
>> into the pad area.
>> Also (more forward-looking) KVM_CAP_ASSIGN_DEV_IRQ identifies guest IRQs
>> by an u32, but again there is quite some padding area available.

> ASSIGN_DEV_IRQ is a deprecated feature. We should not use that API I think.

OK, so do we have other users of the GSI routing beside IRQFD then?

I will go ahead and try to implement some code matching Eric's patches
in kvmtool to test the GSI routing.

Eric, how did you test the irqchip routing on the Midway?

Cheers,
Andre.

> Eric
>>
>> In general I am a bit reluctant to introduce just another level of
>> complexity to the already quite convoluted way of doing IRQs and MSIs on
>> ARM(64), that's why I will investigate if we can use DevID:EvID to refer
>> to an interrupt.
>>
>> So far,
>> Andre.
>>
>>>
>>> Known Issues of this RFC:
>>>
>>> - One of the biggest is the API inconsistencies on ARM. Blame me.
>>>   Routing should apply to KVM_IRQ_LINE ioctl which is not the case yet
>>>   in this series. It only applies to irqfd.
>>>   on x86 typically this KVM_IRQ_LINE is plugged onto irqchip.c kvm_set_irq
>>>   whereas on ARM we inject directly through kvm_vgic_inject_irq
>>>   x on arm/arm64 gsi has a specific structure:
>>>     bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
>>>     field: | irq_type  | vcpu_index |     irq_id     |
>>>     where irq_id matches the Interrupt ID
>>> - for KVM_IRQFD without routing (current implementation) the gsi field
>>>   corresponds to an SPI index = irq_id (above) -32.
>>> - as far as understand qemu integration, gsi is supposed to be within
>>>   [0, KVM_MAX_IRQ_ROUTES]. Difficult to use KVM_IRQ_LINE gsi.
>>> - to be defined what we choose as a convention with irqchip routing is
>>>   applied: gsi -> irqchip input pin.
>>> - Or shouldn't we simply rule out any userspace irqchip routing and stick
>>>   to MSI routing? we could define a fixed identity in-kernel irqchip mapping
>>>   and only offer MSI routing.
>>> - static allocation of chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
>>>   arbitrary put KVM_IRQCHIP_NUM_PINS = 1020 - 32 (SPI count). On s390
>>>   this is even bigger.
>>>
>>> Currently tested on irqchip routing only (Calxeda midway only),
>>> ie NOT TESTED on MSI routing yet.
>>>
>>> This is a very preliminary RFC to ease the discussion.
>>>
>>> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc8-gsi-routing-rfc
>>>
>>> It applies on Andre's [PATCH 00/13] arm64: KVM: GICv3 ITS emulation
>>> (http://www.spinics.net/lists/kvm/msg117402.html)
>>>
>>> Eric Auger (6):
>>>   KVM: api: add kvm_irq_routing_extended_msi
>>>   KVM: kvm_host: add kvm_extended_msi
>>>   KVM: irqchip: convey devid to kvm_set_msi
>>>   KVM: arm/arm64: enable irqchip routing
>>>   KVM: arm/arm64: enable MSI routing
>>>   KVM: arm: implement kvm_set_msi by gsi direct mapping
>>>
>>>  Documentation/virtual/kvm/api.txt | 20 ++++++--
>>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>>  arch/arm/kvm/Kconfig              |  3 ++
>>>  arch/arm/kvm/Makefile             |  2 +-
>>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>>  arch/arm64/kvm/Kconfig            |  2 +
>>>  arch/arm64/kvm/Makefile           |  2 +-
>>>  include/kvm/arm_vgic.h            |  9 ----
>>>  include/linux/kvm_host.h          | 10 ++++
>>>  include/uapi/linux/kvm.h          |  9 ++++
>>>  virt/kvm/arm/vgic.c               | 96 +++++++++++++++++++++++++++------------
>>>  virt/kvm/irqchip.c                | 20 ++++++--
>>>  12 files changed, 128 insertions(+), 48 deletions(-)
>>>
> 

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Salut Andre,
On 06/23/2015 11:03 AM, Andre Przywara wrote:
> Hi Eric,
> 
> I went back reading the code and looked at how the x86 APIC works more
> closely to understand the GSI routing better.
> See below for more ...
> 
> On 22/06/15 10:21, Eric Auger wrote:
>> On 06/22/2015 10:40 AM, Andre Przywara wrote:
>>> Hi Eric,
>>>
>>> I briefly looked over the series, the patches itself look good overall.
>>> I have one or two comments on the actual code, but want to discuss the
>>> general approach first (more a dump of some first thoughts):
>>>
>>> On 18/06/15 18:40, Eric Auger wrote:
>>>> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
>>>> appears to be requested. More specifically MSI routing is needed.
>>>> irqchip routing does not sound to be really useful on arm but usage of
>>>> MSI routing also mandates to integrate irqchip routing. The initial
>>>> implementation of irqfd on arm must be upgraded with the integration
>>>> of kvm irqchip.c code and the implementation of its standard hooks
>>>> in the architecture specific part.
>>>>
>>>> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
>>>> ioctl along with KVM_IRQFD. If the userspace does not define any routing
>>>> table, no irqfd injection can happen. The user-space can use
>>>> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.
>>>>
>>>> for irqchip routing, the convention is, only SPI can be injected and the
>>>> SPI ID corresponds to irqchip.pin + 32. For MSI routing the interrupt ID
>>>> matches the MSI msg data. API evolve to support associating a device ID
>>>> to a routine entry.
>>>
>>> So if I get this right, in a guest ITS case we have now three different
>>> IRQ name spaces:
>>> a) the LPI number, which is guest internal. The ITS driver in the guest
>>> maintains it. We can track assignments and changes when handling the
>>> MAPVI command in the host kernel, but this would stay in the kernel, as
>>> I don't see an efficient way of propagating this to userland.
>>> b) the GSI number, which is used in communication between userland and
>>> the host kernel. The guest kernel does not know about this at all. Also
>>> the ioctl requires us to set the routing for _all_ GSIs, and I read it
>>> that it assumes starting at GSI 0.
>> all injected GSI must effectively have a routing entry in KVM. Starting
>> at 0 that's not requested. At qemu level there's just the constaint gsi
>> fits between [0, max route number].
> 
> Yeah, you are right, I somehow missed that each routing entry has a gsi
> field in it. So we have to allocate all of them at once with one ioctl,
> but they can be sparse.
> 
>>  So we cannot even pretend to have
>>> LPIs here, because we would need at least 8192 empty entries then, not
>>> to speak of the possibly sparse allocation above. So we have a
>>> completely distinct name space here.
>> What is done currently at qemu level for other archs -  if I understand
>> it correctly -  is there is static GSI routing for standard IRQ. For MSI
>> irqfd setup they use spare gsi number not yet used for GSI routing < max
>> route number. So this is sparse for MSI but not for standard IRQs.
>> Effectively we do not plan to have GSI routing for LPIs but only MSI
>> routing.
> 
> That seems to make sense to me. Since we already limit the number of
> SPIs to something sensible with our KVM_DEV_ARM_VGIC_GRP_NR_IRQS, we
> could infer an implicit direct routing for those SPIs. KVM could check
> the IRQ number against vgic.nr_irqs to see whether an IRQ is routed or not.
> Any GSI beyond that number would be an MSI with your enhanced DevID:EvID
> pair in it, which gets injected via the ITS emulation code (or the
> respective GICv2m code).
> 
> That would be the idea, but if it turns out that not routing SPIs but
> only MSIs requires too many changes to the (core) KVM code (haven't
> looked yet),

I am currently prototyping that. Maybe this is not so much change in the
core. I am attempting to build a default irqchip routing table and just
allow userspace to add MSI entries on top of those. This should be the
topic of RFC v2, I think this week.
 we could require routing entries for SPIs as well.
> After all that's what for instance kvmtool sets up for x86, creating
> default 1:1 mappings for ISA and low APIC IRQs and allocating MSIs on
> demand after that.
> 
>>> c) The DevID:EvID pair, which actually identifies an IRQ in all the
>>> three regimes and is the only authoritative ID.
>>>
>>> So that means we need to maintain the connection between all the three,
>>> somehow duplicating the whole ITS mapping again to map GSIs to DevID:EvID.
>> Currently the KVM routing table indeed stores GSI/DevID:EvID mapping.
>>>
>>> So I wonder if we could use DevID:EvID directly.
>>> The KVM_IRQFD ioctl struct has some space, so we could put the DevID
>>> into the pad area.
>>> Also (more forward-looking) KVM_CAP_ASSIGN_DEV_IRQ identifies guest IRQs
>>> by an u32, but again there is quite some padding area available.
> 
>> ASSIGN_DEV_IRQ is a deprecated feature. We should not use that API I think.
> 
> OK, so do we have other users of the GSI routing beside IRQFD then?
Well KVM_IRQ_LINE as well. That's why I would like to unplug userspace
irqchip routing.
> 
> I will go ahead and try to implement some code matching Eric's patches
> in kvmtool to test the GSI routing.
> 
> Eric, how did you test the irqchip routing on the Midway?
I used xgmac passthrough on Midway.
> 
> Cheers,
> Andre.
> 
>> Eric
>>>
>>> In general I am a bit reluctant to introduce just another level of
>>> complexity to the already quite convoluted way of doing IRQs and MSIs on
>>> ARM(64), that's why I will investigate if we can use DevID:EvID to refer
>>> to an interrupt.
>>>
>>> So far,
>>> Andre.
>>>
>>>>
>>>> Known Issues of this RFC:
>>>>
>>>> - One of the biggest is the API inconsistencies on ARM. Blame me.
>>>>   Routing should apply to KVM_IRQ_LINE ioctl which is not the case yet
>>>>   in this series. It only applies to irqfd.
>>>>   on x86 typically this KVM_IRQ_LINE is plugged onto irqchip.c kvm_set_irq
>>>>   whereas on ARM we inject directly through kvm_vgic_inject_irq
>>>>   x on arm/arm64 gsi has a specific structure:
>>>>     bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
>>>>     field: | irq_type  | vcpu_index |     irq_id     |
>>>>     where irq_id matches the Interrupt ID
>>>> - for KVM_IRQFD without routing (current implementation) the gsi field
>>>>   corresponds to an SPI index = irq_id (above) -32.
>>>> - as far as understand qemu integration, gsi is supposed to be within
>>>>   [0, KVM_MAX_IRQ_ROUTES]. Difficult to use KVM_IRQ_LINE gsi.
>>>> - to be defined what we choose as a convention with irqchip routing is
>>>>   applied: gsi -> irqchip input pin.
>>>> - Or shouldn't we simply rule out any userspace irqchip routing and stick
>>>>   to MSI routing? we could define a fixed identity in-kernel irqchip mapping
>>>>   and only offer MSI routing.
>>>> - static allocation of chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
>>>>   arbitrary put KVM_IRQCHIP_NUM_PINS = 1020 - 32 (SPI count). On s390
>>>>   this is even bigger.
>>>>
>>>> Currently tested on irqchip routing only (Calxeda midway only),
>>>> ie NOT TESTED on MSI routing yet.
>>>>
>>>> This is a very preliminary RFC to ease the discussion.
>>>>
>>>> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc8-gsi-routing-rfc
>>>>
>>>> It applies on Andre's [PATCH 00/13] arm64: KVM: GICv3 ITS emulation
>>>> (http://www.spinics.net/lists/kvm/msg117402.html)
>>>>
>>>> Eric Auger (6):
>>>>   KVM: api: add kvm_irq_routing_extended_msi
>>>>   KVM: kvm_host: add kvm_extended_msi
>>>>   KVM: irqchip: convey devid to kvm_set_msi
>>>>   KVM: arm/arm64: enable irqchip routing
>>>>   KVM: arm/arm64: enable MSI routing
>>>>   KVM: arm: implement kvm_set_msi by gsi direct mapping
>>>>
>>>>  Documentation/virtual/kvm/api.txt | 20 ++++++--
>>>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>>>  arch/arm/kvm/Kconfig              |  3 ++
>>>>  arch/arm/kvm/Makefile             |  2 +-
>>>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>>>  arch/arm64/kvm/Kconfig            |  2 +
>>>>  arch/arm64/kvm/Makefile           |  2 +-
>>>>  include/kvm/arm_vgic.h            |  9 ----
>>>>  include/linux/kvm_host.h          | 10 ++++
>>>>  include/uapi/linux/kvm.h          |  9 ++++
>>>>  virt/kvm/arm/vgic.c               | 96 +++++++++++++++++++++++++++------------
>>>>  virt/kvm/irqchip.c                | 20 ++++++--
>>>>  12 files changed, 128 insertions(+), 48 deletions(-)
>>>>
>>

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Favel,
On 06/23/2015 10:50 AM, Pavel Fedin wrote:
>  Hello!
> 
>> I was talking about what
>> is done without this "gsi direct mapping" feature. In that case the low
>> GSI numbers are used by irqchip routing entries. So kvm_irqchip_get_virq
>> returns a gsi beyond the irqchip routed gsi's.
> 
>  But aren't these low numbers just routed by the qemu too? I don't see qemu getting any predefined
> routing from the kernel. 

(some) iqrchip routing is set through kvm_irqchip_add_irq_route.
hw/i386/kvm/ioapic.c
hw/intc/openpic_kvm.c

kvm_irqchip_add_irq_route uses kvm_add_routing_entry and this is this
latter that uses set_gsis which books gsi's.

there is also a default gsi routing table set in kvm archi specific code.

Best Regards

Eric

kvm_init_irq_routing() allocates used_gsi_bitmap as all zeros, and
> kvm_irqchip_get_virq() just starts to give those free numbers, starting from zero. I believe during
> machine init we allocate virq's and route them to irqchip explicitly, don't we ?
>  Sorry for being perhaps ignorant, i haven't studied x86 qemu code too much.

> 
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
> 
> 

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Andre,
On 06/23/2015 11:03 AM, Andre Przywara wrote:
> Hi Eric,
> 
> I went back reading the code and looked at how the x86 APIC works more
> closely to understand the GSI routing better.
> See below for more ...
> 
> On 22/06/15 10:21, Eric Auger wrote:
>> On 06/22/2015 10:40 AM, Andre Przywara wrote:
>>> Hi Eric,
>>>
>>> I briefly looked over the series, the patches itself look good overall.
>>> I have one or two comments on the actual code, but want to discuss the
>>> general approach first (more a dump of some first thoughts):
>>>
>>> On 18/06/15 18:40, Eric Auger wrote:
>>>> With the advent of GICv3 ITS in-kernel emulation, KVM GSI routing
>>>> appears to be requested. More specifically MSI routing is needed.
>>>> irqchip routing does not sound to be really useful on arm but usage of
>>>> MSI routing also mandates to integrate irqchip routing. The initial
>>>> implementation of irqfd on arm must be upgraded with the integration
>>>> of kvm irqchip.c code and the implementation of its standard hooks
>>>> in the architecture specific part.
>>>>
>>>> The series therefore allows and mandates the usage of KVM_SET_GSI_ROUTING
>>>> ioctl along with KVM_IRQFD. If the userspace does not define any routing
>>>> table, no irqfd injection can happen. The user-space can use
>>>> KVM_CAP_IRQ_ROUTING to detect whether a routing table is needed.
>>>>
>>>> for irqchip routing, the convention is, only SPI can be injected and the
>>>> SPI ID corresponds to irqchip.pin + 32. For MSI routing the interrupt ID
>>>> matches the MSI msg data. API evolve to support associating a device ID
>>>> to a routine entry.
>>>
>>> So if I get this right, in a guest ITS case we have now three different
>>> IRQ name spaces:
>>> a) the LPI number, which is guest internal. The ITS driver in the guest
>>> maintains it. We can track assignments and changes when handling the
>>> MAPVI command in the host kernel, but this would stay in the kernel, as
>>> I don't see an efficient way of propagating this to userland.
>>> b) the GSI number, which is used in communication between userland and
>>> the host kernel. The guest kernel does not know about this at all. Also
>>> the ioctl requires us to set the routing for _all_ GSIs, and I read it
>>> that it assumes starting at GSI 0.
>> all injected GSI must effectively have a routing entry in KVM. Starting
>> at 0 that's not requested. At qemu level there's just the constaint gsi
>> fits between [0, max route number].
> 
> Yeah, you are right, I somehow missed that each routing entry has a gsi
> field in it. So we have to allocate all of them at once with one ioctl,
> but they can be sparse.
> 
>>  So we cannot even pretend to have
>>> LPIs here, because we would need at least 8192 empty entries then, not
>>> to speak of the possibly sparse allocation above. So we have a
>>> completely distinct name space here.
>> What is done currently at qemu level for other archs -  if I understand
>> it correctly -  is there is static GSI routing for standard IRQ. For MSI
>> irqfd setup they use spare gsi number not yet used for GSI routing < max
>> route number. So this is sparse for MSI but not for standard IRQs.
>> Effectively we do not plan to have GSI routing for LPIs but only MSI
>> routing.
> 
> That seems to make sense to me. Since we already limit the number of
> SPIs to something sensible with our KVM_DEV_ARM_VGIC_GRP_NR_IRQS, we
> could infer an implicit direct routing for those SPIs. KVM could check
> the IRQ number against vgic.nr_irqs to see whether an IRQ is routed or not.
> Any GSI beyond that number would be an MSI with your enhanced DevID:EvID
> pair in it, which gets injected via the ITS emulation code (or the
> respective GICv2m code).
> 
> That would be the idea, but if it turns out that not routing SPIs but
> only MSIs requires too many changes to the (core) KVM code (haven't
> looked yet), we could require routing entries for SPIs as well.
Well you were right ;-) I forgot the fact KVM_SET_GSI_ROUTING overwrites
the existing routing table with new user-space provided entries and it's
not possible to keep any hidden irqchip routing entries in place without
the user-space being aware of them.

What I can simply do is preventing any non identity mapping for irqchip
routing entries, ie. gsi != irqchip.pin. This fixes the problem of
KVM_IRQ_LINE. This check can happen in arm kvm_set_routing_entry() and
that's it.

Best Regards

Eric
> After all that's what for instance kvmtool sets up for x86, creating
> default 1:1 mappings for ISA and low PIC IRQs and allocating MSIs on
> demand after that.
> 
>>> c) The DevID:EvID pair, which actually identifies an IRQ in all the
>>> three regimes and is the only authoritative ID.
>>>
>>> So that means we need to maintain the connection between all the three,
>>> somehow duplicating the whole ITS mapping again to map GSIs to DevID:EvID.
>> Currently the KVM routing table indeed stores GSI/DevID:EvID mapping.
>>>
>>> So I wonder if we could use DevID:EvID directly.
>>> The KVM_IRQFD ioctl struct has some space, so we could put the DevID
>>> into the pad area.
>>> Also (more forward-looking) KVM_CAP_ASSIGN_DEV_IRQ identifies guest IRQs
>>> by an u32, but again there is quite some padding area available.
> 
>> ASSIGN_DEV_IRQ is a deprecated feature. We should not use that API I think.
> 
> OK, so do we have other users of the GSI routing beside IRQFD then?
> 
> I will go ahead and try to implement some code matching Eric's patches
> in kvmtool to test the GSI routing.
> 
> Eric, how did you test the irqchip routing on the Midway?
> 
> Cheers,
> Andre.
> 
>> Eric
>>>
>>> In general I am a bit reluctant to introduce just another level of
>>> complexity to the already quite convoluted way of doing IRQs and MSIs on
>>> ARM(64), that's why I will investigate if we can use DevID:EvID to refer
>>> to an interrupt.
>>>
>>> So far,
>>> Andre.
>>>
>>>>
>>>> Known Issues of this RFC:
>>>>
>>>> - One of the biggest is the API inconsistencies on ARM. Blame me.
>>>>   Routing should apply to KVM_IRQ_LINE ioctl which is not the case yet
>>>>   in this series. It only applies to irqfd.
>>>>   on x86 typically this KVM_IRQ_LINE is plugged onto irqchip.c kvm_set_irq
>>>>   whereas on ARM we inject directly through kvm_vgic_inject_irq
>>>>   x on arm/arm64 gsi has a specific structure:
>>>>     bits:  | 31 ... 24 | 23  ... 16 | 15    ...    0 |
>>>>     field: | irq_type  | vcpu_index |     irq_id     |
>>>>     where irq_id matches the Interrupt ID
>>>> - for KVM_IRQFD without routing (current implementation) the gsi field
>>>>   corresponds to an SPI index = irq_id (above) -32.
>>>> - as far as understand qemu integration, gsi is supposed to be within
>>>>   [0, KVM_MAX_IRQ_ROUTES]. Difficult to use KVM_IRQ_LINE gsi.
>>>> - to be defined what we choose as a convention with irqchip routing is
>>>>   applied: gsi -> irqchip input pin.
>>>> - Or shouldn't we simply rule out any userspace irqchip routing and stick
>>>>   to MSI routing? we could define a fixed identity in-kernel irqchip mapping
>>>>   and only offer MSI routing.
>>>> - static allocation of chip[KVM_NR_IRQCHIPS][KVM_IRQCHIP_NUM_PINS];
>>>>   arbitrary put KVM_IRQCHIP_NUM_PINS = 1020 - 32 (SPI count). On s390
>>>>   this is even bigger.
>>>>
>>>> Currently tested on irqchip routing only (Calxeda midway only),
>>>> ie NOT TESTED on MSI routing yet.
>>>>
>>>> This is a very preliminary RFC to ease the discussion.
>>>>
>>>> Code can be found at https://git.linaro.org/people/eric.auger/linux.git/shortlog/refs/heads/v4.1-rc8-gsi-routing-rfc
>>>>
>>>> It applies on Andre's [PATCH 00/13] arm64: KVM: GICv3 ITS emulation
>>>> (http://www.spinics.net/lists/kvm/msg117402.html)
>>>>
>>>> Eric Auger (6):
>>>>   KVM: api: add kvm_irq_routing_extended_msi
>>>>   KVM: kvm_host: add kvm_extended_msi
>>>>   KVM: irqchip: convey devid to kvm_set_msi
>>>>   KVM: arm/arm64: enable irqchip routing
>>>>   KVM: arm/arm64: enable MSI routing
>>>>   KVM: arm: implement kvm_set_msi by gsi direct mapping
>>>>
>>>>  Documentation/virtual/kvm/api.txt | 20 ++++++--
>>>>  arch/arm/include/asm/kvm_host.h   |  2 +
>>>>  arch/arm/kvm/Kconfig              |  3 ++
>>>>  arch/arm/kvm/Makefile             |  2 +-
>>>>  arch/arm64/include/asm/kvm_host.h |  1 +
>>>>  arch/arm64/kvm/Kconfig            |  2 +
>>>>  arch/arm64/kvm/Makefile           |  2 +-
>>>>  include/kvm/arm_vgic.h            |  9 ----
>>>>  include/linux/kvm_host.h          | 10 ++++
>>>>  include/uapi/linux/kvm.h          |  9 ++++
>>>>  virt/kvm/arm/vgic.c               | 96 +++++++++++++++++++++++++++------------
>>>>  virt/kvm/irqchip.c                | 20 ++++++--
>>>>  12 files changed, 128 insertions(+), 48 deletions(-)
>>>>
>>

RE: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

 Hello!

> What I can simply do is preventing any non identity mapping for irqchip
> routing entries, ie. gsi != irqchip.pin. This fixes the problem of
> KVM_IRQ_LINE. This check can happen in arm kvm_set_routing_entry() and
> that's it.

 You cannot fix problem of KVM_IRQ_LINE because anyway with this ioctl on ARM you should OR your
value with (1 << 24), so this is actually not plain GSI any more.
 KVM_IRQ_LINE is never used for triggering MSIs, so i think the easiest thing to do is to just leave
it doing what it does.

 As to irqchip routing of irqfd's, why not? By default all GSIs would be SPIs, then this can be
changed by userspace, or reverted back... Who cares? When using direct mapping qemu never calls
routing functions, so it's OK.


Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Pavel,
On 06/24/2015 02:20 PM, Pavel Fedin wrote:
>  Hello!
> 
>> What I can simply do is preventing any non identity mapping for irqchip
>> routing entries, ie. gsi != irqchip.pin. This fixes the problem of
>> KVM_IRQ_LINE. This check can happen in arm kvm_set_routing_entry() and
>> that's it.
> 
>  You cannot fix problem of KVM_IRQ_LINE because anyway with this ioctl on ARM you should OR your
> value with (1 << 24), so this is actually not plain GSI any more.
>  KVM_IRQ_LINE is never used for triggering MSIs, so i think the easiest thing to do is to just leave
> it doing what it does.

Yes I agree with you. in my next version I simply aknowledge the fact
GSI_ROUTING does not apply to KVM_IRQ_LINE on arm/arm64 but only applies
to KVM_IRQFD.
> 
>  As to irqchip routing of irqfd's, why not? By default all GSIs would be SPIs, then this can be
> changed by userspace, or reverted back... Who cares? When using direct mapping qemu never calls
> routing functions, so it's OK.

Indeed either we leave fully open irqchip routing saying that
irqchip.pin corresponds to an SPI ID. But personally I would prefer we
check irqchip routing entries have flat mapping, ie gsi = irqchip.pin
since in any case we don't want/expect the userspace to play with that.

Best Regards

Eric
> 
> 
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
> 
> 

RE: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Pavel Fedin]

 Hi!

> But personally I would prefer we
> check irqchip routing entries have flat mapping, ie gsi = irqchip.pin
> since in any case we don't want/expect the userspace to play with that.

 Why? On x86 userspace perfectly can play with it. We can imagine some very new qemu version in
future which has all arch-specific kludges like "direct mapping" removed and relying fully on
routing which it sets up from scratch, in the same way as x86 qemu does this. Or we can imagine some
new, legacy-free hypervisor implementation. The gsi == irqchip.pin limitation is just what we have
now by default, for backwards compatibility. But by design we were never obliged to stick to this.
 Well, it's just MHO...

Kind regards,
Pavel Fedin
Expert Engineer
Samsung Electronics Research center Russia

Re: [RFC 0/6] KVM: arm/arm64: gsi routing support

[Eric Auger]

Hi Pavel,
On 06/25/2015 10:46 AM, Pavel Fedin wrote:
>  Hi!
> 
>> But personally I would prefer we
>> check irqchip routing entries have flat mapping, ie gsi = irqchip.pin
>> since in any case we don't want/expect the userspace to play with that.
> 
>  Why? On x86 userspace perfectly can play with it. We can imagine some very new qemu version in
> future which has all arch-specific kludges like "direct mapping" removed and relying fully on
> routing which it sets up from scratch, in the same way as x86 qemu does this. Or we can imagine some
> new, legacy-free hypervisor implementation. The gsi == irqchip.pin limitation is just what we have
> now by default, for backwards compatibility. But by design we were never obliged to stick to this.
>  Well, it's just MHO...
OK I will get rid of the test. I think the most important is we clearly
state routing doesn't apply to KVM_IRQ_LINE.

Thanks

Eric


> 
> Kind regards,
> Pavel Fedin
> Expert Engineer
> Samsung Electronics Research center Russia
> 
>