linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support
@ 2020-03-04 20:33 Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
                   ` (24 more replies)
  0 siblings, 25 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

This (now shorter) series expands the existing GICv4 support to deal
with the new GICv4.1 architecture, which comes with a set of major
improvements compared to v4.0:

- One architectural doorbell per vcpu, instead of one doorbell per VLPI

- Doorbell entirely managed by the HW, with an "at most once" delivery
  guarantee per non-residency phase and only when requested by the
  hypervisor

- A shared memory scheme between ITSs and redistributors, allowing for an
  optimised residency sequence (the use of VMOVP becomes less frequent)

- Support for direct virtual SGI delivery (the injection path still involves
  the hypervisor), at the cost of losing the active state on SGIs. It
  shouldn't be a big deal, but some guest operating systems might notice
  (Linux definitely won't care).

On the other hand, public documentation is not available yet, so that's a
bit annoying...

The series is roughly organised in 3 parts:

(0) Fixes
(1) v4.1 doorbell management
(2) Virtual SGI support
(3) Plumbing of virtual SGIs in KVM

Notes:

  - The whole thing is tested on a FVP model, which can be obtained
    free of charge on ARM's developer website. It requires you to
    create an account, unfortunately... You'll need a fix for the
    devicetree that is in the kernel tree (should be merged before
    the 5.6 release).

  - This series has uncovered a behaviour that looks like a HW bug on
    the Cavium ThunderX (aka TX1) platform. I'd very much welcome some
    clarification from the Marvell/Cavium folks on Cc, as well as an
    official erratum number if this happens to be an actual bug.

    [v3 update]
    People have ignored for two months now, and it is fairly obvious
    that support for this machine is slowly bit-rotting. Maybe I'll
    drop the patch and instead start the process of removing all TX1
    support from the kernel (we'd certainly be better off without it).

    [v4 update]
    TX1 is now broken in mainline, and nobody cares. Make of this what
    you want.

  - I'm extremely grateful for Zenghui Yu's huge effort in carefully
    reviewing this rather difficult series (if we ever get to meet
    face to face, drinks are definitely on me!).

  - Unless someone cries wolf, I plan to take this into 5.7.

* From v4 [4]
  - Rebased on v5.6-rc4
  - Fixed locking all over the shop now that we can bypass the ITS
  - Wait on INVALL at the RD level
  - Plentu of cleanups/fixes thanks to Zenghui's careful review effort

* From v3 [3]:
  - Rebased on v5.6-rc1
  - Considerably smaller thanks to the initial patches being merged
  - Small bug fix after the v5.6 merge window

* From v2 [2]:
  - Another bunch of fixes thanks to Zenghui Yu's very careful review
  - HW-accelerated SGIs are now optional thanks to new architected
    discovery/selection bits exposed by KVM and used by the guest kernel
  - Rebased on v5.5-rc2

* From v1 [1]:
  - A bunch of minor reworks after Zenghui Yu's review
  - A workaround for what looks like a new and unexpected TX1 bug
  - A subtle reorder of the series so that some patches can go in early

[1] https://lore.kernel.org/lkml/20190923182606.32100-1-maz@kernel.org/
[2] https://lore.kernel.org/lkml/20191027144234.8395-1-maz@kernel.org/
[3] https://lore.kernel.org/r/20191224111055.11836-1-maz@kernel.org/
[4] https://lore.kernel.org/r/20200214145736.18550-1-maz@kernel.org/

Marc Zyngier (22):
  irqchip/gic-v3: Use SGIs without active state if offered
  irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors
  irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change
    and RD access
  irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the
    same RD
  irqchip/gic-v4.1: Advertise support v4.1 to KVM
  irqchip/gic-v4.1: Map the ITS SGIR register page
  irqchip/gic-v4.1: Plumb skeletal VSGI irqchip
  irqchip/gic-v4.1: Add initial SGI configuration
  irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks
  irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
  irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks
  irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction
    layer
  irqchip/gic-v4.1: Add VSGI allocation/teardown
  irqchip/gic-v4.1: Add VSGI property setup
  irqchip/gic-v4.1: Eagerly vmap vPEs
  KVM: arm64: GICv4.1: Let doorbells be auto-enabled
  KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
  KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  KVM: arm64: GICv4.1: Plumb SGI implementation selection in the
    distributor
  KVM: arm64: GICv4.1: Reload VLPI configuration on distributor
    enable/disable
  KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs
  KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs

Zenghui Yu (1):
  irqchip/gic-v4.1: Wait for completion of redistributor's INVALL
    operation

 arch/arm/include/asm/kvm_host.h        |   1 +
 arch/arm64/include/asm/kvm_emulate.h   |   3 +-
 arch/arm64/include/asm/kvm_host.h      |   1 +
 drivers/irqchip/irq-gic-v3-its.c       | 385 +++++++++++++++++++++++--
 drivers/irqchip/irq-gic-v3.c           |  13 +-
 drivers/irqchip/irq-gic-v4.c           | 134 ++++++++-
 include/kvm/arm_vgic.h                 |   4 +
 include/linux/irqchip/arm-gic-common.h |   2 +
 include/linux/irqchip/arm-gic-v3.h     |  20 +-
 include/linux/irqchip/arm-gic-v4.h     |  25 +-
 virt/kvm/arm/arm.c                     |   8 +
 virt/kvm/arm/vgic/vgic-debug.c         |  14 +-
 virt/kvm/arm/vgic/vgic-mmio-v3.c       |  68 ++++-
 virt/kvm/arm/vgic/vgic-mmio.c          |  88 +++++-
 virt/kvm/arm/vgic/vgic-v3.c            |   6 +-
 virt/kvm/arm/vgic/vgic-v4.c            | 137 +++++++--
 virt/kvm/arm/vgic/vgic.h               |   1 +
 17 files changed, 844 insertions(+), 66 deletions(-)

-- 
2.20.1


^ permalink raw reply	[flat|nested] 114+ messages in thread

* [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  6:30   ` Zenghui Yu
                     ` (2 more replies)
  2020-03-04 20:33 ` [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors Marc Zyngier
                   ` (23 subsequent siblings)
  24 siblings, 3 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

To allow the direct injection of SGIs into a guest, the GICv4.1
architecture has to sacrifice the Active state so that SGIs look
a lot like LPIs (they are injected by the same mechanism).

In order not to break existing software, the architecture gives
offers guests OSs the choice: SGIs with or without an active
state. It is the hypervisors duty to honor the guest's choice.

For this, the architecture offers a discovery bit indicating whether
the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
bit indicating whether the guest wants Active-less SGIs or not
(controlled by GICD_CTLR.nASSGIreq).

A hypervisor not supporting GICv4.1 SGIs would leave nASSGIcap
clear, and a guest not knowing about GICv4.1 SGIs (or definitely
wanting an Active state) would leave nASSGIreq clear (both being
thankfully backward compatible with older revisions of the GIC).

Since Linux is perfectly happy without an active state on SGIs,
inform the hypervisor that we'll use that if offered.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3.c       | 10 ++++++++--
 include/linux/irqchip/arm-gic-v3.h |  2 ++
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index cd76435c4a31..73e87e176d76 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -724,6 +724,7 @@ static void __init gic_dist_init(void)
 	unsigned int i;
 	u64 affinity;
 	void __iomem *base = gic_data.dist_base;
+	u32 val;
 
 	/* Disable the distributor */
 	writel_relaxed(0, base + GICD_CTLR);
@@ -756,9 +757,14 @@ static void __init gic_dist_init(void)
 	/* Now do the common stuff, and wait for the distributor to drain */
 	gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
 
+	val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
+	if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
+		pr_info("Enabling SGIs without active state\n");
+		val |= GICD_CTLR_nASSGIreq;
+	}
+
 	/* Enable distributor with ARE, Group1 */
-	writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
-		       base + GICD_CTLR);
+	writel_relaxed(val, base + GICD_CTLR);
 
 	/*
 	 * Set all global interrupts to the boot CPU only. ARE must be
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 83439bfb6c5b..c29a02678a6f 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -57,6 +57,7 @@
 #define GICD_SPENDSGIR			0x0F20
 
 #define GICD_CTLR_RWP			(1U << 31)
+#define GICD_CTLR_nASSGIreq		(1U << 8)
 #define GICD_CTLR_DS			(1U << 6)
 #define GICD_CTLR_ARE_NS		(1U << 4)
 #define GICD_CTLR_ENABLE_G1A		(1U << 1)
@@ -90,6 +91,7 @@
 #define GICD_TYPER_ESPIS(typer)						\
 	(((typer) & GICD_TYPER_ESPI) ? GICD_TYPER_SPIS((typer) >> 27) : 0)
 
+#define GICD_TYPER2_nASSGIcap		(1U << 8)
 #define GICD_TYPER2_VIL			(1U << 7)
 #define GICD_TYPER2_VID			GENMASK(4, 0)
 
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access Marc Zyngier
                   ` (22 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

In a system that is only sparsly populated with CPUs, we can end-up with
redistributors structures that are not initialized. Let's make sure we
don't try and access those when iterating over them (in this case when
checking we have a L2 VPE table).

Fixes: 4e6437f12d6e ("irqchip/gic-v4.1: Ensure L2 vPE table is allocated at RD level")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 83b1186ffcad..da883a691028 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -2452,6 +2452,10 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
 	if (!gic_rdists->has_rvpeid)
 		return true;
 
+	/* Skip non-present CPUs */
+	if (!base)
+		return true;
+
 	val  = gicr_read_vpropbaser(base + SZ_128K + GICR_VPROPBASER);
 
 	esz  = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val) + 1;
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  6:56   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 04/23] irqchip/gic-v4.1: Wait for completion of redistributor's INVALL operation Marc Zyngier
                   ` (21 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Before GICv4.1, all operations would be serialized with the affinity
changes by virtue of using the same ITS command queue. With v4.1, things
change, as invalidations (and a number of other operations) are issued
using the redistributor MMIO frame.

We must thus make sure that these redistributor accesses cannot race
against aginst the affinity change, or we may end-up talking to the
wrong redistributor.

To ensure this, we expand the irq_to_cpuid() helper to take a spinlock
when the LPI is mapped to a vLPI (a new per-VPE lock) on each operation
that requires mutual exclusion.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its.c   | 56 +++++++++++++++++++++++++-----
 include/linux/irqchip/arm-gic-v4.h |  5 +++
 2 files changed, 53 insertions(+), 8 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index da883a691028..1af713990123 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -239,15 +239,41 @@ static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
 	return NULL;
 }
 
-static int irq_to_cpuid(struct irq_data *d)
+static int vpe_to_cpuid_lock(struct its_vpe *vpe, unsigned long *flags)
+{
+	raw_spin_lock_irqsave(&vpe->vpe_lock, *flags);
+	return vpe->col_idx;
+}
+
+static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long flags)
+{
+	raw_spin_unlock_irqrestore(&vpe->vpe_lock, flags);
+}
+
+static int irq_to_cpuid_lock(struct irq_data *d, unsigned long *flags)
 {
-	struct its_device *its_dev = irq_data_get_irq_chip_data(d);
 	struct its_vlpi_map *map = get_vlpi_map(d);
+	int cpu;
 
-	if (map)
-		return map->vpe->col_idx;
+	if (map) {
+		cpu = vpe_to_cpuid_lock(map->vpe, flags);
+	} else {
+		/* Physical LPIs are already locked via the irq_desc lock */
+		struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+		cpu = its_dev->event_map.col_map[its_get_event_id(d)];
+		/* Keep GCC quiet... */
+		*flags = 0;
+	}
 
-	return its_dev->event_map.col_map[its_get_event_id(d)];
+	return cpu;
+}
+
+static void irq_to_cpuid_unlock(struct irq_data *d, unsigned long flags)
+{
+	struct its_vlpi_map *map = get_vlpi_map(d);
+
+	if (map)
+		vpe_to_cpuid_unlock(map->vpe, flags);
 }
 
 static struct its_collection *valid_col(struct its_collection *col)
@@ -1329,7 +1355,9 @@ static void direct_lpi_inv(struct irq_data *d)
 {
 	struct its_vlpi_map *map = get_vlpi_map(d);
 	void __iomem *rdbase;
+	unsigned long flags;
 	u64 val;
+	int cpu;
 
 	if (map) {
 		struct its_device *its_dev = irq_data_get_irq_chip_data(d);
@@ -1344,10 +1372,12 @@ static void direct_lpi_inv(struct irq_data *d)
 	}
 
 	/* Target the redistributor this LPI is currently routed to */
-	rdbase = per_cpu_ptr(gic_rdists->rdist, irq_to_cpuid(d))->rd_base;
+	cpu = irq_to_cpuid_lock(d, &flags);
+	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVLPIR);
 
 	wait_for_syncr(rdbase);
+	irq_to_cpuid_unlock(d, flags);
 }
 
 static void lpi_update_config(struct irq_data *d, u8 clr, u8 set)
@@ -3486,17 +3516,25 @@ static int its_vpe_set_affinity(struct irq_data *d,
 {
 	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
 	int from, cpu = cpumask_first(mask_val);
+	unsigned long flags;
 
 	/*
 	 * Changing affinity is mega expensive, so let's be as lazy as
 	 * we can and only do it if we really have to. Also, if mapped
 	 * into the proxy device, we need to move the doorbell
 	 * interrupt to its new location.
+	 *
+	 * Another thing is that changing the affinity of a vPE affects
+	 * *other interrupts* such as all the vLPIs that are routed to
+	 * this vPE. This means that the irq_desc lock is not enough to
+	 * protect us, and that we must ensure nobody samples vpe->col_idx
+	 * during the update, hence the lock below which must also be
+	 * taken on any vLPI handling path that evaluates vpe->col_idx.
 	 */
-	if (vpe->col_idx == cpu)
+	from = vpe_to_cpuid_lock(vpe, &flags);
+	if (from == cpu)
 		goto out;
 
-	from = vpe->col_idx;
 	vpe->col_idx = cpu;
 
 	/*
@@ -3512,6 +3550,7 @@ static int its_vpe_set_affinity(struct irq_data *d,
 
 out:
 	irq_data_update_effective_affinity(d, cpumask_of(cpu));
+	vpe_to_cpuid_unlock(vpe, flags);
 
 	return IRQ_SET_MASK_OK_DONE;
 }
@@ -3855,6 +3894,7 @@ static int its_vpe_init(struct its_vpe *vpe)
 		return -ENOMEM;
 	}
 
+	raw_spin_lock_init(&vpe->vpe_lock);
 	vpe->vpe_id = vpe_id;
 	vpe->vpt_page = vpt_page;
 	if (gic_rdists->has_rvpeid)
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index d9c34968467a..439963f4c66a 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -53,6 +53,11 @@ struct its_vpe {
 		};
 	};
 
+	/*
+	 * Ensures mutual exclusion between affinity setting of the
+	 * vPE and vLPI operations using vpe->col_idx.
+	 */
+	raw_spinlock_t		vpe_lock;
 	/*
 	 * This collection ID is used to indirect the target
 	 * redistributor for this VPE. The ID itself isn't involved in
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 04/23] irqchip/gic-v4.1: Wait for completion of redistributor's INVALL operation
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (2 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-20 14:23   ` Auger Eric
  2020-03-04 20:33 ` [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD Marc Zyngier
                   ` (20 subsequent siblings)
  24 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

From: Zenghui Yu <yuzenghui@huawei.com>

In GICv4.1, we emulate a guest-issued INVALL command by a direct write
to GICR_INVALLR.  Before we finish the emulation and go back to guest,
let's make sure the physical invalidate operation is actually completed
and no stale data will be left in redistributor. Per the specification,
this can be achieved by polling the GICR_SYNCR.Busy bit (to zero).

Signed-off-by: Zenghui Yu <yuzenghui@huawei.com>
Signed-off-by: Marc Zyngier <maz@kernel.org>
Link: https://lore.kernel.org/r/20200302092145.899-1-yuzenghui@huawei.com
---
 drivers/irqchip/irq-gic-v3-its.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 1af713990123..c84370245bea 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3827,6 +3827,8 @@ static void its_vpe_4_1_invall(struct its_vpe *vpe)
 	/* Target the redistributor this vPE is currently known on */
 	rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVALLR);
+
+	wait_for_syncr(rdbase);
 }
 
 static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (3 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 04/23] irqchip/gic-v4.1: Wait for completion of redistributor's INVALL operation Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  7:11   ` Zenghui Yu
                     ` (2 more replies)
  2020-03-04 20:33 ` [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM Marc Zyngier
                   ` (19 subsequent siblings)
  24 siblings, 3 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

The GICv4.1 spec says that it is CONTRAINED UNPREDICTABLE to write to
any of the GICR_INV{LPI,ALL}R registers if GICR_SYNCR.Busy == 1.

To deal with it, we must ensure that only a single invalidation can
happen at a time for a given redistributor. Add a per-RD lock to that
effect and take it around the invalidation/syncr-read to deal with this.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its.c   | 6 ++++++
 drivers/irqchip/irq-gic-v3.c       | 1 +
 include/linux/irqchip/arm-gic-v3.h | 1 +
 3 files changed, 8 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c84370245bea..fc5788584df7 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1373,10 +1373,12 @@ static void direct_lpi_inv(struct irq_data *d)
 
 	/* Target the redistributor this LPI is currently routed to */
 	cpu = irq_to_cpuid_lock(d, &flags);
+	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
 	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVLPIR);
 
 	wait_for_syncr(rdbase);
+	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
 	irq_to_cpuid_unlock(d, flags);
 }
 
@@ -3662,9 +3664,11 @@ static void its_vpe_send_inv(struct irq_data *d)
 		void __iomem *rdbase;
 
 		/* Target the redistributor this VPE is currently known on */
+		raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 		rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
 		gic_write_lpir(d->parent_data->hwirq, rdbase + GICR_INVLPIR);
 		wait_for_syncr(rdbase);
+		raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 	} else {
 		its_vpe_send_cmd(vpe, its_send_inv);
 	}
@@ -3825,10 +3829,12 @@ static void its_vpe_4_1_invall(struct its_vpe *vpe)
 	val |= FIELD_PREP(GICR_INVALLR_VPEID, vpe->vpe_id);
 
 	/* Target the redistributor this vPE is currently known on */
+	raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 	rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVALLR);
 
 	wait_for_syncr(rdbase);
+	raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 }
 
 static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 73e87e176d76..ba405becab53 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -835,6 +835,7 @@ static int __gic_populate_rdist(struct redist_region *region, void __iomem *ptr)
 	typer = gic_read_typer(ptr + GICR_TYPER);
 	if ((typer >> 32) == aff) {
 		u64 offset = ptr - region->redist_base;
+		raw_spin_lock_init(&gic_data_rdist()->rd_lock);
 		gic_data_rdist_rd_base() = ptr;
 		gic_data_rdist()->phys_base = region->phys_base + offset;
 
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index c29a02678a6f..b28acfa71f82 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -652,6 +652,7 @@
 
 struct rdists {
 	struct {
+		raw_spinlock_t	rd_lock;
 		void __iomem	*rd_base;
 		struct page	*pend_page;
 		phys_addr_t	phys_base;
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (4 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page Marc Zyngier
                   ` (18 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Tell KVM that we support v4.1. Nothing uses this information so far.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c       | 9 ++++++++-
 drivers/irqchip/irq-gic-v3.c           | 2 ++
 include/linux/irqchip/arm-gic-common.h | 2 ++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index fc5788584df7..bcc1a0957cda 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4870,6 +4870,7 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	struct device_node *of_node;
 	struct its_node *its;
 	bool has_v4 = false;
+	bool has_v4_1 = false;
 	int err;
 
 	gic_rdists = rdists;
@@ -4890,8 +4891,14 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	if (err)
 		return err;
 
-	list_for_each_entry(its, &its_nodes, entry)
+	list_for_each_entry(its, &its_nodes, entry) {
 		has_v4 |= is_v4(its);
+		has_v4_1 |= is_v4_1(its);
+	}
+
+	/* Don't bother with inconsistent systems */
+	if (WARN_ON(!has_v4_1 && rdists->has_rvpeid))
+		rdists->has_rvpeid = false;
 
 	if (has_v4 & rdists->has_vlpis) {
 		if (its_init_vpe_domain() ||
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index ba405becab53..03e4eadefb00 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -1785,6 +1785,7 @@ static void __init gic_of_setup_kvm_info(struct device_node *node)
 		gic_v3_kvm_info.vcpu = r;
 
 	gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
+	gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
 	gic_set_kvm_info(&gic_v3_kvm_info);
 }
 
@@ -2100,6 +2101,7 @@ static void __init gic_acpi_setup_kvm_info(void)
 	}
 
 	gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
+	gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
 	gic_set_kvm_info(&gic_v3_kvm_info);
 }
 
diff --git a/include/linux/irqchip/arm-gic-common.h b/include/linux/irqchip/arm-gic-common.h
index b9850f5f1906..fa8c0455c352 100644
--- a/include/linux/irqchip/arm-gic-common.h
+++ b/include/linux/irqchip/arm-gic-common.h
@@ -32,6 +32,8 @@ struct gic_kvm_info {
 	struct resource vctrl;
 	/* vlpi support */
 	bool		has_v4;
+	/* rvpeid support */
+	bool		has_v4_1;
 };
 
 const struct gic_kvm_info *gic_get_kvm_info(void);
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (5 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip Marc Zyngier
                   ` (17 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

One of the new features of GICv4.1 is to allow virtual SGIs to be
directly signaled to a VPE. For that, the ITS has grown a new
64kB page containing only a single register that is used to
signal a SGI to a given VPE.

Add a second mapping covering this new 64kB range, and take this
opportunity to limit the original mapping to 64kB, which is enough
to cover the span of the ITS registers.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index bcc1a0957cda..54d6fdf7a28e 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -96,6 +96,7 @@ struct its_node {
 	struct mutex		dev_alloc_lock;
 	struct list_head	entry;
 	void __iomem		*base;
+	void __iomem		*sgir_base;
 	phys_addr_t		phys_base;
 	struct its_cmd_block	*cmd_base;
 	struct its_cmd_block	*cmd_write;
@@ -4456,7 +4457,7 @@ static int __init its_probe_one(struct resource *res,
 	struct page *page;
 	int err;
 
-	its_base = ioremap(res->start, resource_size(res));
+	its_base = ioremap(res->start, SZ_64K);
 	if (!its_base) {
 		pr_warn("ITS@%pa: Unable to map ITS registers\n", &res->start);
 		return -ENOMEM;
@@ -4507,6 +4508,13 @@ static int __init its_probe_one(struct resource *res,
 
 		if (is_v4_1(its)) {
 			u32 svpet = FIELD_GET(GITS_TYPER_SVPET, typer);
+
+			its->sgir_base = ioremap(res->start + SZ_128K, SZ_64K);
+			if (!its->sgir_base) {
+				err = -ENOMEM;
+				goto out_free_its;
+			}
+
 			its->mpidr = readl_relaxed(its_base + GITS_MPIDR);
 
 			pr_info("ITS@%pa: Using GICv4.1 mode %08x %08x\n",
@@ -4520,7 +4528,7 @@ static int __init its_probe_one(struct resource *res,
 				get_order(ITS_CMD_QUEUE_SZ));
 	if (!page) {
 		err = -ENOMEM;
-		goto out_free_its;
+		goto out_unmap_sgir;
 	}
 	its->cmd_base = (void *)page_address(page);
 	its->cmd_write = its->cmd_base;
@@ -4587,6 +4595,9 @@ static int __init its_probe_one(struct resource *res,
 	its_free_tables(its);
 out_free_cmd:
 	free_pages((unsigned long)its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
+out_unmap_sgir:
+	if (its->sgir_base)
+		iounmap(its->sgir_base);
 out_free_its:
 	kfree(its);
 out_unmap:
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (6 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration Marc Zyngier
                   ` (16 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Since GICv4.1 has the capability to inject 16 SGIs into each VPE,
and that I'm keen not to invent too many specific interfaces to
manipulate these interrupts, let's pretend that each of these SGIs
is an actual Linux interrupt.

For that matter, let's introduce a minimal irqchip and irqdomain
setup that will get fleshed up in the following patches.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c   | 68 +++++++++++++++++++++++++++++-
 drivers/irqchip/irq-gic-v4.c       |  8 +++-
 include/linux/irqchip/arm-gic-v4.h |  9 +++-
 3 files changed, 81 insertions(+), 4 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 54d6fdf7a28e..112b452fcb40 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3870,6 +3870,67 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
 	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
 };
 
+static int its_sgi_set_affinity(struct irq_data *d,
+				const struct cpumask *mask_val,
+				bool force)
+{
+	return -EINVAL;
+}
+
+static struct irq_chip its_sgi_irq_chip = {
+	.name			= "GICv4.1-sgi",
+	.irq_set_affinity	= its_sgi_set_affinity,
+};
+
+static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
+				    unsigned int virq, unsigned int nr_irqs,
+				    void *args)
+{
+	struct its_vpe *vpe = args;
+	int i;
+
+	/* Yes, we do want 16 SGIs */
+	WARN_ON(nr_irqs != 16);
+
+	for (i = 0; i < 16; i++) {
+		vpe->sgi_config[i].priority = 0;
+		vpe->sgi_config[i].enabled = false;
+		vpe->sgi_config[i].group = false;
+
+		irq_domain_set_hwirq_and_chip(domain, virq + i, i,
+					      &its_sgi_irq_chip, vpe);
+		irq_set_status_flags(virq + i, IRQ_DISABLE_UNLAZY);
+	}
+
+	return 0;
+}
+
+static void its_sgi_irq_domain_free(struct irq_domain *domain,
+				    unsigned int virq,
+				    unsigned int nr_irqs)
+{
+	/* Nothing to do */
+}
+
+static int its_sgi_irq_domain_activate(struct irq_domain *domain,
+				       struct irq_data *d, bool reserve)
+{
+	return 0;
+}
+
+static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
+					  struct irq_data *d)
+{
+	/* Nothing to do */
+}
+
+static struct irq_domain_ops its_sgi_domain_ops = {
+	.alloc		= its_sgi_irq_domain_alloc,
+	.free		= its_sgi_irq_domain_free,
+	.activate	= its_sgi_irq_domain_activate,
+	.deactivate	= its_sgi_irq_domain_deactivate,
+};
+
 static int its_vpe_id_alloc(void)
 {
 	return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
@@ -4912,8 +4973,13 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 		rdists->has_rvpeid = false;
 
 	if (has_v4 & rdists->has_vlpis) {
+		struct irq_domain_ops *sgi_ops = NULL;
+
+		if (has_v4_1)
+			sgi_ops = &its_sgi_domain_ops;
+
 		if (its_init_vpe_domain() ||
-		    its_init_v4(parent_domain, &its_vpe_domain_ops)) {
+		    its_init_v4(parent_domain, &its_vpe_domain_ops, sgi_ops)) {
 			rdists->has_vlpis = false;
 			pr_err("ITS: Disabling GICv4 support\n");
 		}
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 45969927cc81..c01910d53f9e 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -85,6 +85,7 @@
 
 static struct irq_domain *gic_domain;
 static const struct irq_domain_ops *vpe_domain_ops;
+static const struct irq_domain_ops *sgi_domain_ops;
 
 int its_alloc_vcpu_irqs(struct its_vm *vm)
 {
@@ -216,12 +217,15 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
 	return irq_set_vcpu_affinity(irq, &info);
 }
 
-int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops)
+int its_init_v4(struct irq_domain *domain,
+		const struct irq_domain_ops *vpe_ops,
+		const struct irq_domain_ops *sgi_ops)
 {
 	if (domain) {
 		pr_info("ITS: Enabling GICv4 support\n");
 		gic_domain = domain;
-		vpe_domain_ops = ops;
+		vpe_domain_ops = vpe_ops;
+		sgi_domain_ops = sgi_ops;
 		return 0;
 	}
 
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 439963f4c66a..44e8c19e3d56 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -49,6 +49,11 @@ struct its_vpe {
 		};
 		/* GICv4.1 implementations */
 		struct {
+			struct {
+				u8	priority;
+				bool	enabled;
+				bool	group;
+			}			sgi_config[16];
 			atomic_t vmapp_count;
 		};
 	};
@@ -123,6 +128,8 @@ int its_unmap_vlpi(int irq);
 int its_prop_update_vlpi(int irq, u8 config, bool inv);
 
 struct irq_domain_ops;
-int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops);
+int its_init_v4(struct irq_domain *domain,
+		const struct irq_domain_ops *vpe_ops,
+		const struct irq_domain_ops *sgi_ops);
 
 #endif
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (7 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-16 17:53   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks Marc Zyngier
                   ` (15 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

The GICv4.1 ITS has yet another new command (VSGI) which allows
a VPE-targeted SGI to be configured (or have its pending state
cleared). Add support for this command and plumb it into the
activate irqdomain callback so that it is ready to be used.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c   | 79 +++++++++++++++++++++++++++++-
 include/linux/irqchip/arm-gic-v3.h |  3 +-
 2 files changed, 80 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 112b452fcb40..e0db3f906f87 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -380,6 +380,15 @@ struct its_cmd_desc {
 		struct {
 			struct its_vpe *vpe;
 		} its_invdb_cmd;
+
+		struct {
+			struct its_vpe *vpe;
+			u8 sgi;
+			u8 priority;
+			bool enable;
+			bool group;
+			bool clear;
+		} its_vsgi_cmd;
 	};
 };
 
@@ -528,6 +537,31 @@ static void its_encode_db(struct its_cmd_block *cmd, bool db)
 	its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
 }
 
+static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
+{
+	its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
+}
+
+static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 prio)
+{
+	its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
+}
+
+static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
+{
+	its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
+}
+
+static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
+{
+	its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
+}
+
+static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
+{
+	its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
+}
+
 static inline void its_fixup_cmd(struct its_cmd_block *cmd)
 {
 	/* Let's fixup BE commands */
@@ -893,6 +927,26 @@ static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
 	return valid_vpe(its, desc->its_invdb_cmd.vpe);
 }
 
+static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
+					  struct its_cmd_block *cmd,
+					  struct its_cmd_desc *desc)
+{
+	if (WARN_ON(!is_v4_1(its)))
+		return NULL;
+
+	its_encode_cmd(cmd, GITS_CMD_VSGI);
+	its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
+	its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
+	its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
+	its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
+	its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
+	its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
+
+	its_fixup_cmd(cmd);
+
+	return valid_vpe(its, desc->its_vsgi_cmd.vpe);
+}
+
 static u64 its_cmd_ptr_to_offset(struct its_node *its,
 				 struct its_cmd_block *ptr)
 {
@@ -3870,6 +3924,21 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
 	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
 };
 
+static void its_configure_sgi(struct irq_data *d, bool clear)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+	struct its_cmd_desc desc;
+
+	desc.its_vsgi_cmd.vpe = vpe;
+	desc.its_vsgi_cmd.sgi = d->hwirq;
+	desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
+	desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
+	desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
+	desc.its_vsgi_cmd.clear = clear;
+
+	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
+}
+
 static int its_sgi_set_affinity(struct irq_data *d,
 				const struct cpumask *mask_val,
 				bool force)
@@ -3915,13 +3984,21 @@ static void its_sgi_irq_domain_free(struct irq_domain *domain,
 static int its_sgi_irq_domain_activate(struct irq_domain *domain,
 				       struct irq_data *d, bool reserve)
 {
+	/* Write out the initial SGI configuration */
+	its_configure_sgi(d, false);
 	return 0;
 }
 
 static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
 					  struct irq_data *d)
 {
-	/* Nothing to do */
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+	/* First disable the SGI */
+	vpe->sgi_config[d->hwirq].enabled = false;
+	its_configure_sgi(d, false);
+	/* Now clear the potential pending bit (yes, this is clunky) */
+	its_configure_sgi(d, true);
 }
 
 static struct irq_domain_ops its_sgi_domain_ops = {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index b28acfa71f82..fd3be49ac9a5 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -502,8 +502,9 @@
 #define GITS_CMD_VMAPTI			GITS_CMD_GICv4(GITS_CMD_MAPTI)
 #define GITS_CMD_VMOVI			GITS_CMD_GICv4(GITS_CMD_MOVI)
 #define GITS_CMD_VSYNC			GITS_CMD_GICv4(GITS_CMD_SYNC)
-/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
+/* VMOVP, VSGI and INVDB are the odd ones, as they dont have a physical counterpart */
 #define GITS_CMD_VMOVP			GITS_CMD_GICv4(2)
+#define GITS_CMD_VSGI			GITS_CMD_GICv4(3)
 #define GITS_CMD_INVDB			GITS_CMD_GICv4(0xe)
 
 /*
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (8 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-16 18:15   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state " Marc Zyngier
                   ` (14 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Implement mask/unmask for virtual SGIs by calling into the
configuration helper.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index e0db3f906f87..c93f178914ee 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3939,6 +3939,22 @@ static void its_configure_sgi(struct irq_data *d, bool clear)
 	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
 }
 
+static void its_sgi_mask_irq(struct irq_data *d)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+	vpe->sgi_config[d->hwirq].enabled = false;
+	its_configure_sgi(d, false);
+}
+
+static void its_sgi_unmask_irq(struct irq_data *d)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+	vpe->sgi_config[d->hwirq].enabled = true;
+	its_configure_sgi(d, false);
+}
+
 static int its_sgi_set_affinity(struct irq_data *d,
 				const struct cpumask *mask_val,
 				bool force)
@@ -3948,6 +3964,8 @@ static int its_sgi_set_affinity(struct irq_data *d,
 
 static struct irq_chip its_sgi_irq_chip = {
 	.name			= "GICv4.1-sgi",
+	.irq_mask		= its_sgi_mask_irq,
+	.irq_unmask		= its_sgi_unmask_irq,
 	.irq_set_affinity	= its_sgi_set_affinity,
 };
 
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (9 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  7:41   ` Zenghui Yu
                     ` (2 more replies)
  2020-03-04 20:33 ` [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity " Marc Zyngier
                   ` (13 subsequent siblings)
  24 siblings, 3 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

To implement the get/set_irqchip_state callbacks (limited to the
PENDING state), we have to use a particular set of hacks:

- Reading the pending state is done by using a pair of new redistributor
  registers (GICR_VSGIR, GICR_VSGIPENDR), which allow the 16 interrupts
  state to be retrieved.
- Setting the pending state is done by generating it as we'd otherwise do
  for a guest (writing to GITS_SGIR).
- Clearing the pending state is done by emiting a VSGI command with the
  "clear" bit set.

This requires some interesting locking though:
- When talking to the redistributor, we must make sure that the VPE
  affinity doesn't change, hence taking the VPE lock.
- At the same time, we must ensure that nobody accesses the same
  redistributor's GICR_VSGI*R registers for a different VPE, which
  would corrupt the reading of the pending bits. We thus take the
  per-RD spinlock. Much fun.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its.c   | 73 ++++++++++++++++++++++++++++++
 include/linux/irqchip/arm-gic-v3.h | 14 ++++++
 2 files changed, 87 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c93f178914ee..fb2b836c31ff 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3962,11 +3962,84 @@ static int its_sgi_set_affinity(struct irq_data *d,
 	return -EINVAL;
 }
 
+static int its_sgi_set_irqchip_state(struct irq_data *d,
+				     enum irqchip_irq_state which,
+				     bool state)
+{
+	if (which != IRQCHIP_STATE_PENDING)
+		return -EINVAL;
+
+	if (state) {
+		struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+		struct its_node *its = find_4_1_its();
+		u64 val;
+
+		val  = FIELD_PREP(GITS_SGIR_VPEID, vpe->vpe_id);
+		val |= FIELD_PREP(GITS_SGIR_VINTID, d->hwirq);
+		writeq_relaxed(val, its->sgir_base + GITS_SGIR - SZ_128K);
+	} else {
+		its_configure_sgi(d, true);
+	}
+
+	return 0;
+}
+
+static int its_sgi_get_irqchip_state(struct irq_data *d,
+				     enum irqchip_irq_state which, bool *val)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+	void __iomem *base;
+	unsigned long flags;
+	u32 count = 1000000;	/* 1s! */
+	u32 status;
+	int cpu;
+
+	if (which != IRQCHIP_STATE_PENDING)
+		return -EINVAL;
+
+	/*
+	 * Locking galore! We can race against two different events:
+	 *
+	 * - Concurent vPE affinity change: we must make sure it cannot
+         *   happen, or we'll talk to the wrong redistributor. This is
+         *   identical to what happens with vLPIs.
+	 *
+	 * - Concurrent VSGIPENDR access: As it involves accessing two
+         *   MMIO registers, this must be made atomic one way or another.
+	 */
+	cpu = vpe_to_cpuid_lock(vpe, &flags);
+	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
+	base = gic_data_rdist_cpu(cpu)->rd_base + SZ_128K;
+	writel_relaxed(vpe->vpe_id, base + GICR_VSGIR);
+	do {
+		status = readl_relaxed(base + GICR_VSGIPENDR);
+		if (!(status & GICR_VSGIPENDR_BUSY))
+			goto out;
+
+		count--;
+		if (!count) {
+			pr_err_ratelimited("Unable to get SGI status\n");
+			goto out;
+		}
+		cpu_relax();
+		udelay(1);
+	} while(count);
+
+out:
+	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
+	vpe_to_cpuid_unlock(vpe, flags);
+	*val = !!(status & (1 << d->hwirq));
+
+	return 0;
+}
+
 static struct irq_chip its_sgi_irq_chip = {
 	.name			= "GICv4.1-sgi",
 	.irq_mask		= its_sgi_mask_irq,
 	.irq_unmask		= its_sgi_unmask_irq,
 	.irq_set_affinity	= its_sgi_set_affinity,
+	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
+	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
 };
 
 static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index fd3be49ac9a5..830d2abf14b3 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -345,6 +345,15 @@
 #define GICR_VPENDBASER_4_1_VGRP1EN	(1ULL << 58)
 #define GICR_VPENDBASER_4_1_VPEID	GENMASK_ULL(15, 0)
 
+#define GICR_VSGIR			0x0080
+
+#define GICR_VSGIR_VPEID		GENMASK(15, 0)
+
+#define GICR_VSGIPENDR			0x0088
+
+#define GICR_VSGIPENDR_BUSY		(1U << 31)
+#define GICR_VSGIPENDR_PENDING		GENMASK(15, 0)
+
 /*
  * ITS registers, offsets from ITS_base
  */
@@ -368,6 +377,11 @@
 
 #define GITS_TRANSLATER			0x10040
 
+#define GITS_SGIR			0x20020
+
+#define GITS_SGIR_VPEID			GENMASK_ULL(47, 32)
+#define GITS_SGIR_VINTID		GENMASK_ULL(7, 0)
+
 #define GITS_CTLR_ENABLE		(1U << 0)
 #define GITS_CTLR_ImDe			(1U << 1)
 #define	GITS_CTLR_ITS_NUMBER_SHIFT	4
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (10 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state " Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-17 10:35   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer Marc Zyngier
                   ` (12 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Just like for vLPIs, there is some configuration information that cannot
be directly communicated through the normal irqchip API, and we have to
use our good old friend set_vcpu_affinity as a side-band communication
mechanism.

This is used to configure group and priority for a given vSGI.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
---
 drivers/irqchip/irq-gic-v3-its.c   | 18 ++++++++++++++++++
 include/linux/irqchip/arm-gic-v4.h |  5 +++++
 2 files changed, 23 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index fb2b836c31ff..effb0e0b0c9d 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4033,6 +4033,23 @@ static int its_sgi_get_irqchip_state(struct irq_data *d,
 	return 0;
 }
 
+static int its_sgi_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+	struct its_cmd_info *info = vcpu_info;
+
+	switch (info->cmd_type) {
+	case PROP_UPDATE_SGI:
+		vpe->sgi_config[d->hwirq].priority = info->priority;
+		vpe->sgi_config[d->hwirq].group = info->group;
+		its_configure_sgi(d, false);
+		return 0;
+
+	default:
+		return -EINVAL;
+	}
+}
+
 static struct irq_chip its_sgi_irq_chip = {
 	.name			= "GICv4.1-sgi",
 	.irq_mask		= its_sgi_mask_irq,
@@ -4040,6 +4057,7 @@ static struct irq_chip its_sgi_irq_chip = {
 	.irq_set_affinity	= its_sgi_set_affinity,
 	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
 	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
+	.irq_set_vcpu_affinity	= its_sgi_set_vcpu_affinity,
 };
 
 static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 44e8c19e3d56..b4dbf899460b 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -103,6 +103,7 @@ enum its_vcpu_info_cmd_type {
 	SCHEDULE_VPE,
 	DESCHEDULE_VPE,
 	INVALL_VPE,
+	PROP_UPDATE_SGI,
 };
 
 struct its_cmd_info {
@@ -115,6 +116,10 @@ struct its_cmd_info {
 			bool		g0en;
 			bool		g1en;
 		};
+		struct {
+			u8		priority;
+			bool		group;
+		};
 	};
 };
 
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (11 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity " Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  8:20   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown Marc Zyngier
                   ` (11 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

In order to hide some of the differences between v4.0 and v4.1, move
the doorbell management out of the KVM code, and into the GICv4-specific
layer. This allows the calling code to ask for the doorbell when blocking,
and otherwise to leave the doorbell permanently disabled.

This matches the v4.1 code perfectly, and only results in a minor
refactoring of the v4.0 code.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v4.c       | 45 +++++++++++++++++++++++++++---
 include/kvm/arm_vgic.h             |  1 +
 include/linux/irqchip/arm-gic-v4.h |  3 +-
 virt/kvm/arm/vgic/vgic-v3.c        |  4 ++-
 virt/kvm/arm/vgic/vgic-v4.c        | 34 ++++++++++------------
 5 files changed, 61 insertions(+), 26 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index c01910d53f9e..117ba6db023d 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -87,6 +87,11 @@ static struct irq_domain *gic_domain;
 static const struct irq_domain_ops *vpe_domain_ops;
 static const struct irq_domain_ops *sgi_domain_ops;
 
+static bool has_v4_1(void)
+{
+	return !!sgi_domain_ops;
+}
+
 int its_alloc_vcpu_irqs(struct its_vm *vm)
 {
 	int vpe_base_irq, i;
@@ -139,18 +144,50 @@ static int its_send_vpe_cmd(struct its_vpe *vpe, struct its_cmd_info *info)
 	return irq_set_vcpu_affinity(vpe->irq, info);
 }
 
-int its_schedule_vpe(struct its_vpe *vpe, bool on)
+int its_make_vpe_non_resident(struct its_vpe *vpe, bool db)
 {
-	struct its_cmd_info info;
+	struct irq_desc *desc = irq_to_desc(vpe->irq);
+	struct its_cmd_info info = { };
 	int ret;
 
 	WARN_ON(preemptible());
 
-	info.cmd_type = on ? SCHEDULE_VPE : DESCHEDULE_VPE;
+	info.cmd_type = DESCHEDULE_VPE;
+	if (has_v4_1()) {
+		/* GICv4.1 can directly deal with doorbells */
+		info.req_db = db;
+	} else {
+		/* Undo the nested disable_irq() calls... */
+		while (db && irqd_irq_disabled(&desc->irq_data))
+			enable_irq(vpe->irq);
+	}
+
+	ret = its_send_vpe_cmd(vpe, &info);
+	if (!ret)
+		vpe->resident = false;
+
+	return ret;
+}
+
+int its_make_vpe_resident(struct its_vpe *vpe, bool g0en, bool g1en)
+{
+	struct its_cmd_info info = { };
+	int ret;
+
+	WARN_ON(preemptible());
+
+	info.cmd_type = SCHEDULE_VPE;
+	if (has_v4_1()) {
+		info.g0en = g0en;
+		info.g1en = g1en;
+	} else {
+		/* Disabled the doorbell, as we're about to enter the guest */
+		disable_irq_nosync(vpe->irq);
+	}
 
 	ret = its_send_vpe_cmd(vpe, &info);
 	if (!ret)
-		vpe->resident = on;
+		vpe->resident = true;
 
 	return ret;
 }
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 9d53f545a3d5..63457908c9c4 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -70,6 +70,7 @@ struct vgic_global {
 
 	/* Hardware has GICv4? */
 	bool			has_gicv4;
+	bool			has_gicv4_1;
 
 	/* GIC system register CPU interface */
 	struct static_key_false gicv3_cpuif;
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index b4dbf899460b..8b42d9d9b17e 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -125,7 +125,8 @@ struct its_cmd_info {
 
 int its_alloc_vcpu_irqs(struct its_vm *vm);
 void its_free_vcpu_irqs(struct its_vm *vm);
-int its_schedule_vpe(struct its_vpe *vpe, bool on);
+int its_make_vpe_resident(struct its_vpe *vpe, bool g0en, bool g1en);
+int its_make_vpe_non_resident(struct its_vpe *vpe, bool db);
 int its_invall_vpe(struct its_vpe *vpe);
 int its_map_vlpi(int irq, struct its_vlpi_map *map);
 int its_get_vlpi(int irq, struct its_vlpi_map *map);
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index f45635a6f0ec..1bc09b523486 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -595,7 +595,9 @@ int vgic_v3_probe(const struct gic_kvm_info *info)
 	/* GICv4 support? */
 	if (info->has_v4) {
 		kvm_vgic_global_state.has_gicv4 = gicv4_enable;
-		kvm_info("GICv4 support %sabled\n",
+		kvm_vgic_global_state.has_gicv4_1 = info->has_v4_1 && gicv4_enable;
+		kvm_info("GICv4%s support %sabled\n",
+			 kvm_vgic_global_state.has_gicv4_1 ? ".1" : "",
 			 gicv4_enable ? "en" : "dis");
 	}
 
diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index 46f875589c47..1eb0f8c76219 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -67,10 +67,10 @@
  * it. And if we've migrated our vcpu from one CPU to another, we must
  * tell the ITS (so that the messages reach the right redistributor).
  * This is done in two steps: first issue a irq_set_affinity() on the
- * irq corresponding to the vcpu, then call its_schedule_vpe(). You
- * must be in a non-preemptible context. On exit, another call to
- * its_schedule_vpe() tells the redistributor that we're done with the
- * vcpu.
+ * irq corresponding to the vcpu, then call its_make_vpe_resident().
+ * You must be in a non-preemptible context. On exit, a call to
+ * its_make_vpe_non_resident() tells the redistributor that we're done
+ * with the vcpu.
  *
  * Finally, the doorbell handling: Each vcpu is allocated an interrupt
  * which will fire each time a VLPI is made pending whilst the vcpu is
@@ -86,7 +86,8 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
 	struct kvm_vcpu *vcpu = info;
 
 	/* We got the message, no need to fire again */
-	if (!irqd_irq_disabled(&irq_to_desc(irq)->irq_data))
+	if (!kvm_vgic_global_state.has_gicv4_1 &&
+	    !irqd_irq_disabled(&irq_to_desc(irq)->irq_data))
 		disable_irq_nosync(irq);
 
 	vcpu->arch.vgic_cpu.vgic_v3.its_vpe.pending_last = true;
@@ -199,19 +200,11 @@ void vgic_v4_teardown(struct kvm *kvm)
 int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db)
 {
 	struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
-	struct irq_desc *desc = irq_to_desc(vpe->irq);
 
 	if (!vgic_supports_direct_msis(vcpu->kvm) || !vpe->resident)
 		return 0;
 
-	/*
-	 * If blocking, a doorbell is required. Undo the nested
-	 * disable_irq() calls...
-	 */
-	while (need_db && irqd_irq_disabled(&desc->irq_data))
-		enable_irq(vpe->irq);
-
-	return its_schedule_vpe(vpe, false);
+	return its_make_vpe_non_resident(vpe, need_db);
 }
 
 int vgic_v4_load(struct kvm_vcpu *vcpu)
@@ -232,18 +225,19 @@ int vgic_v4_load(struct kvm_vcpu *vcpu)
 	if (err)
 		return err;
 
-	/* Disabled the doorbell, as we're about to enter the guest */
-	disable_irq_nosync(vpe->irq);
-
-	err = its_schedule_vpe(vpe, true);
+	err = its_make_vpe_resident(vpe, false, vcpu->kvm->arch.vgic.enabled);
 	if (err)
 		return err;
 
 	/*
 	 * Now that the VPE is resident, let's get rid of a potential
-	 * doorbell interrupt that would still be pending.
+	 * doorbell interrupt that would still be pending. This is a
+	 * GICv4.0 only "feature"...
 	 */
-	return irq_set_irqchip_state(vpe->irq, IRQCHIP_STATE_PENDING, false);
+	if (!kvm_vgic_global_state.has_gicv4_1)
+		err = irq_set_irqchip_state(vpe->irq, IRQCHIP_STATE_PENDING, false);
+
+	return err;
 }
 
 static struct vgic_its *vgic_get_its(struct kvm *kvm,
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (12 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  8:06   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2020-03-04 20:33 ` [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup Marc Zyngier
                   ` (10 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Allocate per-VPE SGIs when initializing the GIC-specific part of the
VPE data structure.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v4.c       | 68 +++++++++++++++++++++++++++++-
 include/linux/irqchip/arm-gic-v4.h |  2 +
 2 files changed, 69 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 117ba6db023d..99b33f60ac63 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -92,6 +92,47 @@ static bool has_v4_1(void)
 	return !!sgi_domain_ops;
 }
 
+static int its_alloc_vcpu_sgis(struct its_vpe *vpe, int idx)
+{
+	char *name;
+	int sgi_base;
+
+	if (!has_v4_1())
+		return 0;
+
+	name = kasprintf(GFP_KERNEL, "GICv4-sgi-%d", task_pid_nr(current));
+	if (!name)
+		goto err;
+
+	vpe->fwnode = irq_domain_alloc_named_id_fwnode(name, idx);
+	if (!vpe->fwnode)
+		goto err;
+
+	kfree(name);
+	name = NULL;
+
+	vpe->sgi_domain = irq_domain_create_linear(vpe->fwnode, 16,
+						   sgi_domain_ops, vpe);
+	if (!vpe->sgi_domain)
+		goto err;
+
+	sgi_base = __irq_domain_alloc_irqs(vpe->sgi_domain, -1, 16,
+					       NUMA_NO_NODE, vpe,
+					       false, NULL);
+	if (sgi_base <= 0)
+		goto err;
+
+	return 0;
+
+err:
+	if (vpe->sgi_domain)
+		irq_domain_remove(vpe->sgi_domain);
+	if (vpe->fwnode)
+		irq_domain_free_fwnode(vpe->fwnode);
+	kfree(name);
+	return -ENOMEM;
+}
+
 int its_alloc_vcpu_irqs(struct its_vm *vm)
 {
 	int vpe_base_irq, i;
@@ -118,8 +159,13 @@ int its_alloc_vcpu_irqs(struct its_vm *vm)
 	if (vpe_base_irq <= 0)
 		goto err;
 
-	for (i = 0; i < vm->nr_vpes; i++)
+	for (i = 0; i < vm->nr_vpes; i++) {
+		int ret;
 		vm->vpes[i]->irq = vpe_base_irq + i;
+		ret = its_alloc_vcpu_sgis(vm->vpes[i], i);
+		if (ret)
+			goto err;
+	}
 
 	return 0;
 
@@ -132,8 +178,28 @@ int its_alloc_vcpu_irqs(struct its_vm *vm)
 	return -ENOMEM;
 }
 
+static void its_free_sgi_irqs(struct its_vm *vm)
+{
+	int i;
+
+	if (!has_v4_1())
+		return;
+
+	for (i = 0; i < vm->nr_vpes; i++) {
+		unsigned int irq = irq_find_mapping(vm->vpes[i]->sgi_domain, 0);
+
+		if (WARN_ON(!irq))
+			continue;
+
+		irq_domain_free_irqs(irq, 16);
+		irq_domain_remove(vm->vpes[i]->sgi_domain);
+		irq_domain_free_fwnode(vm->vpes[i]->fwnode);
+	}
+}
+
 void its_free_vcpu_irqs(struct its_vm *vm)
 {
+	its_free_sgi_irqs(vm);
 	irq_domain_free_irqs(vm->vpes[0]->irq, vm->nr_vpes);
 	irq_domain_remove(vm->domain);
 	irq_domain_free_fwnode(vm->fwnode);
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 8b42d9d9b17e..0bb111b4a504 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -49,6 +49,8 @@ struct its_vpe {
 		};
 		/* GICv4.1 implementations */
 		struct {
+			struct fwnode_handle	*fwnode;
+			struct irq_domain	*sgi_domain;
 			struct {
 				u8	priority;
 				bool	enabled;
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (13 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  8:09   ` Zenghui Yu
                     ` (2 more replies)
  2020-03-04 20:33 ` [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs Marc Zyngier
                   ` (9 subsequent siblings)
  24 siblings, 3 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Add the SGI configuration entry point for KVM to use.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its.c   |  2 +-
 drivers/irqchip/irq-gic-v4.c       | 13 +++++++++++++
 include/linux/irqchip/arm-gic-v4.h |  3 ++-
 3 files changed, 16 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index effb0e0b0c9d..b65fba67bd85 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4039,7 +4039,7 @@ static int its_sgi_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
 	struct its_cmd_info *info = vcpu_info;
 
 	switch (info->cmd_type) {
-	case PROP_UPDATE_SGI:
+	case PROP_UPDATE_VSGI:
 		vpe->sgi_config[d->hwirq].priority = info->priority;
 		vpe->sgi_config[d->hwirq].group = info->group;
 		its_configure_sgi(d, false);
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 99b33f60ac63..0c18714ae13e 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -320,6 +320,19 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
 	return irq_set_vcpu_affinity(irq, &info);
 }
 
+int its_prop_update_vsgi(int irq, u8 priority, bool group)
+{
+	struct its_cmd_info info = {
+		.cmd_type = PROP_UPDATE_VSGI,
+		{
+			.priority	= priority,
+			.group		= group,
+		},
+	};
+
+	return irq_set_vcpu_affinity(irq, &info);
+}
+
 int its_init_v4(struct irq_domain *domain,
 		const struct irq_domain_ops *vpe_ops,
 		const struct irq_domain_ops *sgi_ops)
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 0bb111b4a504..6976b8331b60 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -105,7 +105,7 @@ enum its_vcpu_info_cmd_type {
 	SCHEDULE_VPE,
 	DESCHEDULE_VPE,
 	INVALL_VPE,
-	PROP_UPDATE_SGI,
+	PROP_UPDATE_VSGI,
 };
 
 struct its_cmd_info {
@@ -134,6 +134,7 @@ int its_map_vlpi(int irq, struct its_vlpi_map *map);
 int its_get_vlpi(int irq, struct its_vlpi_map *map);
 int its_unmap_vlpi(int irq);
 int its_prop_update_vlpi(int irq, u8 config, bool inv);
+int its_prop_update_vsgi(int irq, u8 priority, bool group);
 
 struct irq_domain_ops;
 int its_init_v4(struct irq_domain *domain,
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (14 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  8:12   ` Zenghui Yu
                     ` (2 more replies)
  2020-03-04 20:33 ` [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled Marc Zyngier
                   ` (8 subsequent siblings)
  24 siblings, 3 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Now that we have HW-accelerated SGIs being delivered to VPEs, it
becomes required to map the VPEs on all ITSs instead of relying
on the lazy approach that we would use when using the ITS-list
mechanism.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 drivers/irqchip/irq-gic-v3-its.c | 39 +++++++++++++++++++++++++-------
 1 file changed, 31 insertions(+), 8 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index b65fba67bd85..6277b3e3731f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1586,12 +1586,31 @@ static int its_irq_set_irqchip_state(struct irq_data *d,
 	return 0;
 }
 
+/*
+ * Two favourable cases:
+ *
+ * (a) Either we have a GICv4.1, and all vPEs have to be mapped at all times
+ *     for vSGI delivery
+ *
+ * (b) Or the ITSs do not use a list map, meaning that VMOVP is cheap enough
+ *     and we're better off mapping all VPEs always
+ *
+ * If neither (a) nor (b) is true, then we map vPEs on demand.
+ *
+ */
+static bool gic_requires_eager_mapping(void)
+{
+	if (!its_list_map || gic_rdists->has_rvpeid)
+		return true;
+
+	return false;
+}
+
 static void its_map_vm(struct its_node *its, struct its_vm *vm)
 {
 	unsigned long flags;
 
-	/* Not using the ITS list? Everything is always mapped. */
-	if (!its_list_map)
+	if (gic_requires_eager_mapping())
 		return;
 
 	raw_spin_lock_irqsave(&vmovp_lock, flags);
@@ -1625,7 +1644,7 @@ static void its_unmap_vm(struct its_node *its, struct its_vm *vm)
 	unsigned long flags;
 
 	/* Not using the ITS list? Everything is always mapped. */
-	if (!its_list_map)
+	if (gic_requires_eager_mapping())
 		return;
 
 	raw_spin_lock_irqsave(&vmovp_lock, flags);
@@ -4260,8 +4279,12 @@ static int its_vpe_irq_domain_activate(struct irq_domain *domain,
 	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
 	struct its_node *its;
 
-	/* If we use the list map, we issue VMAPP on demand... */
-	if (its_list_map)
+	/*
+	 * If we use the list map, we issue VMAPP on demand... Unless
+	 * we're on a GICv4.1 and we eagerly map the VPE on all ITSs
+	 * so that VSGIs can work.
+	 */
+	if (!gic_requires_eager_mapping())
 		return 0;
 
 	/* Map the VPE to the first possible CPU */
@@ -4287,10 +4310,10 @@ static void its_vpe_irq_domain_deactivate(struct irq_domain *domain,
 	struct its_node *its;
 
 	/*
-	 * If we use the list map, we unmap the VPE once no VLPIs are
-	 * associated with the VM.
+	 * If we use the list map on GICv4.0, we unmap the VPE once no
+	 * VLPIs are associated with the VM.
 	 */
-	if (its_list_map)
+	if (!gic_requires_eager_mapping())
 		return;
 
 	list_for_each_entry(its, &its_nodes, entry) {
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (15 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-12  8:15   ` Zenghui Yu
  2020-03-17 11:04   ` Auger Eric
  2020-03-04 20:33 ` [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers Marc Zyngier
                   ` (7 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

As GICv4.1 understands the life cycle of doorbells (instead of
just randomly firing them at the most inconvenient time), just
enable them at irq_request time, and be done with it.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 virt/kvm/arm/vgic/vgic-v4.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index 1eb0f8c76219..c2fcde104ea2 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -141,6 +141,7 @@ int vgic_v4_init(struct kvm *kvm)
 
 	kvm_for_each_vcpu(i, vcpu, kvm) {
 		int irq = dist->its_vm.vpes[i]->irq;
+		unsigned long irq_flags = DB_IRQ_FLAGS;
 
 		/*
 		 * Don't automatically enable the doorbell, as we're
@@ -148,8 +149,14 @@ int vgic_v4_init(struct kvm *kvm)
 		 * blocked. Also disable the lazy disabling, as the
 		 * doorbell could kick us out of the guest too
 		 * early...
+		 *
+		 * On GICv4.1, the doorbell is managed in HW and must
+		 * be left enabled.
 		 */
-		irq_set_status_flags(irq, DB_IRQ_FLAGS);
+		if (kvm_vgic_global_state.has_gicv4_1)
+			irq_flags &= ~IRQ_NOAUTOEN;
+		irq_set_status_flags(irq, irq_flags);
+
 		ret = request_irq(irq, vgic_v4_doorbell_handler,
 				  0, "vcpu", vcpu);
 		if (ret) {
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (16 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-18  3:28   ` Zenghui Yu
  2020-03-20  8:11   ` Auger Eric
  2020-03-04 20:33 ` [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts Marc Zyngier
                   ` (6 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Most of the GICv3 emulation code that deals with SGIs now has to be
aware of the v4.1 capabilities in order to benefit from it.

Add such support, keyed on the interrupt having the hw flag set and
being a SGI.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 virt/kvm/arm/vgic/vgic-mmio-v3.c | 15 +++++-
 virt/kvm/arm/vgic/vgic-mmio.c    | 88 ++++++++++++++++++++++++++++++--
 2 files changed, 96 insertions(+), 7 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index ebc218840fc2..de89da76a379 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -6,6 +6,7 @@
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
+#include <linux/interrupt.h>
 #include <kvm/iodev.h>
 #include <kvm/arm_vgic.h>
 
@@ -942,8 +943,18 @@ void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1)
 		 * generate interrupts of either group.
 		 */
 		if (!irq->group || allow_group1) {
-			irq->pending_latch = true;
-			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+			if (!irq->hw) {
+				irq->pending_latch = true;
+				vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+			} else {
+				/* HW SGI? Ask the GIC to inject it */
+				int err;
+				err = irq_set_irqchip_state(irq->host_irq,
+							    IRQCHIP_STATE_PENDING,
+							    true);
+				WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+				raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+			}
 		} else {
 			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 		}
diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
index 97fb2a40e6ba..2199302597fa 100644
--- a/virt/kvm/arm/vgic/vgic-mmio.c
+++ b/virt/kvm/arm/vgic/vgic-mmio.c
@@ -5,6 +5,8 @@
 
 #include <linux/bitops.h>
 #include <linux/bsearch.h>
+#include <linux/interrupt.h>
+#include <linux/irq.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
 #include <kvm/iodev.h>
@@ -59,6 +61,11 @@ unsigned long vgic_mmio_read_group(struct kvm_vcpu *vcpu,
 	return value;
 }
 
+static void vgic_update_vsgi(struct vgic_irq *irq)
+{
+	WARN_ON(its_prop_update_vsgi(irq->host_irq, irq->priority, irq->group));
+}
+
 void vgic_mmio_write_group(struct kvm_vcpu *vcpu, gpa_t addr,
 			   unsigned int len, unsigned long val)
 {
@@ -71,7 +78,12 @@ void vgic_mmio_write_group(struct kvm_vcpu *vcpu, gpa_t addr,
 
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		irq->group = !!(val & BIT(i));
-		vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+			vgic_update_vsgi(irq);
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+		} else {
+			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
+		}
 
 		vgic_put_irq(vcpu->kvm, irq);
 	}
@@ -113,7 +125,21 @@ void vgic_mmio_write_senable(struct kvm_vcpu *vcpu,
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
-		if (vgic_irq_is_mapped_level(irq)) {
+		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+			if (!irq->enabled) {
+				struct irq_data *data;
+
+				irq->enabled = true;
+				data = &irq_to_desc(irq->host_irq)->irq_data;
+				while (irqd_irq_disabled(data))
+					enable_irq(irq->host_irq);
+			}
+
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+			vgic_put_irq(vcpu->kvm, irq);
+
+			continue;
+		} else if (vgic_irq_is_mapped_level(irq)) {
 			bool was_high = irq->line_level;
 
 			/*
@@ -148,6 +174,8 @@ void vgic_mmio_write_cenable(struct kvm_vcpu *vcpu,
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
+		if (irq->hw && vgic_irq_is_sgi(irq->intid) && irq->enabled)
+			disable_irq_nosync(irq->host_irq);
 
 		irq->enabled = false;
 
@@ -167,10 +195,22 @@ unsigned long vgic_mmio_read_pending(struct kvm_vcpu *vcpu,
 	for (i = 0; i < len * 8; i++) {
 		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
 		unsigned long flags;
+		bool val;
 
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
-		if (irq_is_pending(irq))
-			value |= (1U << i);
+		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+			int err;
+
+			val = false;
+			err = irq_get_irqchip_state(irq->host_irq,
+						    IRQCHIP_STATE_PENDING,
+						    &val);
+			WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+		} else {
+			val = irq_is_pending(irq);
+		}
+
+		value |= ((u32)val << i);
 		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 		vgic_put_irq(vcpu->kvm, irq);
@@ -215,6 +255,21 @@ void vgic_mmio_write_spending(struct kvm_vcpu *vcpu,
 		}
 
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
+
+		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+			/* HW SGI? Ask the GIC to inject it */
+			int err;
+			err = irq_set_irqchip_state(irq->host_irq,
+						    IRQCHIP_STATE_PENDING,
+						    true);
+			WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+			vgic_put_irq(vcpu->kvm, irq);
+
+			continue;
+		}
+
 		if (irq->hw)
 			vgic_hw_irq_spending(vcpu, irq, is_uaccess);
 		else
@@ -269,6 +324,20 @@ void vgic_mmio_write_cpending(struct kvm_vcpu *vcpu,
 
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
+		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+			/* HW SGI? Ask the GIC to clear its pending bit */
+			int err;
+			err = irq_set_irqchip_state(irq->host_irq,
+						    IRQCHIP_STATE_PENDING,
+						    false);
+			WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
+
+			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
+			vgic_put_irq(vcpu->kvm, irq);
+
+			continue;
+		}
+
 		if (irq->hw)
 			vgic_hw_irq_cpending(vcpu, irq, is_uaccess);
 		else
@@ -318,8 +387,15 @@ static void vgic_mmio_change_active(struct kvm_vcpu *vcpu, struct vgic_irq *irq,
 
 	raw_spin_lock_irqsave(&irq->irq_lock, flags);
 
-	if (irq->hw) {
+	if (irq->hw && !vgic_irq_is_sgi(irq->intid)) {
 		vgic_hw_irq_change_active(vcpu, irq, active, !requester_vcpu);
+	} else if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+		/*
+		 * GICv4.1 VSGI feature doesn't track an active state,
+		 * so let's not kid ourselves, there is nothing we can
+		 * do here.
+		 */
+		irq->active = false;
 	} else {
 		u32 model = vcpu->kvm->arch.vgic.vgic_model;
 		u8 active_source;
@@ -493,6 +569,8 @@ void vgic_mmio_write_priority(struct kvm_vcpu *vcpu,
 		raw_spin_lock_irqsave(&irq->irq_lock, flags);
 		/* Narrow the priority range to what we actually support */
 		irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
+		if (irq->hw && vgic_irq_is_sgi(irq->intid))
+			vgic_update_vsgi(irq);
 		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
 
 		vgic_put_irq(vcpu->kvm, irq);
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (17 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-19 16:16   ` Auger Eric
  2020-03-20  4:22   ` Zenghui Yu
  2020-03-04 20:33 ` [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor Marc Zyngier
                   ` (5 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

In order to let a guest buy in the new, active-less SGIs, we
need to be able to switch between the two modes.

Handle this by stopping all guest activity, transfer the state
from one mode to the other, and resume the guest. Nothing calls
this code so far, but a later patch will plug it into the MMIO
emulation.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 include/kvm/arm_vgic.h      |  3 ++
 virt/kvm/arm/vgic/vgic-v4.c | 94 +++++++++++++++++++++++++++++++++++++
 virt/kvm/arm/vgic/vgic.h    |  1 +
 3 files changed, 98 insertions(+)

diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 63457908c9c4..69f4164d6477 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -231,6 +231,9 @@ struct vgic_dist {
 	/* distributor enabled */
 	bool			enabled;
 
+	/* Wants SGIs without active state */
+	bool			nassgireq;
+
 	struct vgic_irq		*spis;
 
 	struct vgic_io_device	dist_iodev;
diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index c2fcde104ea2..a65dc1c85363 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -97,6 +97,100 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
 	return IRQ_HANDLED;
 }
 
+static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct vgic_irq *irq)
+{
+	vpe->sgi_config[irq->intid].enabled	= irq->enabled;
+	vpe->sgi_config[irq->intid].group 	= irq->group;
+	vpe->sgi_config[irq->intid].priority	= irq->priority;
+}
+
+static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
+{
+	struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
+	int i;
+
+	/*
+	 * With GICv4.1, every virtual SGI can be directly injected. So
+	 * let's pretend that they are HW interrupts, tied to a host
+	 * IRQ. The SGI code will do its magic.
+	 */
+	for (i = 0; i < VGIC_NR_SGIS; i++) {
+		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
+		struct irq_desc *desc;
+		int ret;
+
+		if (irq->hw) {
+			vgic_put_irq(vcpu->kvm, irq);
+			continue;
+		}
+
+		irq->hw = true;
+		irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
+
+		/* Transfer the full irq state to the vPE */
+		vgic_v4_sync_sgi_config(vpe, irq);
+		desc = irq_to_desc(irq->host_irq);
+		ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
+					      false);
+		if (!WARN_ON(ret)) {
+			/* Transfer pending state */
+			ret = irq_set_irqchip_state(irq->host_irq,
+						    IRQCHIP_STATE_PENDING,
+						    irq->pending_latch);
+			WARN_ON(ret);
+			irq->pending_latch = false;
+		}
+
+		vgic_put_irq(vcpu->kvm, irq);
+	}
+}
+
+static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu)
+{
+	int i;
+
+	for (i = 0; i < VGIC_NR_SGIS; i++) {
+		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
+		struct irq_desc *desc;
+		int ret;
+
+		if (!irq->hw) {
+			vgic_put_irq(vcpu->kvm, irq);
+			continue;
+		}
+
+		irq->hw = false;
+		ret = irq_get_irqchip_state(irq->host_irq,
+					    IRQCHIP_STATE_PENDING,
+					    &irq->pending_latch);
+		WARN_ON(ret);
+
+		desc = irq_to_desc(irq->host_irq);
+		irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
+
+		vgic_put_irq(vcpu->kvm, irq);
+	}
+}
+
+/* Must be called with the kvm lock held */
+void vgic_v4_configure_vsgis(struct kvm *kvm)
+{
+	struct vgic_dist *dist = &kvm->arch.vgic;
+	struct kvm_vcpu *vcpu;
+	int i;
+
+	kvm_arm_halt_guest(kvm);
+
+	kvm_for_each_vcpu(i, vcpu, kvm) {
+		if (dist->nassgireq)
+			vgic_v4_enable_vsgis(vcpu);
+		else
+			vgic_v4_disable_vsgis(vcpu);
+	}
+
+	kvm_arm_resume_guest(kvm);
+}
+
 /**
  * vgic_v4_init - Initialize the GICv4 data structures
  * @kvm:	Pointer to the VM being initialized
diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
index c7fefd6b1c80..769e4802645e 100644
--- a/virt/kvm/arm/vgic/vgic.h
+++ b/virt/kvm/arm/vgic/vgic.h
@@ -316,5 +316,6 @@ void vgic_its_invalidate_cache(struct kvm *kvm);
 bool vgic_supports_direct_msis(struct kvm *kvm);
 int vgic_v4_init(struct kvm *kvm);
 void vgic_v4_teardown(struct kvm *kvm);
+void vgic_v4_configure_vsgis(struct kvm *kvm);
 
 #endif
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (18 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-18  6:34   ` Zenghui Yu
  2020-03-04 20:33 ` [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable Marc Zyngier
                   ` (4 subsequent siblings)
  24 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

The GICv4.1 architecture gives the hypervisor the option to let
the guest choose whether it wants the good old SGIs with an
active state, or the new, HW-based ones that do not have one.

For this, plumb the configuration of SGIs into the GICv3 MMIO
handling, present the GICD_TYPER2.nASSGIcap to the guest,
and handle the GICD_CTLR.nASSGIreq setting.

In order to be able to deal with the restore of a guest, also
apply the GICD_CTLR.nASSGIreq setting at first run so that we
can move the restored SGIs to the HW if that's what the guest
had selected in a previous life.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 virt/kvm/arm/vgic/vgic-mmio-v3.c | 48 ++++++++++++++++++++++++++++++--
 virt/kvm/arm/vgic/vgic-v3.c      |  2 ++
 2 files changed, 48 insertions(+), 2 deletions(-)

diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index de89da76a379..442f3b8c2559 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -3,6 +3,7 @@
  * VGICv3 MMIO handling functions
  */
 
+#include <linux/bitfield.h>
 #include <linux/irqchip/arm-gic-v3.h>
 #include <linux/kvm.h>
 #include <linux/kvm_host.h>
@@ -70,6 +71,8 @@ static unsigned long vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
 		if (vgic->enabled)
 			value |= GICD_CTLR_ENABLE_SS_G1;
 		value |= GICD_CTLR_ARE_NS | GICD_CTLR_DS;
+		if (kvm_vgic_global_state.has_gicv4_1 && vgic->nassgireq)
+			value |= GICD_CTLR_nASSGIreq;
 		break;
 	case GICD_TYPER:
 		value = vgic->nr_spis + VGIC_NR_PRIVATE_IRQS;
@@ -81,6 +84,10 @@ static unsigned long vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
 			value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
 		}
 		break;
+	case GICD_TYPER2:
+		if (kvm_vgic_global_state.has_gicv4_1)
+			value = GICD_TYPER2_nASSGIcap;
+		break;
 	case GICD_IIDR:
 		value = (PRODUCT_ID_KVM << GICD_IIDR_PRODUCT_ID_SHIFT) |
 			(vgic->implementation_rev << GICD_IIDR_REVISION_SHIFT) |
@@ -98,17 +105,43 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,
 				    unsigned long val)
 {
 	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
-	bool was_enabled = dist->enabled;
 
 	switch (addr & 0x0c) {
-	case GICD_CTLR:
+	case GICD_CTLR: {
+		bool was_enabled, is_hwsgi;
+
+		mutex_lock(&vcpu->kvm->lock);
+
+		was_enabled = dist->enabled;
+		is_hwsgi = dist->nassgireq;
+
 		dist->enabled = val & GICD_CTLR_ENABLE_SS_G1;
 
+		/* Not a GICv4.1? No HW SGIs */
+		if (!kvm_vgic_global_state.has_gicv4_1)
+			val &= ~GICD_CTLR_nASSGIreq;
+
+		/* Dist stays enabled? nASSGIreq is RO */
+		if (was_enabled && dist->enabled) {
+			val &= ~GICD_CTLR_nASSGIreq;
+			val |= FIELD_PREP(GICD_CTLR_nASSGIreq, is_hwsgi);
+		}
+
+		/* Switching HW SGIs? */
+		dist->nassgireq = val & GICD_CTLR_nASSGIreq;
+		if (is_hwsgi != dist->nassgireq)
+			vgic_v4_configure_vsgis(vcpu->kvm);
+
 		if (!was_enabled && dist->enabled)
 			vgic_kick_vcpus(vcpu->kvm);
+
+		mutex_unlock(&vcpu->kvm->lock);
 		break;
+	}
 	case GICD_TYPER:
+	case GICD_TYPER2:
 	case GICD_IIDR:
+		/* This is at best for documentation purposes... */
 		return;
 	}
 }
@@ -117,10 +150,21 @@ static int vgic_mmio_uaccess_write_v3_misc(struct kvm_vcpu *vcpu,
 					   gpa_t addr, unsigned int len,
 					   unsigned long val)
 {
+	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
+
 	switch (addr & 0x0c) {
 	case GICD_IIDR:
 		if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
 			return -EINVAL;
+		return 0;
+	case GICD_CTLR:
+		/* Not a GICv4.1? No HW SGIs */
+		if (!kvm_vgic_global_state.has_gicv4_1)
+			val &= ~GICD_CTLR_nASSGIreq;
+
+		dist->enabled = val & GICD_CTLR_ENABLE_SS_G1;
+		dist->nassgireq = val & GICD_CTLR_nASSGIreq;
+		return 0;
 	}
 
 	vgic_mmio_write_v3_misc(vcpu, addr, len, val);
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index 1bc09b523486..2c9fc13e2c59 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -540,6 +540,8 @@ int vgic_v3_map_resources(struct kvm *kvm)
 		goto out;
 	}
 
+	if (kvm_vgic_global_state.has_gicv4_1)
+		vgic_v4_configure_vsgis(kvm);
 	dist->ready = true;
 
 out:
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (19 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-18  3:17   ` Zenghui Yu
  2020-03-04 20:33 ` [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs Marc Zyngier
                   ` (3 subsequent siblings)
  24 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Each time a Group-enable bit gets flipped, the state of these bits
needs to be forwarded to the hardware. This is a pretty heavy
handed operation, requiring all vcpus to reload their GICv4
configuration. It is thus implemented as a new request type.

Of course, we only support Group-1 for now...

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm/include/asm/kvm_host.h   | 1 +
 arch/arm64/include/asm/kvm_host.h | 1 +
 virt/kvm/arm/arm.c                | 8 ++++++++
 virt/kvm/arm/vgic/vgic-mmio-v3.c  | 5 ++++-
 4 files changed, 14 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/asm/kvm_host.h b/arch/arm/include/asm/kvm_host.h
index a827b4d60d38..3da57e863df6 100644
--- a/arch/arm/include/asm/kvm_host.h
+++ b/arch/arm/include/asm/kvm_host.h
@@ -39,6 +39,7 @@
 #define KVM_REQ_IRQ_PENDING	KVM_ARCH_REQ(1)
 #define KVM_REQ_VCPU_RESET	KVM_ARCH_REQ(2)
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
+#define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 
 DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 57fd46acd058..32c8a675e5a4 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -44,6 +44,7 @@
 #define KVM_REQ_IRQ_PENDING	KVM_ARCH_REQ(1)
 #define KVM_REQ_VCPU_RESET	KVM_ARCH_REQ(2)
 #define KVM_REQ_RECORD_STEAL	KVM_ARCH_REQ(3)
+#define KVM_REQ_RELOAD_GICv4	KVM_ARCH_REQ(4)
 
 DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
 
diff --git a/virt/kvm/arm/arm.c b/virt/kvm/arm/arm.c
index eda7b624eab8..4d864f857ac8 100644
--- a/virt/kvm/arm/arm.c
+++ b/virt/kvm/arm/arm.c
@@ -625,6 +625,14 @@ static void check_vcpu_requests(struct kvm_vcpu *vcpu)
 
 		if (kvm_check_request(KVM_REQ_RECORD_STEAL, vcpu))
 			kvm_update_stolen_time(vcpu);
+
+		if (kvm_check_request(KVM_REQ_RELOAD_GICv4, vcpu)) {
+			/* The distributor enable bits were changed */
+			preempt_disable();
+			vgic_v4_put(vcpu, false);
+			vgic_v4_load(vcpu);
+			preempt_enable();
+		}
 	}
 }
 
diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index 442f3b8c2559..48fd9fc229a2 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -132,7 +132,10 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,
 		if (is_hwsgi != dist->nassgireq)
 			vgic_v4_configure_vsgis(vcpu->kvm);
 
-		if (!was_enabled && dist->enabled)
+		if (kvm_vgic_global_state.has_gicv4_1 &&
+		    was_enabled != dist->enabled)
+			kvm_make_all_cpus_request(vcpu->kvm, KVM_REQ_RELOAD_GICv4);
+		else if (!was_enabled && dist->enabled)
 			vgic_kick_vcpus(vcpu->kvm);
 
 		mutex_unlock(&vcpu->kvm->lock);
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (20 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-20  4:23   ` Zenghui Yu
  2020-03-20  8:12   ` Auger Eric
  2020-03-04 20:33 ` [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs Marc Zyngier
                   ` (2 subsequent siblings)
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

Just like for VLPIs, it is beneficial to avoid trapping on WFI when the
vcpu is using the GICv4.1 SGIs.

Add such a check to vcpu_clear_wfx_traps().

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 arch/arm64/include/asm/kvm_emulate.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
index f658dda12364..a30b4eec7cb4 100644
--- a/arch/arm64/include/asm/kvm_emulate.h
+++ b/arch/arm64/include/asm/kvm_emulate.h
@@ -89,7 +89,8 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
 static inline void vcpu_clear_wfx_traps(struct kvm_vcpu *vcpu)
 {
 	vcpu->arch.hcr_el2 &= ~HCR_TWE;
-	if (atomic_read(&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count))
+	if (atomic_read(&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count) ||
+	    vcpu->kvm->arch.vgic.nassgireq)
 		vcpu->arch.hcr_el2 &= ~HCR_TWI;
 	else
 		vcpu->arch.hcr_el2 |= HCR_TWI;
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (21 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs Marc Zyngier
@ 2020-03-04 20:33 ` Marc Zyngier
  2020-03-18  3:19   ` Zenghui Yu
  2020-03-19 15:05   ` Auger Eric
  2020-03-05  3:39 ` [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Zenghui Yu
  2020-03-09  8:17 ` Zenghui Yu
  24 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-04 20:33 UTC (permalink / raw)
  To: linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, Eric Auger, James Morse, Julien Thierry,
	Suzuki K Poulose

The vgic-state debugfs file could do with showing the pending state
of the HW-backed SGIs. Plug it into the low-level code.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 virt/kvm/arm/vgic/vgic-debug.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/virt/kvm/arm/vgic/vgic-debug.c b/virt/kvm/arm/vgic/vgic-debug.c
index cc12fe9b2df3..b13a9e3f99dd 100644
--- a/virt/kvm/arm/vgic/vgic-debug.c
+++ b/virt/kvm/arm/vgic/vgic-debug.c
@@ -178,6 +178,8 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
 			    struct kvm_vcpu *vcpu)
 {
 	char *type;
+	bool pending;
+
 	if (irq->intid < VGIC_NR_SGIS)
 		type = "SGI";
 	else if (irq->intid < VGIC_NR_PRIVATE_IRQS)
@@ -190,6 +192,16 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
 	if (irq->intid ==0 || irq->intid == VGIC_NR_PRIVATE_IRQS)
 		print_header(s, irq, vcpu);
 
+	pending = irq->pending_latch;
+	if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+		int err;
+
+		err = irq_get_irqchip_state(irq->host_irq,
+					    IRQCHIP_STATE_PENDING,
+					    &pending);
+		WARN_ON_ONCE(err);
+	}
+
 	seq_printf(s, "       %s %4d "
 		      "    %2d "
 		      "%d%d%d%d%d%d%d "
@@ -201,7 +213,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
 		      "\n",
 			type, irq->intid,
 			(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
-			irq->pending_latch,
+			pending,
 			irq->line_level,
 			irq->active,
 			irq->enabled,
-- 
2.20.1


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (22 preceding siblings ...)
  2020-03-04 20:33 ` [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs Marc Zyngier
@ 2020-03-05  3:39 ` Zenghui Yu
  2020-03-09  8:17 ` Zenghui Yu
  24 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-05  3:39 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> This (now shorter) series expands the existing GICv4 support to deal
> with the new GICv4.1 architecture, which comes with a set of major
> improvements compared to v4.0:
> 
> - One architectural doorbell per vcpu, instead of one doorbell per VLPI
> 
> - Doorbell entirely managed by the HW, with an "at most once" delivery
>    guarantee per non-residency phase and only when requested by the
>    hypervisor
> 
> - A shared memory scheme between ITSs and redistributors, allowing for an
>    optimised residency sequence (the use of VMOVP becomes less frequent)
> 
> - Support for direct virtual SGI delivery (the injection path still involves
>    the hypervisor), at the cost of losing the active state on SGIs. It
>    shouldn't be a big deal, but some guest operating systems might notice
>    (Linux definitely won't care).
> 
> On the other hand, public documentation is not available yet, so that's a
> bit annoying...
> 
> The series is roughly organised in 3 parts:
> 
> (0) Fixes
> (1) v4.1 doorbell management
> (2) Virtual SGI support
> (3) Plumbing of virtual SGIs in KVM
> 
> Notes:
> 
>    - The whole thing is tested on a FVP model, which can be obtained
>      free of charge on ARM's developer website. It requires you to
>      create an account, unfortunately... You'll need a fix for the
>      devicetree that is in the kernel tree (should be merged before
>      the 5.6 release).
> 
>    - This series has uncovered a behaviour that looks like a HW bug on
>      the Cavium ThunderX (aka TX1) platform. I'd very much welcome some
>      clarification from the Marvell/Cavium folks on Cc, as well as an
>      official erratum number if this happens to be an actual bug.
> 
>      [v3 update]
>      People have ignored for two months now, and it is fairly obvious
>      that support for this machine is slowly bit-rotting. Maybe I'll
>      drop the patch and instead start the process of removing all TX1
>      support from the kernel (we'd certainly be better off without it).
> 
>      [v4 update]
>      TX1 is now broken in mainline, and nobody cares. Make of this what
>      you want.
> 
>    - I'm extremely grateful for Zenghui Yu's huge effort in carefully
>      reviewing this rather difficult series (if we ever get to meet
>      face to face, drinks are definitely on me!).

It's a pleasure to review this work and it's pretty useful for
understanding how Linux works as a GICv4.1-capable hypervisor.
Yay, cheers ;-)!

I'll go through the v4.1 spec one more time before the final
review of this series, as we still have plenty of time to do
some reviews (and even some tests) before the 5.7 MW.

> 
>    - Unless someone cries wolf, I plan to take this into 5.7.

Good news!


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support
  2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
                   ` (23 preceding siblings ...)
  2020-03-05  3:39 ` [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Zenghui Yu
@ 2020-03-09  8:17 ` Zenghui Yu
  2020-03-09  8:46   ` Marc Zyngier
  24 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-09  8:17 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> On the other hand, public documentation is not available yet, so that's a
> bit annoying...

The IHI0069F is now available. People can have a look at:

https://developer.arm.com/docs/ihi0069/latest


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support
  2020-03-09  8:17 ` Zenghui Yu
@ 2020-03-09  8:46   ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-09  8:46 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

On 2020-03-09 08:17, Zenghui Yu wrote:
> On 2020/3/5 4:33, Marc Zyngier wrote:
>> On the other hand, public documentation is not available yet, so 
>> that's a
>> bit annoying...
> 
> The IHI0069F is now available. People can have a look at:
> 
> https://developer.arm.com/docs/ihi0069/latest

Party! ;-)

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
@ 2020-03-12  6:30   ` Zenghui Yu
  2020-03-12  9:28     ` Marc Zyngier
  2020-03-12 17:16   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  6:30 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> To allow the direct injection of SGIs into a guest, the GICv4.1
> architecture has to sacrifice the Active state so that SGIs look
> a lot like LPIs (they are injected by the same mechanism).
> 
> In order not to break existing software, the architecture gives
> offers guests OSs the choice: SGIs with or without an active
> state. It is the hypervisors duty to honor the guest's choice.
> 
> For this, the architecture offers a discovery bit indicating whether
> the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
> bit indicating whether the guest wants Active-less SGIs or not
> (controlled by GICD_CTLR.nASSGIreq).

I still can't find the description of these two bits in IHI0069F.
Are they actually architected and will be available in the future
version of the spec?  I want to confirm it again since this has a
great impact on the KVM code, any pointers?


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access
  2020-03-04 20:33 ` [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access Marc Zyngier
@ 2020-03-12  6:56   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  6:56 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> Before GICv4.1, all operations would be serialized with the affinity
> changes by virtue of using the same ITS command queue. With v4.1, things
> change, as invalidations (and a number of other operations) are issued
> using the redistributor MMIO frame.
> 
> We must thus make sure that these redistributor accesses cannot race
> against aginst the affinity change, or we may end-up talking to the
          ^^^^^^
> wrong redistributor.
> 
> To ensure this, we expand the irq_to_cpuid() helper to take a spinlock
> when the LPI is mapped to a vLPI (a new per-VPE lock) on each operation
> that requires mutual exclusion.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks

> ---
>   drivers/irqchip/irq-gic-v3-its.c   | 56 +++++++++++++++++++++++++-----
>   include/linux/irqchip/arm-gic-v4.h |  5 +++
>   2 files changed, 53 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index da883a691028..1af713990123 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -239,15 +239,41 @@ static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
>   	return NULL;
>   }
>   
> -static int irq_to_cpuid(struct irq_data *d)
> +static int vpe_to_cpuid_lock(struct its_vpe *vpe, unsigned long *flags)
> +{
> +	raw_spin_lock_irqsave(&vpe->vpe_lock, *flags);
> +	return vpe->col_idx;
> +}
> +
> +static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long flags)
> +{
> +	raw_spin_unlock_irqrestore(&vpe->vpe_lock, flags);
> +}
> +
> +static int irq_to_cpuid_lock(struct irq_data *d, unsigned long *flags)
>   {
> -	struct its_device *its_dev = irq_data_get_irq_chip_data(d);
>   	struct its_vlpi_map *map = get_vlpi_map(d);
> +	int cpu;
>   
> -	if (map)
> -		return map->vpe->col_idx;
> +	if (map) {
> +		cpu = vpe_to_cpuid_lock(map->vpe, flags);
> +	} else {
> +		/* Physical LPIs are already locked via the irq_desc lock */
> +		struct its_device *its_dev = irq_data_get_irq_chip_data(d);
> +		cpu = its_dev->event_map.col_map[its_get_event_id(d)];
> +		/* Keep GCC quiet... */
> +		*flags = 0;
> +	}
>   
> -	return its_dev->event_map.col_map[its_get_event_id(d)];
> +	return cpu;
> +}
> +
> +static void irq_to_cpuid_unlock(struct irq_data *d, unsigned long flags)
> +{
> +	struct its_vlpi_map *map = get_vlpi_map(d);
> +
> +	if (map)
> +		vpe_to_cpuid_unlock(map->vpe, flags);
>   }
>   
>   static struct its_collection *valid_col(struct its_collection *col)
> @@ -1329,7 +1355,9 @@ static void direct_lpi_inv(struct irq_data *d)
>   {
>   	struct its_vlpi_map *map = get_vlpi_map(d);
>   	void __iomem *rdbase;
> +	unsigned long flags;
>   	u64 val;
> +	int cpu;
>   
>   	if (map) {
>   		struct its_device *its_dev = irq_data_get_irq_chip_data(d);
> @@ -1344,10 +1372,12 @@ static void direct_lpi_inv(struct irq_data *d)
>   	}
>   
>   	/* Target the redistributor this LPI is currently routed to */
> -	rdbase = per_cpu_ptr(gic_rdists->rdist, irq_to_cpuid(d))->rd_base;
> +	cpu = irq_to_cpuid_lock(d, &flags);
> +	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
>   	gic_write_lpir(val, rdbase + GICR_INVLPIR);
>   
>   	wait_for_syncr(rdbase);
> +	irq_to_cpuid_unlock(d, flags);
>   }
>   
>   static void lpi_update_config(struct irq_data *d, u8 clr, u8 set)
> @@ -3486,17 +3516,25 @@ static int its_vpe_set_affinity(struct irq_data *d,
>   {
>   	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>   	int from, cpu = cpumask_first(mask_val);
> +	unsigned long flags;
>   
>   	/*
>   	 * Changing affinity is mega expensive, so let's be as lazy as
>   	 * we can and only do it if we really have to. Also, if mapped
>   	 * into the proxy device, we need to move the doorbell
>   	 * interrupt to its new location.
> +	 *
> +	 * Another thing is that changing the affinity of a vPE affects
> +	 * *other interrupts* such as all the vLPIs that are routed to
> +	 * this vPE. This means that the irq_desc lock is not enough to
> +	 * protect us, and that we must ensure nobody samples vpe->col_idx
> +	 * during the update, hence the lock below which must also be
> +	 * taken on any vLPI handling path that evaluates vpe->col_idx.
>   	 */
> -	if (vpe->col_idx == cpu)
> +	from = vpe_to_cpuid_lock(vpe, &flags);
> +	if (from == cpu)
>   		goto out;
>   
> -	from = vpe->col_idx;
>   	vpe->col_idx = cpu;
>   
>   	/*
> @@ -3512,6 +3550,7 @@ static int its_vpe_set_affinity(struct irq_data *d,
>   
>   out:
>   	irq_data_update_effective_affinity(d, cpumask_of(cpu));
> +	vpe_to_cpuid_unlock(vpe, flags);
>   
>   	return IRQ_SET_MASK_OK_DONE;
>   }
> @@ -3855,6 +3894,7 @@ static int its_vpe_init(struct its_vpe *vpe)
>   		return -ENOMEM;
>   	}
>   
> +	raw_spin_lock_init(&vpe->vpe_lock);
>   	vpe->vpe_id = vpe_id;
>   	vpe->vpt_page = vpt_page;
>   	if (gic_rdists->has_rvpeid)
> diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
> index d9c34968467a..439963f4c66a 100644
> --- a/include/linux/irqchip/arm-gic-v4.h
> +++ b/include/linux/irqchip/arm-gic-v4.h
> @@ -53,6 +53,11 @@ struct its_vpe {
>   		};
>   	};
>   
> +	/*
> +	 * Ensures mutual exclusion between affinity setting of the
> +	 * vPE and vLPI operations using vpe->col_idx.
> +	 */
> +	raw_spinlock_t		vpe_lock;
>   	/*
>   	 * This collection ID is used to indirect the target
>   	 * redistributor for this VPE. The ID itself isn't involved in
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD
  2020-03-04 20:33 ` [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD Marc Zyngier
@ 2020-03-12  7:11   ` Zenghui Yu
  2020-03-20 14:23   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  7:11 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> The GICv4.1 spec says that it is CONTRAINED UNPREDICTABLE to write to

s/CONTRAINED/CONSTRAINED/

> any of the GICR_INV{LPI,ALL}R registers if GICR_SYNCR.Busy == 1.
> 
> To deal with it, we must ensure that only a single invalidation can
> happen at a time for a given redistributor. Add a per-RD lock to that
> effect and take it around the invalidation/syncr-read to deal with this.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks

> ---
>   drivers/irqchip/irq-gic-v3-its.c   | 6 ++++++
>   drivers/irqchip/irq-gic-v3.c       | 1 +
>   include/linux/irqchip/arm-gic-v3.h | 1 +
>   3 files changed, 8 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index c84370245bea..fc5788584df7 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -1373,10 +1373,12 @@ static void direct_lpi_inv(struct irq_data *d)
>   
>   	/* Target the redistributor this LPI is currently routed to */
>   	cpu = irq_to_cpuid_lock(d, &flags);
> +	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
>   	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
>   	gic_write_lpir(val, rdbase + GICR_INVLPIR);
>   
>   	wait_for_syncr(rdbase);
> +	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
>   	irq_to_cpuid_unlock(d, flags);
>   }
>   
> @@ -3662,9 +3664,11 @@ static void its_vpe_send_inv(struct irq_data *d)
>   		void __iomem *rdbase;
>   
>   		/* Target the redistributor this VPE is currently known on */
> +		raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>   		rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
>   		gic_write_lpir(d->parent_data->hwirq, rdbase + GICR_INVLPIR);
>   		wait_for_syncr(rdbase);
> +		raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>   	} else {
>   		its_vpe_send_cmd(vpe, its_send_inv);
>   	}
> @@ -3825,10 +3829,12 @@ static void its_vpe_4_1_invall(struct its_vpe *vpe)
>   	val |= FIELD_PREP(GICR_INVALLR_VPEID, vpe->vpe_id);
>   
>   	/* Target the redistributor this vPE is currently known on */
> +	raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>   	rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
>   	gic_write_lpir(val, rdbase + GICR_INVALLR);
>   
>   	wait_for_syncr(rdbase);
> +	raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>   }
>   
>   static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 73e87e176d76..ba405becab53 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -835,6 +835,7 @@ static int __gic_populate_rdist(struct redist_region *region, void __iomem *ptr)
>   	typer = gic_read_typer(ptr + GICR_TYPER);
>   	if ((typer >> 32) == aff) {
>   		u64 offset = ptr - region->redist_base;
> +		raw_spin_lock_init(&gic_data_rdist()->rd_lock);
>   		gic_data_rdist_rd_base() = ptr;
>   		gic_data_rdist()->phys_base = region->phys_base + offset;
>   
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index c29a02678a6f..b28acfa71f82 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -652,6 +652,7 @@
>   
>   struct rdists {
>   	struct {
> +		raw_spinlock_t	rd_lock;
>   		void __iomem	*rd_base;
>   		struct page	*pend_page;
>   		phys_addr_t	phys_base;
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state " Marc Zyngier
@ 2020-03-12  7:41   ` Zenghui Yu
  2020-03-16 21:43   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  7:41 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

Some checkpatch errors below:

On 2020/3/5 4:33, Marc Zyngier wrote:
> To implement the get/set_irqchip_state callbacks (limited to the
> PENDING state), we have to use a particular set of hacks:
> 
> - Reading the pending state is done by using a pair of new redistributor
>    registers (GICR_VSGIR, GICR_VSGIPENDR), which allow the 16 interrupts
>    state to be retrieved.
> - Setting the pending state is done by generating it as we'd otherwise do
>    for a guest (writing to GITS_SGIR).
> - Clearing the pending state is done by emiting a VSGI command with the
>    "clear" bit set.
> 
> This requires some interesting locking though:
> - When talking to the redistributor, we must make sure that the VPE
>    affinity doesn't change, hence taking the VPE lock.
> - At the same time, we must ensure that nobody accesses the same
>    redistributor's GICR_VSGI*R registers for a different VPE, which
>    would corrupt the reading of the pending bits. We thus take the
>    per-RD spinlock. Much fun.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>   drivers/irqchip/irq-gic-v3-its.c   | 73 ++++++++++++++++++++++++++++++
>   include/linux/irqchip/arm-gic-v3.h | 14 ++++++
>   2 files changed, 87 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index c93f178914ee..fb2b836c31ff 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -3962,11 +3962,84 @@ static int its_sgi_set_affinity(struct irq_data *d,
>   	return -EINVAL;
>   }
>   
> +static int its_sgi_set_irqchip_state(struct irq_data *d,
> +				     enum irqchip_irq_state which,
> +				     bool state)
> +{
> +	if (which != IRQCHIP_STATE_PENDING)
> +		return -EINVAL;
> +
> +	if (state) {
> +		struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +		struct its_node *its = find_4_1_its();
> +		u64 val;
> +
> +		val  = FIELD_PREP(GITS_SGIR_VPEID, vpe->vpe_id);
> +		val |= FIELD_PREP(GITS_SGIR_VINTID, d->hwirq);
> +		writeq_relaxed(val, its->sgir_base + GITS_SGIR - SZ_128K);
> +	} else {
> +		its_configure_sgi(d, true);
> +	}
> +
> +	return 0;
> +}
> +
> +static int its_sgi_get_irqchip_state(struct irq_data *d,
> +				     enum irqchip_irq_state which, bool *val)
> +{
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +	void __iomem *base;
> +	unsigned long flags;
> +	u32 count = 1000000;	/* 1s! */
> +	u32 status;
> +	int cpu;
> +
> +	if (which != IRQCHIP_STATE_PENDING)
> +		return -EINVAL;
> +
> +	/*
> +	 * Locking galore! We can race against two different events:
> +	 *
> +	 * - Concurent vPE affinity change: we must make sure it cannot
> +         *   happen, or we'll talk to the wrong redistributor. This is
> +         *   identical to what happens with vLPIs.

code indent should use tabs where possible

> +	 *
> +	 * - Concurrent VSGIPENDR access: As it involves accessing two
> +         *   MMIO registers, this must be made atomic one way or another.

The same here.

> +	 */
> +	cpu = vpe_to_cpuid_lock(vpe, &flags);
> +	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
> +	base = gic_data_rdist_cpu(cpu)->rd_base + SZ_128K;
> +	writel_relaxed(vpe->vpe_id, base + GICR_VSGIR);
> +	do {
> +		status = readl_relaxed(base + GICR_VSGIPENDR);
> +		if (!(status & GICR_VSGIPENDR_BUSY))
> +			goto out;
> +
> +		count--;
> +		if (!count) {
> +			pr_err_ratelimited("Unable to get SGI status\n");
> +			goto out;
> +		}
> +		cpu_relax();
> +		udelay(1);
> +	} while(count);

space required before the open parenthesis '('

> +
> +out:
> +	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
> +	vpe_to_cpuid_unlock(vpe, flags);
> +	*val = !!(status & (1 << d->hwirq));
> +
> +	return 0;
> +}
> +
>   static struct irq_chip its_sgi_irq_chip = {
>   	.name			= "GICv4.1-sgi",
>   	.irq_mask		= its_sgi_mask_irq,
>   	.irq_unmask		= its_sgi_unmask_irq,
>   	.irq_set_affinity	= its_sgi_set_affinity,
> +	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
> +	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
>   };
>   
>   static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index fd3be49ac9a5..830d2abf14b3 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -345,6 +345,15 @@
>   #define GICR_VPENDBASER_4_1_VGRP1EN	(1ULL << 58)
>   #define GICR_VPENDBASER_4_1_VPEID	GENMASK_ULL(15, 0)
>   
> +#define GICR_VSGIR			0x0080
> +
> +#define GICR_VSGIR_VPEID		GENMASK(15, 0)
> +
> +#define GICR_VSGIPENDR			0x0088
> +
> +#define GICR_VSGIPENDR_BUSY		(1U << 31)
> +#define GICR_VSGIPENDR_PENDING		GENMASK(15, 0)
> +
>   /*
>    * ITS registers, offsets from ITS_base
>    */
> @@ -368,6 +377,11 @@
>   
>   #define GITS_TRANSLATER			0x10040
>   
> +#define GITS_SGIR			0x20020
> +
> +#define GITS_SGIR_VPEID			GENMASK_ULL(47, 32)
> +#define GITS_SGIR_VINTID		GENMASK_ULL(7, 0)

GENMASK_ULL(3, 0), though not a problem.  Besides,

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown
  2020-03-04 20:33 ` [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown Marc Zyngier
@ 2020-03-12  8:06   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  8:06 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> Allocate per-VPE SGIs when initializing the GIC-specific part of the
> VPE data structure.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup
  2020-03-04 20:33 ` [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup Marc Zyngier
@ 2020-03-12  8:09   ` Zenghui Yu
  2020-03-17 10:30   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  8:09 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> Add the SGI configuration entry point for KVM to use.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs
  2020-03-04 20:33 ` [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs Marc Zyngier
@ 2020-03-12  8:12   ` Zenghui Yu
  2020-03-17  2:49   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  8:12 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> Now that we have HW-accelerated SGIs being delivered to VPEs, it
> becomes required to map the VPEs on all ITSs instead of relying
> on the lazy approach that we would use when using the ITS-list
> mechanism.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled
  2020-03-04 20:33 ` [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled Marc Zyngier
@ 2020-03-12  8:15   ` Zenghui Yu
  2020-03-17 11:04   ` Auger Eric
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  8:15 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> As GICv4.1 understands the life cycle of doorbells (instead of
> just randomly firing them at the most inconvenient time), just
> enable them at irq_request time, and be done with it.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer
  2020-03-04 20:33 ` [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer Marc Zyngier
@ 2020-03-12  8:20   ` Zenghui Yu
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-12  8:20 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> In order to hide some of the differences between v4.0 and v4.1, move
> the doorbell management out of the KVM code, and into the GICv4-specific
> layer. This allows the calling code to ask for the doorbell when blocking,
> and otherwise to leave the doorbell permanently disabled.
> 
> This matches the v4.1 code perfectly, and only results in a minor
> refactoring of the v4.0 code.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-12  6:30   ` Zenghui Yu
@ 2020-03-12  9:28     ` Marc Zyngier
  2020-03-12 12:05       ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-12  9:28 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Zenghui,

On 2020-03-12 06:30, Zenghui Yu wrote:
> Hi Marc,
> 
> On 2020/3/5 4:33, Marc Zyngier wrote:
>> To allow the direct injection of SGIs into a guest, the GICv4.1
>> architecture has to sacrifice the Active state so that SGIs look
>> a lot like LPIs (they are injected by the same mechanism).
>> 
>> In order not to break existing software, the architecture gives
>> offers guests OSs the choice: SGIs with or without an active
>> state. It is the hypervisors duty to honor the guest's choice.
>> 
>> For this, the architecture offers a discovery bit indicating whether
>> the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
>> bit indicating whether the guest wants Active-less SGIs or not
>> (controlled by GICD_CTLR.nASSGIreq).
> 
> I still can't find the description of these two bits in IHI0069F.
> Are they actually architected and will be available in the future
> version of the spec?  I want to confirm it again since this has a
> great impact on the KVM code, any pointers?

Damn. The bits *are* in the engineering spec version 19 (unfortunately
not a public document, but I believe you should have access to it).

If the bits have effectively been removed from the spec, I'll drop the
GICv4.1 code from the 5.7 queue until we find a way to achieve the same
level of support.

I've emailed people inside ARM to find out.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-12  9:28     ` Marc Zyngier
@ 2020-03-12 12:05       ` Marc Zyngier
  2020-03-13  1:39         ` Zenghui Yu
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-12 12:05 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

On 2020-03-12 09:28, Marc Zyngier wrote:
> Hi Zenghui,
> 
> On 2020-03-12 06:30, Zenghui Yu wrote:
>> Hi Marc,
>> 
>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>> To allow the direct injection of SGIs into a guest, the GICv4.1
>>> architecture has to sacrifice the Active state so that SGIs look
>>> a lot like LPIs (they are injected by the same mechanism).
>>> 
>>> In order not to break existing software, the architecture gives
>>> offers guests OSs the choice: SGIs with or without an active
>>> state. It is the hypervisors duty to honor the guest's choice.
>>> 
>>> For this, the architecture offers a discovery bit indicating whether
>>> the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
>>> bit indicating whether the guest wants Active-less SGIs or not
>>> (controlled by GICD_CTLR.nASSGIreq).
>> 
>> I still can't find the description of these two bits in IHI0069F.
>> Are they actually architected and will be available in the future
>> version of the spec?  I want to confirm it again since this has a
>> great impact on the KVM code, any pointers?
> 
> Damn. The bits *are* in the engineering spec version 19 (unfortunately
> not a public document, but I believe you should have access to it).
> 
> If the bits have effectively been removed from the spec, I'll drop the
> GICv4.1 code from the 5.7 queue until we find a way to achieve the same
> level of support.
> 
> I've emailed people inside ARM to find out.

I've now had written confirmation that the bits are still there.

It is just that the current revision of the documentation was cut 
*before*
they made it into the architecture (there seem to be a 6 month delay 
between
the architecture being sampled and the documentation being released).

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
  2020-03-12  6:30   ` Zenghui Yu
@ 2020-03-12 17:16   ` Auger Eric
  2020-03-12 17:23     ` Marc Zyngier
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-12 17:16 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> To allow the direct injection of SGIs into a guest, the GICv4.1
> architecture has to sacrifice the Active state so that SGIs look
> a lot like LPIs (they are injected by the same mechanism).
> 
> In order not to break existing software, the architecture gives
> offers guests OSs the choice: SGIs with or without an active
nit gives offers
> state. It is the hypervisors duty to honor the guest's choice.
> 
> For this, the architecture offers a discovery bit indicating whether
> the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
> bit indicating whether the guest wants Active-less SGIs or not
> (controlled by GICD_CTLR.nASSGIreq).
> 
> A hypervisor not supporting GICv4.1 SGIs would leave nASSGIcap
> clear, and a guest not knowing about GICv4.1 SGIs (or definitely
> wanting an Active state) would leave nASSGIreq clear (both being
> thankfully backward compatible with older revisions of the GIC).
> 
> Since Linux is perfectly happy without an active state on SGIs,
> inform the hypervisor that we'll use that if offered.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3.c       | 10 ++++++++--
>  include/linux/irqchip/arm-gic-v3.h |  2 ++
>  2 files changed, 10 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index cd76435c4a31..73e87e176d76 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -724,6 +724,7 @@ static void __init gic_dist_init(void)
>  	unsigned int i;
>  	u64 affinity;
>  	void __iomem *base = gic_data.dist_base;
> +	u32 val;
>  
>  	/* Disable the distributor */
>  	writel_relaxed(0, base + GICD_CTLR);
> @@ -756,9 +757,14 @@ static void __init gic_dist_init(void)
>  	/* Now do the common stuff, and wait for the distributor to drain */
>  	gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
>  
> +	val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
> +	if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
> +		pr_info("Enabling SGIs without active state\n");
> +		val |= GICD_CTLR_nASSGIreq;
> +	}
> +
>  	/* Enable distributor with ARE, Group1 */
> -	writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
> -		       base + GICD_CTLR);
> +	writel_relaxed(val, base + GICD_CTLR);
>  
>  	/*
>  	 * Set all global interrupts to the boot CPU only. ARE must be
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index 83439bfb6c5b..c29a02678a6f 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -57,6 +57,7 @@
>  #define GICD_SPENDSGIR			0x0F20
>  
>  #define GICD_CTLR_RWP			(1U << 31)
> +#define GICD_CTLR_nASSGIreq		(1U << 8)
I am not able to find this bit in Arm IHI 0069F (ID022020)
same for the bit in GICD_TYPER. Do we still miss part of the spec?

Thanks

Eric
>  #define GICD_CTLR_DS			(1U << 6)
>  #define GICD_CTLR_ARE_NS		(1U << 4)
>  #define GICD_CTLR_ENABLE_G1A		(1U << 1)
> @@ -90,6 +91,7 @@
>  #define GICD_TYPER_ESPIS(typer)						\
>  	(((typer) & GICD_TYPER_ESPI) ? GICD_TYPER_SPIS((typer) >> 27) : 0)
>  
> +#define GICD_TYPER2_nASSGIcap		(1U << 8)
>  #define GICD_TYPER2_VIL			(1U << 7)
>  #define GICD_TYPER2_VID			GENMASK(4, 0)
>  
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-12 17:16   ` Auger Eric
@ 2020-03-12 17:23     ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-12 17:23 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-12 17:16, Auger Eric wrote:
> Hi Marc,

[...]

>> diff --git a/include/linux/irqchip/arm-gic-v3.h 
>> b/include/linux/irqchip/arm-gic-v3.h
>> index 83439bfb6c5b..c29a02678a6f 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -57,6 +57,7 @@
>>  #define GICD_SPENDSGIR			0x0F20
>> 
>>  #define GICD_CTLR_RWP			(1U << 31)
>> +#define GICD_CTLR_nASSGIreq		(1U << 8)
> I am not able to find this bit in Arm IHI 0069F (ID022020)
> same for the bit in GICD_TYPER. Do we still miss part of the spec?

See my response to Zenghui (TL;DR: this addition to the spec missed the
cut-off for revision F and will be added in the next round).

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-12 12:05       ` Marc Zyngier
@ 2020-03-13  1:39         ` Zenghui Yu
  0 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-13  1:39 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/12 20:05, Marc Zyngier wrote:
> On 2020-03-12 09:28, Marc Zyngier wrote:
>> Hi Zenghui,
>>
>> On 2020-03-12 06:30, Zenghui Yu wrote:
>>> Hi Marc,
>>>
>>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>>> To allow the direct injection of SGIs into a guest, the GICv4.1
>>>> architecture has to sacrifice the Active state so that SGIs look
>>>> a lot like LPIs (they are injected by the same mechanism).
>>>>
>>>> In order not to break existing software, the architecture gives
>>>> offers guests OSs the choice: SGIs with or without an active
>>>> state. It is the hypervisors duty to honor the guest's choice.
>>>>
>>>> For this, the architecture offers a discovery bit indicating whether
>>>> the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
>>>> bit indicating whether the guest wants Active-less SGIs or not
>>>> (controlled by GICD_CTLR.nASSGIreq).
>>>
>>> I still can't find the description of these two bits in IHI0069F.
>>> Are they actually architected and will be available in the future
>>> version of the spec?  I want to confirm it again since this has a
>>> great impact on the KVM code, any pointers?
>>
>> Damn. The bits *are* in the engineering spec version 19 (unfortunately
>> not a public document, but I believe you should have access to it).
>>
>> If the bits have effectively been removed from the spec, I'll drop the
>> GICv4.1 code from the 5.7 queue until we find a way to achieve the same
>> level of support.
>>
>> I've emailed people inside ARM to find out.
> 
> I've now had written confirmation that the bits are still there.
> 
> It is just that the current revision of the documentation was cut *before*
> they made it into the architecture (there seem to be a 6 month delay 
> between
> the architecture being sampled and the documentation being released).

I see. Thanks for the confirmation!


Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip
  2020-03-04 20:33 ` [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip Marc Zyngier
@ 2020-03-16 17:10   ` Auger Eric
  2020-03-19 10:03     ` Marc Zyngier
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-16 17:10 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Since GICv4.1 has the capability to inject 16 SGIs into each VPE,
> and that I'm keen not to invent too many specific interfaces to
> manipulate these interrupts, let's pretend that each of these SGIs
> is an actual Linux interrupt.
> 
> For that matter, let's introduce a minimal irqchip and irqdomain
> setup that will get fleshed up in the following patches.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3-its.c   | 68 +++++++++++++++++++++++++++++-
>  drivers/irqchip/irq-gic-v4.c       |  8 +++-
>  include/linux/irqchip/arm-gic-v4.h |  9 +++-
>  3 files changed, 81 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index 54d6fdf7a28e..112b452fcb40 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -3870,6 +3870,67 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
>  	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
>  };
>  
> +static int its_sgi_set_affinity(struct irq_data *d,
> +				const struct cpumask *mask_val,
> +				bool force)
> +{
> +	return -EINVAL;
> +}
> +
> +static struct irq_chip its_sgi_irq_chip = {
> +	.name			= "GICv4.1-sgi",
> +	.irq_set_affinity	= its_sgi_set_affinity,
> +};
nit: const?
> +
> +static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
> +				    unsigned int virq, unsigned int nr_irqs,
> +				    void *args)
> +{
> +	struct its_vpe *vpe = args;
> +	int i;
> +
> +	/* Yes, we do want 16 SGIs */
> +	WARN_ON(nr_irqs != 16);
> +
> +	for (i = 0; i < 16; i++) {
> +		vpe->sgi_config[i].priority = 0;
> +		vpe->sgi_config[i].enabled = false;
> +		vpe->sgi_config[i].group = false;
> +
> +		irq_domain_set_hwirq_and_chip(domain, virq + i, i,
> +					      &its_sgi_irq_chip, vpe);
> +		irq_set_status_flags(virq + i, IRQ_DISABLE_UNLAZY);
> +	}
> +
> +	return 0;
> +}
> +
> +static void its_sgi_irq_domain_free(struct irq_domain *domain,
> +				    unsigned int virq,
> +				    unsigned int nr_irqs)
> +{
> +	/* Nothing to do */
> +}
> +
> +static int its_sgi_irq_domain_activate(struct irq_domain *domain,
> +				       struct irq_data *d, bool reserve)
> +{
> +	return 0;
> +}
> +
> +static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
> +					  struct irq_data *d)
> +{
> +	/* Nothing to do */
> +}
> +
> +static struct irq_domain_ops its_sgi_domain_ops = {
> +	.alloc		= its_sgi_irq_domain_alloc,
> +	.free		= its_sgi_irq_domain_free,
> +	.activate	= its_sgi_irq_domain_activate,
> +	.deactivate	= its_sgi_irq_domain_deactivate,
> +};
nit: const?
> +
>  static int its_vpe_id_alloc(void)
>  {
>  	return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
> @@ -4912,8 +4973,13 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
>  		rdists->has_rvpeid = false;
>  
>  	if (has_v4 & rdists->has_vlpis) {
> +		struct irq_domain_ops *sgi_ops = NULL;
> +
> +		if (has_v4_1)
> +			sgi_ops = &its_sgi_domain_ops;
> +
>  		if (its_init_vpe_domain() ||
> -		    its_init_v4(parent_domain, &its_vpe_domain_ops)) {
> +		    its_init_v4(parent_domain, &its_vpe_domain_ops, sgi_ops)) {
>  			rdists->has_vlpis = false;
>  			pr_err("ITS: Disabling GICv4 support\n");
>  		}
> diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
> index 45969927cc81..c01910d53f9e 100644
> --- a/drivers/irqchip/irq-gic-v4.c
> +++ b/drivers/irqchip/irq-gic-v4.c
> @@ -85,6 +85,7 @@
>  
>  static struct irq_domain *gic_domain;
>  static const struct irq_domain_ops *vpe_domain_ops;
> +static const struct irq_domain_ops *sgi_domain_ops;
>  
>  int its_alloc_vcpu_irqs(struct its_vm *vm)
>  {
> @@ -216,12 +217,15 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
>  	return irq_set_vcpu_affinity(irq, &info);
>  }
>  
> -int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops)
> +int its_init_v4(struct irq_domain *domain,
> +		const struct irq_domain_ops *vpe_ops,
> +		const struct irq_domain_ops *sgi_ops)
>  {
>  	if (domain) {
>  		pr_info("ITS: Enabling GICv4 support\n");
>  		gic_domain = domain;
> -		vpe_domain_ops = ops;
> +		vpe_domain_ops = vpe_ops;
> +		sgi_domain_ops = sgi_ops;
>  		return 0;
>  	}
>  
> diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
> index 439963f4c66a..44e8c19e3d56 100644
> --- a/include/linux/irqchip/arm-gic-v4.h
> +++ b/include/linux/irqchip/arm-gic-v4.h
> @@ -49,6 +49,11 @@ struct its_vpe {
>  		};
>  		/* GICv4.1 implementations */
>  		struct {
> +			struct {
> +				u8	priority;
> +				bool	enabled;
> +				bool	group;
> +			}			sgi_config[16];
>  			atomic_t vmapp_count;
>  		};
>  	};
> @@ -123,6 +128,8 @@ int its_unmap_vlpi(int irq);
>  int its_prop_update_vlpi(int irq, u8 config, bool inv);
>  
>  struct irq_domain_ops;
> -int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops);
> +int its_init_v4(struct irq_domain *domain,
> +		const struct irq_domain_ops *vpe_ops,
> +		const struct irq_domain_ops *sgi_ops);
>  
>  #endif
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors
  2020-03-04 20:33 ` [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors Marc Zyngier
@ 2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-16 17:10 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> In a system that is only sparsly populated with CPUs, we can end-up with
sparsely
> redistributors structures that are not initialized. Let's make sure we
redistributor structures
> don't try and access those when iterating over them (in this case when
> checking we have a L2 VPE table).
> 
> Fixes: 4e6437f12d6e ("irqchip/gic-v4.1: Ensure L2 vPE table is allocated at RD level")
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3-its.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index 83b1186ffcad..da883a691028 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -2452,6 +2452,10 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
>  	if (!gic_rdists->has_rvpeid)
>  		return true;
>  
> +	/* Skip non-present CPUs */
> +	if (!base)
> +		return true;
> +
>  	val  = gicr_read_vpropbaser(base + SZ_128K + GICR_VPROPBASER);
>  
>  	esz  = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val) + 1;
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page
  2020-03-04 20:33 ` [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page Marc Zyngier
@ 2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-16 17:10 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> One of the new features of GICv4.1 is to allow virtual SGIs to be
> directly signaled to a VPE. For that, the ITS has grown a new
> 64kB page containing only a single register that is used to
> signal a SGI to a given VPE.
> 
> Add a second mapping covering this new 64kB range, and take this
> opportunity to limit the original mapping to 64kB, which is enough
> to cover the span of the ITS registers.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  drivers/irqchip/irq-gic-v3-its.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index bcc1a0957cda..54d6fdf7a28e 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -96,6 +96,7 @@ struct its_node {
>  	struct mutex		dev_alloc_lock;
>  	struct list_head	entry;
>  	void __iomem		*base;
> +	void __iomem		*sgir_base;
>  	phys_addr_t		phys_base;
>  	struct its_cmd_block	*cmd_base;
>  	struct its_cmd_block	*cmd_write;
> @@ -4456,7 +4457,7 @@ static int __init its_probe_one(struct resource *res,
>  	struct page *page;
>  	int err;
>  
> -	its_base = ioremap(res->start, resource_size(res));
> +	its_base = ioremap(res->start, SZ_64K);
>  	if (!its_base) {
>  		pr_warn("ITS@%pa: Unable to map ITS registers\n", &res->start);
>  		return -ENOMEM;
> @@ -4507,6 +4508,13 @@ static int __init its_probe_one(struct resource *res,
>  
>  		if (is_v4_1(its)) {
>  			u32 svpet = FIELD_GET(GITS_TYPER_SVPET, typer);
> +
> +			its->sgir_base = ioremap(res->start + SZ_128K, SZ_64K);
> +			if (!its->sgir_base) {
> +				err = -ENOMEM;
> +				goto out_free_its;
> +			}
> +
>  			its->mpidr = readl_relaxed(its_base + GITS_MPIDR);
>  
>  			pr_info("ITS@%pa: Using GICv4.1 mode %08x %08x\n",
> @@ -4520,7 +4528,7 @@ static int __init its_probe_one(struct resource *res,
>  				get_order(ITS_CMD_QUEUE_SZ));
>  	if (!page) {
>  		err = -ENOMEM;
> -		goto out_free_its;
> +		goto out_unmap_sgir;
>  	}
>  	its->cmd_base = (void *)page_address(page);
>  	its->cmd_write = its->cmd_base;
> @@ -4587,6 +4595,9 @@ static int __init its_probe_one(struct resource *res,
>  	its_free_tables(its);
>  out_free_cmd:
>  	free_pages((unsigned long)its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
> +out_unmap_sgir:
> +	if (its->sgir_base)
> +		iounmap(its->sgir_base);
>  out_free_its:
>  	kfree(its);
>  out_unmap:
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM
  2020-03-04 20:33 ` [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM Marc Zyngier
@ 2020-03-16 17:10   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-16 17:10 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Tell KVM that we support v4.1. Nothing uses this information so far.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

> ---
>  drivers/irqchip/irq-gic-v3-its.c       | 9 ++++++++-
>  drivers/irqchip/irq-gic-v3.c           | 2 ++
>  include/linux/irqchip/arm-gic-common.h | 2 ++
>  3 files changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index fc5788584df7..bcc1a0957cda 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -4870,6 +4870,7 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
>  	struct device_node *of_node;
>  	struct its_node *its;
>  	bool has_v4 = false;
> +	bool has_v4_1 = false;
>  	int err;
>  
>  	gic_rdists = rdists;
> @@ -4890,8 +4891,14 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
>  	if (err)
>  		return err;
>  
> -	list_for_each_entry(its, &its_nodes, entry)
> +	list_for_each_entry(its, &its_nodes, entry) {
>  		has_v4 |= is_v4(its);
> +		has_v4_1 |= is_v4_1(its);
> +	}
> +
> +	/* Don't bother with inconsistent systems */
> +	if (WARN_ON(!has_v4_1 && rdists->has_rvpeid))
> +		rdists->has_rvpeid = false;
>  
>  	if (has_v4 & rdists->has_vlpis) {
>  		if (its_init_vpe_domain() ||
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index ba405becab53..03e4eadefb00 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -1785,6 +1785,7 @@ static void __init gic_of_setup_kvm_info(struct device_node *node)
>  		gic_v3_kvm_info.vcpu = r;
>  
>  	gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
> +	gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
>  	gic_set_kvm_info(&gic_v3_kvm_info);
>  }
>  
> @@ -2100,6 +2101,7 @@ static void __init gic_acpi_setup_kvm_info(void)
>  	}
>  
>  	gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
> +	gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
>  	gic_set_kvm_info(&gic_v3_kvm_info);
>  }
>  
> diff --git a/include/linux/irqchip/arm-gic-common.h b/include/linux/irqchip/arm-gic-common.h
> index b9850f5f1906..fa8c0455c352 100644
> --- a/include/linux/irqchip/arm-gic-common.h
> +++ b/include/linux/irqchip/arm-gic-common.h
> @@ -32,6 +32,8 @@ struct gic_kvm_info {
>  	struct resource vctrl;
>  	/* vlpi support */
>  	bool		has_v4;
> +	/* rvpeid support */
> +	bool		has_v4_1;
>  };
>  
>  const struct gic_kvm_info *gic_get_kvm_info(void);
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration
  2020-03-04 20:33 ` [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration Marc Zyngier
@ 2020-03-16 17:53   ` Auger Eric
  2020-03-17  2:02     ` Zenghui Yu
  2020-03-19 10:20     ` Marc Zyngier
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 2 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-16 17:53 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> The GICv4.1 ITS has yet another new command (VSGI) which allows
> a VPE-targeted SGI to be configured (or have its pending state
> cleared). Add support for this command and plumb it into the
> activate irqdomain callback so that it is ready to be used.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3-its.c   | 79 +++++++++++++++++++++++++++++-
>  include/linux/irqchip/arm-gic-v3.h |  3 +-
>  2 files changed, 80 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index 112b452fcb40..e0db3f906f87 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -380,6 +380,15 @@ struct its_cmd_desc {
>  		struct {
>  			struct its_vpe *vpe;
>  		} its_invdb_cmd;
> +
> +		struct {
> +			struct its_vpe *vpe;
> +			u8 sgi;
> +			u8 priority;
> +			bool enable;
> +			bool group;
> +			bool clear;
> +		} its_vsgi_cmd;
>  	};
>  };
>  
> @@ -528,6 +537,31 @@ static void its_encode_db(struct its_cmd_block *cmd, bool db)
>  	its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
>  }
>  
> +static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
> +{
> +	its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
> +}
> +
> +static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 prio)
> +{
> +	its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
> +}
> +
> +static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
> +{
> +	its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
> +}
> +
> +static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
> +{
> +	its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
> +}
> +
> +static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
> +{
> +	its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
> +}
> +
>  static inline void its_fixup_cmd(struct its_cmd_block *cmd)
>  {
>  	/* Let's fixup BE commands */
> @@ -893,6 +927,26 @@ static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
>  	return valid_vpe(its, desc->its_invdb_cmd.vpe);
>  }
>  
> +static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
> +					  struct its_cmd_block *cmd,
> +					  struct its_cmd_desc *desc)
> +{
> +	if (WARN_ON(!is_v4_1(its)))
> +		return NULL;
> +
> +	its_encode_cmd(cmd, GITS_CMD_VSGI);
> +	its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
> +	its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
> +	its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
> +	its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
> +	its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
> +	its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
> +
> +	its_fixup_cmd(cmd);
> +
> +	return valid_vpe(its, desc->its_vsgi_cmd.vpe);
> +}
> +
>  static u64 its_cmd_ptr_to_offset(struct its_node *its,
>  				 struct its_cmd_block *ptr)
>  {
> @@ -3870,6 +3924,21 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
>  	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
>  };
>  
> +static void its_configure_sgi(struct irq_data *d, bool clear)
> +{
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +	struct its_cmd_desc desc;
> +
> +	desc.its_vsgi_cmd.vpe = vpe;
> +	desc.its_vsgi_cmd.sgi = d->hwirq;
> +	desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
> +	desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
> +	desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
> +	desc.its_vsgi_cmd.clear = clear;
> +
> +	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
I see we pick up the first 4.1 ITS with find_4_1_its(). Can it happen
that not all of them have a mapping for that vPEID and if so we should
rather use one that has this mapping?

The spec says:
The ITS controls must only be used on an ITS that has a mapping for that
vPEID.
Where multiple ITSs have a mapping for the vPEID, any ITS with a mapping
may be used.

> +}
> +
>  static int its_sgi_set_affinity(struct irq_data *d,
>  				const struct cpumask *mask_val,
>  				bool force)
> @@ -3915,13 +3984,21 @@ static void its_sgi_irq_domain_free(struct irq_domain *domain,
>  static int its_sgi_irq_domain_activate(struct irq_domain *domain,
>  				       struct irq_data *d, bool reserve)
>  {
> +	/* Write out the initial SGI configuration */
> +	its_configure_sgi(d, false);
>  	return 0;
>  }
>  
>  static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
>  					  struct irq_data *d)
>  {
> -	/* Nothing to do */
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +
> +	/* First disable the SGI */
> +	vpe->sgi_config[d->hwirq].enabled = false;
> +	its_configure_sgi(d, false);
> +	/* Now clear the potential pending bit (yes, this is clunky) */
nit: Without carefuly reading the VSGI cmd notes, it is difficult to
understand why those 2 steps are needed: maybe replace this comment by
something like:
to change the config, clear must be set to false. Then clear is set and
this leaves the config unchanged. Both steps cannot be done concurrently.

"
> +	its_configure_sgi(d, true);
>  }
>  
>  static struct irq_domain_ops its_sgi_domain_ops = {
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index b28acfa71f82..fd3be49ac9a5 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -502,8 +502,9 @@
>  #define GITS_CMD_VMAPTI			GITS_CMD_GICv4(GITS_CMD_MAPTI)
>  #define GITS_CMD_VMOVI			GITS_CMD_GICv4(GITS_CMD_MOVI)
>  #define GITS_CMD_VSYNC			GITS_CMD_GICv4(GITS_CMD_SYNC)
> -/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
> +/* VMOVP, VSGI and INVDB are the odd ones, as they dont have a physical counterpart */
>  #define GITS_CMD_VMOVP			GITS_CMD_GICv4(2)
> +#define GITS_CMD_VSGI			GITS_CMD_GICv4(3)
>  #define GITS_CMD_INVDB			GITS_CMD_GICv4(0xe)
>  
>  /*
> 
Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks Marc Zyngier
@ 2020-03-16 18:15   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-16 18:15 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose



On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Implement mask/unmask for virtual SGIs by calling into the
> configuration helper.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3-its.c | 18 ++++++++++++++++++
>  1 file changed, 18 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index e0db3f906f87..c93f178914ee 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -3939,6 +3939,22 @@ static void its_configure_sgi(struct irq_data *d, bool clear)
>  	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
>  }
>  
> +static void its_sgi_mask_irq(struct irq_data *d)
> +{
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +
> +	vpe->sgi_config[d->hwirq].enabled = false;
> +	its_configure_sgi(d, false);
> +}
> +
> +static void its_sgi_unmask_irq(struct irq_data *d)
> +{
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +
> +	vpe->sgi_config[d->hwirq].enabled = true;
> +	its_configure_sgi(d, false);
> +}
> +
>  static int its_sgi_set_affinity(struct irq_data *d,
>  				const struct cpumask *mask_val,
>  				bool force)
> @@ -3948,6 +3964,8 @@ static int its_sgi_set_affinity(struct irq_data *d,
>  
>  static struct irq_chip its_sgi_irq_chip = {
>  	.name			= "GICv4.1-sgi",
> +	.irq_mask		= its_sgi_mask_irq,
> +	.irq_unmask		= its_sgi_unmask_irq,
>  	.irq_set_affinity	= its_sgi_set_affinity,
>  };
>  
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state " Marc Zyngier
  2020-03-12  7:41   ` Zenghui Yu
@ 2020-03-16 21:43   ` Auger Eric
  2020-03-19 10:27     ` Marc Zyngier
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-16 21:43 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> To implement the get/set_irqchip_state callbacks (limited to the
> PENDING state), we have to use a particular set of hacks:
> 
> - Reading the pending state is done by using a pair of new redistributor
>   registers (GICR_VSGIR, GICR_VSGIPENDR), which allow the 16 interrupts
>   state to be retrieved.
> - Setting the pending state is done by generating it as we'd otherwise do
>   for a guest (writing to GITS_SGIR).
> - Clearing the pending state is done by emiting a VSGI command with the
emitting
>   "clear" bit set.
> 
> This requires some interesting locking though:
> - When talking to the redistributor, we must make sure that the VPE
>   affinity doesn't change, hence taking the VPE lock.
> - At the same time, we must ensure that nobody accesses the same
>   redistributor's GICR_VSGI*R registers for a different VPE, which
GICR_VSGIR
>   would corrupt the reading of the pending bits. We thus take the
>   per-RD spinlock. Much fun.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  drivers/irqchip/irq-gic-v3-its.c   | 73 ++++++++++++++++++++++++++++++
>  include/linux/irqchip/arm-gic-v3.h | 14 ++++++
>  2 files changed, 87 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index c93f178914ee..fb2b836c31ff 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -3962,11 +3962,84 @@ static int its_sgi_set_affinity(struct irq_data *d,
>  	return -EINVAL;
>  }
>  
> +static int its_sgi_set_irqchip_state(struct irq_data *d,
> +				     enum irqchip_irq_state which,
> +				     bool state)
> +{
> +	if (which != IRQCHIP_STATE_PENDING)
> +		return -EINVAL;
> +
> +	if (state) {
> +		struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +		struct its_node *its = find_4_1_its();
> +		u64 val;
> +
> +		val  = FIELD_PREP(GITS_SGIR_VPEID, vpe->vpe_id);
> +		val |= FIELD_PREP(GITS_SGIR_VINTID, d->hwirq);
> +		writeq_relaxed(val, its->sgir_base + GITS_SGIR - SZ_128K);
> +	} else {
> +		its_configure_sgi(d, true);
> +	}
> +
> +	return 0;
> +}
> +
> +static int its_sgi_get_irqchip_state(struct irq_data *d,
> +				     enum irqchip_irq_state which, bool *val)
> +{
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +	void __iomem *base;
> +	unsigned long flags;
> +	u32 count = 1000000;	/* 1s! */
where does it come from? I did not find any info in the spec about this
delay.
> +	u32 status;
> +	int cpu;
> +
> +	if (which != IRQCHIP_STATE_PENDING)
> +		return -EINVAL;
> +
> +	/*
> +	 * Locking galore! We can race against two different events:
> +	 *
> +	 * - Concurent vPE affinity change: we must make sure it cannot
> +         *   happen, or we'll talk to the wrong redistributor. This is
> +         *   identical to what happens with vLPIs.
> +	 *
> +	 * - Concurrent VSGIPENDR access: As it involves accessing two
> +         *   MMIO registers, this must be made atomic one way or another.
> +	 */
> +	cpu = vpe_to_cpuid_lock(vpe, &flags);
> +	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
> +	base = gic_data_rdist_cpu(cpu)->rd_base + SZ_128K;
> +	writel_relaxed(vpe->vpe_id, base + GICR_VSGIR);
> +	do {
> +		status = readl_relaxed(base + GICR_VSGIPENDR);
> +		if (!(status & GICR_VSGIPENDR_BUSY))
> +			goto out;
> +
> +		count--;
> +		if (!count) {
> +			pr_err_ratelimited("Unable to get SGI status\n");
> +			goto out;
> +		}
> +		cpu_relax();
> +		udelay(1);
> +	} while(count);
> +
> +out:
> +	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
> +	vpe_to_cpuid_unlock(vpe, flags);
> +	*val = !!(status & (1 << d->hwirq));
> +
> +	return 0;
cascade an error on timeout?
> +}
> +
>  static struct irq_chip its_sgi_irq_chip = {
>  	.name			= "GICv4.1-sgi",
>  	.irq_mask		= its_sgi_mask_irq,
>  	.irq_unmask		= its_sgi_unmask_irq,
>  	.irq_set_affinity	= its_sgi_set_affinity,
> +	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
> +	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
>  };
>  
>  static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index fd3be49ac9a5..830d2abf14b3 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -345,6 +345,15 @@
>  #define GICR_VPENDBASER_4_1_VGRP1EN	(1ULL << 58)
>  #define GICR_VPENDBASER_4_1_VPEID	GENMASK_ULL(15, 0)
>  
> +#define GICR_VSGIR			0x0080
> +
> +#define GICR_VSGIR_VPEID		GENMASK(15, 0)
> +
> +#define GICR_VSGIPENDR			0x0088
> +
> +#define GICR_VSGIPENDR_BUSY		(1U << 31)
> +#define GICR_VSGIPENDR_PENDING		GENMASK(15, 0)
> +
>  /*
>   * ITS registers, offsets from ITS_base
>   */
> @@ -368,6 +377,11 @@
>  
>  #define GITS_TRANSLATER			0x10040
>  
> +#define GITS_SGIR			0x20020
> +
> +#define GITS_SGIR_VPEID			GENMASK_ULL(47, 32)
> +#define GITS_SGIR_VINTID		GENMASK_ULL(7, 0)
as spotted by Zenghui 3, 0
> +
>  #define GITS_CTLR_ENABLE		(1U << 0)
>  #define GITS_CTLR_ImDe			(1U << 1)
>  #define	GITS_CTLR_ITS_NUMBER_SHIFT	4
> 

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration
  2020-03-16 17:53   ` Auger Eric
@ 2020-03-17  2:02     ` Zenghui Yu
  2020-03-17  8:36       ` Auger Eric
  2020-03-19 10:20     ` Marc Zyngier
  1 sibling, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-17  2:02 UTC (permalink / raw)
  To: Auger Eric, Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020/3/17 1:53, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> The GICv4.1 ITS has yet another new command (VSGI) which allows
>> a VPE-targeted SGI to be configured (or have its pending state
>> cleared). Add support for this command and plumb it into the
>> activate irqdomain callback so that it is ready to be used.
>>
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
>> ---
>>   drivers/irqchip/irq-gic-v3-its.c   | 79 +++++++++++++++++++++++++++++-
>>   include/linux/irqchip/arm-gic-v3.h |  3 +-
>>   2 files changed, 80 insertions(+), 2 deletions(-)
>>
>> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
>> index 112b452fcb40..e0db3f906f87 100644
>> --- a/drivers/irqchip/irq-gic-v3-its.c
>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>> @@ -380,6 +380,15 @@ struct its_cmd_desc {
>>   		struct {
>>   			struct its_vpe *vpe;
>>   		} its_invdb_cmd;
>> +
>> +		struct {
>> +			struct its_vpe *vpe;
>> +			u8 sgi;
>> +			u8 priority;
>> +			bool enable;
>> +			bool group;
>> +			bool clear;
>> +		} its_vsgi_cmd;
>>   	};
>>   };
>>   
>> @@ -528,6 +537,31 @@ static void its_encode_db(struct its_cmd_block *cmd, bool db)
>>   	its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
>>   }
>>   
>> +static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
>> +}
>> +
>> +static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 prio)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
>> +}
>> +
>> +static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
>> +}
>> +
>> +static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
>> +}
>> +
>> +static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
>> +}
>> +
>>   static inline void its_fixup_cmd(struct its_cmd_block *cmd)
>>   {
>>   	/* Let's fixup BE commands */
>> @@ -893,6 +927,26 @@ static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
>>   	return valid_vpe(its, desc->its_invdb_cmd.vpe);
>>   }
>>   
>> +static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
>> +					  struct its_cmd_block *cmd,
>> +					  struct its_cmd_desc *desc)
>> +{
>> +	if (WARN_ON(!is_v4_1(its)))
>> +		return NULL;
>> +
>> +	its_encode_cmd(cmd, GITS_CMD_VSGI);
>> +	its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
>> +	its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
>> +	its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
>> +	its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
>> +	its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
>> +	its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
>> +
>> +	its_fixup_cmd(cmd);
>> +
>> +	return valid_vpe(its, desc->its_vsgi_cmd.vpe);
>> +}
>> +
>>   static u64 its_cmd_ptr_to_offset(struct its_node *its,
>>   				 struct its_cmd_block *ptr)
>>   {
>> @@ -3870,6 +3924,21 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
>>   	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
>>   };
>>   
>> +static void its_configure_sgi(struct irq_data *d, bool clear)
>> +{
>> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>> +	struct its_cmd_desc desc;
>> +
>> +	desc.its_vsgi_cmd.vpe = vpe;
>> +	desc.its_vsgi_cmd.sgi = d->hwirq;
>> +	desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
>> +	desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
>> +	desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
>> +	desc.its_vsgi_cmd.clear = clear;
>> +
>> +	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
> I see we pick up the first 4.1 ITS with find_4_1_its(). Can it happen
> that not all of them have a mapping for that vPEID and if so we should
> rather use one that has this mapping?

It can't happen in GICv4.1, and you may find the answer in patch #16
("Eagerly vmap vPEs").  I also failed to follow this logic the first
time looking at it [*], so I think it may worth adding some comments
on top of find_4_1_its()?

[*] 
https://lore.kernel.org/lkml/c94061be-d029-69c8-d34f-4d21081d5aba@huawei.com/

> 
> The spec says:
> The ITS controls must only be used on an ITS that has a mapping for that
> vPEID.
> Where multiple ITSs have a mapping for the vPEID, any ITS with a mapping
> may be used.
> 
>> +}
>> +
>>   static int its_sgi_set_affinity(struct irq_data *d,
>>   				const struct cpumask *mask_val,
>>   				bool force)
>> @@ -3915,13 +3984,21 @@ static void its_sgi_irq_domain_free(struct irq_domain *domain,
>>   static int its_sgi_irq_domain_activate(struct irq_domain *domain,
>>   				       struct irq_data *d, bool reserve)
>>   {
>> +	/* Write out the initial SGI configuration */
>> +	its_configure_sgi(d, false);
>>   	return 0;
>>   }
>>   
>>   static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
>>   					  struct irq_data *d)
>>   {
>> -	/* Nothing to do */
>> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>> +
>> +	/* First disable the SGI */
>> +	vpe->sgi_config[d->hwirq].enabled = false;
>> +	its_configure_sgi(d, false);
>> +	/* Now clear the potential pending bit (yes, this is clunky) */
> nit: Without carefuly reading the VSGI cmd notes, it is difficult to
> understand why those 2 steps are needed: maybe replace this comment by
> something like:
> to change the config, clear must be set to false. Then clear is set and
> this leaves the config unchanged. Both steps cannot be done concurrently.
> 
> "

I think it makes sense.


Thanks,
Zenghui

>> +	its_configure_sgi(d, true);
>>   }
>>   
>>   static struct irq_domain_ops its_sgi_domain_ops = {
>> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
>> index b28acfa71f82..fd3be49ac9a5 100644
>> --- a/include/linux/irqchip/arm-gic-v3.h
>> +++ b/include/linux/irqchip/arm-gic-v3.h
>> @@ -502,8 +502,9 @@
>>   #define GITS_CMD_VMAPTI			GITS_CMD_GICv4(GITS_CMD_MAPTI)
>>   #define GITS_CMD_VMOVI			GITS_CMD_GICv4(GITS_CMD_MOVI)
>>   #define GITS_CMD_VSYNC			GITS_CMD_GICv4(GITS_CMD_SYNC)
>> -/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
>> +/* VMOVP, VSGI and INVDB are the odd ones, as they dont have a physical counterpart */
>>   #define GITS_CMD_VMOVP			GITS_CMD_GICv4(2)
>> +#define GITS_CMD_VSGI			GITS_CMD_GICv4(3)
>>   #define GITS_CMD_INVDB			GITS_CMD_GICv4(0xe)
>>   
>>   /*
>>
> Thanks
> 
> Eric
> 
> 
> .
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs
  2020-03-04 20:33 ` [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs Marc Zyngier
  2020-03-12  8:12   ` Zenghui Yu
@ 2020-03-17  2:49   ` Zenghui Yu
  2020-03-19 10:55     ` Marc Zyngier
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-17  2:49 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> Now that we have HW-accelerated SGIs being delivered to VPEs, it
> becomes required to map the VPEs on all ITSs instead of relying
> on the lazy approach that we would use when using the ITS-list
> mechanism.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Before GICv4.1, we use vlpi_count to evaluate whether the vPE has been
mapped on the specified ITS, and use this refcount to only issue VMOVP
to those involved ITSes. It's broken after this patch.

We may need to re-evaluate "whether the vPE is mapped" now that we're at
GICv4.1, otherwise *no* VMOVP will be issued on the v4.1 capable machine
(I mean those without single VMOVP support).

What I'm saying is something like below (only an idea, it even can't
compile), any thoughts?


diff --git a/drivers/irqchip/irq-gic-v3-its.c 
b/drivers/irqchip/irq-gic-v3-its.c
index 2e12bc52e3a2..3450b5e847ca 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -198,7 +198,8 @@ static u16 get_its_list(struct its_vm *vm)
  		if (!is_v4(its))
  			continue;

-		if (vm->vlpi_count[its->list_nr])
+		if (vm->vlpi_count[its->list_nr] ||
+		    gic_requires_eager_mapping())
  			__set_bit(its->list_nr, &its_list);
  	}

@@ -1295,7 +1296,8 @@ static void its_send_vmovp(struct its_vpe *vpe)
  		if (!is_v4(its))
  			continue;

-		if (!vpe->its_vm->vlpi_count[its->list_nr])
+		if (!vpe->its_vm->vlpi_count[its->list_nr] &&
+		    !gic_requires_eager_mapping())
  			continue;

  		desc.its_vmovp_cmd.col = &its->collections[col_id];


Thanks,
Zenghui

> ---
>   drivers/irqchip/irq-gic-v3-its.c | 39 +++++++++++++++++++++++++-------
>   1 file changed, 31 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index b65fba67bd85..6277b3e3731f 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -1586,12 +1586,31 @@ static int its_irq_set_irqchip_state(struct irq_data *d,
>   	return 0;
>   }
>   
> +/*
> + * Two favourable cases:
> + *
> + * (a) Either we have a GICv4.1, and all vPEs have to be mapped at all times
> + *     for vSGI delivery
> + *
> + * (b) Or the ITSs do not use a list map, meaning that VMOVP is cheap enough
> + *     and we're better off mapping all VPEs always
> + *
> + * If neither (a) nor (b) is true, then we map vPEs on demand.
> + *
> + */
> +static bool gic_requires_eager_mapping(void)
> +{
> +	if (!its_list_map || gic_rdists->has_rvpeid)
> +		return true;
> +
> +	return false;
> +}
> +
>   static void its_map_vm(struct its_node *its, struct its_vm *vm)
>   {
>   	unsigned long flags;
>   
> -	/* Not using the ITS list? Everything is always mapped. */
> -	if (!its_list_map)
> +	if (gic_requires_eager_mapping())
>   		return;
>   
>   	raw_spin_lock_irqsave(&vmovp_lock, flags);
> @@ -1625,7 +1644,7 @@ static void its_unmap_vm(struct its_node *its, struct its_vm *vm)
>   	unsigned long flags;
>   
>   	/* Not using the ITS list? Everything is always mapped. */
> -	if (!its_list_map)
> +	if (gic_requires_eager_mapping())
>   		return;
>   
>   	raw_spin_lock_irqsave(&vmovp_lock, flags);
> @@ -4260,8 +4279,12 @@ static int its_vpe_irq_domain_activate(struct irq_domain *domain,
>   	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>   	struct its_node *its;
>   
> -	/* If we use the list map, we issue VMAPP on demand... */
> -	if (its_list_map)
> +	/*
> +	 * If we use the list map, we issue VMAPP on demand... Unless
> +	 * we're on a GICv4.1 and we eagerly map the VPE on all ITSs
> +	 * so that VSGIs can work.
> +	 */
> +	if (!gic_requires_eager_mapping())
>   		return 0;
>   
>   	/* Map the VPE to the first possible CPU */
> @@ -4287,10 +4310,10 @@ static void its_vpe_irq_domain_deactivate(struct irq_domain *domain,
>   	struct its_node *its;
>   
>   	/*
> -	 * If we use the list map, we unmap the VPE once no VLPIs are
> -	 * associated with the VM.
> +	 * If we use the list map on GICv4.0, we unmap the VPE once no
> +	 * VLPIs are associated with the VM.
>   	 */
> -	if (its_list_map)
> +	if (!gic_requires_eager_mapping())
>   		return;
>   
>   	list_for_each_entry(its, &its_nodes, entry) {
> 


^ permalink raw reply related	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration
  2020-03-17  2:02     ` Zenghui Yu
@ 2020-03-17  8:36       ` Auger Eric
  0 siblings, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-17  8:36 UTC (permalink / raw)
  To: Zenghui Yu, Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Zenghui,

On 3/17/20 3:02 AM, Zenghui Yu wrote:
> Hi Eric,
> 
> On 2020/3/17 1:53, Auger Eric wrote:
>> Hi Marc,
>>
>> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>>> The GICv4.1 ITS has yet another new command (VSGI) which allows
>>> a VPE-targeted SGI to be configured (or have its pending state
>>> cleared). Add support for this command and plumb it into the
>>> activate irqdomain callback so that it is ready to be used.
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
>>> ---
>>>   drivers/irqchip/irq-gic-v3-its.c   | 79 +++++++++++++++++++++++++++++-
>>>   include/linux/irqchip/arm-gic-v3.h |  3 +-
>>>   2 files changed, 80 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/drivers/irqchip/irq-gic-v3-its.c
>>> b/drivers/irqchip/irq-gic-v3-its.c
>>> index 112b452fcb40..e0db3f906f87 100644
>>> --- a/drivers/irqchip/irq-gic-v3-its.c
>>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>>> @@ -380,6 +380,15 @@ struct its_cmd_desc {
>>>           struct {
>>>               struct its_vpe *vpe;
>>>           } its_invdb_cmd;
>>> +
>>> +        struct {
>>> +            struct its_vpe *vpe;
>>> +            u8 sgi;
>>> +            u8 priority;
>>> +            bool enable;
>>> +            bool group;
>>> +            bool clear;
>>> +        } its_vsgi_cmd;
>>>       };
>>>   };
>>>   @@ -528,6 +537,31 @@ static void its_encode_db(struct its_cmd_block
>>> *cmd, bool db)
>>>       its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
>>>   }
>>>   +static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
>>> +{
>>> +    its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
>>> +}
>>> +
>>> +static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 prio)
>>> +{
>>> +    its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
>>> +}
>>> +
>>> +static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
>>> +{
>>> +    its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
>>> +}
>>> +
>>> +static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
>>> +{
>>> +    its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
>>> +}
>>> +
>>> +static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
>>> +{
>>> +    its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
>>> +}
>>> +
>>>   static inline void its_fixup_cmd(struct its_cmd_block *cmd)
>>>   {
>>>       /* Let's fixup BE commands */
>>> @@ -893,6 +927,26 @@ static struct its_vpe
>>> *its_build_invdb_cmd(struct its_node *its,
>>>       return valid_vpe(its, desc->its_invdb_cmd.vpe);
>>>   }
>>>   +static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
>>> +                      struct its_cmd_block *cmd,
>>> +                      struct its_cmd_desc *desc)
>>> +{
>>> +    if (WARN_ON(!is_v4_1(its)))
>>> +        return NULL;
>>> +
>>> +    its_encode_cmd(cmd, GITS_CMD_VSGI);
>>> +    its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
>>> +    its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
>>> +    its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
>>> +    its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
>>> +    its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
>>> +    its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
>>> +
>>> +    its_fixup_cmd(cmd);
>>> +
>>> +    return valid_vpe(its, desc->its_vsgi_cmd.vpe);
>>> +}
>>> +
>>>   static u64 its_cmd_ptr_to_offset(struct its_node *its,
>>>                    struct its_cmd_block *ptr)
>>>   {
>>> @@ -3870,6 +3924,21 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
>>>       .irq_set_vcpu_affinity    = its_vpe_4_1_set_vcpu_affinity,
>>>   };
>>>   +static void its_configure_sgi(struct irq_data *d, bool clear)
>>> +{
>>> +    struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>>> +    struct its_cmd_desc desc;
>>> +
>>> +    desc.its_vsgi_cmd.vpe = vpe;
>>> +    desc.its_vsgi_cmd.sgi = d->hwirq;
>>> +    desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
>>> +    desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
>>> +    desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
>>> +    desc.its_vsgi_cmd.clear = clear;
>>> +
>>> +    its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd,
>>> &desc);
>> I see we pick up the first 4.1 ITS with find_4_1_its(). Can it happen
>> that not all of them have a mapping for that vPEID and if so we should
>> rather use one that has this mapping?
> 
> It can't happen in GICv4.1, and you may find the answer in patch #16
> ("Eagerly vmap vPEs").  I also failed to follow this logic the first
> time looking at it [*], so I think it may worth adding some comments
> on top of find_4_1_its()?
> 
> [*]
> https://lore.kernel.org/lkml/c94061be-d029-69c8-d34f-4d21081d5aba@huawei.com/

OK thank you for the pointer. Slowly moving forward ... ;-)

Eric
> 
> 
>>
>> The spec says:
>> The ITS controls must only be used on an ITS that has a mapping for that
>> vPEID.
>> Where multiple ITSs have a mapping for the vPEID, any ITS with a mapping
>> may be used.
>>
>>> +}
>>> +
>>>   static int its_sgi_set_affinity(struct irq_data *d,
>>>                   const struct cpumask *mask_val,
>>>                   bool force)
>>> @@ -3915,13 +3984,21 @@ static void its_sgi_irq_domain_free(struct
>>> irq_domain *domain,
>>>   static int its_sgi_irq_domain_activate(struct irq_domain *domain,
>>>                          struct irq_data *d, bool reserve)
>>>   {
>>> +    /* Write out the initial SGI configuration */
>>> +    its_configure_sgi(d, false);
>>>       return 0;
>>>   }
>>>     static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
>>>                         struct irq_data *d)
>>>   {
>>> -    /* Nothing to do */
>>> +    struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>>> +
>>> +    /* First disable the SGI */
>>> +    vpe->sgi_config[d->hwirq].enabled = false;
>>> +    its_configure_sgi(d, false);
>>> +    /* Now clear the potential pending bit (yes, this is clunky) */
>> nit: Without carefuly reading the VSGI cmd notes, it is difficult to
>> understand why those 2 steps are needed: maybe replace this comment by
>> something like:
>> to change the config, clear must be set to false. Then clear is set and
>> this leaves the config unchanged. Both steps cannot be done concurrently.
>>
>> "
> 
> I think it makes sense.
> 
> 
> Thanks,
> Zenghui
> 
>>> +    its_configure_sgi(d, true);
>>>   }
>>>     static struct irq_domain_ops its_sgi_domain_ops = {
>>> diff --git a/include/linux/irqchip/arm-gic-v3.h
>>> b/include/linux/irqchip/arm-gic-v3.h
>>> index b28acfa71f82..fd3be49ac9a5 100644
>>> --- a/include/linux/irqchip/arm-gic-v3.h
>>> +++ b/include/linux/irqchip/arm-gic-v3.h
>>> @@ -502,8 +502,9 @@
>>>   #define GITS_CMD_VMAPTI            GITS_CMD_GICv4(GITS_CMD_MAPTI)
>>>   #define GITS_CMD_VMOVI            GITS_CMD_GICv4(GITS_CMD_MOVI)
>>>   #define GITS_CMD_VSYNC            GITS_CMD_GICv4(GITS_CMD_SYNC)
>>> -/* VMOVP and INVDB are the odd ones, as they dont have a physical
>>> counterpart */
>>> +/* VMOVP, VSGI and INVDB are the odd ones, as they dont have a
>>> physical counterpart */
>>>   #define GITS_CMD_VMOVP            GITS_CMD_GICv4(2)
>>> +#define GITS_CMD_VSGI            GITS_CMD_GICv4(3)
>>>   #define GITS_CMD_INVDB            GITS_CMD_GICv4(0xe)
>>>     /*
>>>
>> Thanks
>>
>> Eric
>>
>>
>> .
>>
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup
  2020-03-04 20:33 ` [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup Marc Zyngier
  2020-03-12  8:09   ` Zenghui Yu
@ 2020-03-17 10:30   ` Auger Eric
  2020-03-19 10:57     ` Marc Zyngier
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-17 10:30 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Add the SGI configuration entry point for KVM to use.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  drivers/irqchip/irq-gic-v3-its.c   |  2 +-
>  drivers/irqchip/irq-gic-v4.c       | 13 +++++++++++++
>  include/linux/irqchip/arm-gic-v4.h |  3 ++-
>  3 files changed, 16 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index effb0e0b0c9d..b65fba67bd85 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -4039,7 +4039,7 @@ static int its_sgi_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
>  	struct its_cmd_info *info = vcpu_info;
>  
>  	switch (info->cmd_type) {
> -	case PROP_UPDATE_SGI:
> +	case PROP_UPDATE_VSGI:
This change rather belongs to
[PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks
>  		vpe->sgi_config[d->hwirq].priority = info->priority;
>  		vpe->sgi_config[d->hwirq].group = info->group;
>  		its_configure_sgi(d, false);
> diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
> index 99b33f60ac63..0c18714ae13e 100644
> --- a/drivers/irqchip/irq-gic-v4.c
> +++ b/drivers/irqchip/irq-gic-v4.c
> @@ -320,6 +320,19 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
>  	return irq_set_vcpu_affinity(irq, &info);
>  }
>  
> +int its_prop_update_vsgi(int irq, u8 priority, bool group)
> +{
> +	struct its_cmd_info info = {
> +		.cmd_type = PROP_UPDATE_VSGI,
> +		{
> +			.priority	= priority,
> +			.group		= group,
> +		},
> +	};
> +
> +	return irq_set_vcpu_affinity(irq, &info);
> +}
> +
>  int its_init_v4(struct irq_domain *domain,
>  		const struct irq_domain_ops *vpe_ops,
>  		const struct irq_domain_ops *sgi_ops)
> diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
> index 0bb111b4a504..6976b8331b60 100644
> --- a/include/linux/irqchip/arm-gic-v4.h
> +++ b/include/linux/irqchip/arm-gic-v4.h
> @@ -105,7 +105,7 @@ enum its_vcpu_info_cmd_type {
>  	SCHEDULE_VPE,
>  	DESCHEDULE_VPE,
>  	INVALL_VPE,
> -	PROP_UPDATE_SGI,
> +	PROP_UPDATE_VSGI,
same
>  };
>  
>  struct its_cmd_info {
> @@ -134,6 +134,7 @@ int its_map_vlpi(int irq, struct its_vlpi_map *map);
>  int its_get_vlpi(int irq, struct its_vlpi_map *map);
>  int its_unmap_vlpi(int irq);
>  int its_prop_update_vlpi(int irq, u8 config, bool inv);
> +int its_prop_update_vsgi(int irq, u8 priority, bool group);
>  
>  struct irq_domain_ops;
>  int its_init_v4(struct irq_domain *domain,
> 
Besides
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity " Marc Zyngier
@ 2020-03-17 10:35   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-17 10:35 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Just like for vLPIs, there is some configuration information that cannot
> be directly communicated through the normal irqchip API, and we have to
> use our good old friend set_vcpu_affinity as a side-band communication
> mechanism.
> 
> This is used to configure group and priority for a given vSGI.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
> ---
>  drivers/irqchip/irq-gic-v3-its.c   | 18 ++++++++++++++++++
>  include/linux/irqchip/arm-gic-v4.h |  5 +++++
>  2 files changed, 23 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index fb2b836c31ff..effb0e0b0c9d 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -4033,6 +4033,23 @@ static int its_sgi_get_irqchip_state(struct irq_data *d,
>  	return 0;
>  }
>  
> +static int its_sgi_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
> +{
> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
> +	struct its_cmd_info *info = vcpu_info;
> +
> +	switch (info->cmd_type) {
> +	case PROP_UPDATE_SGI:
PROP_UPDATE_VSGI directly?
> +		vpe->sgi_config[d->hwirq].priority = info->priority;
> +		vpe->sgi_config[d->hwirq].group = info->group;
> +		its_configure_sgi(d, false);
> +		return 0;
> +
extra line
> +	default:
> +		return -EINVAL;
> +	}
> +}
> +
>  static struct irq_chip its_sgi_irq_chip = {
>  	.name			= "GICv4.1-sgi",
>  	.irq_mask		= its_sgi_mask_irq,
> @@ -4040,6 +4057,7 @@ static struct irq_chip its_sgi_irq_chip = {
>  	.irq_set_affinity	= its_sgi_set_affinity,
>  	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
>  	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
> +	.irq_set_vcpu_affinity	= its_sgi_set_vcpu_affinity,
>  };
>  
>  static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
> diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
> index 44e8c19e3d56..b4dbf899460b 100644
> --- a/include/linux/irqchip/arm-gic-v4.h
> +++ b/include/linux/irqchip/arm-gic-v4.h
> @@ -103,6 +103,7 @@ enum its_vcpu_info_cmd_type {
>  	SCHEDULE_VPE,
>  	DESCHEDULE_VPE,
>  	INVALL_VPE,
> +	PROP_UPDATE_SGI,
>  };
>  
>  struct its_cmd_info {
> @@ -115,6 +116,10 @@ struct its_cmd_info {
>  			bool		g0en;
>  			bool		g1en;
>  		};
> +		struct {
> +			u8		priority;
> +			bool		group;
> +		};
>  	};
>  };
>  
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled
  2020-03-04 20:33 ` [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled Marc Zyngier
  2020-03-12  8:15   ` Zenghui Yu
@ 2020-03-17 11:04   ` Auger Eric
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-17 11:04 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> As GICv4.1 understands the life cycle of doorbells (instead of
> just randomly firing them at the most inconvenient time), just
> enable them at irq_request time, and be done with it.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric

> ---
>  virt/kvm/arm/vgic/vgic-v4.c | 9 ++++++++-
>  1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
> index 1eb0f8c76219..c2fcde104ea2 100644
> --- a/virt/kvm/arm/vgic/vgic-v4.c
> +++ b/virt/kvm/arm/vgic/vgic-v4.c
> @@ -141,6 +141,7 @@ int vgic_v4_init(struct kvm *kvm)
>  
>  	kvm_for_each_vcpu(i, vcpu, kvm) {
>  		int irq = dist->its_vm.vpes[i]->irq;
> +		unsigned long irq_flags = DB_IRQ_FLAGS;
>  
>  		/*
>  		 * Don't automatically enable the doorbell, as we're
> @@ -148,8 +149,14 @@ int vgic_v4_init(struct kvm *kvm)
>  		 * blocked. Also disable the lazy disabling, as the
>  		 * doorbell could kick us out of the guest too
>  		 * early...
> +		 *
> +		 * On GICv4.1, the doorbell is managed in HW and must
> +		 * be left enabled.
>  		 */
> -		irq_set_status_flags(irq, DB_IRQ_FLAGS);
> +		if (kvm_vgic_global_state.has_gicv4_1)
> +			irq_flags &= ~IRQ_NOAUTOEN;
> +		irq_set_status_flags(irq, irq_flags);
> +
>  		ret = request_irq(irq, vgic_v4_doorbell_handler,
>  				  0, "vcpu", vcpu);
>  		if (ret) {
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable
  2020-03-04 20:33 ` [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable Marc Zyngier
@ 2020-03-18  3:17   ` Zenghui Yu
  2020-03-19 12:18     ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-18  3:17 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

On 2020/3/5 4:33, Marc Zyngier wrote:
> Each time a Group-enable bit gets flipped, the state of these bits
> needs to be forwarded to the hardware. This is a pretty heavy
> handed operation, requiring all vcpus to reload their GICv4
> configuration. It is thus implemented as a new request type.

[note to myself]
... and the status are forwarded to HW by programming VGrp{0,1}En
fields of GICR_VPENDBASER when vPEs are made resident next time.

> 
> Of course, we only support Group-1 for now...
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-04 20:33 ` [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs Marc Zyngier
@ 2020-03-18  3:19   ` Zenghui Yu
  2020-03-19 15:05   ` Auger Eric
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-18  3:19 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> The vgic-state debugfs file could do with showing the pending state
> of the HW-backed SGIs. Plug it into the low-level code.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks

> ---
>   virt/kvm/arm/vgic/vgic-debug.c | 14 +++++++++++++-
>   1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-debug.c b/virt/kvm/arm/vgic/vgic-debug.c
> index cc12fe9b2df3..b13a9e3f99dd 100644
> --- a/virt/kvm/arm/vgic/vgic-debug.c
> +++ b/virt/kvm/arm/vgic/vgic-debug.c
> @@ -178,6 +178,8 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
>   			    struct kvm_vcpu *vcpu)
>   {
>   	char *type;
> +	bool pending;
> +
>   	if (irq->intid < VGIC_NR_SGIS)
>   		type = "SGI";
>   	else if (irq->intid < VGIC_NR_PRIVATE_IRQS)
> @@ -190,6 +192,16 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
>   	if (irq->intid ==0 || irq->intid == VGIC_NR_PRIVATE_IRQS)
>   		print_header(s, irq, vcpu);
>   
> +	pending = irq->pending_latch;
> +	if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +		int err;
> +
> +		err = irq_get_irqchip_state(irq->host_irq,
> +					    IRQCHIP_STATE_PENDING,
> +					    &pending);
> +		WARN_ON_ONCE(err);
> +	}
> +
>   	seq_printf(s, "       %s %4d "
>   		      "    %2d "
>   		      "%d%d%d%d%d%d%d "
> @@ -201,7 +213,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
>   		      "\n",
>   			type, irq->intid,
>   			(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
> -			irq->pending_latch,
> +			pending,
>   			irq->line_level,
>   			irq->active,
>   			irq->enabled,
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
  2020-03-04 20:33 ` [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers Marc Zyngier
@ 2020-03-18  3:28   ` Zenghui Yu
  2020-03-20  8:11   ` Auger Eric
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-18  3:28 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> Most of the GICv3 emulation code that deals with SGIs now has to be
> aware of the v4.1 capabilities in order to benefit from it.
> 
> Add such support, keyed on the interrupt having the hw flag set and
> being a SGI.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-04 20:33 ` [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor Marc Zyngier
@ 2020-03-18  6:34   ` Zenghui Yu
  2020-03-19 12:10     ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-18  6:34 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> The GICv4.1 architecture gives the hypervisor the option to let
> the guest choose whether it wants the good old SGIs with an
> active state, or the new, HW-based ones that do not have one.
> 
> For this, plumb the configuration of SGIs into the GICv3 MMIO
> handling, present the GICD_TYPER2.nASSGIcap to the guest,
> and handle the GICD_CTLR.nASSGIreq setting.
> 
> In order to be able to deal with the restore of a guest, also
> apply the GICD_CTLR.nASSGIreq setting at first run so that we
> can move the restored SGIs to the HW if that's what the guest
> had selected in a previous life.

I'm okay with the restore path.  But it seems that we still fail to
save the pending state of vSGI - software pending_latch of HW-based
vSGIs will not be updated (and always be false) because we directly
inject them through ITS, so vgic_v3_uaccess_read_pending() can't
tell the correct pending state to user-space (the correct one should
be latched in HW).

It would be good if we can sync the hardware state into pending_latch
at an appropriate time (just before save), but not sure if we can...

> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>   virt/kvm/arm/vgic/vgic-mmio-v3.c | 48 ++++++++++++++++++++++++++++++--
>   virt/kvm/arm/vgic/vgic-v3.c      |  2 ++
>   2 files changed, 48 insertions(+), 2 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> index de89da76a379..442f3b8c2559 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> @@ -3,6 +3,7 @@
>    * VGICv3 MMIO handling functions
>    */
>   
> +#include <linux/bitfield.h>
>   #include <linux/irqchip/arm-gic-v3.h>
>   #include <linux/kvm.h>
>   #include <linux/kvm_host.h>
> @@ -70,6 +71,8 @@ static unsigned long vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
>   		if (vgic->enabled)
>   			value |= GICD_CTLR_ENABLE_SS_G1;
>   		value |= GICD_CTLR_ARE_NS | GICD_CTLR_DS;
> +		if (kvm_vgic_global_state.has_gicv4_1 && vgic->nassgireq)

Looking at how we handle the GICD_CTLR.nASSGIreq setting, I think
"nassgireq==true" already indicates "has_gicv4_1==true".  So this
can be simplified.

But I wonder that should we use nassgireq to *only* keep track what
the guest had written into the GICD_CTLR.nASSGIreq.  If not, we may
lose the guest-request bit after migration among hosts with different
has_gicv4_1 settings.


The remaining patches all look good to me :-). I will wait for you to
confirm these two concerns.


Thanks,
Zenghui

> +			value |= GICD_CTLR_nASSGIreq;
>   		break;
>   	case GICD_TYPER:
>   		value = vgic->nr_spis + VGIC_NR_PRIVATE_IRQS;
> @@ -81,6 +84,10 @@ static unsigned long vgic_mmio_read_v3_misc(struct kvm_vcpu *vcpu,
>   			value |= (INTERRUPT_ID_BITS_SPIS - 1) << 19;
>   		}
>   		break;
> +	case GICD_TYPER2:
> +		if (kvm_vgic_global_state.has_gicv4_1)
> +			value = GICD_TYPER2_nASSGIcap;
> +		break;
>   	case GICD_IIDR:
>   		value = (PRODUCT_ID_KVM << GICD_IIDR_PRODUCT_ID_SHIFT) |
>   			(vgic->implementation_rev << GICD_IIDR_REVISION_SHIFT) |
> @@ -98,17 +105,43 @@ static void vgic_mmio_write_v3_misc(struct kvm_vcpu *vcpu,
>   				    unsigned long val)
>   {
>   	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> -	bool was_enabled = dist->enabled;
>   
>   	switch (addr & 0x0c) {
> -	case GICD_CTLR:
> +	case GICD_CTLR: {
> +		bool was_enabled, is_hwsgi;
> +
> +		mutex_lock(&vcpu->kvm->lock);
> +
> +		was_enabled = dist->enabled;
> +		is_hwsgi = dist->nassgireq;
> +
>   		dist->enabled = val & GICD_CTLR_ENABLE_SS_G1;
>   
> +		/* Not a GICv4.1? No HW SGIs */
> +		if (!kvm_vgic_global_state.has_gicv4_1)
> +			val &= ~GICD_CTLR_nASSGIreq;
> +
> +		/* Dist stays enabled? nASSGIreq is RO */
> +		if (was_enabled && dist->enabled) {
> +			val &= ~GICD_CTLR_nASSGIreq;
> +			val |= FIELD_PREP(GICD_CTLR_nASSGIreq, is_hwsgi);
> +		}
> +
> +		/* Switching HW SGIs? */
> +		dist->nassgireq = val & GICD_CTLR_nASSGIreq;
> +		if (is_hwsgi != dist->nassgireq)
> +			vgic_v4_configure_vsgis(vcpu->kvm);
> +
>   		if (!was_enabled && dist->enabled)
>   			vgic_kick_vcpus(vcpu->kvm);
> +
> +		mutex_unlock(&vcpu->kvm->lock);
>   		break;
> +	}
>   	case GICD_TYPER:
> +	case GICD_TYPER2:
>   	case GICD_IIDR:
> +		/* This is at best for documentation purposes... */
>   		return;
>   	}
>   }
> @@ -117,10 +150,21 @@ static int vgic_mmio_uaccess_write_v3_misc(struct kvm_vcpu *vcpu,
>   					   gpa_t addr, unsigned int len,
>   					   unsigned long val)
>   {
> +	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> +
>   	switch (addr & 0x0c) {
>   	case GICD_IIDR:
>   		if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
>   			return -EINVAL;
> +		return 0;
> +	case GICD_CTLR:
> +		/* Not a GICv4.1? No HW SGIs */
> +		if (!kvm_vgic_global_state.has_gicv4_1)
> +			val &= ~GICD_CTLR_nASSGIreq;
> +
> +		dist->enabled = val & GICD_CTLR_ENABLE_SS_G1;
> +		dist->nassgireq = val & GICD_CTLR_nASSGIreq;
> +		return 0;
>   	}
>   
>   	vgic_mmio_write_v3_misc(vcpu, addr, len, val);
> diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
> index 1bc09b523486..2c9fc13e2c59 100644
> --- a/virt/kvm/arm/vgic/vgic-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-v3.c
> @@ -540,6 +540,8 @@ int vgic_v3_map_resources(struct kvm *kvm)
>   		goto out;
>   	}
>   
> +	if (kvm_vgic_global_state.has_gicv4_1)
> +		vgic_v4_configure_vsgis(kvm);
>   	dist->ready = true;
>   
>   out:
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip
  2020-03-16 17:10   ` Auger Eric
@ 2020-03-19 10:03     ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 10:03 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-16 17:10, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> Since GICv4.1 has the capability to inject 16 SGIs into each VPE,
>> and that I'm keen not to invent too many specific interfaces to
>> manipulate these interrupts, let's pretend that each of these SGIs
>> is an actual Linux interrupt.
>> 
>> For that matter, let's introduce a minimal irqchip and irqdomain
>> setup that will get fleshed up in the following patches.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
>> ---
>>  drivers/irqchip/irq-gic-v3-its.c   | 68 
>> +++++++++++++++++++++++++++++-
>>  drivers/irqchip/irq-gic-v4.c       |  8 +++-
>>  include/linux/irqchip/arm-gic-v4.h |  9 +++-
>>  3 files changed, 81 insertions(+), 4 deletions(-)
>> 
>> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
>> b/drivers/irqchip/irq-gic-v3-its.c
>> index 54d6fdf7a28e..112b452fcb40 100644
>> --- a/drivers/irqchip/irq-gic-v3-its.c
>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>> @@ -3870,6 +3870,67 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
>>  	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
>>  };
>> 
>> +static int its_sgi_set_affinity(struct irq_data *d,
>> +				const struct cpumask *mask_val,
>> +				bool force)
>> +{
>> +	return -EINVAL;
>> +}
>> +
>> +static struct irq_chip its_sgi_irq_chip = {
>> +	.name			= "GICv4.1-sgi",
>> +	.irq_set_affinity	= its_sgi_set_affinity,
>> +};
> nit: const?

That would create a warning with irq_domain_set_hwirq_and_chip(), which
doesn't take a const argument. I think this is fixable in the long run,
but only as a sweeping tree-wide change.

>> +
>> +static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
>> +				    unsigned int virq, unsigned int nr_irqs,
>> +				    void *args)
>> +{
>> +	struct its_vpe *vpe = args;
>> +	int i;
>> +
>> +	/* Yes, we do want 16 SGIs */
>> +	WARN_ON(nr_irqs != 16);
>> +
>> +	for (i = 0; i < 16; i++) {
>> +		vpe->sgi_config[i].priority = 0;
>> +		vpe->sgi_config[i].enabled = false;
>> +		vpe->sgi_config[i].group = false;
>> +
>> +		irq_domain_set_hwirq_and_chip(domain, virq + i, i,
>> +					      &its_sgi_irq_chip, vpe);
>> +		irq_set_status_flags(virq + i, IRQ_DISABLE_UNLAZY);
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static void its_sgi_irq_domain_free(struct irq_domain *domain,
>> +				    unsigned int virq,
>> +				    unsigned int nr_irqs)
>> +{
>> +	/* Nothing to do */
>> +}
>> +
>> +static int its_sgi_irq_domain_activate(struct irq_domain *domain,
>> +				       struct irq_data *d, bool reserve)
>> +{
>> +	return 0;
>> +}
>> +
>> +static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
>> +					  struct irq_data *d)
>> +{
>> +	/* Nothing to do */
>> +}
>> +
>> +static struct irq_domain_ops its_sgi_domain_ops = {
>> +	.alloc		= its_sgi_irq_domain_alloc,
>> +	.free		= its_sgi_irq_domain_free,
>> +	.activate	= its_sgi_irq_domain_activate,
>> +	.deactivate	= its_sgi_irq_domain_deactivate,
>> +};
> nit: const?

This one can work with a bit of surgery below:

diff --git a/drivers/irqchip/irq-gic-v3-its.c 
b/drivers/irqchip/irq-gic-v3-its.c
index 2e12bc52e3a2..321f73015d6c 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4129,7 +4129,7 @@ static void its_sgi_irq_domain_deactivate(struct 
irq_domain *domain,
  	its_configure_sgi(d, true);
  }

-static struct irq_domain_ops its_sgi_domain_ops = {
+static const struct irq_domain_ops its_sgi_domain_ops = {
  	.alloc		= its_sgi_irq_domain_alloc,
  	.free		= its_sgi_irq_domain_free,
  	.activate	= its_sgi_irq_domain_activate,
@@ -5182,10 +5182,12 @@ int __init its_init(struct fwnode_handle 
*handle, struct rdists *rdists,
  		rdists->has_rvpeid = false;

  	if (has_v4 & rdists->has_vlpis) {
-		struct irq_domain_ops *sgi_ops = NULL;
+		const struct irq_domain_ops *sgi_ops;

  		if (has_v4_1)
  			sgi_ops = &its_sgi_domain_ops;
+		else
+			sgi_ops = NULL;

  		if (its_init_vpe_domain() ||
  		    its_init_v4(parent_domain, &its_vpe_domain_ops, sgi_ops)) {

>> +
>>  static int its_vpe_id_alloc(void)
>>  {
>>  	return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
>> @@ -4912,8 +4973,13 @@ int __init its_init(struct fwnode_handle 
>> *handle, struct rdists *rdists,
>>  		rdists->has_rvpeid = false;
>> 
>>  	if (has_v4 & rdists->has_vlpis) {
>> +		struct irq_domain_ops *sgi_ops = NULL;
>> +
>> +		if (has_v4_1)
>> +			sgi_ops = &its_sgi_domain_ops;
>> +
>>  		if (its_init_vpe_domain() ||
>> -		    its_init_v4(parent_domain, &its_vpe_domain_ops)) {
>> +		    its_init_v4(parent_domain, &its_vpe_domain_ops, sgi_ops)) {
>>  			rdists->has_vlpis = false;
>>  			pr_err("ITS: Disabling GICv4 support\n");
>>  		}
>> diff --git a/drivers/irqchip/irq-gic-v4.c 
>> b/drivers/irqchip/irq-gic-v4.c
>> index 45969927cc81..c01910d53f9e 100644
>> --- a/drivers/irqchip/irq-gic-v4.c
>> +++ b/drivers/irqchip/irq-gic-v4.c
>> @@ -85,6 +85,7 @@
>> 
>>  static struct irq_domain *gic_domain;
>>  static const struct irq_domain_ops *vpe_domain_ops;
>> +static const struct irq_domain_ops *sgi_domain_ops;
>> 
>>  int its_alloc_vcpu_irqs(struct its_vm *vm)
>>  {
>> @@ -216,12 +217,15 @@ int its_prop_update_vlpi(int irq, u8 config, 
>> bool inv)
>>  	return irq_set_vcpu_affinity(irq, &info);
>>  }
>> 
>> -int its_init_v4(struct irq_domain *domain, const struct 
>> irq_domain_ops *ops)
>> +int its_init_v4(struct irq_domain *domain,
>> +		const struct irq_domain_ops *vpe_ops,
>> +		const struct irq_domain_ops *sgi_ops)
>>  {
>>  	if (domain) {
>>  		pr_info("ITS: Enabling GICv4 support\n");
>>  		gic_domain = domain;
>> -		vpe_domain_ops = ops;
>> +		vpe_domain_ops = vpe_ops;
>> +		sgi_domain_ops = sgi_ops;
>>  		return 0;
>>  	}
>> 
>> diff --git a/include/linux/irqchip/arm-gic-v4.h 
>> b/include/linux/irqchip/arm-gic-v4.h
>> index 439963f4c66a..44e8c19e3d56 100644
>> --- a/include/linux/irqchip/arm-gic-v4.h
>> +++ b/include/linux/irqchip/arm-gic-v4.h
>> @@ -49,6 +49,11 @@ struct its_vpe {
>>  		};
>>  		/* GICv4.1 implementations */
>>  		struct {
>> +			struct {
>> +				u8	priority;
>> +				bool	enabled;
>> +				bool	group;
>> +			}			sgi_config[16];
>>  			atomic_t vmapp_count;
>>  		};
>>  	};
>> @@ -123,6 +128,8 @@ int its_unmap_vlpi(int irq);
>>  int its_prop_update_vlpi(int irq, u8 config, bool inv);
>> 
>>  struct irq_domain_ops;
>> -int its_init_v4(struct irq_domain *domain, const struct 
>> irq_domain_ops *ops);
>> +int its_init_v4(struct irq_domain *domain,
>> +		const struct irq_domain_ops *vpe_ops,
>> +		const struct irq_domain_ops *sgi_ops);
>> 
>>  #endif
>> 
> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration
  2020-03-16 17:53   ` Auger Eric
  2020-03-17  2:02     ` Zenghui Yu
@ 2020-03-19 10:20     ` Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 10:20 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-16 17:53, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> The GICv4.1 ITS has yet another new command (VSGI) which allows
>> a VPE-targeted SGI to be configured (or have its pending state
>> cleared). Add support for this command and plumb it into the
>> activate irqdomain callback so that it is ready to be used.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
>> ---
>>  drivers/irqchip/irq-gic-v3-its.c   | 79 
>> +++++++++++++++++++++++++++++-
>>  include/linux/irqchip/arm-gic-v3.h |  3 +-
>>  2 files changed, 80 insertions(+), 2 deletions(-)
>> 
>> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
>> b/drivers/irqchip/irq-gic-v3-its.c
>> index 112b452fcb40..e0db3f906f87 100644
>> --- a/drivers/irqchip/irq-gic-v3-its.c
>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>> @@ -380,6 +380,15 @@ struct its_cmd_desc {
>>  		struct {
>>  			struct its_vpe *vpe;
>>  		} its_invdb_cmd;
>> +
>> +		struct {
>> +			struct its_vpe *vpe;
>> +			u8 sgi;
>> +			u8 priority;
>> +			bool enable;
>> +			bool group;
>> +			bool clear;
>> +		} its_vsgi_cmd;
>>  	};
>>  };
>> 
>> @@ -528,6 +537,31 @@ static void its_encode_db(struct its_cmd_block 
>> *cmd, bool db)
>>  	its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
>>  }
>> 
>> +static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
>> +}
>> +
>> +static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 
>> prio)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
>> +}
>> +
>> +static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
>> +}
>> +
>> +static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
>> +}
>> +
>> +static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
>> +{
>> +	its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
>> +}
>> +
>>  static inline void its_fixup_cmd(struct its_cmd_block *cmd)
>>  {
>>  	/* Let's fixup BE commands */
>> @@ -893,6 +927,26 @@ static struct its_vpe *its_build_invdb_cmd(struct 
>> its_node *its,
>>  	return valid_vpe(its, desc->its_invdb_cmd.vpe);
>>  }
>> 
>> +static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
>> +					  struct its_cmd_block *cmd,
>> +					  struct its_cmd_desc *desc)
>> +{
>> +	if (WARN_ON(!is_v4_1(its)))
>> +		return NULL;
>> +
>> +	its_encode_cmd(cmd, GITS_CMD_VSGI);
>> +	its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
>> +	its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
>> +	its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
>> +	its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
>> +	its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
>> +	its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
>> +
>> +	its_fixup_cmd(cmd);
>> +
>> +	return valid_vpe(its, desc->its_vsgi_cmd.vpe);
>> +}
>> +
>>  static u64 its_cmd_ptr_to_offset(struct its_node *its,
>>  				 struct its_cmd_block *ptr)
>>  {
>> @@ -3870,6 +3924,21 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
>>  	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
>>  };
>> 
>> +static void its_configure_sgi(struct irq_data *d, bool clear)
>> +{
>> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>> +	struct its_cmd_desc desc;
>> +
>> +	desc.its_vsgi_cmd.vpe = vpe;
>> +	desc.its_vsgi_cmd.sgi = d->hwirq;
>> +	desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
>> +	desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
>> +	desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
>> +	desc.its_vsgi_cmd.clear = clear;
>> +
>> +	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
> I see we pick up the first 4.1 ITS with find_4_1_its(). Can it happen
> that not all of them have a mapping for that vPEID and if so we should
> rather use one that has this mapping?
> 
> The spec says:
> The ITS controls must only be used on an ITS that has a mapping for 
> that
> vPEID.
> Where multiple ITSs have a mapping for the vPEID, any ITS with a 
> mapping
> may be used.

As Zenghui pointed out, we eagerly map all the VPEs, as we need the 
vSGIs
to be delivered by the ITS (yes, this is pretty backward, both in the 
series
and the architecture).

I'll add this coment to lift the ambiguity:

	/*
	 * GICv4.1 allows us to send VSGI commands to any ITS as long as the
	 * destination VPE is mapped there. Since we map them eagerly at
	 * activation time, we're pretty sure the first GICv4.1 ITS will do.
	 */

> 
>> +}
>> +
>>  static int its_sgi_set_affinity(struct irq_data *d,
>>  				const struct cpumask *mask_val,
>>  				bool force)
>> @@ -3915,13 +3984,21 @@ static void its_sgi_irq_domain_free(struct 
>> irq_domain *domain,
>>  static int its_sgi_irq_domain_activate(struct irq_domain *domain,
>>  				       struct irq_data *d, bool reserve)
>>  {
>> +	/* Write out the initial SGI configuration */
>> +	its_configure_sgi(d, false);
>>  	return 0;
>>  }
>> 
>>  static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
>>  					  struct irq_data *d)
>>  {
>> -	/* Nothing to do */
>> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>> +
>> +	/* First disable the SGI */
>> +	vpe->sgi_config[d->hwirq].enabled = false;
>> +	its_configure_sgi(d, false);
>> +	/* Now clear the potential pending bit (yes, this is clunky) */
> nit: Without carefuly reading the VSGI cmd notes, it is difficult to
> understand why those 2 steps are needed: maybe replace this comment by
> something like:
> to change the config, clear must be set to false. Then clear is set and
> this leaves the config unchanged. Both steps cannot be done 
> concurrently.

I've added this:

	/*
	 * The VSGI command is awkward:
	 *
	 * - To change the configuration, CLEAR must be set to false,
	 *   leaving the pending bit unchanged.
	 * - To clear the pending bit, CLEAR must be set to true, leaving
	 *   the configuration unchanged.
	 *
	 * You just can't do both at once, hence the two commands below.
	 */

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
  2020-03-16 21:43   ` Auger Eric
@ 2020-03-19 10:27     ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 10:27 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-16 21:43, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> To implement the get/set_irqchip_state callbacks (limited to the
>> PENDING state), we have to use a particular set of hacks:
>> 
>> - Reading the pending state is done by using a pair of new 
>> redistributor
>>   registers (GICR_VSGIR, GICR_VSGIPENDR), which allow the 16 
>> interrupts
>>   state to be retrieved.
>> - Setting the pending state is done by generating it as we'd otherwise 
>> do
>>   for a guest (writing to GITS_SGIR).
>> - Clearing the pending state is done by emiting a VSGI command with 
>> the
> emitting
>>   "clear" bit set.
>> 
>> This requires some interesting locking though:
>> - When talking to the redistributor, we must make sure that the VPE
>>   affinity doesn't change, hence taking the VPE lock.
>> - At the same time, we must ensure that nobody accesses the same
>>   redistributor's GICR_VSGI*R registers for a different VPE, which
> GICR_VSGIR
>>   would corrupt the reading of the pending bits. We thus take the
>>   per-RD spinlock. Much fun.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  drivers/irqchip/irq-gic-v3-its.c   | 73 
>> ++++++++++++++++++++++++++++++
>>  include/linux/irqchip/arm-gic-v3.h | 14 ++++++
>>  2 files changed, 87 insertions(+)
>> 
>> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
>> b/drivers/irqchip/irq-gic-v3-its.c
>> index c93f178914ee..fb2b836c31ff 100644
>> --- a/drivers/irqchip/irq-gic-v3-its.c
>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>> @@ -3962,11 +3962,84 @@ static int its_sgi_set_affinity(struct 
>> irq_data *d,
>>  	return -EINVAL;
>>  }
>> 
>> +static int its_sgi_set_irqchip_state(struct irq_data *d,
>> +				     enum irqchip_irq_state which,
>> +				     bool state)
>> +{
>> +	if (which != IRQCHIP_STATE_PENDING)
>> +		return -EINVAL;
>> +
>> +	if (state) {
>> +		struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>> +		struct its_node *its = find_4_1_its();
>> +		u64 val;
>> +
>> +		val  = FIELD_PREP(GITS_SGIR_VPEID, vpe->vpe_id);
>> +		val |= FIELD_PREP(GITS_SGIR_VINTID, d->hwirq);
>> +		writeq_relaxed(val, its->sgir_base + GITS_SGIR - SZ_128K);
>> +	} else {
>> +		its_configure_sgi(d, true);
>> +	}
>> +
>> +	return 0;
>> +}
>> +
>> +static int its_sgi_get_irqchip_state(struct irq_data *d,
>> +				     enum irqchip_irq_state which, bool *val)
>> +{
>> +	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
>> +	void __iomem *base;
>> +	unsigned long flags;
>> +	u32 count = 1000000;	/* 1s! */
> where does it come from? I did not find any info in the spec about this
> delay.

There is no such thing in the spec. However, these timeouts have proven 
to be
very useful in detecting broken HW (I'm lucky enough to have such 
wonders
at hand...), as well as SW bugs. The ! second value is purely empirical
(if a whole second is not enough for things to move, they're as good as 
dead).

>> +	u32 status;
>> +	int cpu;
>> +
>> +	if (which != IRQCHIP_STATE_PENDING)
>> +		return -EINVAL;
>> +
>> +	/*
>> +	 * Locking galore! We can race against two different events:
>> +	 *
>> +	 * - Concurent vPE affinity change: we must make sure it cannot
>> +         *   happen, or we'll talk to the wrong redistributor. This 
>> is
>> +         *   identical to what happens with vLPIs.
>> +	 *
>> +	 * - Concurrent VSGIPENDR access: As it involves accessing two
>> +         *   MMIO registers, this must be made atomic one way or 
>> another.
>> +	 */
>> +	cpu = vpe_to_cpuid_lock(vpe, &flags);
>> +	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
>> +	base = gic_data_rdist_cpu(cpu)->rd_base + SZ_128K;
>> +	writel_relaxed(vpe->vpe_id, base + GICR_VSGIR);
>> +	do {
>> +		status = readl_relaxed(base + GICR_VSGIPENDR);
>> +		if (!(status & GICR_VSGIPENDR_BUSY))
>> +			goto out;
>> +
>> +		count--;
>> +		if (!count) {
>> +			pr_err_ratelimited("Unable to get SGI status\n");
>> +			goto out;
>> +		}
>> +		cpu_relax();
>> +		udelay(1);
>> +	} while(count);
>> +
>> +out:
>> +	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
>> +	vpe_to_cpuid_unlock(vpe, flags);
>> +	*val = !!(status & (1 << d->hwirq));
>> +
>> +	return 0;
> cascade an error on timeout?

Sure, that's a good idea.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs
  2020-03-17  2:49   ` Zenghui Yu
@ 2020-03-19 10:55     ` Marc Zyngier
  2020-03-20  2:31       ` Zenghui Yu
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 10:55 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

On 2020-03-17 02:49, Zenghui Yu wrote:
> Hi Marc,
> 
> On 2020/3/5 4:33, Marc Zyngier wrote:
>> Now that we have HW-accelerated SGIs being delivered to VPEs, it
>> becomes required to map the VPEs on all ITSs instead of relying
>> on the lazy approach that we would use when using the ITS-list
>> mechanism.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
> 
> Before GICv4.1, we use vlpi_count to evaluate whether the vPE has been
> mapped on the specified ITS, and use this refcount to only issue VMOVP
> to those involved ITSes. It's broken after this patch.
> 
> We may need to re-evaluate "whether the vPE is mapped" now that we're 
> at
> GICv4.1, otherwise *no* VMOVP will be issued on the v4.1 capable 
> machine
> (I mean those without single VMOVP support).
> 
> What I'm saying is something like below (only an idea, it even can't
> compile), any thoughts?
> 
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
> b/drivers/irqchip/irq-gic-v3-its.c
> index 2e12bc52e3a2..3450b5e847ca 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -198,7 +198,8 @@ static u16 get_its_list(struct its_vm *vm)
>  		if (!is_v4(its))
>  			continue;
> 
> -		if (vm->vlpi_count[its->list_nr])
> +		if (vm->vlpi_count[its->list_nr] ||
> +		    gic_requires_eager_mapping())
>  			__set_bit(its->list_nr, &its_list);
>  	}
> 
> @@ -1295,7 +1296,8 @@ static void its_send_vmovp(struct its_vpe *vpe)
>  		if (!is_v4(its))
>  			continue;
> 
> -		if (!vpe->its_vm->vlpi_count[its->list_nr])
> +		if (!vpe->its_vm->vlpi_count[its->list_nr] &&
> +		    !gic_requires_eager_mapping())
>  			continue;
> 
>  		desc.its_vmovp_cmd.col = &its->collections[col_id];

It took me a while to wrap my head around that one, but you're as usual 
spot on.

The use of gic_requires_eager_mapping() is a bit confusing here, as it 
isn't
so much that the VPE is eagerly mapped, but the predicate on which we 
evaluate
the need for a VMOVP. How about this:

diff --git a/drivers/irqchip/irq-gic-v3-its.c 
b/drivers/irqchip/irq-gic-v3-its.c
index cd6451e190c9..348f7a909a69 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -189,6 +189,15 @@ static DEFINE_IDA(its_vpeid_ida);
  #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
  #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)

+/*
+ * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
+ * always have vSGIs mapped.
+ */
+static bool require_its_list_vmovp(struct its_vm *vm, struct its_node 
*its)
+{
+	return (gic_rdists->has_rvpeid || vm->vlpi_count[its->list_nr]);
+}
+
  static u16 get_its_list(struct its_vm *vm)
  {
  	struct its_node *its;
@@ -198,7 +207,7 @@ static u16 get_its_list(struct its_vm *vm)
  		if (!is_v4(its))
  			continue;

-		if (vm->vlpi_count[its->list_nr])
+		if (require_its_list_vmovp(vm, its))
  			__set_bit(its->list_nr, &its_list);
  	}

@@ -1295,7 +1304,7 @@ static void its_send_vmovp(struct its_vpe *vpe)
  		if (!is_v4(its))
  			continue;

-		if (!vpe->its_vm->vlpi_count[its->list_nr])
+		if (!require_its_list_vmovp(vpe->its_vm, its))
  			continue;

  		desc.its_vmovp_cmd.col = &its->collections[col_id];


Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup
  2020-03-17 10:30   ` Auger Eric
@ 2020-03-19 10:57     ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 10:57 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-17 10:30, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> Add the SGI configuration entry point for KVM to use.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  drivers/irqchip/irq-gic-v3-its.c   |  2 +-
>>  drivers/irqchip/irq-gic-v4.c       | 13 +++++++++++++
>>  include/linux/irqchip/arm-gic-v4.h |  3 ++-
>>  3 files changed, 16 insertions(+), 2 deletions(-)
>> 
>> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
>> b/drivers/irqchip/irq-gic-v3-its.c
>> index effb0e0b0c9d..b65fba67bd85 100644
>> --- a/drivers/irqchip/irq-gic-v3-its.c
>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>> @@ -4039,7 +4039,7 @@ static int its_sgi_set_vcpu_affinity(struct 
>> irq_data *d, void *vcpu_info)
>>  	struct its_cmd_info *info = vcpu_info;
>> 
>>  	switch (info->cmd_type) {
>> -	case PROP_UPDATE_SGI:
>> +	case PROP_UPDATE_VSGI:
> This change rather belongs to
> [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI 
> callbacks

Absolutely. I messed up a rebase, obviously.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-18  6:34   ` Zenghui Yu
@ 2020-03-19 12:10     ` Marc Zyngier
  2020-03-19 20:38       ` Auger Eric
  2020-03-20  3:53       ` Zenghui Yu
  0 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 12:10 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Zenghui,

On 2020-03-18 06:34, Zenghui Yu wrote:
> Hi Marc,
> 
> On 2020/3/5 4:33, Marc Zyngier wrote:
>> The GICv4.1 architecture gives the hypervisor the option to let
>> the guest choose whether it wants the good old SGIs with an
>> active state, or the new, HW-based ones that do not have one.
>> 
>> For this, plumb the configuration of SGIs into the GICv3 MMIO
>> handling, present the GICD_TYPER2.nASSGIcap to the guest,
>> and handle the GICD_CTLR.nASSGIreq setting.
>> 
>> In order to be able to deal with the restore of a guest, also
>> apply the GICD_CTLR.nASSGIreq setting at first run so that we
>> can move the restored SGIs to the HW if that's what the guest
>> had selected in a previous life.
> 
> I'm okay with the restore path.  But it seems that we still fail to
> save the pending state of vSGI - software pending_latch of HW-based
> vSGIs will not be updated (and always be false) because we directly
> inject them through ITS, so vgic_v3_uaccess_read_pending() can't
> tell the correct pending state to user-space (the correct one should
> be latched in HW).
> 
> It would be good if we can sync the hardware state into pending_latch
> at an appropriate time (just before save), but not sure if we can...

The problem is to find the "appropriate time". It would require to 
define
a point in the save sequence where we transition the state from HW to
SW. I'm not keen on adding more state than we already have.

But what we can do is to just ask the HW to give us the right state
on userspace access, at all times. How about this:

diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c 
b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index 48fd9fc229a2..281fe7216c59 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -305,8 +305,18 @@ static unsigned long 
vgic_v3_uaccess_read_pending(struct kvm_vcpu *vcpu,
  	 */
  	for (i = 0; i < len * 8; i++) {
  		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
+		bool state = irq->pending_latch;

-		if (irq->pending_latch)
+		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
+			int err;
+
+			err = irq_get_irqchip_state(irq->host_irq,
+						    IRQCHIP_STATE_PENDING,
+						    &state);
+			WARN_ON(err);
+		}
+
+		if (state)
  			value |= (1U << i);

  		vgic_put_irq(vcpu->kvm, irq);

I can add this to "KVM: arm64: GICv4.1: Add direct injection capability 
to SGI registers".

> 
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>   virt/kvm/arm/vgic/vgic-mmio-v3.c | 48 
>> ++++++++++++++++++++++++++++++--
>>   virt/kvm/arm/vgic/vgic-v3.c      |  2 ++
>>   2 files changed, 48 insertions(+), 2 deletions(-)
>> 
>> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c 
>> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> index de89da76a379..442f3b8c2559 100644
>> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> @@ -3,6 +3,7 @@
>>    * VGICv3 MMIO handling functions
>>    */
>>   +#include <linux/bitfield.h>
>>   #include <linux/irqchip/arm-gic-v3.h>
>>   #include <linux/kvm.h>
>>   #include <linux/kvm_host.h>
>> @@ -70,6 +71,8 @@ static unsigned long vgic_mmio_read_v3_misc(struct 
>> kvm_vcpu *vcpu,
>>   		if (vgic->enabled)
>>   			value |= GICD_CTLR_ENABLE_SS_G1;
>>   		value |= GICD_CTLR_ARE_NS | GICD_CTLR_DS;
>> +		if (kvm_vgic_global_state.has_gicv4_1 && vgic->nassgireq)
> 
> Looking at how we handle the GICD_CTLR.nASSGIreq setting, I think
> "nassgireq==true" already indicates "has_gicv4_1==true".  So this
> can be simplified.

Indeed. I've now dropped the has_gicv4.1 check.

> But I wonder that should we use nassgireq to *only* keep track what
> the guest had written into the GICD_CTLR.nASSGIreq.  If not, we may
> lose the guest-request bit after migration among hosts with different
> has_gicv4_1 settings.

I'm unsure of what you're suggesting here. If userspace tries to set
GICD_CTLR.nASSGIreq on a non-4.1 host, this bit will not latch.
Userspace can check that at restore time. Or we could fail the
userspace write, which is a bit odd (the bit is otherwise RES0).

Could you clarify your proposal?

> The remaining patches all look good to me :-). I will wait for you to
> confirm these two concerns.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable
  2020-03-18  3:17   ` Zenghui Yu
@ 2020-03-19 12:18     ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 12:18 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

On 2020-03-18 03:17, Zenghui Yu wrote:
> On 2020/3/5 4:33, Marc Zyngier wrote:
>> Each time a Group-enable bit gets flipped, the state of these bits
>> needs to be forwarded to the hardware. This is a pretty heavy
>> handed operation, requiring all vcpus to reload their GICv4
>> configuration. It is thus implemented as a new request type.
> 
> [note to myself]
> ... and the status are forwarded to HW by programming VGrp{0,1}En
> fields of GICR_VPENDBASER when vPEs are made resident next time.

I've added something based on this comment to the commit message.

Thanks!

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-04 20:33 ` [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs Marc Zyngier
  2020-03-18  3:19   ` Zenghui Yu
@ 2020-03-19 15:05   ` Auger Eric
  2020-03-19 15:21     ` Marc Zyngier
  1 sibling, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-19 15:05 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> The vgic-state debugfs file could do with showing the pending state
> of the HW-backed SGIs. Plug it into the low-level code.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  virt/kvm/arm/vgic/vgic-debug.c | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-debug.c b/virt/kvm/arm/vgic/vgic-debug.c
> index cc12fe9b2df3..b13a9e3f99dd 100644
> --- a/virt/kvm/arm/vgic/vgic-debug.c
> +++ b/virt/kvm/arm/vgic/vgic-debug.c
> @@ -178,6 +178,8 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
>  			    struct kvm_vcpu *vcpu)
>  {
>  	char *type;
> +	bool pending;
nit: can be directly initialized to irq->pending_latch
> +
>  	if (irq->intid < VGIC_NR_SGIS)
>  		type = "SGI";
>  	else if (irq->intid < VGIC_NR_PRIVATE_IRQS)
> @@ -190,6 +192,16 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
>  	if (irq->intid ==0 || irq->intid == VGIC_NR_PRIVATE_IRQS)
>  		print_header(s, irq, vcpu);
>  
> +	pending = irq->pending_latch;
> +	if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +		int err;
> +
> +		err = irq_get_irqchip_state(irq->host_irq,
> +					    IRQCHIP_STATE_PENDING,
> +					    &pending);
> +		WARN_ON_ONCE(err);
> +	}
> +
>  	seq_printf(s, "       %s %4d "
>  		      "    %2d "
>  		      "%d%d%d%d%d%d%d "
> @@ -201,7 +213,7 @@ static void print_irq_state(struct seq_file *s, struct vgic_irq *irq,
>  		      "\n",
>  			type, irq->intid,
>  			(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
> -			irq->pending_latch,
> +			pending,
>  			irq->line_level,
>  			irq->active,
>  			irq->enabled,
> 
The patch looks good to me but I am now lost about how we retrieve the
pending stat of other hw mapped interrupts. Looks we use
irq->pending_latch always. Is that correct?

For the patch:
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-19 15:05   ` Auger Eric
@ 2020-03-19 15:21     ` Marc Zyngier
  2020-03-19 15:43       ` Auger Eric
  2020-03-20  4:38       ` Zenghui Yu
  0 siblings, 2 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 15:21 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-19 15:05, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> The vgic-state debugfs file could do with showing the pending state
>> of the HW-backed SGIs. Plug it into the low-level code.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  virt/kvm/arm/vgic/vgic-debug.c | 14 +++++++++++++-
>>  1 file changed, 13 insertions(+), 1 deletion(-)
>> 
>> diff --git a/virt/kvm/arm/vgic/vgic-debug.c 
>> b/virt/kvm/arm/vgic/vgic-debug.c
>> index cc12fe9b2df3..b13a9e3f99dd 100644
>> --- a/virt/kvm/arm/vgic/vgic-debug.c
>> +++ b/virt/kvm/arm/vgic/vgic-debug.c
>> @@ -178,6 +178,8 @@ static void print_irq_state(struct seq_file *s, 
>> struct vgic_irq *irq,
>>  			    struct kvm_vcpu *vcpu)
>>  {
>>  	char *type;
>> +	bool pending;
> nit: can be directly initialized to irq->pending_latch
>> +
>>  	if (irq->intid < VGIC_NR_SGIS)
>>  		type = "SGI";
>>  	else if (irq->intid < VGIC_NR_PRIVATE_IRQS)
>> @@ -190,6 +192,16 @@ static void print_irq_state(struct seq_file *s, 
>> struct vgic_irq *irq,
>>  	if (irq->intid ==0 || irq->intid == VGIC_NR_PRIVATE_IRQS)
>>  		print_header(s, irq, vcpu);
>> 
>> +	pending = irq->pending_latch;
>> +	if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
>> +		int err;
>> +
>> +		err = irq_get_irqchip_state(irq->host_irq,
>> +					    IRQCHIP_STATE_PENDING,
>> +					    &pending);
>> +		WARN_ON_ONCE(err);
>> +	}
>> +
>>  	seq_printf(s, "       %s %4d "
>>  		      "    %2d "
>>  		      "%d%d%d%d%d%d%d "
>> @@ -201,7 +213,7 @@ static void print_irq_state(struct seq_file *s, 
>> struct vgic_irq *irq,
>>  		      "\n",
>>  			type, irq->intid,
>>  			(irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
>> -			irq->pending_latch,
>> +			pending,
>>  			irq->line_level,
>>  			irq->active,
>>  			irq->enabled,
>> 
> The patch looks good to me but I am now lost about how we retrieve the
> pending stat of other hw mapped interrupts. Looks we use
> irq->pending_latch always. Is that correct?

Correct. GICv4.0 doesn't give us an architectural way to look at the
vLPI pending state (there isn't even a guarantee about when the GIC
will stop writing to memory, if it ever does).

With GICv4.1, you can introspect the HW state for SGIs. You can also
look at the vLPI state by peeking at the virtual pending table, but
you'd need to unmap the VPE first, which I obviously don't want to do
for this debug interface, specially as it can be used whilst the guest
is up and running.

In the future, we'll have to implement that in order to support guest
save/restore from a GICv4.1 system. I haven't given much thought to it
though.

> For the patch:
> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-19 15:21     ` Marc Zyngier
@ 2020-03-19 15:43       ` Auger Eric
  2020-03-19 16:16         ` Marc Zyngier
  2020-03-20  4:38       ` Zenghui Yu
  1 sibling, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-19 15:43 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/19/20 4:21 PM, Marc Zyngier wrote:
> Hi Eric,
> 
> On 2020-03-19 15:05, Auger Eric wrote:
>> Hi Marc,
>>
>> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>>> The vgic-state debugfs file could do with showing the pending state
>>> of the HW-backed SGIs. Plug it into the low-level code.
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>  virt/kvm/arm/vgic/vgic-debug.c | 14 +++++++++++++-
>>>  1 file changed, 13 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-debug.c
>>> b/virt/kvm/arm/vgic/vgic-debug.c
>>> index cc12fe9b2df3..b13a9e3f99dd 100644
>>> --- a/virt/kvm/arm/vgic/vgic-debug.c
>>> +++ b/virt/kvm/arm/vgic/vgic-debug.c
>>> @@ -178,6 +178,8 @@ static void print_irq_state(struct seq_file *s,
>>> struct vgic_irq *irq,
>>>                  struct kvm_vcpu *vcpu)
>>>  {
>>>      char *type;
>>> +    bool pending;
>> nit: can be directly initialized to irq->pending_latch
>>> +
>>>      if (irq->intid < VGIC_NR_SGIS)
>>>          type = "SGI";
>>>      else if (irq->intid < VGIC_NR_PRIVATE_IRQS)
>>> @@ -190,6 +192,16 @@ static void print_irq_state(struct seq_file *s,
>>> struct vgic_irq *irq,
>>>      if (irq->intid ==0 || irq->intid == VGIC_NR_PRIVATE_IRQS)
>>>          print_header(s, irq, vcpu);
>>>
>>> +    pending = irq->pending_latch;
>>> +    if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
>>> +        int err;
>>> +
>>> +        err = irq_get_irqchip_state(irq->host_irq,
>>> +                        IRQCHIP_STATE_PENDING,
>>> +                        &pending);
>>> +        WARN_ON_ONCE(err);
>>> +    }
>>> +
>>>      seq_printf(s, "       %s %4d "
>>>                "    %2d "
>>>                "%d%d%d%d%d%d%d "
>>> @@ -201,7 +213,7 @@ static void print_irq_state(struct seq_file *s,
>>> struct vgic_irq *irq,
>>>                "\n",
>>>              type, irq->intid,
>>>              (irq->target_vcpu) ? irq->target_vcpu->vcpu_id : -1,
>>> -            irq->pending_latch,
>>> +            pending,
>>>              irq->line_level,
>>>              irq->active,
>>>              irq->enabled,
>>>
>> The patch looks good to me but I am now lost about how we retrieve the
>> pending stat of other hw mapped interrupts. Looks we use
>> irq->pending_latch always. Is that correct?
> 
> Correct. GICv4.0 doesn't give us an architectural way to look at the
> vLPI pending state (there isn't even a guarantee about when the GIC
> will stop writing to memory, if it ever does).
> 
> With GICv4.1, you can introspect the HW state for SGIs. You can also
> look at the vLPI state by peeking at the virtual pending table, but
> you'd need to unmap the VPE first, which I obviously don't want to do
> for this debug interface, specially as it can be used whilst the guest
> is up and running.
OK for vLPIs, what about other HW mapped IRQs (arch timer?)

Eric
> 
> In the future, we'll have to implement that in order to support guest
> save/restore from a GICv4.1 system. I haven't given much thought to it
> though.
> 
>> For the patch:
>> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> 
> Thanks,
> 
>         M.


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-19 15:43       ` Auger Eric
@ 2020-03-19 16:16         ` Marc Zyngier
  2020-03-19 16:17           ` Auger Eric
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 16:16 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-19 15:43, Auger Eric wrote:
> Hi Marc,
> 
> On 3/19/20 4:21 PM, Marc Zyngier wrote:
>> Hi Eric,

[...]

>>> The patch looks good to me but I am now lost about how we retrieve 
>>> the
>>> pending stat of other hw mapped interrupts. Looks we use
>>> irq->pending_latch always. Is that correct?
>> 
>> Correct. GICv4.0 doesn't give us an architectural way to look at the
>> vLPI pending state (there isn't even a guarantee about when the GIC
>> will stop writing to memory, if it ever does).
>> 
>> With GICv4.1, you can introspect the HW state for SGIs. You can also
>> look at the vLPI state by peeking at the virtual pending table, but
>> you'd need to unmap the VPE first, which I obviously don't want to do
>> for this debug interface, specially as it can be used whilst the guest
>> is up and running.
> OK for vLPIs, what about other HW mapped IRQs (arch timer?)

Different kind of HW. With those, the injection is still virtual, so the
SW pending bit is still very much valid. You can actually try and make
the timer interrupt pending, it should show up.

What the irq->hw bit means is "this virtual interrupt is somehow related
to the host_irq". How this is interpreted is completely 
context-dependent.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  2020-03-04 20:33 ` [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts Marc Zyngier
@ 2020-03-19 16:16   ` Auger Eric
  2020-03-19 19:52     ` Marc Zyngier
  2020-03-20  4:22   ` Zenghui Yu
  1 sibling, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-19 16:16 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> In order to let a guest buy in the new, active-less SGIs, we
> need to be able to switch between the two modes.
> 
> Handle this by stopping all guest activity, transfer the state
> from one mode to the other, and resume the guest. Nothing calls
> this code so far, but a later patch will plug it into the MMIO
> emulation.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  include/kvm/arm_vgic.h      |  3 ++
>  virt/kvm/arm/vgic/vgic-v4.c | 94 +++++++++++++++++++++++++++++++++++++
>  virt/kvm/arm/vgic/vgic.h    |  1 +
>  3 files changed, 98 insertions(+)
> 
> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
> index 63457908c9c4..69f4164d6477 100644
> --- a/include/kvm/arm_vgic.h
> +++ b/include/kvm/arm_vgic.h
> @@ -231,6 +231,9 @@ struct vgic_dist {
>  	/* distributor enabled */
>  	bool			enabled;
>  
> +	/* Wants SGIs without active state */
> +	bool			nassgireq;
> +
>  	struct vgic_irq		*spis;
>  
>  	struct vgic_io_device	dist_iodev;
> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
> index c2fcde104ea2..a65dc1c85363 100644
> --- a/virt/kvm/arm/vgic/vgic-v4.c
> +++ b/virt/kvm/arm/vgic/vgic-v4.c
> @@ -97,6 +97,100 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
>  	return IRQ_HANDLED;
>  }
>  
> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct vgic_irq *irq)
> +{
> +	vpe->sgi_config[irq->intid].enabled	= irq->enabled;
> +	vpe->sgi_config[irq->intid].group 	= irq->group;
> +	vpe->sgi_config[irq->intid].priority	= irq->priority;
> +}
> +
> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
> +{
> +	struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
> +	int i;
> +
> +	/*
> +	 * With GICv4.1, every virtual SGI can be directly injected. So
> +	 * let's pretend that they are HW interrupts, tied to a host
> +	 * IRQ. The SGI code will do its magic.
> +	 */
> +	for (i = 0; i < VGIC_NR_SGIS; i++) {
> +		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
> +		struct irq_desc *desc;
> +		int ret;
Is is safe without holding the irq->irq_lock?
> +
> +		if (irq->hw) {
> +			vgic_put_irq(vcpu->kvm, irq);
> +			continue;
> +		}
> +
> +		irq->hw = true;
> +		irq->host_irq = irq_find_mapping(vpe->sgi_domain, i);
> +
> +		/* Transfer the full irq state to the vPE */
> +		vgic_v4_sync_sgi_config(vpe, irq);
> +		desc = irq_to_desc(irq->host_irq);
> +		ret = irq_domain_activate_irq(irq_desc_get_irq_data(desc),
> +					      false);
> +		if (!WARN_ON(ret)) {
> +			/* Transfer pending state */
> +			ret = irq_set_irqchip_state(irq->host_irq,
> +						    IRQCHIP_STATE_PENDING,
> +						    irq->pending_latch);
> +			WARN_ON(ret);
> +			irq->pending_latch = false;
> +		}
> +
> +		vgic_put_irq(vcpu->kvm, irq);
> +	}
> +}
> +
> +static void vgic_v4_disable_vsgis(struct kvm_vcpu *vcpu)
> +{
> +	int i;
> +
> +	for (i = 0; i < VGIC_NR_SGIS; i++) {
> +		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
> +		struct irq_desc *desc;
> +		int ret;
> +
> +		if (!irq->hw) {
> +			vgic_put_irq(vcpu->kvm, irq);
> +			continue;
> +		}
> +
> +		irq->hw = false;
> +		ret = irq_get_irqchip_state(irq->host_irq,
> +					    IRQCHIP_STATE_PENDING,
> +					    &irq->pending_latch);
> +		WARN_ON(ret);
> +
> +		desc = irq_to_desc(irq->host_irq);
> +		irq_domain_deactivate_irq(irq_desc_get_irq_data(desc));
> +
> +		vgic_put_irq(vcpu->kvm, irq);
> +	}
> +}
> +
> +/* Must be called with the kvm lock held */
> +void vgic_v4_configure_vsgis(struct kvm *kvm)
> +{
> +	struct vgic_dist *dist = &kvm->arch.vgic;
> +	struct kvm_vcpu *vcpu;
> +	int i;
> +
> +	kvm_arm_halt_guest(kvm);
> +
> +	kvm_for_each_vcpu(i, vcpu, kvm) {
> +		if (dist->nassgireq)
> +			vgic_v4_enable_vsgis(vcpu);
> +		else
> +			vgic_v4_disable_vsgis(vcpu);
> +	}
> +
> +	kvm_arm_resume_guest(kvm);
> +}
> +
>  /**
>   * vgic_v4_init - Initialize the GICv4 data structures
>   * @kvm:	Pointer to the VM being initialized
> diff --git a/virt/kvm/arm/vgic/vgic.h b/virt/kvm/arm/vgic/vgic.h
> index c7fefd6b1c80..769e4802645e 100644
> --- a/virt/kvm/arm/vgic/vgic.h
> +++ b/virt/kvm/arm/vgic/vgic.h
> @@ -316,5 +316,6 @@ void vgic_its_invalidate_cache(struct kvm *kvm);
>  bool vgic_supports_direct_msis(struct kvm *kvm);
>  int vgic_v4_init(struct kvm *kvm);
>  void vgic_v4_teardown(struct kvm *kvm);
> +void vgic_v4_configure_vsgis(struct kvm *kvm);
>  
>  #endif
> 
Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-19 16:16         ` Marc Zyngier
@ 2020-03-19 16:17           ` Auger Eric
  0 siblings, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-19 16:17 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi,

On 3/19/20 5:16 PM, Marc Zyngier wrote:
> Hi Eric,
> 
> On 2020-03-19 15:43, Auger Eric wrote:
>> Hi Marc,
>>
>> On 3/19/20 4:21 PM, Marc Zyngier wrote:
>>> Hi Eric,
> 
> [...]
> 
>>>> The patch looks good to me but I am now lost about how we retrieve the
>>>> pending stat of other hw mapped interrupts. Looks we use
>>>> irq->pending_latch always. Is that correct?
>>>
>>> Correct. GICv4.0 doesn't give us an architectural way to look at the
>>> vLPI pending state (there isn't even a guarantee about when the GIC
>>> will stop writing to memory, if it ever does).
>>>
>>> With GICv4.1, you can introspect the HW state for SGIs. You can also
>>> look at the vLPI state by peeking at the virtual pending table, but
>>> you'd need to unmap the VPE first, which I obviously don't want to do
>>> for this debug interface, specially as it can be used whilst the guest
>>> is up and running.
>> OK for vLPIs, what about other HW mapped IRQs (arch timer?)
> 
> Different kind of HW. With those, the injection is still virtual, so the
> SW pending bit is still very much valid. You can actually try and make
> the timer interrupt pending, it should show up.
> 
> What the irq->hw bit means is "this virtual interrupt is somehow related
> to the host_irq". How this is interpreted is completely context-dependent.
OK thank you for refreshing my memories ;-)

Eric
> 
> Thanks,
> 
>         M.


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  2020-03-19 16:16   ` Auger Eric
@ 2020-03-19 19:52     ` Marc Zyngier
  2020-03-19 20:13       ` Auger Eric
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-19 19:52 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-19 16:16, Auger Eric wrote:
> Hi Marc,
> 
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> In order to let a guest buy in the new, active-less SGIs, we
>> need to be able to switch between the two modes.
>> 
>> Handle this by stopping all guest activity, transfer the state
>> from one mode to the other, and resume the guest. Nothing calls
>> this code so far, but a later patch will plug it into the MMIO
>> emulation.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  include/kvm/arm_vgic.h      |  3 ++
>>  virt/kvm/arm/vgic/vgic-v4.c | 94 
>> +++++++++++++++++++++++++++++++++++++
>>  virt/kvm/arm/vgic/vgic.h    |  1 +
>>  3 files changed, 98 insertions(+)
>> 
>> diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
>> index 63457908c9c4..69f4164d6477 100644
>> --- a/include/kvm/arm_vgic.h
>> +++ b/include/kvm/arm_vgic.h
>> @@ -231,6 +231,9 @@ struct vgic_dist {
>>  	/* distributor enabled */
>>  	bool			enabled;
>> 
>> +	/* Wants SGIs without active state */
>> +	bool			nassgireq;
>> +
>>  	struct vgic_irq		*spis;
>> 
>>  	struct vgic_io_device	dist_iodev;
>> diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
>> index c2fcde104ea2..a65dc1c85363 100644
>> --- a/virt/kvm/arm/vgic/vgic-v4.c
>> +++ b/virt/kvm/arm/vgic/vgic-v4.c
>> @@ -97,6 +97,100 @@ static irqreturn_t vgic_v4_doorbell_handler(int 
>> irq, void *info)
>>  	return IRQ_HANDLED;
>>  }
>> 
>> +static void vgic_v4_sync_sgi_config(struct its_vpe *vpe, struct 
>> vgic_irq *irq)
>> +{
>> +	vpe->sgi_config[irq->intid].enabled	= irq->enabled;
>> +	vpe->sgi_config[irq->intid].group 	= irq->group;
>> +	vpe->sgi_config[irq->intid].priority	= irq->priority;
>> +}
>> +
>> +static void vgic_v4_enable_vsgis(struct kvm_vcpu *vcpu)
>> +{
>> +	struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
>> +	int i;
>> +
>> +	/*
>> +	 * With GICv4.1, every virtual SGI can be directly injected. So
>> +	 * let's pretend that they are HW interrupts, tied to a host
>> +	 * IRQ. The SGI code will do its magic.
>> +	 */
>> +	for (i = 0; i < VGIC_NR_SGIS; i++) {
>> +		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, i);
>> +		struct irq_desc *desc;
>> +		int ret;
> Is is safe without holding the irq->irq_lock?

The assumption here is that we're coming vgic_v4_configure_vsgis(), 
which starts
by stopping the whole guest. My guess is that it should be safe enough, 
but
maybe you are thinking of something else?

Thanks,

          M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  2020-03-19 19:52     ` Marc Zyngier
@ 2020-03-19 20:13       ` Auger Eric
  2020-03-20  9:17         ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-19 20:13 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Lorenzo Pieralisi, Jason Cooper, kvm, Suzuki K Poulose,
	linux-kernel, Robert Richter, James Morse, Julien Thierry,
	Zenghui Yu, Thomas Gleixner, kvmarm, linux-arm-kernel

Hi Marc,

On 3/19/20 8:52 PM, Marc Zyngier wrote:
> The assumption here is that we're coming vgic_v4_configure_vsgis(),
> which starts
> by stopping the whole guest. My guess is that it should be safe enough, but
> maybe you are thinking of something else?
I don't have a specific case in mind. Just preferred asking to make
sure. Usually when touching those fields we take the lock (that's also
the case in vgic_debug_show for instance).

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-19 12:10     ` Marc Zyngier
@ 2020-03-19 20:38       ` Auger Eric
  2020-03-20  3:08         ` Zenghui Yu
  2020-03-20  3:53       ` Zenghui Yu
  1 sibling, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-19 20:38 UTC (permalink / raw)
  To: Marc Zyngier, Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, James Morse,
	Julien Thierry, Suzuki K Poulose

Hi Marc,
On 3/19/20 1:10 PM, Marc Zyngier wrote:
> Hi Zenghui,
> 
> On 2020-03-18 06:34, Zenghui Yu wrote:
>> Hi Marc,
>>
>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>> The GICv4.1 architecture gives the hypervisor the option to let
>>> the guest choose whether it wants the good old SGIs with an
>>> active state, or the new, HW-based ones that do not have one.
>>>
>>> For this, plumb the configuration of SGIs into the GICv3 MMIO
>>> handling, present the GICD_TYPER2.nASSGIcap to the guest,
>>> and handle the GICD_CTLR.nASSGIreq setting.
>>>
>>> In order to be able to deal with the restore of a guest, also
>>> apply the GICD_CTLR.nASSGIreq setting at first run so that we
>>> can move the restored SGIs to the HW if that's what the guest
>>> had selected in a previous life.
>>
>> I'm okay with the restore path.  But it seems that we still fail to
>> save the pending state of vSGI - software pending_latch of HW-based
>> vSGIs will not be updated (and always be false) because we directly
>> inject them through ITS, so vgic_v3_uaccess_read_pending() can't
>> tell the correct pending state to user-space (the correct one should
>> be latched in HW).
>>
>> It would be good if we can sync the hardware state into pending_latch
>> at an appropriate time (just before save), but not sure if we can...
> 
> The problem is to find the "appropriate time". It would require to define
> a point in the save sequence where we transition the state from HW to
> SW. I'm not keen on adding more state than we already have.

may be we could use a dedicated device group/attr as we have for the ITS
save tables? the user space would choose.

Thanks

Eric
> 
> But what we can do is to just ask the HW to give us the right state
> on userspace access, at all times. How about this:
> 
> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c
> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> index 48fd9fc229a2..281fe7216c59 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> @@ -305,8 +305,18 @@ static unsigned long
> vgic_v3_uaccess_read_pending(struct kvm_vcpu *vcpu,
>       */
>      for (i = 0; i < len * 8; i++) {
>          struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
> +        bool state = irq->pending_latch;
> 
> -        if (irq->pending_latch)
> +        if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +            int err;
> +
> +            err = irq_get_irqchip_state(irq->host_irq,
> +                            IRQCHIP_STATE_PENDING,
> +                            &state);
> +            WARN_ON(err);
> +        }
> +
> +        if (state)
>              value |= (1U << i);
> 
>          vgic_put_irq(vcpu->kvm, irq);
> 
> I can add this to "KVM: arm64: GICv4.1: Add direct injection capability
> to SGI registers".
> 
>>
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>   virt/kvm/arm/vgic/vgic-mmio-v3.c | 48 ++++++++++++++++++++++++++++++--
>>>   virt/kvm/arm/vgic/vgic-v3.c      |  2 ++
>>>   2 files changed, 48 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> index de89da76a379..442f3b8c2559 100644
>>> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> @@ -3,6 +3,7 @@
>>>    * VGICv3 MMIO handling functions
>>>    */
>>>   +#include <linux/bitfield.h>
>>>   #include <linux/irqchip/arm-gic-v3.h>
>>>   #include <linux/kvm.h>
>>>   #include <linux/kvm_host.h>
>>> @@ -70,6 +71,8 @@ static unsigned long vgic_mmio_read_v3_misc(struct
>>> kvm_vcpu *vcpu,
>>>           if (vgic->enabled)
>>>               value |= GICD_CTLR_ENABLE_SS_G1;
>>>           value |= GICD_CTLR_ARE_NS | GICD_CTLR_DS;
>>> +        if (kvm_vgic_global_state.has_gicv4_1 && vgic->nassgireq)
>>
>> Looking at how we handle the GICD_CTLR.nASSGIreq setting, I think
>> "nassgireq==true" already indicates "has_gicv4_1==true".  So this
>> can be simplified.
> 
> Indeed. I've now dropped the has_gicv4.1 check.
> 
>> But I wonder that should we use nassgireq to *only* keep track what
>> the guest had written into the GICD_CTLR.nASSGIreq.  If not, we may
>> lose the guest-request bit after migration among hosts with different
>> has_gicv4_1 settings.
> 
> I'm unsure of what you're suggesting here. If userspace tries to set
> GICD_CTLR.nASSGIreq on a non-4.1 host, this bit will not latch.
> Userspace can check that at restore time. Or we could fail the
> userspace write, which is a bit odd (the bit is otherwise RES0).
> 
> Could you clarify your proposal?
> 
>> The remaining patches all look good to me :-). I will wait for you to
>> confirm these two concerns.
> 
> Thanks,
> 
>         M.


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs
  2020-03-19 10:55     ` Marc Zyngier
@ 2020-03-20  2:31       ` Zenghui Yu
  0 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20  2:31 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/19 18:55, Marc Zyngier wrote:
> On 2020-03-17 02:49, Zenghui Yu wrote:
>> Hi Marc,
>>
>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>> Now that we have HW-accelerated SGIs being delivered to VPEs, it
>>> becomes required to map the VPEs on all ITSs instead of relying
>>> on the lazy approach that we would use when using the ITS-list
>>> mechanism.
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>
>> Before GICv4.1, we use vlpi_count to evaluate whether the vPE has been
>> mapped on the specified ITS, and use this refcount to only issue VMOVP
>> to those involved ITSes. It's broken after this patch.
>>
>> We may need to re-evaluate "whether the vPE is mapped" now that we're at
>> GICv4.1, otherwise *no* VMOVP will be issued on the v4.1 capable machine
>> (I mean those without single VMOVP support).
>>
>> What I'm saying is something like below (only an idea, it even can't
>> compile), any thoughts?
>>
>>
>> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
>> b/drivers/irqchip/irq-gic-v3-its.c
>> index 2e12bc52e3a2..3450b5e847ca 100644
>> --- a/drivers/irqchip/irq-gic-v3-its.c
>> +++ b/drivers/irqchip/irq-gic-v3-its.c
>> @@ -198,7 +198,8 @@ static u16 get_its_list(struct its_vm *vm)
>>          if (!is_v4(its))
>>              continue;
>>
>> -        if (vm->vlpi_count[its->list_nr])
>> +        if (vm->vlpi_count[its->list_nr] ||
>> +            gic_requires_eager_mapping())
>>              __set_bit(its->list_nr, &its_list);
>>      }
>>
>> @@ -1295,7 +1296,8 @@ static void its_send_vmovp(struct its_vpe *vpe)
>>          if (!is_v4(its))
>>              continue;
>>
>> -        if (!vpe->its_vm->vlpi_count[its->list_nr])
>> +        if (!vpe->its_vm->vlpi_count[its->list_nr] &&
>> +            !gic_requires_eager_mapping())
>>              continue;
>>
>>          desc.its_vmovp_cmd.col = &its->collections[col_id];
> 
> It took me a while to wrap my head around that one, but you're as usual 
> spot on.
> 
> The use of gic_requires_eager_mapping() is a bit confusing here, as it 
> isn't
> so much that the VPE is eagerly mapped, but the predicate on which we 
> evaluate
> the need for a VMOVP. How about this:
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c 
> b/drivers/irqchip/irq-gic-v3-its.c
> index cd6451e190c9..348f7a909a69 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -189,6 +189,15 @@ static DEFINE_IDA(its_vpeid_ida);
>   #define gic_data_rdist_rd_base()    (gic_data_rdist()->rd_base)
>   #define gic_data_rdist_vlpi_base()    (gic_data_rdist_rd_base() + 
> SZ_128K)
> 
> +/*
> + * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
> + * always have vSGIs mapped.
> + */
> +static bool require_its_list_vmovp(struct its_vm *vm, struct its_node 
> *its)
> +{
> +    return (gic_rdists->has_rvpeid || vm->vlpi_count[its->list_nr]);
> +}
> +
>   static u16 get_its_list(struct its_vm *vm)
>   {
>       struct its_node *its;
> @@ -198,7 +207,7 @@ static u16 get_its_list(struct its_vm *vm)
>           if (!is_v4(its))
>               continue;
> 
> -        if (vm->vlpi_count[its->list_nr])
> +        if (require_its_list_vmovp(vm, its))
>               __set_bit(its->list_nr, &its_list);
>       }
> 
> @@ -1295,7 +1304,7 @@ static void its_send_vmovp(struct its_vpe *vpe)
>           if (!is_v4(its))
>               continue;
> 
> -        if (!vpe->its_vm->vlpi_count[its->list_nr])
> +        if (!require_its_list_vmovp(vpe->its_vm, its))
>               continue;
> 
>           desc.its_vmovp_cmd.col = &its->collections[col_id];

Indeed this looks much clearer. We're actually evaluating the need
for issuing VMOVP to a specified ITS, on system using its_list_map
feature (though we evaluate it by checking whether the vPE is mapped
on this ITS).


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-19 20:38       ` Auger Eric
@ 2020-03-20  3:08         ` Zenghui Yu
  2020-03-20  7:59           ` Auger Eric
  0 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20  3:08 UTC (permalink / raw)
  To: Auger Eric, Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, James Morse,
	Julien Thierry, Suzuki K Poulose

On 2020/3/20 4:38, Auger Eric wrote:
> Hi Marc,
> On 3/19/20 1:10 PM, Marc Zyngier wrote:
>> Hi Zenghui,
>>
>> On 2020-03-18 06:34, Zenghui Yu wrote:
>>> Hi Marc,
>>>
>>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>>> The GICv4.1 architecture gives the hypervisor the option to let
>>>> the guest choose whether it wants the good old SGIs with an
>>>> active state, or the new, HW-based ones that do not have one.
>>>>
>>>> For this, plumb the configuration of SGIs into the GICv3 MMIO
>>>> handling, present the GICD_TYPER2.nASSGIcap to the guest,
>>>> and handle the GICD_CTLR.nASSGIreq setting.
>>>>
>>>> In order to be able to deal with the restore of a guest, also
>>>> apply the GICD_CTLR.nASSGIreq setting at first run so that we
>>>> can move the restored SGIs to the HW if that's what the guest
>>>> had selected in a previous life.
>>>
>>> I'm okay with the restore path.  But it seems that we still fail to
>>> save the pending state of vSGI - software pending_latch of HW-based
>>> vSGIs will not be updated (and always be false) because we directly
>>> inject them through ITS, so vgic_v3_uaccess_read_pending() can't
>>> tell the correct pending state to user-space (the correct one should
>>> be latched in HW).
>>>
>>> It would be good if we can sync the hardware state into pending_latch
>>> at an appropriate time (just before save), but not sure if we can...
>>
>> The problem is to find the "appropriate time". It would require to define
>> a point in the save sequence where we transition the state from HW to
>> SW. I'm not keen on adding more state than we already have.
> 
> may be we could use a dedicated device group/attr as we have for the ITS
> save tables? the user space would choose.

It means that userspace will be aware of some form of GICv4.1 details
(e.g., get/set vSGI state at HW level) that KVM has implemented.
Is it something that userspace required to know? I'm open to this ;-)

> 
> Thanks
> 
> Eric
>>
>> But what we can do is to just ask the HW to give us the right state
>> on userspace access, at all times. How about this:
>>
>> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> index 48fd9fc229a2..281fe7216c59 100644
>> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> @@ -305,8 +305,18 @@ static unsigned long
>> vgic_v3_uaccess_read_pending(struct kvm_vcpu *vcpu,
>>        */
>>       for (i = 0; i < len * 8; i++) {
>>           struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
>> +        bool state = irq->pending_latch;
>>
>> -        if (irq->pending_latch)
>> +        if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
>> +            int err;
>> +
>> +            err = irq_get_irqchip_state(irq->host_irq,
>> +                            IRQCHIP_STATE_PENDING,
>> +                            &state);
>> +            WARN_ON(err);
>> +        }
>> +
>> +        if (state)
>>               value |= (1U << i);
>>
>>           vgic_put_irq(vcpu->kvm, irq);

Anyway this looks good to me and will do the right thing on a userspace
save.

>>
>> I can add this to "KVM: arm64: GICv4.1: Add direct injection capability
>> to SGI registers".

Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-19 12:10     ` Marc Zyngier
  2020-03-19 20:38       ` Auger Eric
@ 2020-03-20  3:53       ` Zenghui Yu
  2020-03-20  9:01         ` Marc Zyngier
  1 sibling, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20  3:53 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/19 20:10, Marc Zyngier wrote:
>> But I wonder that should we use nassgireq to *only* keep track what
>> the guest had written into the GICD_CTLR.nASSGIreq.  If not, we may
>> lose the guest-request bit after migration among hosts with different
>> has_gicv4_1 settings.
> 
> I'm unsure of what you're suggesting here. If userspace tries to set
> GICD_CTLR.nASSGIreq on a non-4.1 host, this bit will not latch.

This is exactly what I *was* concerning about.

> Userspace can check that at restore time. Or we could fail the
> userspace write, which is a bit odd (the bit is otherwise RES0).
> 
> Could you clarify your proposal?

Let's assume two hosts below. 'has_gicv4_1' is true on host-A while
it is false on host-B because of lack of HW support or the kernel
parameter "kvm-arm.vgic_v4_enable=0".

If we migrate guest through A->B->A, we may end-up lose the initial
guest-request "nASSGIreq=1" and don't use direct vSGI delivery for
this guest when it's migrated back to host-A.

This can be "fixed" by keep track of what guest had written into
nASSGIreq. And we need to evaluate the need for using direct vSGI
for a specified guest by 'has_gicv4_1 && nassgireq'.

But if it's expected that "if userspace tries to set nASSGIreq on
a non-4.1 host, this bit will not latch", then this shouldn't be
a problem at all.

Anyway this is not a big deal to me and I won't complain about it
in the future ;-) Either way, for this patch:

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  2020-03-04 20:33 ` [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts Marc Zyngier
  2020-03-19 16:16   ` Auger Eric
@ 2020-03-20  4:22   ` Zenghui Yu
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20  4:22 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> In order to let a guest buy in the new, active-less SGIs, we
> need to be able to switch between the two modes.
> 
> Handle this by stopping all guest activity, transfer the state
> from one mode to the other, and resume the guest. Nothing calls
> this code so far, but a later patch will plug it into the MMIO
> emulation.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs
  2020-03-04 20:33 ` [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs Marc Zyngier
@ 2020-03-20  4:23   ` Zenghui Yu
  2020-03-20  8:12   ` Auger Eric
  1 sibling, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20  4:23 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Eric Auger, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/5 4:33, Marc Zyngier wrote:
> Just like for VLPIs, it is beneficial to avoid trapping on WFI when the
> vcpu is using the GICv4.1 SGIs.
> 
> Add such a check to vcpu_clear_wfx_traps().
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>

Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>


Thanks

> ---
>   arch/arm64/include/asm/kvm_emulate.h | 3 ++-
>   1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index f658dda12364..a30b4eec7cb4 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -89,7 +89,8 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
>   static inline void vcpu_clear_wfx_traps(struct kvm_vcpu *vcpu)
>   {
>   	vcpu->arch.hcr_el2 &= ~HCR_TWE;
> -	if (atomic_read(&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count))
> +	if (atomic_read(&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count) ||
> +	    vcpu->kvm->arch.vgic.nassgireq)
>   		vcpu->arch.hcr_el2 &= ~HCR_TWI;
>   	else
>   		vcpu->arch.hcr_el2 |= HCR_TWI;
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-19 15:21     ` Marc Zyngier
  2020-03-19 15:43       ` Auger Eric
@ 2020-03-20  4:38       ` Zenghui Yu
  2020-03-20  9:09         ` Marc Zyngier
  1 sibling, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20  4:38 UTC (permalink / raw)
  To: Marc Zyngier, Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, James Morse,
	Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/19 23:21, Marc Zyngier wrote:
> With GICv4.1, you can introspect the HW state for SGIs. You can also
> look at the vLPI state by peeking at the virtual pending table, but
> you'd need to unmap the VPE first,

Out of curiosity, could you please point me to the "unmap the VPE"
requirement in the v4.1 spec? I'd like to have a look.


Thanks!
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-20  3:08         ` Zenghui Yu
@ 2020-03-20  7:59           ` Auger Eric
  2020-03-20  9:46             ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-20  7:59 UTC (permalink / raw)
  To: Zenghui Yu, Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, James Morse,
	Julien Thierry, Suzuki K Poulose

Hi Zenghui,

On 3/20/20 4:08 AM, Zenghui Yu wrote:
> On 2020/3/20 4:38, Auger Eric wrote:
>> Hi Marc,
>> On 3/19/20 1:10 PM, Marc Zyngier wrote:
>>> Hi Zenghui,
>>>
>>> On 2020-03-18 06:34, Zenghui Yu wrote:
>>>> Hi Marc,
>>>>
>>>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>>>> The GICv4.1 architecture gives the hypervisor the option to let
>>>>> the guest choose whether it wants the good old SGIs with an
>>>>> active state, or the new, HW-based ones that do not have one.
>>>>>
>>>>> For this, plumb the configuration of SGIs into the GICv3 MMIO
>>>>> handling, present the GICD_TYPER2.nASSGIcap to the guest,
>>>>> and handle the GICD_CTLR.nASSGIreq setting.
>>>>>
>>>>> In order to be able to deal with the restore of a guest, also
>>>>> apply the GICD_CTLR.nASSGIreq setting at first run so that we
>>>>> can move the restored SGIs to the HW if that's what the guest
>>>>> had selected in a previous life.
>>>>
>>>> I'm okay with the restore path.  But it seems that we still fail to
>>>> save the pending state of vSGI - software pending_latch of HW-based
>>>> vSGIs will not be updated (and always be false) because we directly
>>>> inject them through ITS, so vgic_v3_uaccess_read_pending() can't
>>>> tell the correct pending state to user-space (the correct one should
>>>> be latched in HW).
>>>>
>>>> It would be good if we can sync the hardware state into pending_latch
>>>> at an appropriate time (just before save), but not sure if we can...
>>>
>>> The problem is to find the "appropriate time". It would require to
>>> define
>>> a point in the save sequence where we transition the state from HW to
>>> SW. I'm not keen on adding more state than we already have.
>>
>> may be we could use a dedicated device group/attr as we have for the ITS
>> save tables? the user space would choose.
> 
> It means that userspace will be aware of some form of GICv4.1 details
> (e.g., get/set vSGI state at HW level) that KVM has implemented.
> Is it something that userspace required to know? I'm open to this ;-)
Not sure we would be obliged to expose fine details. This could be a
generic save/restore device group/attr whose implementation at KVM level
could differ depending on the version being implemented, no?

Thanks

Eric
> 
>>
>> Thanks
>>
>> Eric
>>>
>>> But what we can do is to just ask the HW to give us the right state
>>> on userspace access, at all times. How about this:
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> index 48fd9fc229a2..281fe7216c59 100644
>>> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> @@ -305,8 +305,18 @@ static unsigned long
>>> vgic_v3_uaccess_read_pending(struct kvm_vcpu *vcpu,
>>>        */
>>>       for (i = 0; i < len * 8; i++) {
>>>           struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid
>>> + i);
>>> +        bool state = irq->pending_latch;
>>>
>>> -        if (irq->pending_latch)
>>> +        if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
>>> +            int err;
>>> +
>>> +            err = irq_get_irqchip_state(irq->host_irq,
>>> +                            IRQCHIP_STATE_PENDING,
>>> +                            &state);
>>> +            WARN_ON(err);
>>> +        }
>>> +
>>> +        if (state)
>>>               value |= (1U << i);
>>>
>>>           vgic_put_irq(vcpu->kvm, irq);
> 
> Anyway this looks good to me and will do the right thing on a userspace
> save.
> 
>>>
>>> I can add this to "KVM: arm64: GICv4.1: Add direct injection capability
>>> to SGI registers".
> 
> Thanks,
> Zenghui
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
  2020-03-04 20:33 ` [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers Marc Zyngier
  2020-03-18  3:28   ` Zenghui Yu
@ 2020-03-20  8:11   ` Auger Eric
  2020-03-20 10:05     ` Marc Zyngier
  1 sibling, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-20  8:11 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,
On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Most of the GICv3 emulation code that deals with SGIs now has to be
> aware of the v4.1 capabilities in order to benefit from it.
> 
> Add such support, keyed on the interrupt having the hw flag set and
> being a SGI.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---
>  virt/kvm/arm/vgic/vgic-mmio-v3.c | 15 +++++-
>  virt/kvm/arm/vgic/vgic-mmio.c    | 88 ++++++++++++++++++++++++++++++--
>  2 files changed, 96 insertions(+), 7 deletions(-)
> 
> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> index ebc218840fc2..de89da76a379 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> @@ -6,6 +6,7 @@
>  #include <linux/irqchip/arm-gic-v3.h>
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
> +#include <linux/interrupt.h>
>  #include <kvm/iodev.h>
>  #include <kvm/arm_vgic.h>
>  
> @@ -942,8 +943,18 @@ void vgic_v3_dispatch_sgi(struct kvm_vcpu *vcpu, u64 reg, bool allow_group1)
>  		 * generate interrupts of either group.
>  		 */
>  		if (!irq->group || allow_group1) {
> -			irq->pending_latch = true;
> -			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
> +			if (!irq->hw) {
> +				irq->pending_latch = true;
> +				vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
> +			} else {
> +				/* HW SGI? Ask the GIC to inject it */
> +				int err;
nit: add line
> +				err = irq_set_irqchip_state(irq->host_irq,
> +							    IRQCHIP_STATE_PENDING,
> +							    true);
> +				WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
> +				raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
> +			}
>  		} else {
>  			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
>  		}
> diff --git a/virt/kvm/arm/vgic/vgic-mmio.c b/virt/kvm/arm/vgic/vgic-mmio.c
> index 97fb2a40e6ba..2199302597fa 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio.c
> @@ -5,6 +5,8 @@
>  
>  #include <linux/bitops.h>
>  #include <linux/bsearch.h>
> +#include <linux/interrupt.h>
> +#include <linux/irq.h>
>  #include <linux/kvm.h>
>  #include <linux/kvm_host.h>
>  #include <kvm/iodev.h>
> @@ -59,6 +61,11 @@ unsigned long vgic_mmio_read_group(struct kvm_vcpu *vcpu,
>  	return value;
>  }
>  
> +static void vgic_update_vsgi(struct vgic_irq *irq)
> +{
> +	WARN_ON(its_prop_update_vsgi(irq->host_irq, irq->priority, irq->group));
> +}
> +
>  void vgic_mmio_write_group(struct kvm_vcpu *vcpu, gpa_t addr,
>  			   unsigned int len, unsigned long val)
>  {
> @@ -71,7 +78,12 @@ void vgic_mmio_write_group(struct kvm_vcpu *vcpu, gpa_t addr,
>  
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
>  		irq->group = !!(val & BIT(i));
> -		vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +			vgic_update_vsgi(irq);
> +			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
> +		} else {
> +			vgic_queue_irq_unlock(vcpu->kvm, irq, flags);
> +		}
>  
>  		vgic_put_irq(vcpu->kvm, irq);
>  	}
> @@ -113,7 +125,21 @@ void vgic_mmio_write_senable(struct kvm_vcpu *vcpu,
>  		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
>  
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
> -		if (vgic_irq_is_mapped_level(irq)) {
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +			if (!irq->enabled) {
> +				struct irq_data *data;
> +
> +				irq->enabled = true;
> +				data = &irq_to_desc(irq->host_irq)->irq_data;
> +				while (irqd_irq_disabled(data))
> +					enable_irq(irq->host_irq);
could you explain me why the while() is requested?
> +			}
> +
> +			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
> +			vgic_put_irq(vcpu->kvm, irq);
> +
> +			continue;
> +		} else if (vgic_irq_is_mapped_level(irq)) {
>  			bool was_high = irq->line_level;
>  
>  			/*
> @@ -148,6 +174,8 @@ void vgic_mmio_write_cenable(struct kvm_vcpu *vcpu,
>  		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
>  
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid) && irq->enabled)
> +			disable_irq_nosync(irq->host_irq);
>  
>  		irq->enabled = false;
>  
> @@ -167,10 +195,22 @@ unsigned long vgic_mmio_read_pending(struct kvm_vcpu *vcpu,
>  	for (i = 0; i < len * 8; i++) {
>  		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
>  		unsigned long flags;
> +		bool val;
>  
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
> -		if (irq_is_pending(irq))
> -			value |= (1U << i);
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +			int err;
> +
> +			val = false;
> +			err = irq_get_irqchip_state(irq->host_irq,
> +						    IRQCHIP_STATE_PENDING,
> +						    &val);
> +			WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
> +		} else {
> +			val = irq_is_pending(irq);
> +		}
> +
> +		value |= ((u32)val << i);
>  		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
>  
>  		vgic_put_irq(vcpu->kvm, irq);
> @@ -215,6 +255,21 @@ void vgic_mmio_write_spending(struct kvm_vcpu *vcpu,
>  		}
>  
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +			/* HW SGI? Ask the GIC to inject it */
> +			int err;
nit: extra line
> +			err = irq_set_irqchip_state(irq->host_irq,
> +						    IRQCHIP_STATE_PENDING,
> +						    true);
> +			WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
> +
> +			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
> +			vgic_put_irq(vcpu->kvm, irq);
> +
> +			continue;
> +		}
> +
>  		if (irq->hw)
>  			vgic_hw_irq_spending(vcpu, irq, is_uaccess);
>  		else
> @@ -269,6 +324,20 @@ void vgic_mmio_write_cpending(struct kvm_vcpu *vcpu,
>  
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
>  
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +			/* HW SGI? Ask the GIC to clear its pending bit */
> +			int err;
> +			err = irq_set_irqchip_state(irq->host_irq,
> +						    IRQCHIP_STATE_PENDING,
> +						    false);
> +			WARN_RATELIMIT(err, "IRQ %d", irq->host_irq);
> +
> +			raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
> +			vgic_put_irq(vcpu->kvm, irq);
> +
> +			continue;
> +		}
> +
>  		if (irq->hw)
>  			vgic_hw_irq_cpending(vcpu, irq, is_uaccess);
>  		else
> @@ -318,8 +387,15 @@ static void vgic_mmio_change_active(struct kvm_vcpu *vcpu, struct vgic_irq *irq,
>  
>  	raw_spin_lock_irqsave(&irq->irq_lock, flags);
>  
> -	if (irq->hw) {
> +	if (irq->hw && !vgic_irq_is_sgi(irq->intid)) {
>  		vgic_hw_irq_change_active(vcpu, irq, active, !requester_vcpu);
> +	} else if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
> +		/*
> +		 * GICv4.1 VSGI feature doesn't track an active state,
> +		 * so let's not kid ourselves, there is nothing we can
> +		 * do here.
> +		 */
> +		irq->active = false;
>  	} else {
>  		u32 model = vcpu->kvm->arch.vgic.vgic_model;
>  		u8 active_source;
> @@ -493,6 +569,8 @@ void vgic_mmio_write_priority(struct kvm_vcpu *vcpu,
>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
>  		/* Narrow the priority range to what we actually support */
>  		irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
> +		if (irq->hw && vgic_irq_is_sgi(irq->intid))
> +			vgic_update_vsgi(irq);
>  		raw_spin_unlock_irqrestore(&irq->irq_lock, flags);
>  
>  		vgic_put_irq(vcpu->kvm, irq);
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs
  2020-03-04 20:33 ` [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs Marc Zyngier
  2020-03-20  4:23   ` Zenghui Yu
@ 2020-03-20  8:12   ` Auger Eric
  1 sibling, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-20  8:12 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> Just like for VLPIs, it is beneficial to avoid trapping on WFI when the
> vcpu is using the GICv4.1 SGIs.
> 
> Add such a check to vcpu_clear_wfx_traps().
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric
> ---
>  arch/arm64/include/asm/kvm_emulate.h | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_emulate.h b/arch/arm64/include/asm/kvm_emulate.h
> index f658dda12364..a30b4eec7cb4 100644
> --- a/arch/arm64/include/asm/kvm_emulate.h
> +++ b/arch/arm64/include/asm/kvm_emulate.h
> @@ -89,7 +89,8 @@ static inline unsigned long *vcpu_hcr(struct kvm_vcpu *vcpu)
>  static inline void vcpu_clear_wfx_traps(struct kvm_vcpu *vcpu)
>  {
>  	vcpu->arch.hcr_el2 &= ~HCR_TWE;
> -	if (atomic_read(&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count))
> +	if (atomic_read(&vcpu->arch.vgic_cpu.vgic_v3.its_vpe.vlpi_count) ||
> +	    vcpu->kvm->arch.vgic.nassgireq)
>  		vcpu->arch.hcr_el2 &= ~HCR_TWI;
>  	else
>  		vcpu->arch.hcr_el2 |= HCR_TWI;
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-20  3:53       ` Zenghui Yu
@ 2020-03-20  9:01         ` Marc Zyngier
  2020-03-23  8:11           ` Zenghui Yu
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20  9:01 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Zenghui,

On 2020-03-20 03:53, Zenghui Yu wrote:
> Hi Marc,
> 
> On 2020/3/19 20:10, Marc Zyngier wrote:
>>> But I wonder that should we use nassgireq to *only* keep track what
>>> the guest had written into the GICD_CTLR.nASSGIreq.  If not, we may
>>> lose the guest-request bit after migration among hosts with different
>>> has_gicv4_1 settings.
>> 
>> I'm unsure of what you're suggesting here. If userspace tries to set
>> GICD_CTLR.nASSGIreq on a non-4.1 host, this bit will not latch.
> 
> This is exactly what I *was* concerning about.
> 
>> Userspace can check that at restore time. Or we could fail the
>> userspace write, which is a bit odd (the bit is otherwise RES0).
>> 
>> Could you clarify your proposal?
> 
> Let's assume two hosts below. 'has_gicv4_1' is true on host-A while
> it is false on host-B because of lack of HW support or the kernel
> parameter "kvm-arm.vgic_v4_enable=0".
> 
> If we migrate guest through A->B->A, we may end-up lose the initial
> guest-request "nASSGIreq=1" and don't use direct vSGI delivery for
> this guest when it's migrated back to host-A.

My point above is that we shouldn't allow the A->B migration the first
place, and fail it as quickly as possible. We don't know what the guest
has observed in terms of GIC capability, and it may not have enabled the
new flavour of SGIs just yet.

> This can be "fixed" by keep track of what guest had written into
> nASSGIreq. And we need to evaluate the need for using direct vSGI
> for a specified guest by 'has_gicv4_1 && nassgireq'.

It feels odd. It means we have more state than the HW normally has.
I have an alternative proposal, see below.

> But if it's expected that "if userspace tries to set nASSGIreq on
> a non-4.1 host, this bit will not latch", then this shouldn't be
> a problem at all.

Well, that is the semantics of the RES0 bit. It applies from both
guest and userspace.

And actually, maybe we can handle that pretty cheaply. If userspace
tries to restore GICD_TYPER2 to a value that isn't what KVM can
offer, we just return an error. Exactly like we do for GICD_IIDR.
Hence the following patch:

diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c 
b/virt/kvm/arm/vgic/vgic-mmio-v3.c
index 28b639fd1abc..e72dcc454247 100644
--- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
+++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
@@ -156,6 +156,7 @@ static int vgic_mmio_uaccess_write_v3_misc(struct 
kvm_vcpu *vcpu,
  	struct vgic_dist *dist = &vcpu->kvm->arch.vgic;

  	switch (addr & 0x0c) {
+	case GICD_TYPER2:
  	case GICD_IIDR:
  		if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
  			return -EINVAL;

Being a RO register, writing something that isn't compatible with the
possible behaviour of the hypervisor will just return an error.

What do you think?

> Anyway this is not a big deal to me and I won't complain about it
> in the future ;-) Either way, for this patch:
> 
> Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>

Thanks a lot!

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-20  4:38       ` Zenghui Yu
@ 2020-03-20  9:09         ` Marc Zyngier
  2020-03-20 11:35           ` Zenghui Yu
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20  9:09 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: Auger Eric, linux-arm-kernel, kvmarm, kvm, linux-kernel,
	Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Zenghui,

On 2020-03-20 04:38, Zenghui Yu wrote:
> Hi Marc,
> 
> On 2020/3/19 23:21, Marc Zyngier wrote:
>> With GICv4.1, you can introspect the HW state for SGIs. You can also
>> look at the vLPI state by peeking at the virtual pending table, but
>> you'd need to unmap the VPE first,
> 
> Out of curiosity, could you please point me to the "unmap the VPE"
> requirement in the v4.1 spec? I'd like to have a look.

Sure. See IHI0069F, 5.3.19 (VMAPP GICv4.1), "Caching of virtual LPI data
structures", and the bit that says:

"A VMAPP with {V,Alloc}=={0,1} cleans and invalidates any caching of the
Virtual Pending Table and Virtual Configuration Table associated with 
the
vPEID held in the GIC"

which is what was crucially missing from the GICv4.0 spec (it doesn't 
say
when the GIC is done writing to memory).

Side note: it'd be good to know what the rules are for your own GICv4
implementations, so that we can at least make sure the current code is 
safe.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts
  2020-03-19 20:13       ` Auger Eric
@ 2020-03-20  9:17         ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20  9:17 UTC (permalink / raw)
  To: Auger Eric
  Cc: Lorenzo Pieralisi, Jason Cooper, kvm, Suzuki K Poulose,
	linux-kernel, Robert Richter, James Morse, Julien Thierry,
	Zenghui Yu, Thomas Gleixner, kvmarm, linux-arm-kernel

Hi Eric,

On 2020-03-19 20:13, Auger Eric wrote:
> Hi Marc,
> 
> On 3/19/20 8:52 PM, Marc Zyngier wrote:
>> The assumption here is that we're coming vgic_v4_configure_vsgis(),
>> which starts
>> by stopping the whole guest. My guess is that it should be safe 
>> enough, but
>> maybe you are thinking of something else?
> I don't have a specific case in mind. Just preferred asking to make
> sure. Usually when touching those fields we take the lock (that's also
> the case in vgic_debug_show for instance).

Ah, good thing you mention the debug interface. I think it is the only
thing that can run behind our back... Fair enough, I'll add the locking.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-20  7:59           ` Auger Eric
@ 2020-03-20  9:46             ` Marc Zyngier
  2020-03-20 11:09               ` Auger Eric
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20  9:46 UTC (permalink / raw)
  To: Auger Eric
  Cc: Zenghui Yu, linux-arm-kernel, kvmarm, kvm, linux-kernel,
	Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

On 2020-03-20 07:59, Auger Eric wrote:
> Hi Zenghui,
> 
> On 3/20/20 4:08 AM, Zenghui Yu wrote:
>> On 2020/3/20 4:38, Auger Eric wrote:
>>> Hi Marc,
>>> On 3/19/20 1:10 PM, Marc Zyngier wrote:
>>>> Hi Zenghui,
>>>> 
>>>> On 2020-03-18 06:34, Zenghui Yu wrote:
>>>>> Hi Marc,
>>>>> 
>>>>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>>>>> The GICv4.1 architecture gives the hypervisor the option to let
>>>>>> the guest choose whether it wants the good old SGIs with an
>>>>>> active state, or the new, HW-based ones that do not have one.
>>>>>> 
>>>>>> For this, plumb the configuration of SGIs into the GICv3 MMIO
>>>>>> handling, present the GICD_TYPER2.nASSGIcap to the guest,
>>>>>> and handle the GICD_CTLR.nASSGIreq setting.
>>>>>> 
>>>>>> In order to be able to deal with the restore of a guest, also
>>>>>> apply the GICD_CTLR.nASSGIreq setting at first run so that we
>>>>>> can move the restored SGIs to the HW if that's what the guest
>>>>>> had selected in a previous life.
>>>>> 
>>>>> I'm okay with the restore path.  But it seems that we still fail to
>>>>> save the pending state of vSGI - software pending_latch of HW-based
>>>>> vSGIs will not be updated (and always be false) because we directly
>>>>> inject them through ITS, so vgic_v3_uaccess_read_pending() can't
>>>>> tell the correct pending state to user-space (the correct one 
>>>>> should
>>>>> be latched in HW).
>>>>> 
>>>>> It would be good if we can sync the hardware state into 
>>>>> pending_latch
>>>>> at an appropriate time (just before save), but not sure if we 
>>>>> can...
>>>> 
>>>> The problem is to find the "appropriate time". It would require to
>>>> define
>>>> a point in the save sequence where we transition the state from HW 
>>>> to
>>>> SW. I'm not keen on adding more state than we already have.
>>> 
>>> may be we could use a dedicated device group/attr as we have for the 
>>> ITS
>>> save tables? the user space would choose.
>> 
>> It means that userspace will be aware of some form of GICv4.1 details
>> (e.g., get/set vSGI state at HW level) that KVM has implemented.
>> Is it something that userspace required to know? I'm open to this ;-)
> Not sure we would be obliged to expose fine details. This could be a
> generic save/restore device group/attr whose implementation at KVM 
> level
> could differ depending on the version being implemented, no?

What prevents us from hooking this synchronization to the current 
behaviour
of KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES? After all, this is already the 
point
where we synchronize the KVM view of the pending state with userspace.
Here, it's just a matter of picking the information from some other 
place
(i.e. the host's virtual pending table).

The thing we need though is the guarantee that the guest isn't going to
get more vLPIs at that stage, as they would be lost. This effectively
assumes that we can also save/restore the state of the signalling 
devices,
and I don't know if we're quite there yet.

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
  2020-03-20  8:11   ` Auger Eric
@ 2020-03-20 10:05     ` Marc Zyngier
  2020-03-20 10:56       ` Auger Eric
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20 10:05 UTC (permalink / raw)
  To: Auger Eric
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Eric,

On 2020-03-20 08:11, Auger Eric wrote:
> Hi Marc,
> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>> Most of the GICv3 emulation code that deals with SGIs now has to be
>> aware of the v4.1 capabilities in order to benefit from it.
>> 
>> Add such support, keyed on the interrupt having the hw flag set and
>> being a SGI.
>> 
>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>> ---
>>  virt/kvm/arm/vgic/vgic-mmio-v3.c | 15 +++++-
>>  virt/kvm/arm/vgic/vgic-mmio.c    | 88 
>> ++++++++++++++++++++++++++++++--
>>  2 files changed, 96 insertions(+), 7 deletions(-)
>> 

[...]

>> @@ -113,7 +125,21 @@ void vgic_mmio_write_senable(struct kvm_vcpu 
>> *vcpu,
>>  		struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid + i);
>> 
>>  		raw_spin_lock_irqsave(&irq->irq_lock, flags);
>> -		if (vgic_irq_is_mapped_level(irq)) {
>> +		if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
>> +			if (!irq->enabled) {
>> +				struct irq_data *data;
>> +
>> +				irq->enabled = true;
>> +				data = &irq_to_desc(irq->host_irq)->irq_data;
>> +				while (irqd_irq_disabled(data))
>> +					enable_irq(irq->host_irq);
> could you explain me why the while() is requested?

Ah, interesting question: disable_irq() (and its variants) can nest. 
This
means that if you have done two disable_irq(), you must do two 
enable_irq()
to get back to the interrupt being enabled.

The locking should ensure that this nesting doesn't happen, but I'm 
paranoid
(see the GICv4.0 doorbell handling). It also makes it easier to reason 
about
the initial state.

[...]

> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks!

          M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers
  2020-03-20 10:05     ` Marc Zyngier
@ 2020-03-20 10:56       ` Auger Eric
  0 siblings, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-20 10:56 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Zenghui Yu,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 3/20/20 11:05 AM, Marc Zyngier wrote:
> Hi Eric,
> 
> On 2020-03-20 08:11, Auger Eric wrote:
>> Hi Marc,
>> On 3/4/20 9:33 PM, Marc Zyngier wrote:
>>> Most of the GICv3 emulation code that deals with SGIs now has to be
>>> aware of the v4.1 capabilities in order to benefit from it.
>>>
>>> Add such support, keyed on the interrupt having the hw flag set and
>>> being a SGI.
>>>
>>> Signed-off-by: Marc Zyngier <maz@kernel.org>
>>> ---
>>>  virt/kvm/arm/vgic/vgic-mmio-v3.c | 15 +++++-
>>>  virt/kvm/arm/vgic/vgic-mmio.c    | 88 ++++++++++++++++++++++++++++++--
>>>  2 files changed, 96 insertions(+), 7 deletions(-)
>>>
> 
> [...]
> 
>>> @@ -113,7 +125,21 @@ void vgic_mmio_write_senable(struct kvm_vcpu *vcpu,
>>>          struct vgic_irq *irq = vgic_get_irq(vcpu->kvm, vcpu, intid +
>>> i);
>>>
>>>          raw_spin_lock_irqsave(&irq->irq_lock, flags);
>>> -        if (vgic_irq_is_mapped_level(irq)) {
>>> +        if (irq->hw && vgic_irq_is_sgi(irq->intid)) {
>>> +            if (!irq->enabled) {
>>> +                struct irq_data *data;
>>> +
>>> +                irq->enabled = true;
>>> +                data = &irq_to_desc(irq->host_irq)->irq_data;
>>> +                while (irqd_irq_disabled(data))
>>> +                    enable_irq(irq->host_irq);
>> could you explain me why the while() is requested?
> 
> Ah, interesting question: disable_irq() (and its variants) can nest. This
> means that if you have done two disable_irq(), you must do two enable_irq()
> to get back to the interrupt being enabled.
> 
> The locking should ensure that this nesting doesn't happen, but I'm
> paranoid
> (see the GICv4.0 doorbell handling). It also makes it easier to reason
> about
> the initial state.

OK! thank you for this explanation.

Thanks

Eric
> 
> [...]
> 
>> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> 
> Thanks!
> 
>          M.


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-20  9:46             ` Marc Zyngier
@ 2020-03-20 11:09               ` Auger Eric
  2020-03-20 11:20                 ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Auger Eric @ 2020-03-20 11:09 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Lorenzo Pieralisi, Jason Cooper, kvm, Suzuki K Poulose,
	linux-kernel, Robert Richter, James Morse, Julien Thierry,
	Zenghui Yu, Thomas Gleixner, kvmarm, linux-arm-kernel

Hi Marc,

On 3/20/20 10:46 AM, Marc Zyngier wrote:
> On 2020-03-20 07:59, Auger Eric wrote:
>> Hi Zenghui,
>>
>> On 3/20/20 4:08 AM, Zenghui Yu wrote:
>>> On 2020/3/20 4:38, Auger Eric wrote:
>>>> Hi Marc,
>>>> On 3/19/20 1:10 PM, Marc Zyngier wrote:
>>>>> Hi Zenghui,
>>>>>
>>>>> On 2020-03-18 06:34, Zenghui Yu wrote:
>>>>>> Hi Marc,
>>>>>>
>>>>>> On 2020/3/5 4:33, Marc Zyngier wrote:
>>>>>>> The GICv4.1 architecture gives the hypervisor the option to let
>>>>>>> the guest choose whether it wants the good old SGIs with an
>>>>>>> active state, or the new, HW-based ones that do not have one.
>>>>>>>
>>>>>>> For this, plumb the configuration of SGIs into the GICv3 MMIO
>>>>>>> handling, present the GICD_TYPER2.nASSGIcap to the guest,
>>>>>>> and handle the GICD_CTLR.nASSGIreq setting.
>>>>>>>
>>>>>>> In order to be able to deal with the restore of a guest, also
>>>>>>> apply the GICD_CTLR.nASSGIreq setting at first run so that we
>>>>>>> can move the restored SGIs to the HW if that's what the guest
>>>>>>> had selected in a previous life.
>>>>>>
>>>>>> I'm okay with the restore path.  But it seems that we still fail to
>>>>>> save the pending state of vSGI - software pending_latch of HW-based
>>>>>> vSGIs will not be updated (and always be false) because we directly
>>>>>> inject them through ITS, so vgic_v3_uaccess_read_pending() can't
>>>>>> tell the correct pending state to user-space (the correct one should
>>>>>> be latched in HW).
>>>>>>
>>>>>> It would be good if we can sync the hardware state into pending_latch
>>>>>> at an appropriate time (just before save), but not sure if we can...
>>>>>
>>>>> The problem is to find the "appropriate time". It would require to
>>>>> define
>>>>> a point in the save sequence where we transition the state from HW to
>>>>> SW. I'm not keen on adding more state than we already have.
>>>>
>>>> may be we could use a dedicated device group/attr as we have for the
>>>> ITS
>>>> save tables? the user space would choose.
>>>
>>> It means that userspace will be aware of some form of GICv4.1 details
>>> (e.g., get/set vSGI state at HW level) that KVM has implemented.
>>> Is it something that userspace required to know? I'm open to this ;-)
>> Not sure we would be obliged to expose fine details. This could be a
>> generic save/restore device group/attr whose implementation at KVM level
>> could differ depending on the version being implemented, no?
> 
> What prevents us from hooking this synchronization to the current behaviour
> of KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES? After all, this is already the
> point
> where we synchronize the KVM view of the pending state with userspace.
> Here, it's just a matter of picking the information from some other place
> (i.e. the host's virtual pending table).
agreed
> 
> The thing we need though is the guarantee that the guest isn't going to
> get more vLPIs at that stage, as they would be lost. This effectively
> assumes that we can also save/restore the state of the signalling devices,
> and I don't know if we're quite there yet.
On QEMU, when KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES is called, the VM is
stopped.
See cddafd8f353d ("hw/intc/arm_gicv3_its: Implement state save/restore")
So I think it should work, no?

Thanks

Eric

> 
> Thanks,
> 
>         M.


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-20 11:09               ` Auger Eric
@ 2020-03-20 11:20                 ` Marc Zyngier
  0 siblings, 0 replies; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20 11:20 UTC (permalink / raw)
  To: Auger Eric
  Cc: Lorenzo Pieralisi, Jason Cooper, kvm, Suzuki K Poulose,
	linux-kernel, Robert Richter, James Morse, Julien Thierry,
	Zenghui Yu, Thomas Gleixner, kvmarm, linux-arm-kernel

Hi Eric,

On 2020-03-20 11:09, Auger Eric wrote:
> Hi Marc,

[...]

>>>> It means that userspace will be aware of some form of GICv4.1 
>>>> details
>>>> (e.g., get/set vSGI state at HW level) that KVM has implemented.
>>>> Is it something that userspace required to know? I'm open to this 
>>>> ;-)
>>> Not sure we would be obliged to expose fine details. This could be a
>>> generic save/restore device group/attr whose implementation at KVM 
>>> level
>>> could differ depending on the version being implemented, no?
>> 
>> What prevents us from hooking this synchronization to the current 
>> behaviour
>> of KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES? After all, this is already 
>> the
>> point
>> where we synchronize the KVM view of the pending state with userspace.
>> Here, it's just a matter of picking the information from some other 
>> place
>> (i.e. the host's virtual pending table).
> agreed
>> 
>> The thing we need though is the guarantee that the guest isn't going 
>> to
>> get more vLPIs at that stage, as they would be lost. This effectively
>> assumes that we can also save/restore the state of the signalling 
>> devices,
>> and I don't know if we're quite there yet.
> On QEMU, when KVM_DEV_ARM_VGIC_SAVE_PENDING_TABLES is called, the VM is
> stopped.
> See cddafd8f353d ("hw/intc/arm_gicv3_its: Implement state 
> save/restore")
> So I think it should work, no?

The guest being stopped is a good start. But my concern is on the device 
side.

If the device is still active (generating interrupts), these interrupts 
will
be dropped because the vPE will have been unmapped from the ITS in order 
to
clean the ITS caches and make sure the virtual pending table is up to 
date.

In turn, restoring the guest may lead to a lockup because we would have 
lost
these interrupts. What does QEMU on x86 do in this case?

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-20  9:09         ` Marc Zyngier
@ 2020-03-20 11:35           ` Zenghui Yu
  2020-03-20 11:46             ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20 11:35 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Auger Eric, linux-arm-kernel, kvmarm, kvm, linux-kernel,
	Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/20 17:09, Marc Zyngier wrote:
> Hi Zenghui,
> 
> On 2020-03-20 04:38, Zenghui Yu wrote:
>> Hi Marc,
>>
>> On 2020/3/19 23:21, Marc Zyngier wrote:
>>> With GICv4.1, you can introspect the HW state for SGIs. You can also
>>> look at the vLPI state by peeking at the virtual pending table, but
>>> you'd need to unmap the VPE first,
>>
>> Out of curiosity, could you please point me to the "unmap the VPE"
>> requirement in the v4.1 spec? I'd like to have a look.
> 
> Sure. See IHI0069F, 5.3.19 (VMAPP GICv4.1), "Caching of virtual LPI data
> structures", and the bit that says:
> 
> "A VMAPP with {V,Alloc}=={0,1} cleans and invalidates any caching of the
> Virtual Pending Table and Virtual Configuration Table associated with the
> vPEID held in the GIC"
> 
> which is what was crucially missing from the GICv4.0 spec (it doesn't say
> when the GIC is done writing to memory).

OK. Thanks for the pointer!

> 
> Side note: it'd be good to know what the rules are for your own GICv4
> implementations, so that we can at least make sure the current code is 
> safe.

As far as I know, there will be some clean and invalidate operations
when v4.0 VPENDBASER.Valid gets programmed. But not sure about behaviors
on VMAPP (unmap), it may be a totally v4.1 stuff. I'll have a talk with
our SOC team.

But how can the current code be unsafe? Is anywhere in the current code
will peek/poke the vpt (whilst GIC continues writing things into it)?


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-20 11:35           ` Zenghui Yu
@ 2020-03-20 11:46             ` Marc Zyngier
  2020-03-20 12:09               ` Zenghui Yu
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-20 11:46 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: Auger Eric, linux-arm-kernel, kvmarm, kvm, linux-kernel,
	Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

On 2020-03-20 11:35, Zenghui Yu wrote:
> Hi Marc,
> 
> On 2020/3/20 17:09, Marc Zyngier wrote:
>> Hi Zenghui,
>> 
>> On 2020-03-20 04:38, Zenghui Yu wrote:
>>> Hi Marc,
>>> 
>>> On 2020/3/19 23:21, Marc Zyngier wrote:
>>>> With GICv4.1, you can introspect the HW state for SGIs. You can also
>>>> look at the vLPI state by peeking at the virtual pending table, but
>>>> you'd need to unmap the VPE first,
>>> 
>>> Out of curiosity, could you please point me to the "unmap the VPE"
>>> requirement in the v4.1 spec? I'd like to have a look.
>> 
>> Sure. See IHI0069F, 5.3.19 (VMAPP GICv4.1), "Caching of virtual LPI 
>> data
>> structures", and the bit that says:
>> 
>> "A VMAPP with {V,Alloc}=={0,1} cleans and invalidates any caching of 
>> the
>> Virtual Pending Table and Virtual Configuration Table associated with 
>> the
>> vPEID held in the GIC"
>> 
>> which is what was crucially missing from the GICv4.0 spec (it doesn't 
>> say
>> when the GIC is done writing to memory).
> 
> OK. Thanks for the pointer!
> 
>> 
>> Side note: it'd be good to know what the rules are for your own GICv4
>> implementations, so that we can at least make sure the current code is 
>> safe.
> 
> As far as I know, there will be some clean and invalidate operations
> when v4.0 VPENDBASER.Valid gets programmed.

Interesting. The ideal behaviour would be that the VPT is up-to-date and
the caches clean when Valid is cleared (and once Dirty flips to 0).

> But not sure about behaviors
> on VMAPP (unmap), it may be a totally v4.1 stuff. I'll have a talk with
> our SOC team.

The VMAPP stuff is purely v4.1.

> But how can the current code be unsafe? Is anywhere in the current code
> will peek/poke the vpt (whilst GIC continues writing things into it)?

No. But on VM termination, the memory will be freed, and will eventually 
be
reallocated. If the GIC can still write to that memory after it has been
freed, you end-up with memory corruption... Which is why I'm curious of
what ensures that on your implementation.

I'd also like to know the same thing about the QC implementation, but
there's nobody left to find out...

Thanks,

          M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs
  2020-03-20 11:46             ` Marc Zyngier
@ 2020-03-20 12:09               ` Zenghui Yu
  0 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-20 12:09 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Auger Eric, linux-arm-kernel, kvmarm, kvm, linux-kernel,
	Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/20 19:46, Marc Zyngier wrote:
>>> Side note: it'd be good to know what the rules are for your own GICv4
>>> implementations, so that we can at least make sure the current code 
>>> is safe.
>>
>> As far as I know, there will be some clean and invalidate operations
>> when v4.0 VPENDBASER.Valid gets programmed.
> 
> Interesting. The ideal behaviour would be that the VPT is up-to-date and
> the caches clean when Valid is cleared (and once Dirty flips to 0).
> 
>> But not sure about behaviors
>> on VMAPP (unmap), it may be a totally v4.1 stuff. I'll have a talk with
>> our SOC team.
> 
> The VMAPP stuff is purely v4.1.
> 
>> But how can the current code be unsafe? Is anywhere in the current code
>> will peek/poke the vpt (whilst GIC continues writing things into it)?
> 
> No. But on VM termination, the memory will be freed, and will eventually be
> reallocated. If the GIC can still write to that memory after it has been
> freed, you end-up with memory corruption... Which is why I'm curious of
> what ensures that on your implementation.

Ah, I got it. I will check it with HiSilicon people next week and go
back to you if the code becomes unsafe due to the incomplete GICv4.


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD
  2020-03-04 20:33 ` [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD Marc Zyngier
  2020-03-12  7:11   ` Zenghui Yu
@ 2020-03-20 14:23   ` Auger Eric
  2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-20 14:23 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> The GICv4.1 spec says that it is CONTRAINED UNPREDICTABLE to write to
> any of the GICR_INV{LPI,ALL}R registers if GICR_SYNCR.Busy == 1.
> 
> To deal with it, we must ensure that only a single invalidation can
> happen at a time for a given redistributor. Add a per-RD lock to that
> effect and take it around the invalidation/syncr-read to deal with this.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric
> ---
>  drivers/irqchip/irq-gic-v3-its.c   | 6 ++++++
>  drivers/irqchip/irq-gic-v3.c       | 1 +
>  include/linux/irqchip/arm-gic-v3.h | 1 +
>  3 files changed, 8 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index c84370245bea..fc5788584df7 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -1373,10 +1373,12 @@ static void direct_lpi_inv(struct irq_data *d)
>  
>  	/* Target the redistributor this LPI is currently routed to */
>  	cpu = irq_to_cpuid_lock(d, &flags);
> +	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
>  	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
>  	gic_write_lpir(val, rdbase + GICR_INVLPIR);
>  
>  	wait_for_syncr(rdbase);
> +	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
>  	irq_to_cpuid_unlock(d, flags);
>  }
>  
> @@ -3662,9 +3664,11 @@ static void its_vpe_send_inv(struct irq_data *d)
>  		void __iomem *rdbase;
>  
>  		/* Target the redistributor this VPE is currently known on */
> +		raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>  		rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
>  		gic_write_lpir(d->parent_data->hwirq, rdbase + GICR_INVLPIR);
>  		wait_for_syncr(rdbase);
> +		raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>  	} else {
>  		its_vpe_send_cmd(vpe, its_send_inv);
>  	}
> @@ -3825,10 +3829,12 @@ static void its_vpe_4_1_invall(struct its_vpe *vpe)
>  	val |= FIELD_PREP(GICR_INVALLR_VPEID, vpe->vpe_id);
>  
>  	/* Target the redistributor this vPE is currently known on */
> +	raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>  	rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
>  	gic_write_lpir(val, rdbase + GICR_INVALLR);
>  
>  	wait_for_syncr(rdbase);
> +	raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
>  }
>  
>  static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
> diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
> index 73e87e176d76..ba405becab53 100644
> --- a/drivers/irqchip/irq-gic-v3.c
> +++ b/drivers/irqchip/irq-gic-v3.c
> @@ -835,6 +835,7 @@ static int __gic_populate_rdist(struct redist_region *region, void __iomem *ptr)
>  	typer = gic_read_typer(ptr + GICR_TYPER);
>  	if ((typer >> 32) == aff) {
>  		u64 offset = ptr - region->redist_base;
> +		raw_spin_lock_init(&gic_data_rdist()->rd_lock);
>  		gic_data_rdist_rd_base() = ptr;
>  		gic_data_rdist()->phys_base = region->phys_base + offset;
>  
> diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
> index c29a02678a6f..b28acfa71f82 100644
> --- a/include/linux/irqchip/arm-gic-v3.h
> +++ b/include/linux/irqchip/arm-gic-v3.h
> @@ -652,6 +652,7 @@
>  
>  struct rdists {
>  	struct {
> +		raw_spinlock_t	rd_lock;
>  		void __iomem	*rd_base;
>  		struct page	*pend_page;
>  		phys_addr_t	phys_base;
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 04/23] irqchip/gic-v4.1: Wait for completion of redistributor's INVALL operation
  2020-03-04 20:33 ` [PATCH v5 04/23] irqchip/gic-v4.1: Wait for completion of redistributor's INVALL operation Marc Zyngier
@ 2020-03-20 14:23   ` Auger Eric
  0 siblings, 0 replies; 114+ messages in thread
From: Auger Eric @ 2020-03-20 14:23 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvmarm, kvm, linux-kernel
  Cc: Lorenzo Pieralisi, Jason Cooper, Robert Richter, Thomas Gleixner,
	Zenghui Yu, James Morse, Julien Thierry, Suzuki K Poulose

Hi,

On 3/4/20 9:33 PM, Marc Zyngier wrote:
> From: Zenghui Yu <yuzenghui@huawei.com>
> 
> In GICv4.1, we emulate a guest-issued INVALL command by a direct write
> to GICR_INVALLR.  Before we finish the emulation and go back to guest,
> let's make sure the physical invalidate operation is actually completed
> and no stale data will be left in redistributor. Per the specification,
> this can be achieved by polling the GICR_SYNCR.Busy bit (to zero).
> 
> Signed-off-by: Zenghui Yu <yuzenghui@huawei.com>
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> Link: https://lore.kernel.org/r/20200302092145.899-1-yuzenghui@huawei.com
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric
> ---
>  drivers/irqchip/irq-gic-v3-its.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
> index 1af713990123..c84370245bea 100644
> --- a/drivers/irqchip/irq-gic-v3-its.c
> +++ b/drivers/irqchip/irq-gic-v3-its.c
> @@ -3827,6 +3827,8 @@ static void its_vpe_4_1_invall(struct its_vpe *vpe)
>  	/* Target the redistributor this vPE is currently known on */
>  	rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
>  	gic_write_lpir(val, rdbase + GICR_INVALLR);
> +
> +	wait_for_syncr(rdbase);
>  }
>  
>  static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
> 


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-20  9:01         ` Marc Zyngier
@ 2020-03-23  8:11           ` Zenghui Yu
  2020-03-23  8:25             ` Marc Zyngier
  0 siblings, 1 reply; 114+ messages in thread
From: Zenghui Yu @ 2020-03-23  8:11 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/20 17:01, Marc Zyngier wrote:
> Hi Zenghui,
> 
> On 2020-03-20 03:53, Zenghui Yu wrote:
>> Hi Marc,
>>
>> On 2020/3/19 20:10, Marc Zyngier wrote:
>>>> But I wonder that should we use nassgireq to *only* keep track what
>>>> the guest had written into the GICD_CTLR.nASSGIreq.  If not, we may
>>>> lose the guest-request bit after migration among hosts with different
>>>> has_gicv4_1 settings.
>>>
>>> I'm unsure of what you're suggesting here. If userspace tries to set
>>> GICD_CTLR.nASSGIreq on a non-4.1 host, this bit will not latch.
>>
>> This is exactly what I *was* concerning about.
>>
>>> Userspace can check that at restore time. Or we could fail the
>>> userspace write, which is a bit odd (the bit is otherwise RES0).
>>>
>>> Could you clarify your proposal?
>>
>> Let's assume two hosts below. 'has_gicv4_1' is true on host-A while
>> it is false on host-B because of lack of HW support or the kernel
>> parameter "kvm-arm.vgic_v4_enable=0".
>>
>> If we migrate guest through A->B->A, we may end-up lose the initial
>> guest-request "nASSGIreq=1" and don't use direct vSGI delivery for
>> this guest when it's migrated back to host-A.
> 
> My point above is that we shouldn't allow the A->B migration the first
> place, and fail it as quickly as possible. We don't know what the guest
> has observed in terms of GIC capability, and it may not have enabled the
> new flavour of SGIs just yet.

Indeed. I didn't realize it.

> 
>> This can be "fixed" by keep track of what guest had written into
>> nASSGIreq. And we need to evaluate the need for using direct vSGI
>> for a specified guest by 'has_gicv4_1 && nassgireq'.
> 
> It feels odd. It means we have more state than the HW normally has.
> I have an alternative proposal, see below.
> 
>> But if it's expected that "if userspace tries to set nASSGIreq on
>> a non-4.1 host, this bit will not latch", then this shouldn't be
>> a problem at all.
> 
> Well, that is the semantics of the RES0 bit. It applies from both
> guest and userspace.
> 
> And actually, maybe we can handle that pretty cheaply. If userspace
> tries to restore GICD_TYPER2 to a value that isn't what KVM can
> offer, we just return an error. Exactly like we do for GICD_IIDR.
> Hence the following patch:
> 
> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c 
> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> index 28b639fd1abc..e72dcc454247 100644
> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
> @@ -156,6 +156,7 @@ static int vgic_mmio_uaccess_write_v3_misc(struct 
> kvm_vcpu *vcpu,
>       struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
> 
>       switch (addr & 0x0c) {
> +    case GICD_TYPER2:
>       case GICD_IIDR:
>           if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
>               return -EINVAL;
> 
> Being a RO register, writing something that isn't compatible with the
> possible behaviour of the hypervisor will just return an error.

This is really a nice point to address my concern! I'm happy to see
this in v6 now.

> 
> What do you think?

I agreed with you, with a bit nervous though. Some old guests (which
have no knowledge about GICv4.1 vSGIs and don't care about nASSGIcap
at all) will also fail to migrate from A to B, just because now we
present two different (unused) GICD_TYPER2 registers to them.

Is it a little unfair to them :-) ?


Thanks,
Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-23  8:11           ` Zenghui Yu
@ 2020-03-23  8:25             ` Marc Zyngier
  2020-03-23 12:40               ` Zenghui Yu
  0 siblings, 1 reply; 114+ messages in thread
From: Marc Zyngier @ 2020-03-23  8:25 UTC (permalink / raw)
  To: Zenghui Yu
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Zenghui,

[...]

>> And actually, maybe we can handle that pretty cheaply. If userspace
>> tries to restore GICD_TYPER2 to a value that isn't what KVM can
>> offer, we just return an error. Exactly like we do for GICD_IIDR.
>> Hence the following patch:
>> 
>> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c 
>> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> index 28b639fd1abc..e72dcc454247 100644
>> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>> @@ -156,6 +156,7 @@ static int vgic_mmio_uaccess_write_v3_misc(struct 
>> kvm_vcpu *vcpu,
>>       struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>> 
>>       switch (addr & 0x0c) {
>> +    case GICD_TYPER2:
>>       case GICD_IIDR:
>>           if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
>>               return -EINVAL;
>> 
>> Being a RO register, writing something that isn't compatible with the
>> possible behaviour of the hypervisor will just return an error.
> 
> This is really a nice point to address my concern! I'm happy to see
> this in v6 now.
> 
>> 
>> What do you think?
> 
> I agreed with you, with a bit nervous though. Some old guests (which
> have no knowledge about GICv4.1 vSGIs and don't care about nASSGIcap
> at all) will also fail to migrate from A to B, just because now we
> present two different (unused) GICD_TYPER2 registers to them.
> 
> Is it a little unfair to them :-) ?

I never pretended to be fair! ;-)

I'm happy to prevent migrating from a v4.1 system (A) to a v4.0
system (B). As soon as the guest has run, it isn't safe to do so
(it may have read TYPER2, and now knows about vSGIs). We *could*
track this and find ways to migrate this state as well, but it
feels fragile.

Migrating from B to A is more appealing. It should be possible to
do so without much difficulty (just check that the nASSGIcap bit
is either 0 or equal to KVM's view of the capability).

But overall I seriously doubt we can easily migrate guests across
very different HW. We've been talking about this for years, and
we still don't have a good solution for it given the diversity
of the ecosystem... :-/

Thanks,

         M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 114+ messages in thread

* Re: [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor
  2020-03-23  8:25             ` Marc Zyngier
@ 2020-03-23 12:40               ` Zenghui Yu
  0 siblings, 0 replies; 114+ messages in thread
From: Zenghui Yu @ 2020-03-23 12:40 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: linux-arm-kernel, kvmarm, kvm, linux-kernel, Lorenzo Pieralisi,
	Jason Cooper, Robert Richter, Thomas Gleixner, Eric Auger,
	James Morse, Julien Thierry, Suzuki K Poulose

Hi Marc,

On 2020/3/23 16:25, Marc Zyngier wrote:
> Hi Zenghui,
> 
> [...]
> 
>>> And actually, maybe we can handle that pretty cheaply. If userspace
>>> tries to restore GICD_TYPER2 to a value that isn't what KVM can
>>> offer, we just return an error. Exactly like we do for GICD_IIDR.
>>> Hence the following patch:
>>>
>>> diff --git a/virt/kvm/arm/vgic/vgic-mmio-v3.c 
>>> b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> index 28b639fd1abc..e72dcc454247 100644
>>> --- a/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> +++ b/virt/kvm/arm/vgic/vgic-mmio-v3.c
>>> @@ -156,6 +156,7 @@ static int vgic_mmio_uaccess_write_v3_misc(struct 
>>> kvm_vcpu *vcpu,
>>>       struct vgic_dist *dist = &vcpu->kvm->arch.vgic;
>>>
>>>       switch (addr & 0x0c) {
>>> +    case GICD_TYPER2:
>>>       case GICD_IIDR:
>>>           if (val != vgic_mmio_read_v3_misc(vcpu, addr, len))
>>>               return -EINVAL;
>>>
>>> Being a RO register, writing something that isn't compatible with the
>>> possible behaviour of the hypervisor will just return an error.
>>
>> This is really a nice point to address my concern! I'm happy to see
>> this in v6 now.
>>
>>>
>>> What do you think?
>>
>> I agreed with you, with a bit nervous though. Some old guests (which
>> have no knowledge about GICv4.1 vSGIs and don't care about nASSGIcap
>> at all) will also fail to migrate from A to B, just because now we
>> present two different (unused) GICD_TYPER2 registers to them.
>>
>> Is it a little unfair to them :-) ?
> 
> I never pretended to be fair! ;-)
> 
> I'm happy to prevent migrating from a v4.1 system (A) to a v4.0
> system (B). As soon as the guest has run, it isn't safe to do so
> (it may have read TYPER2, and now knows about vSGIs). We *could*
> track this and find ways to migrate this state as well, but it
> feels fragile.
> 
> Migrating from B to A is more appealing. It should be possible to
> do so without much difficulty (just check that the nASSGIcap bit
> is either 0 or equal to KVM's view of the capability).
> 
> But overall I seriously doubt we can easily migrate guests across
> very different HW. We've been talking about this for years, and
> we still don't have a good solution for it given the diversity
> of the ecosystem... :-/

Fair enough. Thanks for your detailed explanation!


Zenghui


^ permalink raw reply	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Eagerly vmap vPEs
  2020-03-04 20:33 ` [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs Marc Zyngier
  2020-03-12  8:12   ` Zenghui Yu
  2020-03-17  2:49   ` Zenghui Yu
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     009384b38034111bf2c0c7bfb2740f5bd45c176c
Gitweb:        https://git.kernel.org/tip/009384b38034111bf2c0c7bfb2740f5bd45c176c
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:23 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:15:51 

irqchip/gic-v4.1: Eagerly vmap vPEs

Now that we have HW-accelerated SGIs being delivered to VPEs, it
becomes required to map the VPEs on all ITSs instead of relying
on the lazy approach that we would use when using the ITS-list
mechanism.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-17-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c | 52 +++++++++++++++++++++++++------
 1 file changed, 42 insertions(+), 10 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index aae5332..1259f7f 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -189,6 +189,15 @@ static DEFINE_IDA(its_vpeid_ida);
 #define gic_data_rdist_rd_base()	(gic_data_rdist()->rd_base)
 #define gic_data_rdist_vlpi_base()	(gic_data_rdist_rd_base() + SZ_128K)
 
+/*
+ * Skip ITSs that have no vLPIs mapped, unless we're on GICv4.1, as we
+ * always have vSGIs mapped.
+ */
+static bool require_its_list_vmovp(struct its_vm *vm, struct its_node *its)
+{
+	return (gic_rdists->has_rvpeid || vm->vlpi_count[its->list_nr]);
+}
+
 static u16 get_its_list(struct its_vm *vm)
 {
 	struct its_node *its;
@@ -198,7 +207,7 @@ static u16 get_its_list(struct its_vm *vm)
 		if (!is_v4(its))
 			continue;
 
-		if (vm->vlpi_count[its->list_nr])
+		if (require_its_list_vmovp(vm, its))
 			__set_bit(its->list_nr, &its_list);
 	}
 
@@ -1295,7 +1304,7 @@ static void its_send_vmovp(struct its_vpe *vpe)
 		if (!is_v4(its))
 			continue;
 
-		if (!vpe->its_vm->vlpi_count[its->list_nr])
+		if (!require_its_list_vmovp(vpe->its_vm, its))
 			continue;
 
 		desc.its_vmovp_cmd.col = &its->collections[col_id];
@@ -1586,12 +1595,31 @@ static int its_irq_set_irqchip_state(struct irq_data *d,
 	return 0;
 }
 
+/*
+ * Two favourable cases:
+ *
+ * (a) Either we have a GICv4.1, and all vPEs have to be mapped at all times
+ *     for vSGI delivery
+ *
+ * (b) Or the ITSs do not use a list map, meaning that VMOVP is cheap enough
+ *     and we're better off mapping all VPEs always
+ *
+ * If neither (a) nor (b) is true, then we map vPEs on demand.
+ *
+ */
+static bool gic_requires_eager_mapping(void)
+{
+	if (!its_list_map || gic_rdists->has_rvpeid)
+		return true;
+
+	return false;
+}
+
 static void its_map_vm(struct its_node *its, struct its_vm *vm)
 {
 	unsigned long flags;
 
-	/* Not using the ITS list? Everything is always mapped. */
-	if (!its_list_map)
+	if (gic_requires_eager_mapping())
 		return;
 
 	raw_spin_lock_irqsave(&vmovp_lock, flags);
@@ -1625,7 +1653,7 @@ static void its_unmap_vm(struct its_node *its, struct its_vm *vm)
 	unsigned long flags;
 
 	/* Not using the ITS list? Everything is always mapped. */
-	if (!its_list_map)
+	if (gic_requires_eager_mapping())
 		return;
 
 	raw_spin_lock_irqsave(&vmovp_lock, flags);
@@ -4282,8 +4310,12 @@ static int its_vpe_irq_domain_activate(struct irq_domain *domain,
 	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
 	struct its_node *its;
 
-	/* If we use the list map, we issue VMAPP on demand... */
-	if (its_list_map)
+	/*
+	 * If we use the list map, we issue VMAPP on demand... Unless
+	 * we're on a GICv4.1 and we eagerly map the VPE on all ITSs
+	 * so that VSGIs can work.
+	 */
+	if (!gic_requires_eager_mapping())
 		return 0;
 
 	/* Map the VPE to the first possible CPU */
@@ -4309,10 +4341,10 @@ static void its_vpe_irq_domain_deactivate(struct irq_domain *domain,
 	struct its_node *its;
 
 	/*
-	 * If we use the list map, we unmap the VPE once no VLPIs are
-	 * associated with the VM.
+	 * If we use the list map on GICv4.0, we unmap the VPE once no
+	 * VLPIs are associated with the VM.
 	 */
-	if (its_list_map)
+	if (!gic_requires_eager_mapping())
 		return;
 
 	list_for_each_entry(its, &its_nodes, entry) {

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Add VSGI property setup
  2020-03-04 20:33 ` [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup Marc Zyngier
  2020-03-12  8:09   ` Zenghui Yu
  2020-03-17 10:30   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     d50676f5ce8481b98f9bbc1514b5d3f8747dd3c2
Gitweb:        https://git.kernel.org/tip/d50676f5ce8481b98f9bbc1514b5d3f8747dd3c2
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:22 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:15:51 

irqchip/gic-v4.1: Add VSGI property setup

Add the SGI configuration entry point for KVM to use.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-16-maz@kernel.org
---
 drivers/irqchip/irq-gic-v4.c       | 13 +++++++++++++
 include/linux/irqchip/arm-gic-v4.h |  1 +
 2 files changed, 14 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 99b33f6..0c18714 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -320,6 +320,19 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
 	return irq_set_vcpu_affinity(irq, &info);
 }
 
+int its_prop_update_vsgi(int irq, u8 priority, bool group)
+{
+	struct its_cmd_info info = {
+		.cmd_type = PROP_UPDATE_VSGI,
+		{
+			.priority	= priority,
+			.group		= group,
+		},
+	};
+
+	return irq_set_vcpu_affinity(irq, &info);
+}
+
 int its_init_v4(struct irq_domain *domain,
 		const struct irq_domain_ops *vpe_ops,
 		const struct irq_domain_ops *sgi_ops)
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index b120a01..6976b83 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -134,6 +134,7 @@ int its_map_vlpi(int irq, struct its_vlpi_map *map);
 int its_get_vlpi(int irq, struct its_vlpi_map *map);
 int its_unmap_vlpi(int irq);
 int its_prop_update_vlpi(int irq, u8 config, bool inv);
+int its_prop_update_vsgi(int irq, u8 priority, bool group);
 
 struct irq_domain_ops;
 int its_init_v4(struct irq_domain *domain,

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Add VSGI allocation/teardown
  2020-03-04 20:33 ` [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown Marc Zyngier
  2020-03-12  8:06   ` Zenghui Yu
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     6d31b6ff985dbd144b2c4d519cf573b8f81865d9
Gitweb:        https://git.kernel.org/tip/6d31b6ff985dbd144b2c4d519cf573b8f81865d9
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:21 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:15:51 

irqchip/gic-v4.1: Add VSGI allocation/teardown

Allocate per-VPE SGIs when initializing the GIC-specific part of the
VPE data structure.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-15-maz@kernel.org
---
 drivers/irqchip/irq-gic-v4.c       | 68 ++++++++++++++++++++++++++++-
 include/linux/irqchip/arm-gic-v4.h |  2 +-
 2 files changed, 69 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 117ba6d..99b33f6 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -92,6 +92,47 @@ static bool has_v4_1(void)
 	return !!sgi_domain_ops;
 }
 
+static int its_alloc_vcpu_sgis(struct its_vpe *vpe, int idx)
+{
+	char *name;
+	int sgi_base;
+
+	if (!has_v4_1())
+		return 0;
+
+	name = kasprintf(GFP_KERNEL, "GICv4-sgi-%d", task_pid_nr(current));
+	if (!name)
+		goto err;
+
+	vpe->fwnode = irq_domain_alloc_named_id_fwnode(name, idx);
+	if (!vpe->fwnode)
+		goto err;
+
+	kfree(name);
+	name = NULL;
+
+	vpe->sgi_domain = irq_domain_create_linear(vpe->fwnode, 16,
+						   sgi_domain_ops, vpe);
+	if (!vpe->sgi_domain)
+		goto err;
+
+	sgi_base = __irq_domain_alloc_irqs(vpe->sgi_domain, -1, 16,
+					       NUMA_NO_NODE, vpe,
+					       false, NULL);
+	if (sgi_base <= 0)
+		goto err;
+
+	return 0;
+
+err:
+	if (vpe->sgi_domain)
+		irq_domain_remove(vpe->sgi_domain);
+	if (vpe->fwnode)
+		irq_domain_free_fwnode(vpe->fwnode);
+	kfree(name);
+	return -ENOMEM;
+}
+
 int its_alloc_vcpu_irqs(struct its_vm *vm)
 {
 	int vpe_base_irq, i;
@@ -118,8 +159,13 @@ int its_alloc_vcpu_irqs(struct its_vm *vm)
 	if (vpe_base_irq <= 0)
 		goto err;
 
-	for (i = 0; i < vm->nr_vpes; i++)
+	for (i = 0; i < vm->nr_vpes; i++) {
+		int ret;
 		vm->vpes[i]->irq = vpe_base_irq + i;
+		ret = its_alloc_vcpu_sgis(vm->vpes[i], i);
+		if (ret)
+			goto err;
+	}
 
 	return 0;
 
@@ -132,8 +178,28 @@ err:
 	return -ENOMEM;
 }
 
+static void its_free_sgi_irqs(struct its_vm *vm)
+{
+	int i;
+
+	if (!has_v4_1())
+		return;
+
+	for (i = 0; i < vm->nr_vpes; i++) {
+		unsigned int irq = irq_find_mapping(vm->vpes[i]->sgi_domain, 0);
+
+		if (WARN_ON(!irq))
+			continue;
+
+		irq_domain_free_irqs(irq, 16);
+		irq_domain_remove(vm->vpes[i]->sgi_domain);
+		irq_domain_free_fwnode(vm->vpes[i]->fwnode);
+	}
+}
+
 void its_free_vcpu_irqs(struct its_vm *vm)
 {
+	its_free_sgi_irqs(vm);
 	irq_domain_free_irqs(vm->vpes[0]->irq, vm->nr_vpes);
 	irq_domain_remove(vm->domain);
 	irq_domain_free_fwnode(vm->fwnode);
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 34ed4b5..b120a01 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -49,6 +49,8 @@ struct its_vpe {
 		};
 		/* GICv4.1 implementations */
 		struct {
+			struct fwnode_handle	*fwnode;
+			struct irq_domain	*sgi_domain;
 			struct {
 				u8	priority;
 				bool	enabled;

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer
  2020-03-04 20:33 ` [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer Marc Zyngier
  2020-03-12  8:20   ` Zenghui Yu
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     ae699ad348cdcd416cbf28e8a02fc468780161f7
Gitweb:        https://git.kernel.org/tip/ae699ad348cdcd416cbf28e8a02fc468780161f7
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:20 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:15:51 

irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer

In order to hide some of the differences between v4.0 and v4.1, move
the doorbell management out of the KVM code, and into the GICv4-specific
layer. This allows the calling code to ask for the doorbell when blocking,
and otherwise to leave the doorbell permanently disabled.

This matches the v4.1 code perfectly, and only results in a minor
refactoring of the v4.0 code.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-14-maz@kernel.org
---
 drivers/irqchip/irq-gic-v4.c       | 45 ++++++++++++++++++++++++++---
 include/kvm/arm_vgic.h             |  1 +-
 include/linux/irqchip/arm-gic-v4.h |  3 +-
 virt/kvm/arm/vgic/vgic-v3.c        |  4 ++-
 virt/kvm/arm/vgic/vgic-v4.c        | 34 +++++++++-------------
 5 files changed, 61 insertions(+), 26 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index c01910d..117ba6d 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -87,6 +87,11 @@ static struct irq_domain *gic_domain;
 static const struct irq_domain_ops *vpe_domain_ops;
 static const struct irq_domain_ops *sgi_domain_ops;
 
+static bool has_v4_1(void)
+{
+	return !!sgi_domain_ops;
+}
+
 int its_alloc_vcpu_irqs(struct its_vm *vm)
 {
 	int vpe_base_irq, i;
@@ -139,18 +144,50 @@ static int its_send_vpe_cmd(struct its_vpe *vpe, struct its_cmd_info *info)
 	return irq_set_vcpu_affinity(vpe->irq, info);
 }
 
-int its_schedule_vpe(struct its_vpe *vpe, bool on)
+int its_make_vpe_non_resident(struct its_vpe *vpe, bool db)
 {
-	struct its_cmd_info info;
+	struct irq_desc *desc = irq_to_desc(vpe->irq);
+	struct its_cmd_info info = { };
 	int ret;
 
 	WARN_ON(preemptible());
 
-	info.cmd_type = on ? SCHEDULE_VPE : DESCHEDULE_VPE;
+	info.cmd_type = DESCHEDULE_VPE;
+	if (has_v4_1()) {
+		/* GICv4.1 can directly deal with doorbells */
+		info.req_db = db;
+	} else {
+		/* Undo the nested disable_irq() calls... */
+		while (db && irqd_irq_disabled(&desc->irq_data))
+			enable_irq(vpe->irq);
+	}
+
+	ret = its_send_vpe_cmd(vpe, &info);
+	if (!ret)
+		vpe->resident = false;
+
+	return ret;
+}
+
+int its_make_vpe_resident(struct its_vpe *vpe, bool g0en, bool g1en)
+{
+	struct its_cmd_info info = { };
+	int ret;
+
+	WARN_ON(preemptible());
+
+	info.cmd_type = SCHEDULE_VPE;
+	if (has_v4_1()) {
+		info.g0en = g0en;
+		info.g1en = g1en;
+	} else {
+		/* Disabled the doorbell, as we're about to enter the guest */
+		disable_irq_nosync(vpe->irq);
+	}
 
 	ret = its_send_vpe_cmd(vpe, &info);
 	if (!ret)
-		vpe->resident = on;
+		vpe->resident = true;
 
 	return ret;
 }
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index 9d53f54..6345790 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -70,6 +70,7 @@ struct vgic_global {
 
 	/* Hardware has GICv4? */
 	bool			has_gicv4;
+	bool			has_gicv4_1;
 
 	/* GIC system register CPU interface */
 	struct static_key_false gicv3_cpuif;
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 1b34100..34ed4b5 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -125,7 +125,8 @@ struct its_cmd_info {
 
 int its_alloc_vcpu_irqs(struct its_vm *vm);
 void its_free_vcpu_irqs(struct its_vm *vm);
-int its_schedule_vpe(struct its_vpe *vpe, bool on);
+int its_make_vpe_resident(struct its_vpe *vpe, bool g0en, bool g1en);
+int its_make_vpe_non_resident(struct its_vpe *vpe, bool db);
 int its_invall_vpe(struct its_vpe *vpe);
 int its_map_vlpi(int irq, struct its_vlpi_map *map);
 int its_get_vlpi(int irq, struct its_vlpi_map *map);
diff --git a/virt/kvm/arm/vgic/vgic-v3.c b/virt/kvm/arm/vgic/vgic-v3.c
index f45635a..1bc09b5 100644
--- a/virt/kvm/arm/vgic/vgic-v3.c
+++ b/virt/kvm/arm/vgic/vgic-v3.c
@@ -595,7 +595,9 @@ int vgic_v3_probe(const struct gic_kvm_info *info)
 	/* GICv4 support? */
 	if (info->has_v4) {
 		kvm_vgic_global_state.has_gicv4 = gicv4_enable;
-		kvm_info("GICv4 support %sabled\n",
+		kvm_vgic_global_state.has_gicv4_1 = info->has_v4_1 && gicv4_enable;
+		kvm_info("GICv4%s support %sabled\n",
+			 kvm_vgic_global_state.has_gicv4_1 ? ".1" : "",
 			 gicv4_enable ? "en" : "dis");
 	}
 
diff --git a/virt/kvm/arm/vgic/vgic-v4.c b/virt/kvm/arm/vgic/vgic-v4.c
index 46f8755..1eb0f8c 100644
--- a/virt/kvm/arm/vgic/vgic-v4.c
+++ b/virt/kvm/arm/vgic/vgic-v4.c
@@ -67,10 +67,10 @@
  * it. And if we've migrated our vcpu from one CPU to another, we must
  * tell the ITS (so that the messages reach the right redistributor).
  * This is done in two steps: first issue a irq_set_affinity() on the
- * irq corresponding to the vcpu, then call its_schedule_vpe(). You
- * must be in a non-preemptible context. On exit, another call to
- * its_schedule_vpe() tells the redistributor that we're done with the
- * vcpu.
+ * irq corresponding to the vcpu, then call its_make_vpe_resident().
+ * You must be in a non-preemptible context. On exit, a call to
+ * its_make_vpe_non_resident() tells the redistributor that we're done
+ * with the vcpu.
  *
  * Finally, the doorbell handling: Each vcpu is allocated an interrupt
  * which will fire each time a VLPI is made pending whilst the vcpu is
@@ -86,7 +86,8 @@ static irqreturn_t vgic_v4_doorbell_handler(int irq, void *info)
 	struct kvm_vcpu *vcpu = info;
 
 	/* We got the message, no need to fire again */
-	if (!irqd_irq_disabled(&irq_to_desc(irq)->irq_data))
+	if (!kvm_vgic_global_state.has_gicv4_1 &&
+	    !irqd_irq_disabled(&irq_to_desc(irq)->irq_data))
 		disable_irq_nosync(irq);
 
 	vcpu->arch.vgic_cpu.vgic_v3.its_vpe.pending_last = true;
@@ -199,19 +200,11 @@ void vgic_v4_teardown(struct kvm *kvm)
 int vgic_v4_put(struct kvm_vcpu *vcpu, bool need_db)
 {
 	struct its_vpe *vpe = &vcpu->arch.vgic_cpu.vgic_v3.its_vpe;
-	struct irq_desc *desc = irq_to_desc(vpe->irq);
 
 	if (!vgic_supports_direct_msis(vcpu->kvm) || !vpe->resident)
 		return 0;
 
-	/*
-	 * If blocking, a doorbell is required. Undo the nested
-	 * disable_irq() calls...
-	 */
-	while (need_db && irqd_irq_disabled(&desc->irq_data))
-		enable_irq(vpe->irq);
-
-	return its_schedule_vpe(vpe, false);
+	return its_make_vpe_non_resident(vpe, need_db);
 }
 
 int vgic_v4_load(struct kvm_vcpu *vcpu)
@@ -232,18 +225,19 @@ int vgic_v4_load(struct kvm_vcpu *vcpu)
 	if (err)
 		return err;
 
-	/* Disabled the doorbell, as we're about to enter the guest */
-	disable_irq_nosync(vpe->irq);
-
-	err = its_schedule_vpe(vpe, true);
+	err = its_make_vpe_resident(vpe, false, vcpu->kvm->arch.vgic.enabled);
 	if (err)
 		return err;
 
 	/*
 	 * Now that the VPE is resident, let's get rid of a potential
-	 * doorbell interrupt that would still be pending.
+	 * doorbell interrupt that would still be pending. This is a
+	 * GICv4.0 only "feature"...
 	 */
-	return irq_set_irqchip_state(vpe->irq, IRQCHIP_STATE_PENDING, false);
+	if (!kvm_vgic_global_state.has_gicv4_1)
+		err = irq_set_irqchip_state(vpe->irq, IRQCHIP_STATE_PENDING, false);
+
+	return err;
 }
 
 static struct vgic_its *vgic_get_its(struct kvm *kvm,

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state " Marc Zyngier
  2020-03-12  7:41   ` Zenghui Yu
  2020-03-16 21:43   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     7017ff0ee1de9d45fafee88a4e7890cce92f482e
Gitweb:        https://git.kernel.org/tip/7017ff0ee1de9d45fafee88a4e7890cce92f482e
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:18 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:05:09 

irqchip/gic-v4.1: Plumb get/set_irqchip_state SGI callbacks

To implement the get/set_irqchip_state callbacks (limited to the
PENDING state), we have to use a particular set of hacks:

- Reading the pending state is done by using a pair of new redistributor
  registers (GICR_VSGIR, GICR_VSGIPENDR), which allow the 16 interrupts
  state to be retrieved.
- Setting the pending state is done by generating it as we'd otherwise do
  for a guest (writing to GITS_SGIR).
- Clearing the pending state is done by emitting a VSGI command with the
  "clear" bit set.

This requires some interesting locking though:
- When talking to the redistributor, we must make sure that the VPE
  affinity doesn't change, hence taking the VPE lock.
- At the same time, we must ensure that nobody accesses the same
  redistributor's GICR_VSGIR registers for a different VPE, which
  would corrupt the reading of the pending bits. We thus take the
  per-RD spinlock. Much fun.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-12-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c   | 77 +++++++++++++++++++++++++++++-
 include/linux/irqchip/arm-gic-v3.h | 14 +++++-
 2 files changed, 91 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index bc6666a..c614f0c 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3972,11 +3972,88 @@ static int its_sgi_set_affinity(struct irq_data *d,
 	return IRQ_SET_MASK_OK;
 }
 
+static int its_sgi_set_irqchip_state(struct irq_data *d,
+				     enum irqchip_irq_state which,
+				     bool state)
+{
+	if (which != IRQCHIP_STATE_PENDING)
+		return -EINVAL;
+
+	if (state) {
+		struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+		struct its_node *its = find_4_1_its();
+		u64 val;
+
+		val  = FIELD_PREP(GITS_SGIR_VPEID, vpe->vpe_id);
+		val |= FIELD_PREP(GITS_SGIR_VINTID, d->hwirq);
+		writeq_relaxed(val, its->sgir_base + GITS_SGIR - SZ_128K);
+	} else {
+		its_configure_sgi(d, true);
+	}
+
+	return 0;
+}
+
+static int its_sgi_get_irqchip_state(struct irq_data *d,
+				     enum irqchip_irq_state which, bool *val)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+	void __iomem *base;
+	unsigned long flags;
+	u32 count = 1000000;	/* 1s! */
+	u32 status;
+	int cpu;
+
+	if (which != IRQCHIP_STATE_PENDING)
+		return -EINVAL;
+
+	/*
+	 * Locking galore! We can race against two different events:
+	 *
+	 * - Concurent vPE affinity change: we must make sure it cannot
+	 *   happen, or we'll talk to the wrong redistributor. This is
+	 *   identical to what happens with vLPIs.
+	 *
+	 * - Concurrent VSGIPENDR access: As it involves accessing two
+	 *   MMIO registers, this must be made atomic one way or another.
+	 */
+	cpu = vpe_to_cpuid_lock(vpe, &flags);
+	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
+	base = gic_data_rdist_cpu(cpu)->rd_base + SZ_128K;
+	writel_relaxed(vpe->vpe_id, base + GICR_VSGIR);
+	do {
+		status = readl_relaxed(base + GICR_VSGIPENDR);
+		if (!(status & GICR_VSGIPENDR_BUSY))
+			goto out;
+
+		count--;
+		if (!count) {
+			pr_err_ratelimited("Unable to get SGI status\n");
+			goto out;
+		}
+		cpu_relax();
+		udelay(1);
+	} while (count);
+
+out:
+	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
+	vpe_to_cpuid_unlock(vpe, flags);
+
+	if (!count)
+		return -ENXIO;
+
+	*val = !!(status & (1 << d->hwirq));
+
+	return 0;
+}
+
 static struct irq_chip its_sgi_irq_chip = {
 	.name			= "GICv4.1-sgi",
 	.irq_mask		= its_sgi_mask_irq,
 	.irq_unmask		= its_sgi_unmask_irq,
 	.irq_set_affinity	= its_sgi_set_affinity,
+	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
+	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
 };
 
 static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index fd3be49..590cdbe 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -345,6 +345,15 @@
 #define GICR_VPENDBASER_4_1_VGRP1EN	(1ULL << 58)
 #define GICR_VPENDBASER_4_1_VPEID	GENMASK_ULL(15, 0)
 
+#define GICR_VSGIR			0x0080
+
+#define GICR_VSGIR_VPEID		GENMASK(15, 0)
+
+#define GICR_VSGIPENDR			0x0088
+
+#define GICR_VSGIPENDR_BUSY		(1U << 31)
+#define GICR_VSGIPENDR_PENDING		GENMASK(15, 0)
+
 /*
  * ITS registers, offsets from ITS_base
  */
@@ -368,6 +377,11 @@
 
 #define GITS_TRANSLATER			0x10040
 
+#define GITS_SGIR			0x20020
+
+#define GITS_SGIR_VPEID			GENMASK_ULL(47, 32)
+#define GITS_SGIR_VINTID		GENMASK_ULL(3, 0)
+
 #define GITS_CTLR_ENABLE		(1U << 0)
 #define GITS_CTLR_ImDe			(1U << 1)
 #define	GITS_CTLR_ITS_NUMBER_SHIFT	4

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity " Marc Zyngier
  2020-03-17 10:35   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     05d32df13c6b3c0850b68928048536e9a736d520
Gitweb:        https://git.kernel.org/tip/05d32df13c6b3c0850b68928048536e9a736d520
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:19 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:15:51 

irqchip/gic-v4.1: Plumb set_vcpu_affinity SGI callbacks

Just like for vLPIs, there is some configuration information that cannot
be directly communicated through the normal irqchip API, and we have to
use our good old friend set_vcpu_affinity as a side-band communication
mechanism.

This is used to configure group and priority for a given vSGI.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-13-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c   | 18 ++++++++++++++++++
 include/linux/irqchip/arm-gic-v4.h |  5 +++++
 2 files changed, 23 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c614f0c..aae5332 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4047,6 +4047,23 @@ out:
 	return 0;
 }
 
+static int its_sgi_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+	struct its_cmd_info *info = vcpu_info;
+
+	switch (info->cmd_type) {
+	case PROP_UPDATE_VSGI:
+		vpe->sgi_config[d->hwirq].priority = info->priority;
+		vpe->sgi_config[d->hwirq].group = info->group;
+		its_configure_sgi(d, false);
+		return 0;
+
+	default:
+		return -EINVAL;
+	}
+}
+
 static struct irq_chip its_sgi_irq_chip = {
 	.name			= "GICv4.1-sgi",
 	.irq_mask		= its_sgi_mask_irq,
@@ -4054,6 +4071,7 @@ static struct irq_chip its_sgi_irq_chip = {
 	.irq_set_affinity	= its_sgi_set_affinity,
 	.irq_set_irqchip_state	= its_sgi_set_irqchip_state,
 	.irq_get_irqchip_state	= its_sgi_get_irqchip_state,
+	.irq_set_vcpu_affinity	= its_sgi_set_vcpu_affinity,
 };
 
 static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 44e8c19..1b34100 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -103,6 +103,7 @@ enum its_vcpu_info_cmd_type {
 	SCHEDULE_VPE,
 	DESCHEDULE_VPE,
 	INVALL_VPE,
+	PROP_UPDATE_VSGI,
 };
 
 struct its_cmd_info {
@@ -115,6 +116,10 @@ struct its_cmd_info {
 			bool		g0en;
 			bool		g1en;
 		};
+		struct {
+			u8		priority;
+			bool		group;
+		};
 	};
 };
 

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip
  2020-03-04 20:33 ` [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     166cba71818cd49d7d815fdc6f97c63395e94fc5
Gitweb:        https://git.kernel.org/tip/166cba71818cd49d7d815fdc6f97c63395e94fc5
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:15 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:05:04 

irqchip/gic-v4.1: Plumb skeletal VSGI irqchip

Since GICv4.1 has the capability to inject 16 SGIs into each VPE,
and that I'm keen not to invent too many specific interfaces to
manipulate these interrupts, let's pretend that each of these SGIs
is an actual Linux interrupt.

For that matter, let's introduce a minimal irqchip and irqdomain
setup that will get fleshed up in the following patches.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-9-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c   | 75 ++++++++++++++++++++++++++++-
 drivers/irqchip/irq-gic-v4.c       |  8 ++-
 include/linux/irqchip/arm-gic-v4.h |  9 ++-
 3 files changed, 88 insertions(+), 4 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 54d6fdf..c9c8121 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3870,6 +3870,72 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
 	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
 };
 
+static int its_sgi_set_affinity(struct irq_data *d,
+				const struct cpumask *mask_val,
+				bool force)
+{
+	/*
+	 * There is no notion of affinity for virtual SGIs, at least
+	 * not on the host (since they can only be targetting a vPE).
+	 * Tell the kernel we've done whatever it asked for.
+	 */
+	return IRQ_SET_MASK_OK;
+}
+
+static struct irq_chip its_sgi_irq_chip = {
+	.name			= "GICv4.1-sgi",
+	.irq_set_affinity	= its_sgi_set_affinity,
+};
+
+static int its_sgi_irq_domain_alloc(struct irq_domain *domain,
+				    unsigned int virq, unsigned int nr_irqs,
+				    void *args)
+{
+	struct its_vpe *vpe = args;
+	int i;
+
+	/* Yes, we do want 16 SGIs */
+	WARN_ON(nr_irqs != 16);
+
+	for (i = 0; i < 16; i++) {
+		vpe->sgi_config[i].priority = 0;
+		vpe->sgi_config[i].enabled = false;
+		vpe->sgi_config[i].group = false;
+
+		irq_domain_set_hwirq_and_chip(domain, virq + i, i,
+					      &its_sgi_irq_chip, vpe);
+		irq_set_status_flags(virq + i, IRQ_DISABLE_UNLAZY);
+	}
+
+	return 0;
+}
+
+static void its_sgi_irq_domain_free(struct irq_domain *domain,
+				    unsigned int virq,
+				    unsigned int nr_irqs)
+{
+	/* Nothing to do */
+}
+
+static int its_sgi_irq_domain_activate(struct irq_domain *domain,
+				       struct irq_data *d, bool reserve)
+{
+	return 0;
+}
+
+static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
+					  struct irq_data *d)
+{
+	/* Nothing to do */
+}
+
+static const struct irq_domain_ops its_sgi_domain_ops = {
+	.alloc		= its_sgi_irq_domain_alloc,
+	.free		= its_sgi_irq_domain_free,
+	.activate	= its_sgi_irq_domain_activate,
+	.deactivate	= its_sgi_irq_domain_deactivate,
+};
+
 static int its_vpe_id_alloc(void)
 {
 	return ida_simple_get(&its_vpeid_ida, 0, ITS_MAX_VPEID, GFP_KERNEL);
@@ -4912,8 +4978,15 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 		rdists->has_rvpeid = false;
 
 	if (has_v4 & rdists->has_vlpis) {
+		const struct irq_domain_ops *sgi_ops;
+
+		if (has_v4_1)
+			sgi_ops = &its_sgi_domain_ops;
+		else
+			sgi_ops = NULL;
+
 		if (its_init_vpe_domain() ||
-		    its_init_v4(parent_domain, &its_vpe_domain_ops)) {
+		    its_init_v4(parent_domain, &its_vpe_domain_ops, sgi_ops)) {
 			rdists->has_vlpis = false;
 			pr_err("ITS: Disabling GICv4 support\n");
 		}
diff --git a/drivers/irqchip/irq-gic-v4.c b/drivers/irqchip/irq-gic-v4.c
index 4596992..c01910d 100644
--- a/drivers/irqchip/irq-gic-v4.c
+++ b/drivers/irqchip/irq-gic-v4.c
@@ -85,6 +85,7 @@
 
 static struct irq_domain *gic_domain;
 static const struct irq_domain_ops *vpe_domain_ops;
+static const struct irq_domain_ops *sgi_domain_ops;
 
 int its_alloc_vcpu_irqs(struct its_vm *vm)
 {
@@ -216,12 +217,15 @@ int its_prop_update_vlpi(int irq, u8 config, bool inv)
 	return irq_set_vcpu_affinity(irq, &info);
 }
 
-int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops)
+int its_init_v4(struct irq_domain *domain,
+		const struct irq_domain_ops *vpe_ops,
+		const struct irq_domain_ops *sgi_ops)
 {
 	if (domain) {
 		pr_info("ITS: Enabling GICv4 support\n");
 		gic_domain = domain;
-		vpe_domain_ops = ops;
+		vpe_domain_ops = vpe_ops;
+		sgi_domain_ops = sgi_ops;
 		return 0;
 	}
 
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index 439963f..44e8c19 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -49,6 +49,11 @@ struct its_vpe {
 		};
 		/* GICv4.1 implementations */
 		struct {
+			struct {
+				u8	priority;
+				bool	enabled;
+				bool	group;
+			}			sgi_config[16];
 			atomic_t vmapp_count;
 		};
 	};
@@ -123,6 +128,8 @@ int its_unmap_vlpi(int irq);
 int its_prop_update_vlpi(int irq, u8 config, bool inv);
 
 struct irq_domain_ops;
-int its_init_v4(struct irq_domain *domain, const struct irq_domain_ops *ops);
+int its_init_v4(struct irq_domain *domain,
+		const struct irq_domain_ops *vpe_ops,
+		const struct irq_domain_ops *sgi_ops);
 
 #endif

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Add initial SGI configuration
  2020-03-04 20:33 ` [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration Marc Zyngier
  2020-03-16 17:53   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     e252cf8a34d92adf41124cb59b19b49d25395548
Gitweb:        https://git.kernel.org/tip/e252cf8a34d92adf41124cb59b19b49d25395548
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:16 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:05:08 

irqchip/gic-v4.1: Add initial SGI configuration

The GICv4.1 ITS has yet another new command (VSGI) which allows
a VPE-targeted SGI to be configured (or have its pending state
cleared). Add support for this command and plumb it into the
activate irqdomain callback so that it is ready to be used.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-10-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c   | 92 ++++++++++++++++++++++++++++-
 include/linux/irqchip/arm-gic-v3.h |  3 +-
 2 files changed, 93 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c9c8121..28c856a 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -380,6 +380,15 @@ struct its_cmd_desc {
 		struct {
 			struct its_vpe *vpe;
 		} its_invdb_cmd;
+
+		struct {
+			struct its_vpe *vpe;
+			u8 sgi;
+			u8 priority;
+			bool enable;
+			bool group;
+			bool clear;
+		} its_vsgi_cmd;
 	};
 };
 
@@ -528,6 +537,31 @@ static void its_encode_db(struct its_cmd_block *cmd, bool db)
 	its_mask_encode(&cmd->raw_cmd[2], db, 63, 63);
 }
 
+static void its_encode_sgi_intid(struct its_cmd_block *cmd, u8 sgi)
+{
+	its_mask_encode(&cmd->raw_cmd[0], sgi, 35, 32);
+}
+
+static void its_encode_sgi_priority(struct its_cmd_block *cmd, u8 prio)
+{
+	its_mask_encode(&cmd->raw_cmd[0], prio >> 4, 23, 20);
+}
+
+static void its_encode_sgi_group(struct its_cmd_block *cmd, bool grp)
+{
+	its_mask_encode(&cmd->raw_cmd[0], grp, 10, 10);
+}
+
+static void its_encode_sgi_clear(struct its_cmd_block *cmd, bool clr)
+{
+	its_mask_encode(&cmd->raw_cmd[0], clr, 9, 9);
+}
+
+static void its_encode_sgi_enable(struct its_cmd_block *cmd, bool en)
+{
+	its_mask_encode(&cmd->raw_cmd[0], en, 8, 8);
+}
+
 static inline void its_fixup_cmd(struct its_cmd_block *cmd)
 {
 	/* Let's fixup BE commands */
@@ -893,6 +927,26 @@ static struct its_vpe *its_build_invdb_cmd(struct its_node *its,
 	return valid_vpe(its, desc->its_invdb_cmd.vpe);
 }
 
+static struct its_vpe *its_build_vsgi_cmd(struct its_node *its,
+					  struct its_cmd_block *cmd,
+					  struct its_cmd_desc *desc)
+{
+	if (WARN_ON(!is_v4_1(its)))
+		return NULL;
+
+	its_encode_cmd(cmd, GITS_CMD_VSGI);
+	its_encode_vpeid(cmd, desc->its_vsgi_cmd.vpe->vpe_id);
+	its_encode_sgi_intid(cmd, desc->its_vsgi_cmd.sgi);
+	its_encode_sgi_priority(cmd, desc->its_vsgi_cmd.priority);
+	its_encode_sgi_group(cmd, desc->its_vsgi_cmd.group);
+	its_encode_sgi_clear(cmd, desc->its_vsgi_cmd.clear);
+	its_encode_sgi_enable(cmd, desc->its_vsgi_cmd.enable);
+
+	its_fixup_cmd(cmd);
+
+	return valid_vpe(its, desc->its_vsgi_cmd.vpe);
+}
+
 static u64 its_cmd_ptr_to_offset(struct its_node *its,
 				 struct its_cmd_block *ptr)
 {
@@ -3870,6 +3924,26 @@ static struct irq_chip its_vpe_4_1_irq_chip = {
 	.irq_set_vcpu_affinity	= its_vpe_4_1_set_vcpu_affinity,
 };
 
+static void its_configure_sgi(struct irq_data *d, bool clear)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+	struct its_cmd_desc desc;
+
+	desc.its_vsgi_cmd.vpe = vpe;
+	desc.its_vsgi_cmd.sgi = d->hwirq;
+	desc.its_vsgi_cmd.priority = vpe->sgi_config[d->hwirq].priority;
+	desc.its_vsgi_cmd.enable = vpe->sgi_config[d->hwirq].enabled;
+	desc.its_vsgi_cmd.group = vpe->sgi_config[d->hwirq].group;
+	desc.its_vsgi_cmd.clear = clear;
+
+	/*
+	 * GICv4.1 allows us to send VSGI commands to any ITS as long as the
+	 * destination VPE is mapped there. Since we map them eagerly at
+	 * activation time, we're pretty sure the first GICv4.1 ITS will do.
+	 */
+	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
+}
+
 static int its_sgi_set_affinity(struct irq_data *d,
 				const struct cpumask *mask_val,
 				bool force)
@@ -3920,13 +3994,29 @@ static void its_sgi_irq_domain_free(struct irq_domain *domain,
 static int its_sgi_irq_domain_activate(struct irq_domain *domain,
 				       struct irq_data *d, bool reserve)
 {
+	/* Write out the initial SGI configuration */
+	its_configure_sgi(d, false);
 	return 0;
 }
 
 static void its_sgi_irq_domain_deactivate(struct irq_domain *domain,
 					  struct irq_data *d)
 {
-	/* Nothing to do */
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+	/*
+	 * The VSGI command is awkward:
+	 *
+	 * - To change the configuration, CLEAR must be set to false,
+	 *   leaving the pending bit unchanged.
+	 * - To clear the pending bit, CLEAR must be set to true, leaving
+	 *   the configuration unchanged.
+	 *
+	 * You just can't do both at once, hence the two commands below.
+	 */
+	vpe->sgi_config[d->hwirq].enabled = false;
+	its_configure_sgi(d, false);
+	its_configure_sgi(d, true);
 }
 
 static const struct irq_domain_ops its_sgi_domain_ops = {
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index b28acfa..fd3be49 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -502,8 +502,9 @@
 #define GITS_CMD_VMAPTI			GITS_CMD_GICv4(GITS_CMD_MAPTI)
 #define GITS_CMD_VMOVI			GITS_CMD_GICv4(GITS_CMD_MOVI)
 #define GITS_CMD_VSYNC			GITS_CMD_GICv4(GITS_CMD_SYNC)
-/* VMOVP and INVDB are the odd ones, as they dont have a physical counterpart */
+/* VMOVP, VSGI and INVDB are the odd ones, as they dont have a physical counterpart */
 #define GITS_CMD_VMOVP			GITS_CMD_GICv4(2)
+#define GITS_CMD_VSGI			GITS_CMD_GICv4(3)
 #define GITS_CMD_INVDB			GITS_CMD_GICv4(0xe)
 
 /*

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks
  2020-03-04 20:33 ` [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks Marc Zyngier
  2020-03-16 18:15   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     b4e8d644ec623cbb66f192a7fefbd0a66e314be8
Gitweb:        https://git.kernel.org/tip/b4e8d644ec623cbb66f192a7fefbd0a66e314be8
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:17 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Tue, 24 Mar 2020 12:05:09 

irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks

Implement mask/unmask for virtual SGIs by calling into the
configuration helper.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-11-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 28c856a..bc6666a 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -3944,6 +3944,22 @@ static void its_configure_sgi(struct irq_data *d, bool clear)
 	its_send_single_vcommand(find_4_1_its(), its_build_vsgi_cmd, &desc);
 }
 
+static void its_sgi_mask_irq(struct irq_data *d)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+	vpe->sgi_config[d->hwirq].enabled = false;
+	its_configure_sgi(d, false);
+}
+
+static void its_sgi_unmask_irq(struct irq_data *d)
+{
+	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
+
+	vpe->sgi_config[d->hwirq].enabled = true;
+	its_configure_sgi(d, false);
+}
+
 static int its_sgi_set_affinity(struct irq_data *d,
 				const struct cpumask *mask_val,
 				bool force)
@@ -3958,6 +3974,8 @@ static int its_sgi_set_affinity(struct irq_data *d,
 
 static struct irq_chip its_sgi_irq_chip = {
 	.name			= "GICv4.1-sgi",
+	.irq_mask		= its_sgi_mask_irq,
+	.irq_unmask		= its_sgi_unmask_irq,
 	.irq_set_affinity	= its_sgi_set_affinity,
 };
 

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Map the ITS SGIR register page
  2020-03-04 20:33 ` [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     5e46a48413a6660955de7e56f9f364f2b890381c
Gitweb:        https://git.kernel.org/tip/5e46a48413a6660955de7e56f9f364f2b890381c
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:14 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Fri, 20 Mar 2020 17:48:38 

irqchip/gic-v4.1: Map the ITS SGIR register page

One of the new features of GICv4.1 is to allow virtual SGIs to be
directly signaled to a VPE. For that, the ITS has grown a new
64kB page containing only a single register that is used to
signal a SGI to a given VPE.

Add a second mapping covering this new 64kB range, and take this
opportunity to limit the original mapping to 64kB, which is enough
to cover the span of the ITS registers.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-8-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index bcc1a09..54d6fdf 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -96,6 +96,7 @@ struct its_node {
 	struct mutex		dev_alloc_lock;
 	struct list_head	entry;
 	void __iomem		*base;
+	void __iomem		*sgir_base;
 	phys_addr_t		phys_base;
 	struct its_cmd_block	*cmd_base;
 	struct its_cmd_block	*cmd_write;
@@ -4456,7 +4457,7 @@ static int __init its_probe_one(struct resource *res,
 	struct page *page;
 	int err;
 
-	its_base = ioremap(res->start, resource_size(res));
+	its_base = ioremap(res->start, SZ_64K);
 	if (!its_base) {
 		pr_warn("ITS@%pa: Unable to map ITS registers\n", &res->start);
 		return -ENOMEM;
@@ -4507,6 +4508,13 @@ static int __init its_probe_one(struct resource *res,
 
 		if (is_v4_1(its)) {
 			u32 svpet = FIELD_GET(GITS_TYPER_SVPET, typer);
+
+			its->sgir_base = ioremap(res->start + SZ_128K, SZ_64K);
+			if (!its->sgir_base) {
+				err = -ENOMEM;
+				goto out_free_its;
+			}
+
 			its->mpidr = readl_relaxed(its_base + GITS_MPIDR);
 
 			pr_info("ITS@%pa: Using GICv4.1 mode %08x %08x\n",
@@ -4520,7 +4528,7 @@ static int __init its_probe_one(struct resource *res,
 				get_order(ITS_CMD_QUEUE_SZ));
 	if (!page) {
 		err = -ENOMEM;
-		goto out_free_its;
+		goto out_unmap_sgir;
 	}
 	its->cmd_base = (void *)page_address(page);
 	its->cmd_write = its->cmd_base;
@@ -4587,6 +4595,9 @@ out_free_tables:
 	its_free_tables(its);
 out_free_cmd:
 	free_pages((unsigned long)its->cmd_base, get_order(ITS_CMD_QUEUE_SZ));
+out_unmap_sgir:
+	if (its->sgir_base)
+		iounmap(its->sgir_base);
 out_free_its:
 	kfree(its);
 out_unmap:

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Advertise support v4.1 to KVM
  2020-03-04 20:33 ` [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     3c40706d05fdea421e991da50e72a29d41131a66
Gitweb:        https://git.kernel.org/tip/3c40706d05fdea421e991da50e72a29d41131a66
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:13 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Fri, 20 Mar 2020 17:48:38 

irqchip/gic-v4.1: Advertise support v4.1 to KVM

Tell KVM that we support v4.1. Nothing uses this information so far.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-7-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c       |  9 ++++++++-
 drivers/irqchip/irq-gic-v3.c           |  2 ++
 include/linux/irqchip/arm-gic-common.h |  2 ++
 3 files changed, 12 insertions(+), 1 deletion(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index fc57885..bcc1a09 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -4870,6 +4870,7 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	struct device_node *of_node;
 	struct its_node *its;
 	bool has_v4 = false;
+	bool has_v4_1 = false;
 	int err;
 
 	gic_rdists = rdists;
@@ -4890,8 +4891,14 @@ int __init its_init(struct fwnode_handle *handle, struct rdists *rdists,
 	if (err)
 		return err;
 
-	list_for_each_entry(its, &its_nodes, entry)
+	list_for_each_entry(its, &its_nodes, entry) {
 		has_v4 |= is_v4(its);
+		has_v4_1 |= is_v4_1(its);
+	}
+
+	/* Don't bother with inconsistent systems */
+	if (WARN_ON(!has_v4_1 && rdists->has_rvpeid))
+		rdists->has_rvpeid = false;
 
 	if (has_v4 & rdists->has_vlpis) {
 		if (its_init_vpe_domain() ||
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index 0f716c2..8c5de59 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -1764,6 +1764,7 @@ static void __init gic_of_setup_kvm_info(struct device_node *node)
 		gic_v3_kvm_info.vcpu = r;
 
 	gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
+	gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
 	gic_set_kvm_info(&gic_v3_kvm_info);
 }
 
@@ -2079,6 +2080,7 @@ static void __init gic_acpi_setup_kvm_info(void)
 	}
 
 	gic_v3_kvm_info.has_v4 = gic_data.rdists.has_vlpis;
+	gic_v3_kvm_info.has_v4_1 = gic_data.rdists.has_rvpeid;
 	gic_set_kvm_info(&gic_v3_kvm_info);
 }
 
diff --git a/include/linux/irqchip/arm-gic-common.h b/include/linux/irqchip/arm-gic-common.h
index b9850f5..fa8c045 100644
--- a/include/linux/irqchip/arm-gic-common.h
+++ b/include/linux/irqchip/arm-gic-common.h
@@ -32,6 +32,8 @@ struct gic_kvm_info {
 	struct resource vctrl;
 	/* vlpi support */
 	bool		has_v4;
+	/* rvpeid support */
+	bool		has_v4_1;
 };
 
 const struct gic_kvm_info *gic_get_kvm_info(void);

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD
  2020-03-04 20:33 ` [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD Marc Zyngier
  2020-03-12  7:11   ` Zenghui Yu
  2020-03-20 14:23   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     9058a4e980648e7d068a7f7726a8ea4c67d0e88a
Gitweb:        https://git.kernel.org/tip/9058a4e980648e7d068a7f7726a8ea4c67d0e88a
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:12 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Fri, 20 Mar 2020 17:48:21 

irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD

The GICv4.1 spec says that it is CONTRAINED UNPREDICTABLE to write to
any of the GICR_INV{LPI,ALL}R registers if GICR_SYNCR.Busy == 1.

To deal with it, we must ensure that only a single invalidation can
happen at a time for a given redistributor. Add a per-RD lock to that
effect and take it around the invalidation/syncr-read to deal with this.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-6-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c   | 6 ++++++
 drivers/irqchip/irq-gic-v3.c       | 1 +
 include/linux/irqchip/arm-gic-v3.h | 1 +
 3 files changed, 8 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index c843702..fc57885 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -1373,10 +1373,12 @@ static void direct_lpi_inv(struct irq_data *d)
 
 	/* Target the redistributor this LPI is currently routed to */
 	cpu = irq_to_cpuid_lock(d, &flags);
+	raw_spin_lock(&gic_data_rdist_cpu(cpu)->rd_lock);
 	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVLPIR);
 
 	wait_for_syncr(rdbase);
+	raw_spin_unlock(&gic_data_rdist_cpu(cpu)->rd_lock);
 	irq_to_cpuid_unlock(d, flags);
 }
 
@@ -3662,9 +3664,11 @@ static void its_vpe_send_inv(struct irq_data *d)
 		void __iomem *rdbase;
 
 		/* Target the redistributor this VPE is currently known on */
+		raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 		rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
 		gic_write_lpir(d->parent_data->hwirq, rdbase + GICR_INVLPIR);
 		wait_for_syncr(rdbase);
+		raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 	} else {
 		its_vpe_send_cmd(vpe, its_send_inv);
 	}
@@ -3825,10 +3829,12 @@ static void its_vpe_4_1_invall(struct its_vpe *vpe)
 	val |= FIELD_PREP(GICR_INVALLR_VPEID, vpe->vpe_id);
 
 	/* Target the redistributor this vPE is currently known on */
+	raw_spin_lock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 	rdbase = per_cpu_ptr(gic_rdists->rdist, vpe->col_idx)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVALLR);
 
 	wait_for_syncr(rdbase);
+	raw_spin_unlock(&gic_data_rdist_cpu(vpe->col_idx)->rd_lock);
 }
 
 static int its_vpe_4_1_set_vcpu_affinity(struct irq_data *d, void *vcpu_info)
diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index b6b0f86..0f716c2 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -834,6 +834,7 @@ static int __gic_populate_rdist(struct redist_region *region, void __iomem *ptr)
 	typer = gic_read_typer(ptr + GICR_TYPER);
 	if ((typer >> 32) == aff) {
 		u64 offset = ptr - region->redist_base;
+		raw_spin_lock_init(&gic_data_rdist()->rd_lock);
 		gic_data_rdist_rd_base() = ptr;
 		gic_data_rdist()->phys_base = region->phys_base + offset;
 
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index c29a026..b28acfa 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -652,6 +652,7 @@
 
 struct rdists {
 	struct {
+		raw_spinlock_t	rd_lock;
 		void __iomem	*rd_base;
 		struct page	*pend_page;
 		phys_addr_t	phys_base;

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access
  2020-03-04 20:33 ` [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access Marc Zyngier
  2020-03-12  6:56   ` Zenghui Yu
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     f3a059219bc718ccc3bf3ff894f089b7e9a93139
Gitweb:        https://git.kernel.org/tip/f3a059219bc718ccc3bf3ff894f089b7e9a93139
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:10 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Thu, 19 Mar 2020 11:21:58 

irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access

Before GICv4.1, all operations would be serialized with the affinity
changes by virtue of using the same ITS command queue. With v4.1, things
change, as invalidations (and a number of other operations) are issued
using the redistributor MMIO frame.

We must thus make sure that these redistributor accesses cannot race
against aginst the affinity change, or we may end-up talking to the
wrong redistributor.

To ensure this, we expand the irq_to_cpuid() helper to take a spinlock
when the LPI is mapped to a vLPI (a new per-VPE lock) on each operation
that requires mutual exclusion.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-4-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c   | 56 ++++++++++++++++++++++++-----
 include/linux/irqchip/arm-gic-v4.h |  5 +++-
 2 files changed, 53 insertions(+), 8 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index da883a6..1af7139 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -239,15 +239,41 @@ static struct its_vlpi_map *get_vlpi_map(struct irq_data *d)
 	return NULL;
 }
 
-static int irq_to_cpuid(struct irq_data *d)
+static int vpe_to_cpuid_lock(struct its_vpe *vpe, unsigned long *flags)
+{
+	raw_spin_lock_irqsave(&vpe->vpe_lock, *flags);
+	return vpe->col_idx;
+}
+
+static void vpe_to_cpuid_unlock(struct its_vpe *vpe, unsigned long flags)
+{
+	raw_spin_unlock_irqrestore(&vpe->vpe_lock, flags);
+}
+
+static int irq_to_cpuid_lock(struct irq_data *d, unsigned long *flags)
 {
-	struct its_device *its_dev = irq_data_get_irq_chip_data(d);
 	struct its_vlpi_map *map = get_vlpi_map(d);
+	int cpu;
 
-	if (map)
-		return map->vpe->col_idx;
+	if (map) {
+		cpu = vpe_to_cpuid_lock(map->vpe, flags);
+	} else {
+		/* Physical LPIs are already locked via the irq_desc lock */
+		struct its_device *its_dev = irq_data_get_irq_chip_data(d);
+		cpu = its_dev->event_map.col_map[its_get_event_id(d)];
+		/* Keep GCC quiet... */
+		*flags = 0;
+	}
 
-	return its_dev->event_map.col_map[its_get_event_id(d)];
+	return cpu;
+}
+
+static void irq_to_cpuid_unlock(struct irq_data *d, unsigned long flags)
+{
+	struct its_vlpi_map *map = get_vlpi_map(d);
+
+	if (map)
+		vpe_to_cpuid_unlock(map->vpe, flags);
 }
 
 static struct its_collection *valid_col(struct its_collection *col)
@@ -1329,7 +1355,9 @@ static void direct_lpi_inv(struct irq_data *d)
 {
 	struct its_vlpi_map *map = get_vlpi_map(d);
 	void __iomem *rdbase;
+	unsigned long flags;
 	u64 val;
+	int cpu;
 
 	if (map) {
 		struct its_device *its_dev = irq_data_get_irq_chip_data(d);
@@ -1344,10 +1372,12 @@ static void direct_lpi_inv(struct irq_data *d)
 	}
 
 	/* Target the redistributor this LPI is currently routed to */
-	rdbase = per_cpu_ptr(gic_rdists->rdist, irq_to_cpuid(d))->rd_base;
+	cpu = irq_to_cpuid_lock(d, &flags);
+	rdbase = per_cpu_ptr(gic_rdists->rdist, cpu)->rd_base;
 	gic_write_lpir(val, rdbase + GICR_INVLPIR);
 
 	wait_for_syncr(rdbase);
+	irq_to_cpuid_unlock(d, flags);
 }
 
 static void lpi_update_config(struct irq_data *d, u8 clr, u8 set)
@@ -3486,17 +3516,25 @@ static int its_vpe_set_affinity(struct irq_data *d,
 {
 	struct its_vpe *vpe = irq_data_get_irq_chip_data(d);
 	int from, cpu = cpumask_first(mask_val);
+	unsigned long flags;
 
 	/*
 	 * Changing affinity is mega expensive, so let's be as lazy as
 	 * we can and only do it if we really have to. Also, if mapped
 	 * into the proxy device, we need to move the doorbell
 	 * interrupt to its new location.
+	 *
+	 * Another thing is that changing the affinity of a vPE affects
+	 * *other interrupts* such as all the vLPIs that are routed to
+	 * this vPE. This means that the irq_desc lock is not enough to
+	 * protect us, and that we must ensure nobody samples vpe->col_idx
+	 * during the update, hence the lock below which must also be
+	 * taken on any vLPI handling path that evaluates vpe->col_idx.
 	 */
-	if (vpe->col_idx == cpu)
+	from = vpe_to_cpuid_lock(vpe, &flags);
+	if (from == cpu)
 		goto out;
 
-	from = vpe->col_idx;
 	vpe->col_idx = cpu;
 
 	/*
@@ -3512,6 +3550,7 @@ static int its_vpe_set_affinity(struct irq_data *d,
 
 out:
 	irq_data_update_effective_affinity(d, cpumask_of(cpu));
+	vpe_to_cpuid_unlock(vpe, flags);
 
 	return IRQ_SET_MASK_OK_DONE;
 }
@@ -3855,6 +3894,7 @@ static int its_vpe_init(struct its_vpe *vpe)
 		return -ENOMEM;
 	}
 
+	raw_spin_lock_init(&vpe->vpe_lock);
 	vpe->vpe_id = vpe_id;
 	vpe->vpt_page = vpt_page;
 	if (gic_rdists->has_rvpeid)
diff --git a/include/linux/irqchip/arm-gic-v4.h b/include/linux/irqchip/arm-gic-v4.h
index d9c3496..439963f 100644
--- a/include/linux/irqchip/arm-gic-v4.h
+++ b/include/linux/irqchip/arm-gic-v4.h
@@ -54,6 +54,11 @@ struct its_vpe {
 	};
 
 	/*
+	 * Ensures mutual exclusion between affinity setting of the
+	 * vPE and vLPI operations using vpe->col_idx.
+	 */
+	raw_spinlock_t		vpe_lock;
+	/*
 	 * This collection ID is used to indirect the target
 	 * redistributor for this VPE. The ID itself isn't involved in
 	 * programming of the ITS.

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors
  2020-03-04 20:33 ` [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors Marc Zyngier
  2020-03-16 17:10   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  1 sibling, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, Eric Auger, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     28d160de5194c68ff534443d2a8b6f1d10d57c58
Gitweb:        https://git.kernel.org/tip/28d160de5194c68ff534443d2a8b6f1d10d57c58
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:09 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Thu, 19 Mar 2020 11:21:58 

irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors

In a system that is only sparsly populated with CPUs, we can end-up with
redistributors structures that are not initialized. Let's make sure we
don't try and access those when iterating over them (in this case when
checking we have a L2 VPE table).

Fixes: 4e6437f12d6e ("irqchip/gic-v4.1: Ensure L2 vPE table is allocated at RD level")
Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
Link: https://lore.kernel.org/r/20200304203330.4967-3-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3-its.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/irqchip/irq-gic-v3-its.c b/drivers/irqchip/irq-gic-v3-its.c
index 83b1186..da883a6 100644
--- a/drivers/irqchip/irq-gic-v3-its.c
+++ b/drivers/irqchip/irq-gic-v3-its.c
@@ -2452,6 +2452,10 @@ static bool allocate_vpe_l2_table(int cpu, u32 id)
 	if (!gic_rdists->has_rvpeid)
 		return true;
 
+	/* Skip non-present CPUs */
+	if (!base)
+		return true;
+
 	val  = gicr_read_vpropbaser(base + SZ_128K + GICR_VPROPBASER);
 
 	esz  = FIELD_GET(GICR_VPROPBASER_4_1_ENTRY_SIZE, val) + 1;

^ permalink raw reply related	[flat|nested] 114+ messages in thread

* [tip: irq/core] irqchip/gic-v3: Use SGIs without active state if offered
  2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
  2020-03-12  6:30   ` Zenghui Yu
  2020-03-12 17:16   ` Auger Eric
@ 2020-03-29 20:26   ` tip-bot2 for Marc Zyngier
  2 siblings, 0 replies; 114+ messages in thread
From: tip-bot2 for Marc Zyngier @ 2020-03-29 20:26 UTC (permalink / raw)
  To: linux-tip-commits; +Cc: Marc Zyngier, Zenghui Yu, x86, LKML

The following commit has been merged into the irq/core branch of tip:

Commit-ID:     0b04758b002bde9434053be2fff8064ac3d9d8bb
Gitweb:        https://git.kernel.org/tip/0b04758b002bde9434053be2fff8064ac3d9d8bb
Author:        Marc Zyngier <maz@kernel.org>
AuthorDate:    Wed, 04 Mar 2020 20:33:08 
Committer:     Marc Zyngier <maz@kernel.org>
CommitterDate: Thu, 19 Mar 2020 11:11:21 

irqchip/gic-v3: Use SGIs without active state if offered

To allow the direct injection of SGIs into a guest, the GICv4.1
architecture has to sacrifice the Active state so that SGIs look
a lot like LPIs (they are injected by the same mechanism).

In order not to break existing software, the architecture gives
offers guests OSs the choice: SGIs with or without an active
state. It is the hypervisors duty to honor the guest's choice.

For this, the architecture offers a discovery bit indicating whether
the GIC supports GICv4.1 SGIs (GICD_TYPER2.nASSGIcap), and another
bit indicating whether the guest wants Active-less SGIs or not
(controlled by GICD_CTLR.nASSGIreq).

A hypervisor not supporting GICv4.1 SGIs would leave nASSGIcap
clear, and a guest not knowing about GICv4.1 SGIs (or definitely
wanting an Active state) would leave nASSGIreq clear (both being
thankfully backward compatible with older revisions of the GIC).

Since Linux is perfectly happy without an active state on SGIs,
inform the hypervisor that we'll use that if offered.

Signed-off-by: Marc Zyngier <maz@kernel.org>
Reviewed-by: Zenghui Yu <yuzenghui@huawei.com>
Link: https://lore.kernel.org/r/20200304203330.4967-2-maz@kernel.org
---
 drivers/irqchip/irq-gic-v3.c       | 10 ++++++++--
 include/linux/irqchip/arm-gic-v3.h |  2 ++
 2 files changed, 10 insertions(+), 2 deletions(-)

diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c
index c1f7af9..b6b0f86 100644
--- a/drivers/irqchip/irq-gic-v3.c
+++ b/drivers/irqchip/irq-gic-v3.c
@@ -723,6 +723,7 @@ static void __init gic_dist_init(void)
 	unsigned int i;
 	u64 affinity;
 	void __iomem *base = gic_data.dist_base;
+	u32 val;
 
 	/* Disable the distributor */
 	writel_relaxed(0, base + GICD_CTLR);
@@ -755,9 +756,14 @@ static void __init gic_dist_init(void)
 	/* Now do the common stuff, and wait for the distributor to drain */
 	gic_dist_config(base, GIC_LINE_NR, gic_dist_wait_for_rwp);
 
+	val = GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1;
+	if (gic_data.rdists.gicd_typer2 & GICD_TYPER2_nASSGIcap) {
+		pr_info("Enabling SGIs without active state\n");
+		val |= GICD_CTLR_nASSGIreq;
+	}
+
 	/* Enable distributor with ARE, Group1 */
-	writel_relaxed(GICD_CTLR_ARE_NS | GICD_CTLR_ENABLE_G1A | GICD_CTLR_ENABLE_G1,
-		       base + GICD_CTLR);
+	writel_relaxed(val, base + GICD_CTLR);
 
 	/*
 	 * Set all global interrupts to the boot CPU only. ARE must be
diff --git a/include/linux/irqchip/arm-gic-v3.h b/include/linux/irqchip/arm-gic-v3.h
index 83439bf..c29a026 100644
--- a/include/linux/irqchip/arm-gic-v3.h
+++ b/include/linux/irqchip/arm-gic-v3.h
@@ -57,6 +57,7 @@
 #define GICD_SPENDSGIR			0x0F20
 
 #define GICD_CTLR_RWP			(1U << 31)
+#define GICD_CTLR_nASSGIreq		(1U << 8)
 #define GICD_CTLR_DS			(1U << 6)
 #define GICD_CTLR_ARE_NS		(1U << 4)
 #define GICD_CTLR_ENABLE_G1A		(1U << 1)
@@ -90,6 +91,7 @@
 #define GICD_TYPER_ESPIS(typer)						\
 	(((typer) & GICD_TYPER_ESPI) ? GICD_TYPER_SPIS((typer) >> 27) : 0)
 
+#define GICD_TYPER2_nASSGIcap		(1U << 8)
 #define GICD_TYPER2_VIL			(1U << 7)
 #define GICD_TYPER2_VID			GENMASK(4, 0)
 

^ permalink raw reply related	[flat|nested] 114+ messages in thread

end of thread, other threads:[~2020-03-29 20:28 UTC | newest]

Thread overview: 114+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-03-04 20:33 [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 01/23] irqchip/gic-v3: Use SGIs without active state if offered Marc Zyngier
2020-03-12  6:30   ` Zenghui Yu
2020-03-12  9:28     ` Marc Zyngier
2020-03-12 12:05       ` Marc Zyngier
2020-03-13  1:39         ` Zenghui Yu
2020-03-12 17:16   ` Auger Eric
2020-03-12 17:23     ` Marc Zyngier
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 02/23] irqchip/gic-v4.1: Skip absent CPUs while iterating over redistributors Marc Zyngier
2020-03-16 17:10   ` Auger Eric
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 03/23] irqchip/gic-v4.1: Ensure mutual exclusion between vPE affinity change and RD access Marc Zyngier
2020-03-12  6:56   ` Zenghui Yu
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 04/23] irqchip/gic-v4.1: Wait for completion of redistributor's INVALL operation Marc Zyngier
2020-03-20 14:23   ` Auger Eric
2020-03-04 20:33 ` [PATCH v5 05/23] irqchip/gic-v4.1: Ensure mutual exclusion betwen invalidations on the same RD Marc Zyngier
2020-03-12  7:11   ` Zenghui Yu
2020-03-20 14:23   ` Auger Eric
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 06/23] irqchip/gic-v4.1: Advertise support v4.1 to KVM Marc Zyngier
2020-03-16 17:10   ` Auger Eric
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 07/23] irqchip/gic-v4.1: Map the ITS SGIR register page Marc Zyngier
2020-03-16 17:10   ` Auger Eric
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 08/23] irqchip/gic-v4.1: Plumb skeletal VSGI irqchip Marc Zyngier
2020-03-16 17:10   ` Auger Eric
2020-03-19 10:03     ` Marc Zyngier
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 09/23] irqchip/gic-v4.1: Add initial SGI configuration Marc Zyngier
2020-03-16 17:53   ` Auger Eric
2020-03-17  2:02     ` Zenghui Yu
2020-03-17  8:36       ` Auger Eric
2020-03-19 10:20     ` Marc Zyngier
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 10/23] irqchip/gic-v4.1: Plumb mask/unmask SGI callbacks Marc Zyngier
2020-03-16 18:15   ` Auger Eric
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 11/23] irqchip/gic-v4.1: Plumb get/set_irqchip_state " Marc Zyngier
2020-03-12  7:41   ` Zenghui Yu
2020-03-16 21:43   ` Auger Eric
2020-03-19 10:27     ` Marc Zyngier
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 12/23] irqchip/gic-v4.1: Plumb set_vcpu_affinity " Marc Zyngier
2020-03-17 10:35   ` Auger Eric
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 13/23] irqchip/gic-v4.1: Move doorbell management to the GICv4 abstraction layer Marc Zyngier
2020-03-12  8:20   ` Zenghui Yu
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 14/23] irqchip/gic-v4.1: Add VSGI allocation/teardown Marc Zyngier
2020-03-12  8:06   ` Zenghui Yu
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 15/23] irqchip/gic-v4.1: Add VSGI property setup Marc Zyngier
2020-03-12  8:09   ` Zenghui Yu
2020-03-17 10:30   ` Auger Eric
2020-03-19 10:57     ` Marc Zyngier
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 16/23] irqchip/gic-v4.1: Eagerly vmap vPEs Marc Zyngier
2020-03-12  8:12   ` Zenghui Yu
2020-03-17  2:49   ` Zenghui Yu
2020-03-19 10:55     ` Marc Zyngier
2020-03-20  2:31       ` Zenghui Yu
2020-03-29 20:26   ` [tip: irq/core] " tip-bot2 for Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 17/23] KVM: arm64: GICv4.1: Let doorbells be auto-enabled Marc Zyngier
2020-03-12  8:15   ` Zenghui Yu
2020-03-17 11:04   ` Auger Eric
2020-03-04 20:33 ` [PATCH v5 18/23] KVM: arm64: GICv4.1: Add direct injection capability to SGI registers Marc Zyngier
2020-03-18  3:28   ` Zenghui Yu
2020-03-20  8:11   ` Auger Eric
2020-03-20 10:05     ` Marc Zyngier
2020-03-20 10:56       ` Auger Eric
2020-03-04 20:33 ` [PATCH v5 19/23] KVM: arm64: GICv4.1: Allow SGIs to switch between HW and SW interrupts Marc Zyngier
2020-03-19 16:16   ` Auger Eric
2020-03-19 19:52     ` Marc Zyngier
2020-03-19 20:13       ` Auger Eric
2020-03-20  9:17         ` Marc Zyngier
2020-03-20  4:22   ` Zenghui Yu
2020-03-04 20:33 ` [PATCH v5 20/23] KVM: arm64: GICv4.1: Plumb SGI implementation selection in the distributor Marc Zyngier
2020-03-18  6:34   ` Zenghui Yu
2020-03-19 12:10     ` Marc Zyngier
2020-03-19 20:38       ` Auger Eric
2020-03-20  3:08         ` Zenghui Yu
2020-03-20  7:59           ` Auger Eric
2020-03-20  9:46             ` Marc Zyngier
2020-03-20 11:09               ` Auger Eric
2020-03-20 11:20                 ` Marc Zyngier
2020-03-20  3:53       ` Zenghui Yu
2020-03-20  9:01         ` Marc Zyngier
2020-03-23  8:11           ` Zenghui Yu
2020-03-23  8:25             ` Marc Zyngier
2020-03-23 12:40               ` Zenghui Yu
2020-03-04 20:33 ` [PATCH v5 21/23] KVM: arm64: GICv4.1: Reload VLPI configuration on distributor enable/disable Marc Zyngier
2020-03-18  3:17   ` Zenghui Yu
2020-03-19 12:18     ` Marc Zyngier
2020-03-04 20:33 ` [PATCH v5 22/23] KVM: arm64: GICv4.1: Allow non-trapping WFI when using HW SGIs Marc Zyngier
2020-03-20  4:23   ` Zenghui Yu
2020-03-20  8:12   ` Auger Eric
2020-03-04 20:33 ` [PATCH v5 23/23] KVM: arm64: GICv4.1: Expose HW-based SGIs in debugfs Marc Zyngier
2020-03-18  3:19   ` Zenghui Yu
2020-03-19 15:05   ` Auger Eric
2020-03-19 15:21     ` Marc Zyngier
2020-03-19 15:43       ` Auger Eric
2020-03-19 16:16         ` Marc Zyngier
2020-03-19 16:17           ` Auger Eric
2020-03-20  4:38       ` Zenghui Yu
2020-03-20  9:09         ` Marc Zyngier
2020-03-20 11:35           ` Zenghui Yu
2020-03-20 11:46             ` Marc Zyngier
2020-03-20 12:09               ` Zenghui Yu
2020-03-05  3:39 ` [PATCH v5 00/23] irqchip/gic-v4: GICv4.1 architecture support Zenghui Yu
2020-03-09  8:17 ` Zenghui Yu
2020-03-09  8:46   ` Marc Zyngier

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).