[RFC PATCH 00/49] New VGIC(-v2) implementation

[RFC PATCH 00/49] New VGIC(-v2) implementation

[Andre Przywara]

tl;dr: More preparatory patches from patch 07, actual new VGIC starting
at patch 20.
=============

During development of the Dom0 ITS MSI support last year we realised
that the existing GIC interrupt controller emulation has some shortcomings.
After some tries to fix those in the existing code, it was agreed upon
that the problems are fundamental and a new implementation based on the
"new VGIC" in KVM is the best choice.
This is the first drop of this new VGIC implementation. It lives in the
xen/arch/arm/vgic/ directory and is written to be a compile time option,
so people can choose whether to use the new VGIC or the existing
implementation. This is just for a transitional period, the old VGIC is
expected to be removed after confidence in the new implementation has grown.

This series starts with some GICv3 redistributor cleanup, which I posted
before. I need to incorporate the comments from the list, but for now I left
those patches as it from the previous post.

Starting with patch 07 there are some more cleanups and preparations for
the existing VGIC/GIC code. A big part of those patches are preparations to
properly support level triggered interrupts. This is one of the biggest
problems in the existing VGIC, which only correctly emulates edge triggered
IRQs. This affects both arch code and some users like the timer and the
event channel.

Starting with patch 20 we plumb in the new VGIC then. This is done in a
new directory, with all the files actually not wired into the build system
until the very last patch. The idea is to split the series into reviewable
chunks without resorting to nasty hacks to keep bisectability.
The code was forked from Linux' virt/kvm/arm/vgic/, as of 4.14-rc7, plus
some recent changes to improve support for level triggered and hardware
mapped interrupts, which is what we use heavily in Dom0. The code was
heavily adapted to fit into Xen, starting with using the Xen coding style
and using Xen structure and variable names (struct domain instead of
struct kvm, for instance). Where interfacing functions were similar enough,
they were changed over to the existing Xen name and prototypes (for instance
kvm_vgic_create() was renamed to domain_vgic_register()). As far as possible
the code layout and split was re-used from KVM, so patches in Linux should
be relatively easy to port into Xen. Due to the mentioned changes this can
not be done easily in an automatic way, but it should be not too complicated
to extract the gist of the patch and re-apply this to our code base.

The actual VGIC code splits into several parts:
- The core is the struct vgic_irq, which holds every information about a
virtual IRQ, including a per-IRQ lock. Also there is on (ordered) per-VCPU
list (ap_list), which links the interrupts to be considered by a VCPU.
There are functions to deal with queuing and removing IRQs from those lists
safely, obeying the locking order. (patches 20-23)
- There are functions to push vIRQs on a VCPU list to the list registers,
and handle their state changes. (patches 24-26)
- The distributor MMIO emulation is using separate functions per register,
also having read and write split. (patches 27-37)
- There are functions to deal with Xen specialities. (patches 38-43)
- The data structures and the wiring of the emulation into the hypervisor
  and the guests are done in vgic-init.c. (patches 44-47)
- Finally patch 49 enables the build of the new VGIC. This requires to
  increase the size limit for struct vcpu in patch 48.

Please consider this series the first post that it is. It was briefly tested
on a Juno and a Midway (with and without CONFIG_NEW_VGIC). It does not cover
GICv3 yet, though supporting this should be relatively straightforward.
In fact I have most of the code here already, but I wanted to wait for
initial feedback before applying similar transformations to the *-v3.c files.
Also this does not include ITS support, though the code itself is more ready
for that than the old VGIC ever was.
I did some semi-automatic translation of coding style, so I am sure there
are quite some leftovers that slipped through, including identifiers with
kvm_ in their name ;-)

I would appreciate if somehow could have a look at those patches, patch 20
and following are surely the most interesting ones to have a look at.

Cheers,
Andre

Andre Przywara (49):
  tools: ARM: vGICv3: avoid inserting optional DT properties
  ARM: vGICv3: drop GUEST_GICV3_RDIST_REGIONS symbol
  ARM: GICv3: use hardware GICv3 redistributor regions for Dom0
  ARM: GICv3: simplify GICv3 redistributor stride handling
  ARM: vGICv3: always use architected redist stride
  ARM: vGICv3: remove rdist_stride from VGIC structure
  ARM: VGIC: move gic_remove_from_lr_pending() prototype
  ARM: VGIC: move max_vcpus VGIC limit to struct arch_domain
  ARM: VGIC: change to level-IRQ compatible IRQ injection interface
  ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist
  ARM: VGIC: reorder prototypes in vgic.h
  ARM: VGIC: introduce gic_get_nr_lrs()
  ARM: VGIC: Add hypervisor base address to vgic_v2_setup_hw()
  ARM: VGIC: extend GIC CPU interface definitions
  ARM: GIC: Allow tweaking the active state of an IRQ
  ARM: GIC: allow reading pending state of a hardware IRQ
  ARM: timer: Handle level triggered IRQs correctly
  ARM: evtchn: Handle level triggered IRQs correctly
  ARM: vPL011: Use the VGIC's level triggered IRQs handling if available
  ARM: new VGIC: Add data structure definitions
  ARM: new VGIC: Add acccessor to new struct vgic_irq instance
  ARM: new VGIC: Implement virtual IRQ injection
  ARM: new VGIC: Add IRQ sorting
  ARM: new VGIC: Add IRQ sync/flush framework
  ARM: new VGIC: Add GICv2 world switch backend
  ARM: new VGIC: Implement vgic_vcpu_pending_irq
  ARM: new VGIC: Add MMIO handling framework
  ARM: new VGIC: Add GICv2 MMIO handling framework
  ARM: new VGIC: Add CTLR, TYPER and IIDR handlers
  ARM: new VGIC: Add ENABLE registers handlers
  ARM: new VGIC: Add PENDING registers handlers
  ARM: new VGIC: Add ACTIVE registers handlers
  ARM: new VGIC: Add PRIORITY registers handlers
  ARM: new VGIC: Add CONFIG registers handlers
  ARM: new VGIC: Add TARGET registers handlers
  ARM: new VGIC: Add SGIR register handler
  ARM: new VGIC: Add SGIPENDR register handlers
  ARM: new VGIC: handle hardware mapped IRQs
  ARM: new VGIC: Add event channel IRQ handling
  ARM: new VGIC: Handle virtual IRQ allocation/reservation
  ARM: new VGIC: dump virtual IRQ info
  ARM: new VGIC: provide system register emulation stub
  ARM: new VGIC: Add preliminary stub implementations
  ARM: new VGIC: vgic-init: register VGIC
  ARM: new VGIC: vgic-init: implement vgic_init
  ARM: new VGIC: vgic-init: implement map_resources
  ARM: new VGIC: Add vgic_v2_enable
  ARM: allocate two pages for struct vcpu
  ARM: VGIC: wire new VGIC(-v2) files into Xen build system

 tools/libxl/libxl_arm.c           |   8 -
 xen/arch/arm/Kconfig              |   6 +-
 xen/arch/arm/Makefile             |  10 +-
 xen/arch/arm/domain.c             |  25 +-
 xen/arch/arm/gic-v2.c             |  20 +-
 xen/arch/arm/gic-v3-lpi.c         |   2 +-
 xen/arch/arm/gic-v3.c             |  72 ++--
 xen/arch/arm/gic-vgic.c           |  10 +-
 xen/arch/arm/gic.c                |  15 +
 xen/arch/arm/irq.c                |   2 +-
 xen/arch/arm/time.c               |  36 +-
 xen/arch/arm/traps.c              |   2 +
 xen/arch/arm/vgic-v2.c            |   3 +-
 xen/arch/arm/vgic-v3.c            |  45 +-
 xen/arch/arm/vgic.c               |  46 +-
 xen/arch/arm/vgic/vgic-init.c     | 259 ++++++++++++
 xen/arch/arm/vgic/vgic-mmio-v2.c  | 297 +++++++++++++
 xen/arch/arm/vgic/vgic-mmio.c     | 589 ++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h     | 194 +++++++++
 xen/arch/arm/vgic/vgic-v2.c       | 339 +++++++++++++++
 xen/arch/arm/vgic/vgic.c          | 871 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h          |  69 +++
 xen/arch/arm/vpl011.c             |   6 +-
 xen/arch/arm/vtimer.c             |   4 +-
 xen/common/Makefile               |   1 +
 xen/common/list_sort.c            | 170 ++++++++
 xen/include/asm-arm/arm_vgic.h    | 269 ++++++++++++
 xen/include/asm-arm/domain.h      |  92 +---
 xen/include/asm-arm/event.h       |   1 +
 xen/include/asm-arm/gic.h         |  31 +-
 xen/include/asm-arm/gic_v3_defs.h |   5 +
 xen/include/asm-arm/vgic.h        | 170 ++++++--
 xen/include/public/arch-arm.h     |   3 -
 xen/include/xen/list_sort.h       |  11 +
 xen/include/xen/timer.h           |   2 +
 35 files changed, 3456 insertions(+), 229 deletions(-)
 create mode 100644 xen/arch/arm/vgic/vgic-init.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.h
 create mode 100644 xen/arch/arm/vgic/vgic-v2.c
 create mode 100644 xen/arch/arm/vgic/vgic.c
 create mode 100644 xen/arch/arm/vgic/vgic.h
 create mode 100644 xen/common/list_sort.c
 create mode 100644 xen/include/asm-arm/arm_vgic.h
 create mode 100644 xen/include/xen/list_sort.h

-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 01/49] tools: ARM: vGICv3: avoid inserting optional DT properties

[Andre Przywara]

When creating a GICv3 devicetree node, we currently insert the
redistributor-stride and #redistributor-regions properties, with fixed
values which are actually the architected ones. But those properties are
optional and only needed to cover for broken platforms, where the values
differ from the architected one. This will never be the case for the
constructed DomU memory map.
So we drop those properties altogether and provide a clean and architected
GICv3 DT node for DomUs.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 tools/libxl/libxl_arm.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
index 3e46554301..b5bba3cd33 100644
--- a/tools/libxl/libxl_arm.c
+++ b/tools/libxl/libxl_arm.c
@@ -524,14 +524,6 @@ static int make_gicv3_node(libxl__gc *gc, void *fdt)
     res = fdt_property(fdt, "interrupt-controller", NULL, 0);
     if (res) return res;
 
-    res = fdt_property_cell(fdt, "redistributor-stride",
-                            GUEST_GICV3_RDIST_STRIDE);
-    if (res) return res;
-
-    res = fdt_property_cell(fdt, "#redistributor-regions",
-                            GUEST_GICV3_RDIST_REGIONS);
-    if (res) return res;
-
     res = fdt_property_regs(gc, fdt, ROOT_ADDRESS_CELLS, ROOT_SIZE_CELLS,
                             2,
                             gicd_base, gicd_size,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 02/49] ARM: vGICv3: drop GUEST_GICV3_RDIST_REGIONS symbol

[Andre Przywara]

Architecturally there is only one GICv3 redistributor region.
Drop the symbol which suggested that was a delibarate choice for Xen
guests, instead hard code the "1" in the appropriate places, along with
a comment to explain the reasons.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic-v3.c        | 17 ++++++++++++-----
 xen/include/public/arch-arm.h |  1 -
 2 files changed, 12 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 2ad8a6be62..12338c6b21 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1632,8 +1632,18 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
 
 static inline unsigned int vgic_v3_rdist_count(struct domain *d)
 {
-    return is_hardware_domain(d) ? vgic_v3_hw.nr_rdist_regions :
-               GUEST_GICV3_RDIST_REGIONS;
+    /*
+     * Architecturally there is only one GICv3 redistributor region.
+     * The GICv3 DT binding provisions for multiple regions, since there are
+     * platforms out there which break this architectural assumption.
+     * ACPI does not support this workaround at all.
+     * For Dom0 we have to live with the MMIO layout the hardware provides,
+     * so we have to copy the multiple regions - as the first region may not
+     * provide enough space to hold all redistributors we need.
+     * However DomU get a constructed memory map, so we can go with
+     * the architected single redistributor region.
+     */
+    return is_hardware_domain(d) ? vgic_v3_hw.nr_rdist_regions : 1;
 }
 
 static int vgic_v3_domain_init(struct domain *d)
@@ -1692,9 +1702,6 @@ static int vgic_v3_domain_init(struct domain *d)
     {
         d->arch.vgic.dbase = GUEST_GICV3_GICD_BASE;
 
-        /* XXX: Only one Re-distributor region mapped for the guest */
-        BUILD_BUG_ON(GUEST_GICV3_RDIST_REGIONS != 1);
-
         d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
 
         /* The first redistributor should contain enough space for all CPUs */
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 05fd11ca38..ca79ab6284 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -402,7 +402,6 @@ typedef uint64_t xen_callback_t;
 #define GUEST_GICV3_GICD_SIZE      xen_mk_ullong(0x00010000)
 
 #define GUEST_GICV3_RDIST_STRIDE   xen_mk_ullong(0x00020000)
-#define GUEST_GICV3_RDIST_REGIONS  1
 
 #define GUEST_GICV3_GICR0_BASE     xen_mk_ullong(0x03020000) /* vCPU0..127 */
 #define GUEST_GICV3_GICR0_SIZE     xen_mk_ullong(0x01000000)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 03/49] ARM: GICv3: use hardware GICv3 redistributor regions for Dom0

[Andre Przywara]

The code to generate the DT node or MADT table for Dom0 reaches into the
domain's VGIC structure to learn the number of redistributor regions and
their base addresses.
Since those values are copied from the hardware, we can as well use
those hardware values directly when setting up the hardware domain.

This avoids the hardware GIC code to reference vGIC data structures,
making this variable VGIC internal.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v3.c | 14 ++++++--------
 1 file changed, 6 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 25c30bb9ea..bdca77417c 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1167,8 +1167,7 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
     if ( res )
         return res;
 
-    res = fdt_property_cell(fdt, "#redistributor-regions",
-                            d->arch.vgic.nr_regions);
+    res = fdt_property_cell(fdt, "#redistributor-regions", gicv3.rdist_count);
     if ( res )
         return res;
 
@@ -1178,7 +1177,7 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
      * CPU interface and virtual cpu interfaces accessesed as System registers
      * So cells are created only for Distributor and rdist regions
      */
-    new_len = new_len * (d->arch.vgic.nr_regions + 1);
+    new_len = new_len * (gicv3.rdist_count + 1);
 
     hw_reg = dt_get_property(gic, "reg", &len);
     if ( !hw_reg )
@@ -1406,13 +1405,13 @@ static int gicv3_make_hwdom_madt(const struct domain *d, u32 offset)
 
     /* Add Generic Redistributor */
     size = sizeof(struct acpi_madt_generic_redistributor);
-    for ( i = 0; i < d->arch.vgic.nr_regions; i++ )
+    for ( i = 0; i < gicv3.rdist_count; i++ )
     {
         gicr = (struct acpi_madt_generic_redistributor *)(base_ptr + table_len);
         gicr->header.type = ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR;
         gicr->header.length = size;
-        gicr->base_address = d->arch.vgic.rdist_regions[i].base;
-        gicr->length = d->arch.vgic.rdist_regions[i].size;
+        gicr->base_address = gicv3.rdist_regions[i].base;
+        gicr->length = gicv3.rdist_regions[i].size;
         table_len += size;
     }
 
@@ -1425,8 +1424,7 @@ static unsigned long gicv3_get_hwdom_extra_madt_size(const struct domain *d)
 {
     unsigned long size;
 
-    size = sizeof(struct acpi_madt_generic_redistributor)
-           * d->arch.vgic.nr_regions;
+    size = sizeof(struct acpi_madt_generic_redistributor) * gicv3.rdist_count;
 
     size += sizeof(struct acpi_madt_generic_translator)
             * vgic_v3_its_count(d);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 04/49] ARM: GICv3: simplify GICv3 redistributor stride handling

[Andre Przywara]

Instead of hard coding the architected redistributor stride into the
code, lets use a clear #define to the two values for GICv3 and GICv4 and
clarify the algorithm to determine the needed stride value.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v3.c             | 18 ++++++++++--------
 xen/include/asm-arm/gic_v3_defs.h |  5 +++++
 2 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index bdca77417c..7837d93dc1 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -690,6 +690,15 @@ static int __init gicv3_populate_rdist(void)
         do {
             typer = readq_relaxed(ptr + GICR_TYPER);
 
+            /* Set the architectural redist size if not overridden by DT. */
+            if ( !gicv3.rdist_stride )
+            {
+                if ( typer & GICR_TYPER_VLPIS )
+                    gicv3.rdist_stride = GICV4_GICR_SIZE;
+                else
+                    gicv3.rdist_stride = GICV3_GICR_SIZE;
+            }
+
             if ( (typer >> 32) == aff )
             {
                 this_cpu(rbase) = ptr;
@@ -732,14 +741,7 @@ static int __init gicv3_populate_rdist(void)
             if ( gicv3.rdist_regions[i].single_rdist )
                 break;
 
-            if ( gicv3.rdist_stride )
-                ptr += gicv3.rdist_stride;
-            else
-            {
-                ptr += SZ_64K * 2; /* Skip RD_base + SGI_base */
-                if ( typer & GICR_TYPER_VLPIS )
-                    ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */
-            }
+            ptr += gicv3.rdist_stride;
 
         } while ( !(typer & GICR_TYPER_LAST) );
     }
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
index 65c9dc47cf..412e41afed 100644
--- a/xen/include/asm-arm/gic_v3_defs.h
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -18,6 +18,8 @@
 #ifndef __ASM_ARM_GIC_V3_DEFS_H__
 #define __ASM_ARM_GIC_V3_DEFS_H__
 
+#include <xen/sizes.h>
+
 /*
  * Additional registers defined in GIC v3.
  * Common GICD registers are defined in gic.h
@@ -68,6 +70,9 @@
 #define GICV3_GICD_IIDR_VAL          0x34c
 #define GICV3_GICR_IIDR_VAL          GICV3_GICD_IIDR_VAL
 
+#define GICV3_GICR_SIZE              (2 * SZ_64K)
+#define GICV4_GICR_SIZE              (4 * SZ_64K)
+
 #define GICR_CTLR                    (0x0000)
 #define GICR_IIDR                    (0x0004)
 #define GICR_TYPER                   (0x0008)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 05/49] ARM: vGICv3: always use architected redist stride

[Andre Przywara]

The redistributor-stride property in a GICv3 DT node is only there to
cover broken platforms where this value deviates from the architected one.
Since we emulate the GICv3 distributor even for Dom0, we don't need to
copy the broken behaviour. All the special handling for Dom0s using
GICv3 is just for using the hardware's memory map, which is unaffected
by the redistributor stride - it can never be smaller than the
architected two pages.
Remove the redistributor-stride property from Dom0's DT node and also
remove the code that tried to reuse the hardware value for Dom0's GICv3
emulation.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v3.c  |  5 -----
 xen/arch/arm/vgic-v3.c | 14 ++++++--------
 2 files changed, 6 insertions(+), 13 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 7837d93dc1..02c85e4c0c 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1164,11 +1164,6 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
     if ( res )
         return res;
 
-    res = fdt_property_cell(fdt, "redistributor-stride",
-                            d->arch.vgic.rdist_stride);
-    if ( res )
-        return res;
-
     res = fdt_property_cell(fdt, "#redistributor-regions", gicv3.rdist_count);
     if ( res )
         return res;
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 12338c6b21..e45bbc6dcf 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1024,10 +1024,9 @@ static struct vcpu *get_vcpu_from_rdist(struct domain *d,
     paddr_t gpa, uint32_t *offset)
 {
     struct vcpu *v;
-    uint32_t stride = d->arch.vgic.rdist_stride;
     unsigned int vcpu_id;
 
-    vcpu_id = region->first_cpu + ((gpa - region->base) / stride);
+    vcpu_id = region->first_cpu + ((gpa - region->base) / GICV3_GICR_SIZE);
     if ( unlikely(vcpu_id >= d->max_vcpus) )
         return NULL;
 
@@ -1586,7 +1585,6 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
 
     /* Convenient alias */
     struct domain *d = v->domain;
-    uint32_t rdist_stride = d->arch.vgic.rdist_stride;
 
     /*
      * Find the region where the re-distributor lives. For this purpose,
@@ -1602,11 +1600,11 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
 
     /* Get the base address of the redistributor */
     rdist_base = region->base;
-    rdist_base += (v->vcpu_id - region->first_cpu) * rdist_stride;
+    rdist_base += (v->vcpu_id - region->first_cpu) * GICV3_GICR_SIZE;
 
     /* Check if a valid region was found for the re-distributor */
     if ( (rdist_base < region->base) ||
-         ((rdist_base + rdist_stride) > (region->base + region->size)) )
+         ((rdist_base + GICV3_GICR_SIZE) > (region->base + region->size)) )
     {
         dprintk(XENLOG_ERR,
                 "d%u: Unable to find a re-distributor for VCPU %u\n",
@@ -1622,7 +1620,7 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
      * VGIC_V3_RDIST_LAST flags.
      * Note that we are assuming max_vcpus will never change.
      */
-    last_cpu = (region->size / rdist_stride) + region->first_cpu - 1;
+    last_cpu = (region->size / GICV3_GICR_SIZE) + region->first_cpu - 1;
 
     if ( v->vcpu_id == last_cpu || (v->vcpu_id == (d->max_vcpus - 1)) )
         v->arch.vgic.flags |= VGIC_V3_RDIST_LAST;
@@ -1693,7 +1691,7 @@ static int vgic_v3_domain_init(struct domain *d)
             /* Set the first CPU handled by this region */
             d->arch.vgic.rdist_regions[i].first_cpu = first_cpu;
 
-            first_cpu += size / d->arch.vgic.rdist_stride;
+            first_cpu += size / GICV3_GICR_SIZE;
         }
 
         d->arch.vgic.intid_bits = vgic_v3_hw.intid_bits;
@@ -1705,7 +1703,7 @@ static int vgic_v3_domain_init(struct domain *d)
         d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
 
         /* The first redistributor should contain enough space for all CPUs */
-        BUILD_BUG_ON((GUEST_GICV3_GICR0_SIZE / GUEST_GICV3_RDIST_STRIDE) < MAX_VIRT_CPUS);
+        BUILD_BUG_ON((GUEST_GICV3_GICR0_SIZE / GICV3_GICR_SIZE) < MAX_VIRT_CPUS);
         d->arch.vgic.rdist_regions[0].base = GUEST_GICV3_GICR0_BASE;
         d->arch.vgic.rdist_regions[0].size = GUEST_GICV3_GICR0_SIZE;
         d->arch.vgic.rdist_regions[0].first_cpu = 0;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 06/49] ARM: vGICv3: remove rdist_stride from VGIC structure

[Andre Przywara]

The last patch removed the usage of the hardware's redistributor-stride
value from our (Dom0) GICv3 emulation. This means we no longer need to
store this value in the VGIC data structure.
Remove that variable and every code snippet that handled that, instead
simply always use the architected value.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v3.c         |  3 +--
 xen/arch/arm/vgic-v3.c        | 14 --------------
 xen/include/asm-arm/domain.h  |  1 -
 xen/include/asm-arm/vgic.h    |  1 -
 xen/include/public/arch-arm.h |  2 --
 5 files changed, 1 insertion(+), 20 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 02c85e4c0c..ea14ab4028 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1682,8 +1682,7 @@ static int __init gicv3_init(void)
     reg = readl_relaxed(GICD + GICD_TYPER);
     intid_bits = GICD_TYPE_ID_BITS(reg);
 
-    vgic_v3_setup_hw(dbase, gicv3.rdist_count, gicv3.rdist_regions,
-                     gicv3.rdist_stride, intid_bits);
+    vgic_v3_setup_hw(dbase, gicv3.rdist_count, gicv3.rdist_regions, intid_bits);
     gicv3_init_v2();
 
     spin_lock_init(&gicv3.lock);
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index e45bbc6dcf..9b1b62744c 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -58,21 +58,18 @@ static struct {
     /* Re-distributor regions */
     unsigned int nr_rdist_regions;
     const struct rdist_region *regions;
-    uint32_t rdist_stride; /* Re-distributor stride */
     unsigned int intid_bits;  /* Number of interrupt ID bits */
 } vgic_v3_hw;
 
 void vgic_v3_setup_hw(paddr_t dbase,
                       unsigned int nr_rdist_regions,
                       const struct rdist_region *regions,
-                      uint32_t rdist_stride,
                       unsigned int intid_bits)
 {
     vgic_v3_hw.enabled = true;
     vgic_v3_hw.dbase = dbase;
     vgic_v3_hw.nr_rdist_regions = nr_rdist_regions;
     vgic_v3_hw.regions = regions;
-    vgic_v3_hw.rdist_stride = rdist_stride;
     vgic_v3_hw.intid_bits = intid_bits;
 }
 
@@ -1672,15 +1669,6 @@ static int vgic_v3_domain_init(struct domain *d)
 
         d->arch.vgic.dbase = vgic_v3_hw.dbase;
 
-        d->arch.vgic.rdist_stride = vgic_v3_hw.rdist_stride;
-        /*
-         * If the stride is not set, the default stride for GICv3 is 2 * 64K:
-         *     - first 64k page for Control and Physical LPIs
-         *     - second 64k page for Control and Generation of SGIs
-         */
-        if ( !d->arch.vgic.rdist_stride )
-            d->arch.vgic.rdist_stride = 2 * SZ_64K;
-
         for ( i = 0; i < vgic_v3_hw.nr_rdist_regions; i++ )
         {
             paddr_t size = vgic_v3_hw.regions[i].size;
@@ -1700,8 +1688,6 @@ static int vgic_v3_domain_init(struct domain *d)
     {
         d->arch.vgic.dbase = GUEST_GICV3_GICD_BASE;
 
-        d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
-
         /* The first redistributor should contain enough space for all CPUs */
         BUILD_BUG_ON((GUEST_GICV3_GICR0_SIZE / GICV3_GICR_SIZE) < MAX_VIRT_CPUS);
         d->arch.vgic.rdist_regions[0].base = GUEST_GICV3_GICR0_BASE;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 4fe189b1c3..3eda7196ff 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -108,7 +108,6 @@ struct arch_domain
             unsigned int first_cpu;         /* First CPU handled */
         } *rdist_regions;
         int nr_regions;                     /* Number of rdist regions */
-        uint32_t rdist_stride;              /* Re-Distributor stride */
         unsigned long int nr_lpis;
         uint64_t rdist_propbase;
         struct rb_root its_devices;         /* Devices mapped to an ITS */
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 6ea9f140a7..d61b54867b 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -261,7 +261,6 @@ struct rdist_region;
 void vgic_v3_setup_hw(paddr_t dbase,
                       unsigned int nr_rdist_regions,
                       const struct rdist_region *regions,
-                      uint32_t rdist_stride,
                       unsigned int intid_bits);
 #endif
 
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index ca79ab6284..3bca165fbf 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -401,8 +401,6 @@ typedef uint64_t xen_callback_t;
 #define GUEST_GICV3_GICD_BASE      xen_mk_ullong(0x03001000)
 #define GUEST_GICV3_GICD_SIZE      xen_mk_ullong(0x00010000)
 
-#define GUEST_GICV3_RDIST_STRIDE   xen_mk_ullong(0x00020000)
-
 #define GUEST_GICV3_GICR0_BASE     xen_mk_ullong(0x03020000) /* vCPU0..127 */
 #define GUEST_GICV3_GICR0_SIZE     xen_mk_ullong(0x01000000)
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 07/49] ARM: VGIC: move gic_remove_from_lr_pending() prototype

[Andre Przywara]

The prototype for gic_remove_from_lr_pending() is the last function in
gic.h which references a VGIC data structure.
Move it over to vgic.h, so that we can remove the inclusion of vgic.h
from gic.h. We add it to asm/domain.h instead, where it is actually
needed.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/include/asm-arm/domain.h | 1 +
 xen/include/asm-arm/gic.h    | 2 --
 xen/include/asm-arm/vgic.h   | 1 +
 3 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 3eda7196ff..1dd9683d25 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -8,6 +8,7 @@
 #include <asm/vfp.h>
 #include <asm/mmio.h>
 #include <asm/gic.h>
+#include <asm/vgic.h>
 #include <public/hvm/params.h>
 #include <xen/serial.h>
 #include <xen/rbtree.h>
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 497f195bc1..1d382b0ade 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -156,7 +156,6 @@
 #ifndef __ASSEMBLY__
 #include <xen/device_tree.h>
 #include <xen/irq.h>
-#include <asm-arm/vgic.h>
 
 #define DT_COMPAT_GIC_CORTEX_A15 "arm,cortex-a15-gic"
 
@@ -245,7 +244,6 @@ extern void init_maintenance_interrupt(void);
 extern void gic_raise_guest_irq(struct vcpu *v, unsigned int irq,
         unsigned int priority);
 extern void gic_raise_inflight_irq(struct vcpu *v, unsigned int virtual_irq);
-extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 
 /* Accept an interrupt from the GIC and dispatch its handler */
 extern void gic_interrupt(struct cpu_user_regs *regs, int is_fiq);
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index d61b54867b..d03298e12c 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -205,6 +205,7 @@ extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
 extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
 extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
 extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
+extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
 extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
 extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 08/49] ARM: VGIC: move max_vcpus VGIC limit to struct arch_domain

[Andre Przywara]

The VGIC model used for a domain (GICv2 or GICv3) determines the maximum
number of VCPUs for that guest, as GICv2 can only handle 8 processors.
In the moment we carry this per-VGIC-model limit in the vgic_ops,
alongside the model specific functions. That makes some sense, but
exposes some current VGIC implementation details to generic Xen code.
Add a new arch specific field in our domain structure to hold this vcpu limit,
and initialize it when we set the ops. This allows us to plug in the new
VGIC later without also needing to carry some ops structure.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/domain.c        | 5 ++---
 xen/arch/arm/vgic.c          | 3 ++-
 xen/include/asm-arm/domain.h | 1 +
 3 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index a010443bfd..9ad4cd0a6e 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -975,11 +975,10 @@ unsigned int domain_max_vcpus(const struct domain *d)
      * allocation when the vgic_ops haven't been initialised yet,
      * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
      */
-    if ( !d->arch.vgic.handler )
+    if ( !d->arch.max_vcpus )
         return MAX_VIRT_CPUS;
     else
-        return min_t(unsigned int, MAX_VIRT_CPUS,
-                     d->arch.vgic.handler->max_vcpus);
+        return d->arch.max_vcpus;
 }
 
 /*
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 9921769b15..5f47aa84a9 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -166,7 +166,8 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
 
 void register_vgic_ops(struct domain *d, const struct vgic_ops *ops)
 {
-   d->arch.vgic.handler = ops;
+    d->arch.vgic.handler = ops;
+    d->arch.max_vcpus = ops->max_vcpus;
 }
 
 void domain_vgic_free(struct domain *d)
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 1dd9683d25..2fef32eaee 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -149,6 +149,7 @@ struct arch_domain
 #ifdef CONFIG_SBSA_VUART_CONSOLE
     struct vpl011 vpl011;
 #endif
+    unsigned int max_vcpus;
 
 }  __cacheline_aligned;
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 09/49] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Andre Przywara]

At the moment vgic_vcpu_inject_irq() is the interface for Xen internal
code and virtual devices to inject IRQs into a guest. This interface has
two shortcomings:
1) It requires a VCPU pointer, which we may not know (and don't need!)
for shared interrupts. A second function (vgic_vcpu_inject_spi()), was
there to work around this issue.
2) This interface only really supports edge triggered IRQs, which is
what the Xen VGIC emulates only anyway. However this needs to and will
change, so we need to add the desired level (high or low) to the
interface.
This replaces the existing injection call (taking a VCPU and and IRQ
parameter) with a new one, taking domain, VCPU, IRQ and level parameters.
The VCPU can be NULL in case we don't know and don't care.
We change all call sites to use this new interface. This still doesn't
give us the missing level IRQ handling, but at least prepares the callers
to do the right thing later automatically.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/domain.c      |  4 ++--
 xen/arch/arm/gic-v3-lpi.c  |  2 +-
 xen/arch/arm/irq.c         |  2 +-
 xen/arch/arm/time.c        |  2 +-
 xen/arch/arm/vgic.c        | 43 +++++++++++++++++++++++++------------------
 xen/arch/arm/vpl011.c      |  2 +-
 xen/arch/arm/vtimer.c      |  4 ++--
 xen/include/asm-arm/vgic.h |  4 ++--
 8 files changed, 35 insertions(+), 28 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 9ad4cd0a6e..e76cfdfe83 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -952,14 +952,14 @@ void vcpu_mark_events_pending(struct vcpu *v)
     if ( already_pending )
         return;
 
-    vgic_vcpu_inject_irq(v, v->domain->arch.evtchn_irq);
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
 }
 
 /* The ARM spec declares that even if local irqs are masked in
  * the CPSR register, an irq should wake up a cpu from WFI anyway.
  * For this reason we need to check for irqs that need delivery,
  * ignoring the CPSR register, *after* calling SCHEDOP_block to
- * avoid races with vgic_vcpu_inject_irq.
+ * avoid races with vgic_inject_irq.
  */
 void vcpu_block_unless_event_pending(struct vcpu *v)
 {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 84582157b8..efd5cd62fb 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -153,7 +153,7 @@ void vgic_vcpu_inject_lpi(struct domain *d, unsigned int virq)
     if ( vcpu_id >= d->max_vcpus )
           return;
 
-    vgic_vcpu_inject_irq(d->vcpu[vcpu_id], virq);
+    vgic_inject_irq(d, d->vcpu[vcpu_id], virq, true);
 }
 
 /*
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 29af10e82c..aa4e832cae 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -225,7 +225,7 @@ void do_IRQ(struct cpu_user_regs *regs, unsigned int irq, int is_fiq)
          * The irq cannot be a PPI, we only support delivery of SPIs to
          * guests.
 	 */
-        vgic_vcpu_inject_spi(info->d, info->virq);
+        vgic_inject_irq(info->d, NULL, info->virq, true);
         goto out_no_end;
     }
 
diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index 36f640f0c1..c11fcfeadd 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -260,7 +260,7 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
 
     current->arch.virt_timer.ctl = READ_SYSREG32(CNTV_CTL_EL0);
     WRITE_SYSREG32(current->arch.virt_timer.ctl | CNTx_CTL_MASK, CNTV_CTL_EL0);
-    vgic_vcpu_inject_irq(current, current->arch.virt_timer.irq);
+    vgic_inject_irq(current->domain, current, current->arch.virt_timer.irq, true);
 }
 
 /*
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 5f47aa84a9..2fc6e19625 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -285,7 +285,7 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
         vgic_remove_irq_from_queues(old, p);
         irq_set_affinity(p->desc, cpumask_of(new->processor));
         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
-        vgic_vcpu_inject_irq(new, irq);
+        vgic_inject_irq(new->domain, new, irq, true);
         return true;
     }
     /* if the IRQ is in a GICH_LR register, set GIC_IRQ_GUEST_MIGRATING
@@ -444,7 +444,7 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
                         sgir, target->list);
                 continue;
             }
-            vgic_vcpu_inject_irq(d->vcpu[vcpuid], virq);
+            vgic_inject_irq(d, d->vcpu[vcpuid], virq, true);
         }
         break;
     case SGI_TARGET_OTHERS:
@@ -453,12 +453,12 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
         {
             if ( i != current->vcpu_id && d->vcpu[i] != NULL &&
                  is_vcpu_online(d->vcpu[i]) )
-                vgic_vcpu_inject_irq(d->vcpu[i], virq);
+                vgic_inject_irq(d, d->vcpu[i], virq, true);
         }
         break;
     case SGI_TARGET_SELF:
         perfc_incr(vgic_sgi_self);
-        vgic_vcpu_inject_irq(d->vcpu[current->vcpu_id], virq);
+        vgic_inject_irq(d, current, virq, true);
         break;
     default:
         gprintk(XENLOG_WARNING,
@@ -518,13 +518,29 @@ void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p)
     gic_remove_from_lr_pending(v, p);
 }
 
-void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
+int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                    bool level)
 {
     uint8_t priority;
     struct pending_irq *iter, *n;
     unsigned long flags;
     bool running;
 
+    /*
+     * For edge triggered interrupts we always ignore a "falling edge".
+     * For level triggered interrupts we shouldn't, but do anyways.
+     */
+    if ( !level )
+        return 0;
+
+    if ( !v )
+    {
+        /* The IRQ needs to be an SPI if no vCPU is specified. */
+        ASSERT(virq >= 32 && virq <= vgic_num_irqs(d));
+
+        v = vgic_get_target_vcpu(d->vcpu[0], virq);
+    };
+
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
 
     n = irq_to_pending(v, virq);
@@ -532,14 +548,14 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
     if ( unlikely(!n) )
     {
         spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
-        return;
+        return 0;
     }
 
     /* vcpu offline */
     if ( test_bit(_VPF_down, &v->pause_flags) )
     {
         spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
-        return;
+        return 0;
     }
 
     set_bit(GIC_IRQ_GUEST_QUEUED, &n->status);
@@ -576,22 +592,13 @@ out:
         perfc_incr(vgic_cross_cpu_intr_inject);
         smp_send_event_check_mask(cpumask_of(v->processor));
     }
-}
-
-void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq)
-{
-    struct vcpu *v;
 
-    /* the IRQ needs to be an SPI */
-    ASSERT(virq >= 32 && virq <= vgic_num_irqs(d));
-
-    v = vgic_get_target_vcpu(d->vcpu[0], virq);
-    vgic_vcpu_inject_irq(v, virq);
+    return 0;
 }
 
 void arch_evtchn_inject(struct vcpu *v)
 {
-    vgic_vcpu_inject_irq(v, v->domain->arch.evtchn_irq);
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
 }
 
 bool vgic_evtchn_irq_pending(struct vcpu *v)
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 7788c2fc32..5dcf4bec18 100644
--- a/xen/arch/arm/vpl011.c
+++ b/xen/arch/arm/vpl011.c
@@ -68,7 +68,7 @@ static void vpl011_update_interrupt_status(struct domain *d)
      * status bit has been set since the last time.
      */
     if ( uartmis & ~vpl011->shadow_uartmis )
-        vgic_vcpu_inject_spi(d, GUEST_VPL011_SPI);
+        vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, true);
 
     vpl011->shadow_uartmis = uartmis;
 }
diff --git a/xen/arch/arm/vtimer.c b/xen/arch/arm/vtimer.c
index f52a723a5f..8164f6c7f1 100644
--- a/xen/arch/arm/vtimer.c
+++ b/xen/arch/arm/vtimer.c
@@ -46,7 +46,7 @@ static void phys_timer_expired(void *data)
     if ( !(t->ctl & CNTx_CTL_MASK) )
     {
         perfc_incr(vtimer_phys_inject);
-        vgic_vcpu_inject_irq(t->v, t->irq);
+        vgic_inject_irq(t->v->domain, t->v, t->irq, true);
     }
     else
         perfc_incr(vtimer_phys_masked);
@@ -56,7 +56,7 @@ static void virt_timer_expired(void *data)
 {
     struct vtimer *t = data;
     t->ctl |= CNTx_CTL_MASK;
-    vgic_vcpu_inject_irq(t->v, t->irq);
+    vgic_inject_irq(t->v->domain, t->v, t->irq, true);
     perfc_incr(vtimer_virt_inject);
 }
 
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index d03298e12c..b75fdeb068 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -202,8 +202,8 @@ extern int domain_vgic_init(struct domain *d, unsigned int nr_spis);
 extern void domain_vgic_free(struct domain *d);
 extern int vcpu_vgic_init(struct vcpu *v);
 extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
-extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
-extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
+extern int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                           bool level);
 extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
 extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 10/49] ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist

[Andre Przywara]

Currently we describe the VGIC specific fields in an structure
*embedded* in struct arch_domain and struct arch_vcpu. These members
there are however related to the current VGIC implementation, and will
be substantially different in the future.
To allow coexistence of two implementations, move the definition of these
embedded structures into vgic.h, and just use the opaque type in the arch
specific structures.
This allows easy switching between different implementations later.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/include/asm-arm/domain.h | 85 +-----------------------------------------
 xen/include/asm-arm/vgic.h   | 88 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 83 deletions(-)

diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 2fef32eaee..968ffb0c81 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -74,57 +74,7 @@ struct arch_domain
         uint64_t offset;
     } virt_timer_base;
 
-    struct {
-        /* Version of the vGIC */
-        enum gic_version version;
-        /* GIC HW version specific vGIC driver handler */
-        const struct vgic_ops *handler;
-        /*
-         * Covers access to other members of this struct _except_ for
-         * shared_irqs where each member contains its own locking.
-         *
-         * If both class of lock is required then this lock must be
-         * taken first. If multiple rank locks are required (including
-         * the per-vcpu private_irqs rank) then they must be taken in
-         * rank order.
-         */
-        spinlock_t lock;
-        uint32_t ctlr;
-        int nr_spis; /* Number of SPIs */
-        unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
-        struct vgic_irq_rank *shared_irqs;
-        /*
-         * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
-         * struct arch_vcpu.
-         */
-        struct pending_irq *pending_irqs;
-        /* Base address for guest GIC */
-        paddr_t dbase; /* Distributor base address */
-#ifdef CONFIG_HAS_GICV3
-        /* GIC V3 addressing */
-        /* List of contiguous occupied by the redistributors */
-        struct vgic_rdist_region {
-            paddr_t base;                   /* Base address */
-            paddr_t size;                   /* Size */
-            unsigned int first_cpu;         /* First CPU handled */
-        } *rdist_regions;
-        int nr_regions;                     /* Number of rdist regions */
-        unsigned long int nr_lpis;
-        uint64_t rdist_propbase;
-        struct rb_root its_devices;         /* Devices mapped to an ITS */
-        spinlock_t its_devices_lock;        /* Protects the its_devices tree */
-        struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
-        rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
-        struct list_head vits_list;         /* List of virtual ITSes */
-        unsigned int intid_bits;
-        /*
-         * TODO: if there are more bool's being added below, consider
-         * a flags variable instead.
-         */
-        bool rdists_enabled;                /* Is any redistributor enabled? */
-        bool has_its;
-#endif
-    } vgic;
+    struct vgic_dist vgic;
 
     struct vuart {
 #define VUART_BUF_SIZE 128
@@ -248,38 +198,7 @@ struct arch_vcpu
     union gic_state_data gic;
     uint64_t lr_mask;
 
-    struct {
-        /*
-         * SGIs and PPIs are per-VCPU, SPIs are domain global and in
-         * struct arch_domain.
-         */
-        struct pending_irq pending_irqs[32];
-        struct vgic_irq_rank *private_irqs;
-
-        /* This list is ordered by IRQ priority and it is used to keep
-         * track of the IRQs that the VGIC injected into the guest.
-         * Depending on the availability of LR registers, the IRQs might
-         * actually be in an LR, and therefore injected into the guest,
-         * or queued in gic.lr_pending.
-         * As soon as an IRQ is EOI'd by the guest and removed from the
-         * corresponding LR it is also removed from this list. */
-        struct list_head inflight_irqs;
-        /* lr_pending is used to queue IRQs (struct pending_irq) that the
-         * vgic tried to inject in the guest (calling gic_set_guest_irq) but
-         * no LRs were available at the time.
-         * As soon as an LR is freed we remove the first IRQ from this
-         * list and write it to the LR register.
-         * lr_pending is a subset of vgic.inflight_irqs. */
-        struct list_head lr_pending;
-        spinlock_t lock;
-
-        /* GICv3: redistributor base and flags for this vCPU */
-        paddr_t rdist_base;
-        uint64_t rdist_pendbase;
-#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
-#define VGIC_V3_LPIS_ENABLED    (1 << 1)
-        uint8_t flags;
-    } vgic;
+    struct vgic_cpu vgic;
 
     /* Timer registers  */
     uint32_t cntkctl;
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index b75fdeb068..4e1c37f091 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -19,6 +19,9 @@
 #define __ASM_ARM_VGIC_H__
 
 #include <xen/bitops.h>
+#include <xen/radix-tree.h>
+#include <xen/rbtree.h>
+#include <asm/gic.h>
 #include <asm/mmio.h>
 #include <asm/vreg.h>
 
@@ -123,6 +126,91 @@ struct vgic_irq_rank {
     uint8_t vcpu[32];
 };
 
+struct vgic_dist {
+    /* Version of the vGIC */
+    enum gic_version version;
+    /* GIC HW version specific vGIC driver handler */
+    const struct vgic_ops *handler;
+    /*
+     * Covers access to other members of this struct _except_ for
+     * shared_irqs where each member contains its own locking.
+     *
+     * If both class of lock is required then this lock must be
+     * taken first. If multiple rank locks are required (including
+     * the per-vcpu private_irqs rank) then they must be taken in
+     * rank order.
+     */
+    spinlock_t lock;
+    uint32_t ctlr;
+    int nr_spis; /* Number of SPIs */
+    unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
+    struct vgic_irq_rank *shared_irqs;
+    /*
+     * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
+     * struct arch_vcpu.
+     */
+    struct pending_irq *pending_irqs;
+    /* Base address for guest GIC */
+    paddr_t dbase; /* Distributor base address */
+#ifdef CONFIG_HAS_GICV3
+    /* GIC V3 addressing */
+    /* List of contiguous occupied by the redistributors */
+    struct vgic_rdist_region {
+        paddr_t base;                   /* Base address */
+        paddr_t size;                   /* Size */
+        unsigned int first_cpu;         /* First CPU handled */
+    } *rdist_regions;
+    int nr_regions;                     /* Number of rdist regions */
+    unsigned long int nr_lpis;
+    uint64_t rdist_propbase;
+    struct rb_root its_devices;         /* Devices mapped to an ITS */
+    spinlock_t its_devices_lock;        /* Protects the its_devices tree */
+    struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
+    rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
+    struct list_head vits_list;         /* List of virtual ITSes */
+    unsigned int intid_bits;
+    /*
+     * TODO: if there are more bool's being added below, consider
+     * a flags variable instead.
+     */
+    bool rdists_enabled;                /* Is any redistributor enabled? */
+    bool has_its;
+#endif
+};
+
+struct vgic_cpu {
+    /*
+     * SGIs and PPIs are per-VCPU, SPIs are domain global and in
+     * struct arch_domain.
+     */
+    struct pending_irq pending_irqs[32];
+    struct vgic_irq_rank *private_irqs;
+
+    /* This list is ordered by IRQ priority and it is used to keep
+     * track of the IRQs that the VGIC injected into the guest.
+     * Depending on the availability of LR registers, the IRQs might
+     * actually be in an LR, and therefore injected into the guest,
+     * or queued in gic.lr_pending.
+     * As soon as an IRQ is EOI'd by the guest and removed from the
+     * corresponding LR it is also removed from this list. */
+    struct list_head inflight_irqs;
+    /* lr_pending is used to queue IRQs (struct pending_irq) that the
+     * vgic tried to inject in the guest (calling gic_set_guest_irq) but
+     * no LRs were available at the time.
+     * As soon as an LR is freed we remove the first IRQ from this
+     * list and write it to the LR register.
+     * lr_pending is a subset of vgic.inflight_irqs. */
+    struct list_head lr_pending;
+    spinlock_t lock;
+
+    /* GICv3: redistributor base and flags for this vCPU */
+    paddr_t rdist_base;
+    uint64_t rdist_pendbase;
+#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
+#define VGIC_V3_LPIS_ENABLED    (1 << 1)
+    uint8_t flags;
+};
+
 struct sgi_target {
     uint8_t aff1;
     uint16_t list;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 11/49] ARM: VGIC: reorder prototypes in vgic.h

[Andre Przywara]

Currently vgic.h both contains prototypes used by Xen arch code outside
of the actual VGIC (for instance vgic_vcpu_inject_irq()), and prototypes
for functions used by the VGIC internally.
Group them to later allow an easy split with one #ifdef.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/include/asm-arm/vgic.h | 73 ++++++++++++++++++++++++----------------------
 1 file changed, 38 insertions(+), 35 deletions(-)

diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 4e1c37f091..8c39ff1402 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -279,19 +279,8 @@ enum gic_sgi_mode;
  */
 #define REG_RANK_INDEX(b, n, s) ((((n) >> s) & ((b)-1)) % 32)
 
-/*
- * In the moment vgic_num_irqs() just covers SPIs and the private IRQs,
- * as it's mostly used for allocating the pending_irq and irq_desc array,
- * in which LPIs don't participate.
- */
-#define vgic_num_irqs(d)        ((d)->arch.vgic.nr_spis + 32)
 
-extern int domain_vgic_init(struct domain *d, unsigned int nr_spis);
-extern void domain_vgic_free(struct domain *d);
-extern int vcpu_vgic_init(struct vcpu *v);
 extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
-extern int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
-                           bool level);
 extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
 extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
@@ -307,29 +296,40 @@ extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
 int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
 
-bool vgic_evtchn_irq_pending(struct vcpu *v);
-struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
-                                      unsigned int virq);
-int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
-                        struct irq_desc *desc, bool connect);
-
-extern int domain_vgic_register(struct domain *d, int *mmio_count);
-extern int vcpu_vgic_free(struct vcpu *v);
 extern bool vgic_to_sgi(struct vcpu *v, register_t sgir,
                         enum gic_sgi_mode irqmode, int virq,
                         const struct sgi_target *target);
 extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq);
 
-/* Reserve a specific guest vIRQ */
-extern bool vgic_reserve_virq(struct domain *d, unsigned int virq);
+void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
+                      paddr_t vbase, uint32_t aliased_offset);
+
+#ifdef CONFIG_HAS_GICV3
+struct rdist_region;
+void vgic_v3_setup_hw(paddr_t dbase,
+                      unsigned int nr_rdist_regions,
+                      const struct rdist_region *regions,
+                      unsigned int intid_bits);
+#endif
+
+/*** Common VGIC functions used by Xen arch code ****/
 
 /*
- * Allocate a guest VIRQ
- *  - spi == 0 => allocate a PPI. It will be the same on every vCPU
- *  - spi == 1 => allocate an SPI
+ * In the moment vgic_num_irqs() just covers SPIs and the private IRQs,
+ * as it's mostly used for allocating the pending_irq and irq_desc array,
+ * in which LPIs don't participate.
  */
-extern int vgic_allocate_virq(struct domain *d, bool spi);
+#define vgic_num_irqs(d)        ((d)->arch.vgic.nr_spis + 32)
 
+/*
+ * Allocate a guest VIRQ
+ *  - is_spi == 0 => allocate a PPI. It will be the same on every vCPU
+ *  - is_spi == 1 => allocate an SPI
+ */
+extern int vgic_allocate_virq(struct domain *d, bool is_spi);
+/* Reserve a specific guest vIRQ */
+extern bool vgic_reserve_virq(struct domain *d, unsigned int virq);
+extern void vgic_free_virq(struct domain *d, unsigned int virq);
 static inline int vgic_allocate_ppi(struct domain *d)
 {
     return vgic_allocate_virq(d, false /* ppi */);
@@ -340,18 +340,21 @@ static inline int vgic_allocate_spi(struct domain *d)
     return vgic_allocate_virq(d, true /* spi */);
 }
 
-extern void vgic_free_virq(struct domain *d, unsigned int virq);
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq);
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                        struct irq_desc *desc, bool connect);
 
-void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
-                      paddr_t vbase, uint32_t aliased_offset);
+bool vgic_evtchn_irq_pending(struct vcpu *v);
 
-#ifdef CONFIG_HAS_GICV3
-struct rdist_region;
-void vgic_v3_setup_hw(paddr_t dbase,
-                      unsigned int nr_rdist_regions,
-                      const struct rdist_region *regions,
-                      unsigned int intid_bits);
-#endif
+int domain_vgic_register(struct domain *d, int *mmio_count);
+int domain_vgic_init(struct domain *d, unsigned int nr_spis);
+void domain_vgic_free(struct domain *d);
+int vcpu_vgic_init(struct vcpu *vcpu);
+int vcpu_vgic_free(struct vcpu *vcpu);
+
+int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                    bool level);
 
 #endif /* __ASM_ARM_VGIC_H__ */
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 12/49] ARM: VGIC: introduce gic_get_nr_lrs()

[Andre Przywara]

So far the number of list registers (LRs) a GIC implements is only
needed in the hardware facing side of the VGIC code (gic-vgic.c).
The new VGIC will need this information in more and multiple places, so
export a function that returns the number.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-vgic.c   | 10 +++++-----
 xen/arch/arm/gic.c        |  5 +++++
 xen/include/asm-arm/gic.h |  1 +
 3 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index d273863556..c92626e4ee 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -25,7 +25,7 @@
 #include <asm/gic.h>
 #include <asm/vgic.h>
 
-#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->info->nr_lrs) - 1))
+#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_get_nr_lrs()) - 1))
 
 #undef GIC_DEBUG
 
@@ -110,7 +110,7 @@ static unsigned int gic_find_unused_lr(struct vcpu *v,
                                        struct pending_irq *p,
                                        unsigned int lr)
 {
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
     unsigned long *lr_mask = (unsigned long *) &this_cpu(lr_mask);
     struct gic_lr lr_val;
 
@@ -137,7 +137,7 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq,
         unsigned int priority)
 {
     int i;
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
     struct pending_irq *p = irq_to_pending(v, virtual_irq);
 
     ASSERT(spin_is_locked(&v->arch.vgic.lock));
@@ -251,7 +251,7 @@ void gic_clear_lrs(struct vcpu *v)
 {
     int i = 0;
     unsigned long flags;
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
 
     /* The idle domain has no LRs to be cleared. Since gic_restore_state
      * doesn't write any LR registers for the idle domain they could be
@@ -278,7 +278,7 @@ static void gic_restore_pending_irqs(struct vcpu *v)
     struct pending_irq *p, *t, *p_r;
     struct list_head *inflight_r;
     unsigned long flags;
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
     int lrs = nr_lrs;
 
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 968e46fabb..89873c1df4 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -47,6 +47,11 @@ void register_gic_ops(const struct gic_hw_operations *ops)
     gic_hw_ops = ops;
 }
 
+int gic_get_nr_lrs(void)
+{
+    return gic_hw_ops->info->nr_lrs;
+}
+
 static void clear_cpu_lr_mask(void)
 {
     this_cpu(lr_mask) = 0ULL;
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 1d382b0ade..c1f027d703 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -222,6 +222,7 @@ enum gic_version {
 DECLARE_PER_CPU(uint64_t, lr_mask);
 
 extern enum gic_version gic_hw_version(void);
+extern int gic_get_nr_lrs(void);
 
 /* Program the IRQ type into the GIC */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 13/49] ARM: VGIC: Add hypervisor base address to vgic_v2_setup_hw()

[Andre Przywara]

The new VGIC will need to know the hypervisor base address at some
point, which is private to the hardware facing part of the VGIC so far.
Add a parameter to vgic_v2_setup_hw() to pass this address on, so a VGIC
implementation can make use of it.
The current VGIC ignores this new parameter.

TODO: add proper value for GICv2 on GICv3 emulation!

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v2.c      | 3 ++-
 xen/arch/arm/gic-v3.c      | 3 ++-
 xen/arch/arm/vgic-v2.c     | 3 ++-
 xen/include/asm-arm/vgic.h | 3 ++-
 4 files changed, 8 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 2b271ba322..7a18abecfa 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -1207,7 +1207,8 @@ static int __init gicv2_init(void)
     if ( !gicv2.map_hbase )
         panic("GICv2: Failed to ioremap for GIC Virtual interface\n");
 
-    vgic_v2_setup_hw(dbase, cbase, csize, vbase, aliased_offset);
+    vgic_v2_setup_hw(dbase, cbase, csize, vbase, gicv2.map_hbase,
+                     aliased_offset);
 
     /* Global settings: interrupt distributor */
     spin_lock_init(&gicv2.lock);
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index ea14ab4028..08d4703687 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1238,7 +1238,8 @@ static void __init gicv3_init_v2(void)
     printk("GICv3 compatible with GICv2 cbase %#"PRIpaddr" vbase %#"PRIpaddr"\n",
            cbase, vbase);
 
-    vgic_v2_setup_hw(dbase, cbase, csize, vbase, 0);
+    /* TODO: provide the proper HBASE address! */
+    vgic_v2_setup_hw(dbase, cbase, csize, vbase, NULL, 0);
 }
 
 static void __init gicv3_ioremap_distributor(paddr_t dist_paddr)
diff --git a/xen/arch/arm/vgic-v2.c b/xen/arch/arm/vgic-v2.c
index 646d1f3d12..96d543c005 100644
--- a/xen/arch/arm/vgic-v2.c
+++ b/xen/arch/arm/vgic-v2.c
@@ -47,7 +47,8 @@ static struct {
 } vgic_v2_hw;
 
 void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
-                      paddr_t vbase, uint32_t aliased_offset)
+                      paddr_t vbase, void __iomem *hbase,
+                      uint32_t aliased_offset)
 {
     vgic_v2_hw.enabled = true;
     vgic_v2_hw.dbase = dbase;
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 8c39ff1402..85ad2aca79 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -302,7 +302,8 @@ extern bool vgic_to_sgi(struct vcpu *v, register_t sgir,
 extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq);
 
 void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
-                      paddr_t vbase, uint32_t aliased_offset);
+                      paddr_t vbase, void __iomem *hbase,
+                      uint32_t aliased_offset);
 
 #ifdef CONFIG_HAS_GICV3
 struct rdist_region;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 14/49] ARM: VGIC: extend GIC CPU interface definitions

[Andre Przywara]

The new VGIC will shortly use more bits of the GICC_CTLR register, so
add the respective definitions from the manual.
Also add a missing definition for GICV_PMR_PRIORITY_MASK.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v2.c     |  2 +-
 xen/include/asm-arm/gic.h | 18 ++++++++++++++++--
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 7a18abecfa..2e35892881 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -358,7 +358,7 @@ static void gicv2_cpu_init(void)
     /* Finest granularity of priority */
     writel_gicc(0x0, GICC_BPR);
     /* Turn on delivery */
-    writel_gicc(GICC_CTL_ENABLE|GICC_CTL_EOI, GICC_CTLR);
+    writel_gicc(GICC_CTL_ENABLE0|GICC_CTL_EOI, GICC_CTLR);
 }
 
 static void gicv2_cpu_disable(void)
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index c1f027d703..c4c68c7770 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -77,6 +77,7 @@
 #define GICC_EOIR       (0x0010)
 #define GICC_RPR        (0x0014)
 #define GICC_HPPIR      (0x0018)
+#define GICC_ABPR       (0x001c)
 #define GICC_APR        (0x00D0)
 #define GICC_NSAPR      (0x00E0)
 #define GICC_IIDR       (0x00FC)
@@ -102,8 +103,18 @@
 #define GICD_TYPE_SEC   0x400
 #define GICD_TYPER_DVIS (1U << 18)
 
-#define GICC_CTL_ENABLE 0x1
-#define GICC_CTL_EOI    (0x1 << 9)
+#define GICC_CTL_ENABLE0_SHIFT  0
+#define GICC_CTL_ENABLE0        (1U << GICC_CTL_ENABLE0_SHIFT)
+#define GICC_CTL_ENABLE1_SHIFT  1
+#define GICC_CTL_ENABLE1        (1U << GICC_CTL_ENABLE1)
+#define GICC_CTL_AC_SHIFT       2
+#define GICC_CTL_AC             (1U << GICC_CTL_AC_SHIFT)
+#define GICC_CTL_FIQEN_SHIFT    3
+#define GICC_CTL_FIQEN          (1U << GICC_CTL_FIQEN_SHIFT)
+#define GICC_CTL_CBPR_SHIFT     4
+#define GICC_CTL_CBPR           (1U << GICC_CTL_CBPR_SHIFT)
+#define GICC_CTL_EOI_SHIFT      9
+#define GICC_CTL_EOI            (1U << GICC_CTL_EOI_SHIFT)
 
 #define GICC_IA_IRQ       0x03ff
 #define GICC_IA_CPU_MASK  0x1c00
@@ -127,6 +138,9 @@
 #define GICH_MISR_VGRP1E  (1 << 6)
 #define GICH_MISR_VGRP1D  (1 << 7)
 
+#define GICV_PMR_PRIORITY_SHIFT		3
+#define GICV_PMR_PRIORITY_MASK		(0x1f << GICV_PMR_PRIORITY_SHIFT)
+
 /*
  * The minimum GICC_BPR is required to be in the range 0-3. We set
  * GICC_BPR to 0 but we must expect that it might be 3. This means we
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 15/49] ARM: GIC: Allow tweaking the active state of an IRQ

[Andre Przywara]

When playing around with hardware mapped, level triggered virtual IRQs,
there is the need to explicitly set the active state of an interrupt at
some point in time.
To prepare the GIC for that, we introduce a set_active_state() function
to let the VGIC manipulate the state of an associated hardware IRQ.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v2.c     |  9 +++++++++
 xen/arch/arm/gic-v3.c     | 16 ++++++++++++++++
 xen/arch/arm/gic.c        |  5 +++++
 xen/include/asm-arm/gic.h |  5 +++++
 4 files changed, 35 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 2e35892881..5339f69fbc 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -235,6 +235,14 @@ static unsigned int gicv2_read_irq(void)
     return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
 }
 
+static void gicv2_set_active_state(int irq, bool active)
+{
+    if (active)
+        writel_gicd(1U << (irq % 32), GICD_ISACTIVER + (irq / 32) * 4);
+    else
+        writel_gicd(1U << (irq % 32), GICD_ICACTIVER + (irq / 32) * 4);
+}
+
 static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int type)
 {
     uint32_t cfg, actual, edgebit;
@@ -1241,6 +1249,7 @@ const static struct gic_hw_operations gicv2_ops = {
     .eoi_irq             = gicv2_eoi_irq,
     .deactivate_irq      = gicv2_dir_irq,
     .read_irq            = gicv2_read_irq,
+    .set_active_state    = gicv2_set_active_state,
     .set_irq_type        = gicv2_set_irq_type,
     .set_irq_priority    = gicv2_set_irq_priority,
     .send_SGI            = gicv2_send_SGI,
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 08d4703687..595eaef43a 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -475,6 +475,21 @@ static unsigned int gicv3_read_irq(void)
     return irq;
 }
 
+static void gicv3_set_active_state(int irq, bool active)
+{
+    void __iomem *base;
+
+    if ( irq >= NR_GIC_LOCAL_IRQS)
+        base = GICD + (irq / 32) * 4;
+    else
+        base = GICD_RDIST_SGI_BASE;
+
+    if ( active )
+        writel(1U << (irq % 32), base + GICD_ISACTIVER);
+    else
+        writel(1U << (irq % 32), base + GICD_ICACTIVER);
+}
+
 static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
 {
      uint64_t mpidr = cpu_logical_map(cpu);
@@ -1722,6 +1737,7 @@ static const struct gic_hw_operations gicv3_ops = {
     .eoi_irq             = gicv3_eoi_irq,
     .deactivate_irq      = gicv3_dir_irq,
     .read_irq            = gicv3_read_irq,
+    .set_active_state    = gicv3_set_active_state,
     .set_irq_type        = gicv3_set_irq_type,
     .set_irq_priority    = gicv3_set_irq_priority,
     .send_SGI            = gicv3_send_sgi,
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 89873c1df4..dfc2108c4d 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -92,6 +92,11 @@ void gic_restore_state(struct vcpu *v)
     isb();
 }
 
+void gic_set_active_state(int irq, bool state)
+{
+    gic_hw_ops->set_active_state(irq, state);
+}
+
 /* desc->irq needs to be disabled before calling this function */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type)
 {
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index c4c68c7770..d330860580 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -238,6 +238,9 @@ DECLARE_PER_CPU(uint64_t, lr_mask);
 extern enum gic_version gic_hw_version(void);
 extern int gic_get_nr_lrs(void);
 
+/* Force the state of an IRQ to active. */
+void gic_set_active_state(int irq, bool state);
+
 /* Program the IRQ type into the GIC */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
 
@@ -347,6 +350,8 @@ struct gic_hw_operations {
     void (*deactivate_irq)(struct irq_desc *irqd);
     /* Read IRQ id and Ack */
     unsigned int (*read_irq)(void);
+    /* Force the state of an IRQ to active */
+    void (*set_active_state)(int irq, bool state);
     /* Set IRQ type */
     void (*set_irq_type)(struct irq_desc *desc, unsigned int type);
     /* Set IRQ priority */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 16/49] ARM: GIC: allow reading pending state of a hardware IRQ

[Andre Przywara]

To synchronize level triggered interrupts which are mapped into a guest,
we need to update the virtual line level at certain points in time.
For a hardware mapped interrupt the GIC is the only place where we can
easily access this information.
Implement a gic_hw_operations member to return the pending state of a
particular interrupt. Due to hardware limitations this only works for
private interrupts of the current CPU, so there is not CPU field in the
prototype.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/gic-v2.c     |  6 ++++++
 xen/arch/arm/gic-v3.c     | 13 +++++++++++++
 xen/arch/arm/gic.c        |  5 +++++
 xen/include/asm-arm/gic.h |  5 +++++
 4 files changed, 29 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 5339f69fbc..30081640ac 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -514,6 +514,11 @@ static unsigned int gicv2_read_apr(int apr_reg)
    return readl_gich(GICH_APR);
 }
 
+bool gicv2_read_pending_state(int irq)
+{
+    return readl_gicd(GICD_ISPENDR + (irq / 32) * 4) & (1U << (irq % 32));
+}
+
 static void gicv2_irq_enable(struct irq_desc *desc)
 {
     unsigned long flags;
@@ -1261,6 +1266,7 @@ const static struct gic_hw_operations gicv2_ops = {
     .write_lr            = gicv2_write_lr,
     .read_vmcr_priority  = gicv2_read_vmcr_priority,
     .read_apr            = gicv2_read_apr,
+    .read_pending_state  = gicv2_read_pending_state,
     .make_hwdom_dt_node  = gicv2_make_hwdom_dt_node,
     .make_hwdom_madt     = gicv2_make_hwdom_madt,
     .get_hwdom_extra_madt_size = gicv2_get_hwdom_extra_madt_size,
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 595eaef43a..2cbfeb8e03 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1081,6 +1081,18 @@ static unsigned int gicv3_read_apr(int apr_reg)
     }
 }
 
+static bool gicv3_read_pending_state(int irq)
+{
+    void __iomem *base;
+
+    if ( irq >= NR_GIC_LOCAL_IRQS)
+        base = GICD + (irq / 32) * 4;
+    else
+        base = GICD_RDIST_SGI_BASE;
+
+    return readl(base + GICD_ISPENDR) & (1U << (irq % 32));
+}
+
 static void gicv3_irq_enable(struct irq_desc *desc)
 {
     unsigned long flags;
@@ -1749,6 +1761,7 @@ static const struct gic_hw_operations gicv3_ops = {
     .write_lr            = gicv3_write_lr,
     .read_vmcr_priority  = gicv3_read_vmcr_priority,
     .read_apr            = gicv3_read_apr,
+    .read_pending_state  = gicv3_read_pending_state,
     .secondary_init      = gicv3_secondary_cpu_init,
     .make_hwdom_dt_node  = gicv3_make_hwdom_dt_node,
     .make_hwdom_madt     = gicv3_make_hwdom_madt,
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index dfc2108c4d..ce9ab2367e 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -116,6 +116,11 @@ static void gic_set_irq_priority(struct irq_desc *desc, unsigned int priority)
     gic_hw_ops->set_irq_priority(desc, priority);
 }
 
+bool gic_read_pending_state(int irq)
+{
+    return gic_hw_ops->read_pending_state(irq);
+}
+
 /* Program the GIC to route an interrupt to the host (i.e. Xen)
  * - needs to be called with desc.lock held
  */
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index d330860580..d7fd18fd47 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -244,6 +244,9 @@ void gic_set_active_state(int irq, bool state);
 /* Program the IRQ type into the GIC */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
 
+/* Read the pending state of an interrupt from the distributor. */
+bool gic_read_pending_state(int irq);
+
 /* Program the GIC to route an interrupt */
 extern void gic_route_irq_to_xen(struct irq_desc *desc, unsigned int priority);
 extern int gic_route_irq_to_guest(struct domain *, unsigned int virq,
@@ -376,6 +379,8 @@ struct gic_hw_operations {
     unsigned int (*read_vmcr_priority)(void);
     /* Read APRn register */
     unsigned int (*read_apr)(int apr_reg);
+    /* Query the pending state of an interrupt at the distributor level. */
+    bool (*read_pending_state)(int irq);
     /* Secondary CPU init */
     int (*secondary_init)(void);
     /* Create GIC node for the hardware domain */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 17/49] ARM: timer: Handle level triggered IRQs correctly

[Andre Przywara]

The ARM Generic Timer uses a level-sensitive interrupt semantic. We
easily catch when the line goes high, as this triggers the hardware IRQ.
However we have to sync the state of the interrupt condition at certain
points to catch when the line goes low and we can remove the vtimer vIRQ
from the vGIC (and the LR).
The VGIC in Xen so far only implemented edge triggered vIRQs, really, so
we need to add new functionality to re-sample the interrupt state.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/time.c     | 34 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/traps.c    |  1 +
 xen/include/xen/timer.h |  2 ++
 3 files changed, 37 insertions(+)

diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index c11fcfeadd..98ebb4305d 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -263,6 +263,40 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
     vgic_inject_irq(current->domain, current, current->arch.virt_timer.irq, true);
 }
 
+/**
+ * vtimer_sync() - update the state of the virtual timer after a guest run
+ * @vcpu: The VCPU to sync the arch timer state
+ *
+ * After returning from a guest, update the state of the virtual interrupt
+ * line, to model the level triggered interrupt correctly.
+ * If the guest has handled a timer interrupt, the virtual interrupt line
+ * needs to be lowered explicitly. vgic_inject_irq() takes care of that.
+ */
+void vtimer_sync(struct vcpu *vcpu)
+{
+    struct vtimer *vtimer = &vcpu->arch.virt_timer;
+    bool level;
+
+    vtimer->ctl = READ_SYSREG32(CNTV_CTL_EL0);
+    vtimer->cval = READ_SYSREG64(CNTV_CVAL_EL0);
+
+    /*
+     * Technically we should mask with 0x7 here, to catch if the timer
+     * interrupt is masked. However Xen always masks the timer upon entering
+     * the hypervisor, leaving it up to the guest to un-mask it.
+     * So we would always read a "low" level, despite the condition being
+     * actually "high". Igoring the mask bit solves this (for now).
+     * Another possible check would be to compare the value of CNTVCT_EL0
+     * against vtimer->cval and derive the interrupt state from that.
+     *
+     * TODO: The proper fix for this is to make vtimer vIRQ hardware mapped,
+     * but this requires reworking the arch timer to implement this.
+     */
+    level = (vtimer->ctl & 0x5) == (CNTx_CTL_ENABLE | CNTx_CTL_PENDING);
+
+    vgic_inject_irq(vcpu->domain, vcpu, vtimer->irq, level);
+}
+
 /*
  * Arch timer interrupt really ought to be level triggered, since the
  * design of the timer/comparator mechanism is based around that
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 1cba7e584d..2d770a14a5 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2024,6 +2024,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
         if ( current->arch.hcr_el2 & HCR_VA )
             current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
 
+        vtimer_sync(current);
         gic_clear_lrs(current);
     }
 }
diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
index 4513260b0d..eddbbf3903 100644
--- a/xen/include/xen/timer.h
+++ b/xen/include/xen/timer.h
@@ -94,6 +94,8 @@ DECLARE_PER_CPU(s_time_t, timer_deadline);
 /* Arch-defined function to reprogram timer hardware for new deadline. */
 int reprogram_timer(s_time_t timeout);
 
+void vtimer_sync(struct vcpu *vcpu);
+
 /* Calculate the aligned first tick time for a given periodic timer. */
 s_time_t align_timer(s_time_t firsttick, uint64_t period);
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 18/49] ARM: evtchn: Handle level triggered IRQs correctly

[Andre Przywara]

The event channel IRQ has level triggered semantics, however the current
VGIC treats everything as edge triggered.
To correctly process those IRQs, we have to lower the (virtual) IRQ line
at some point in time, depending on whether ther interrupt condition
still prevails.
Check the per-VCPU evtchn_upcall_pending variable to make the interrupt
line match its status, and call this function upon every hypervisor
entry.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/domain.c       | 7 +++++++
 xen/arch/arm/traps.c        | 1 +
 xen/include/asm-arm/event.h | 1 +
 3 files changed, 9 insertions(+)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index e76cfdfe83..87bd493924 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -955,6 +955,13 @@ void vcpu_mark_events_pending(struct vcpu *v)
     vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
 }
 
+void vcpu_update_evtchn_irq(struct vcpu *v)
+{
+    bool pending = vcpu_info(v, evtchn_upcall_pending);
+
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, pending);
+}
+
 /* The ARM spec declares that even if local irqs are masked in
  * the CPSR register, an irq should wake up a cpu from WFI anyway.
  * For this reason we need to check for irqs that need delivery,
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 2d770a14a5..f57ef2141a 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2025,6 +2025,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
             current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
 
         vtimer_sync(current);
+        vcpu_update_evtchn_irq(current);
         gic_clear_lrs(current);
     }
 }
diff --git a/xen/include/asm-arm/event.h b/xen/include/asm-arm/event.h
index e8c2a6cb44..87ef76e3d5 100644
--- a/xen/include/asm-arm/event.h
+++ b/xen/include/asm-arm/event.h
@@ -6,6 +6,7 @@
 
 void vcpu_kick(struct vcpu *v);
 void vcpu_mark_events_pending(struct vcpu *v);
+void vcpu_update_evtchn_irq(struct vcpu *v);
 void vcpu_block_unless_event_pending(struct vcpu *v);
 
 static inline int vcpu_event_delivery_is_enabled(struct vcpu *v)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 19/49] ARM: vPL011: Use the VGIC's level triggered IRQs handling if available

[Andre Przywara]

The emulated ARM SBSA UART is using level triggered IRQ semantics,
however the current VGIC can only handle edge triggered IRQs, really.
Disable the existing workaround for this problem in case we have the
new VGIC in place, which can properly handle level triggered IRQs.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vpl011.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 5dcf4bec18..197ece8873 100644
--- a/xen/arch/arm/vpl011.c
+++ b/xen/arch/arm/vpl011.c
@@ -54,6 +54,7 @@ static void vpl011_update_interrupt_status(struct domain *d)
      */
     ASSERT(spin_is_locked(&vpl011->lock));
 
+#ifndef CONFIG_NEW_VGIC
     /*
      * TODO: PL011 interrupts are level triggered which means
      * that interrupt needs to be set/clear instead of being
@@ -71,6 +72,9 @@ static void vpl011_update_interrupt_status(struct domain *d)
         vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, true);
 
     vpl011->shadow_uartmis = uartmis;
+#else
+    vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, !!uartmis);
+#endif
 }
 
 static uint8_t vpl011_read_data(struct domain *d)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 20/49] ARM: new VGIC: Add data structure definitions

[Andre Przywara]

Add a new header file for the new and improved GIC implementation.
The big change is that we now have a struct vgic_irq per IRQ instead
of spreading all the information over various bitmaps in the ranks.

We include this new header conditionally from within the old header
file for the time being to avoid touching all the users.

This is based on Linux commit b18b57787f5e, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/include/asm-arm/arm_vgic.h | 269 +++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/domain.h   |   4 +
 xen/include/asm-arm/vgic.h     |   6 +
 3 files changed, 279 insertions(+)
 create mode 100644 xen/include/asm-arm/arm_vgic.h

diff --git a/xen/include/asm-arm/arm_vgic.h b/xen/include/asm-arm/arm_vgic.h
new file mode 100644
index 0000000000..865e9ee5bc
--- /dev/null
+++ b/xen/include/asm-arm/arm_vgic.h
@@ -0,0 +1,269 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __KVM_ARM_VGIC_H
+#define __KVM_ARM_VGIC_H
+
+#include <asm/atomic.h>
+#include <asm/mmio.h>
+#include <xen/spinlock.h>
+#include <xen/list.h>
+
+#define VGIC_V3_MAX_CPUS        255
+#define VGIC_V2_MAX_CPUS        8
+#define VGIC_NR_IRQS_LEGACY     256
+#define VGIC_NR_SGIS            16
+#define VGIC_NR_PPIS            16
+#define VGIC_NR_PRIVATE_IRQS    (VGIC_NR_SGIS + VGIC_NR_PPIS)
+#define VGIC_MAX_PRIVATE        (VGIC_NR_PRIVATE_IRQS - 1)
+#define VGIC_MAX_SPI            1019
+#define VGIC_MAX_RESERVED       1023
+#define VGIC_MIN_LPI            8192
+
+#define irq_is_ppi(irq) ((irq) >= VGIC_NR_SGIS && (irq) < VGIC_NR_PRIVATE_IRQS)
+#define irq_is_spi(irq) ((irq) >= VGIC_NR_PRIVATE_IRQS && \
+                         (irq) <= VGIC_MAX_SPI)
+
+enum vgic_type {
+    VGIC_V2,        /* Good ol' GICv2 */
+    VGIC_V3,        /* New fancy GICv3 */
+};
+
+#define VGIC_V2_MAX_LRS         (1 << 6)
+#define VGIC_V3_MAX_LRS         16
+#define VGIC_V3_LR_INDEX(lr)    (VGIC_V3_MAX_LRS - 1 - lr)
+
+enum vgic_irq_config {
+    VGIC_CONFIG_EDGE = 0,
+    VGIC_CONFIG_LEVEL
+};
+
+struct vgic_irq {
+    spinlock_t irq_lock;        /* Protects the content of the struct */
+    struct list_head lpi_list;  /* Used to link all LPIs together */
+    struct list_head ap_list;
+
+    struct vcpu *vcpu;          /*
+                                 * SGIs and PPIs: The VCPU
+                                 * SPIs and LPIs: The VCPU whose ap_list
+                                 * this is queued on.
+                                 */
+
+    struct vcpu *target_vcpu;   /*
+                                 * The VCPU that this interrupt should
+                                 * be sent to, as a result of the
+                                 * targets reg (v2) or the affinity reg (v3).
+                                 */
+
+    u32 intid;                  /* Guest visible INTID */
+    bool line_level;            /* Level only */
+    bool pending_latch;         /*
+                                 * The pending latch state used to
+                                 * calculate the pending state for both
+                                 * level and edge triggered IRQs.
+                                 */
+    bool active;                /* not used for LPIs */
+    bool enabled;
+    bool hw;                    /* Tied to HW IRQ */
+    atomic_t refcount;          /* Used for LPIs */
+    u32 hwintid;                /* HW INTID number */
+    union
+    {
+        u8 targets;             /* GICv2 target VCPUs mask */
+        u32 mpidr;              /* GICv3 target VCPU */
+    };
+    u8 source;                  /* GICv2 SGIs only */
+    u8 priority;
+    enum vgic_irq_config config;    /* Level or edge */
+};
+
+struct vgic_register_region;
+struct vgic_its;
+
+enum iodev_type {
+    IODEV_CPUIF,
+    IODEV_DIST,
+    IODEV_REDIST,
+    IODEV_ITS
+};
+
+struct vgic_io_device {
+    paddr_t base_addr;
+    union
+    {
+        struct vcpu *redist_vcpu;
+        struct vgic_its *its;
+    };
+    const struct vgic_register_region *regions;
+    enum iodev_type iodev_type;
+    int nr_regions;
+};
+
+struct vgic_its {
+    /* The base address of the ITS control register frame */
+    paddr_t     vgic_its_base;
+
+    bool        enabled;
+    struct vgic_io_device       iodev;
+
+    /* These registers correspond to GITS_BASER{0,1} */
+    u64         baser_device_table;
+    u64         baser_coll_table;
+
+    /* Protects the command queue */
+    spinlock_t  cmd_lock;
+    u64         cbaser;
+    u32         creadr;
+    u32         cwriter;
+
+    /* migration ABI revision in use */
+    u32         abi_rev;
+
+    /* Protects the device and collection lists */
+    spinlock_t  its_lock;
+    struct list_head            device_list;
+    struct list_head            collection_list;
+};
+
+struct vgic_dist {
+    bool                ready;
+    bool                initialized;
+
+    /* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
+    u32                 version;
+
+    /* Do injected MSIs require an additional device ID? */
+    bool                msis_require_devid;
+
+    int                 nr_spis;
+
+    /* base addresses in guest physical address space: */
+    paddr_t             vgic_dist_base;     /* distributor */
+    union
+    {
+        /* either a GICv2 CPU interface */
+        paddr_t         vgic_cpu_base;
+        /* or a number of GICv3 redistributor regions */
+        struct
+        {
+            paddr_t     vgic_redist_base;
+            paddr_t     vgic_redist_free_offset;
+        };
+    };
+
+    /* distributor enabled */
+    bool                enabled;
+
+    struct vgic_irq     *spis;
+    unsigned long       *allocated_irqs; /* bitmap of IRQs allocated */
+
+    struct vgic_io_device   dist_iodev;
+
+    bool                has_its;
+
+    /*
+     * Contains the attributes and gpa of the LPI configuration table.
+     * Since we report GICR_TYPER.CommonLPIAff as 0b00, we can share
+     * one address across all redistributors.
+     * GICv3 spec: 6.1.2 "LPI Configuration tables"
+     */
+    u64                 propbaser;
+
+    /* Protects the lpi_list and the count value below. */
+    spinlock_t          lpi_list_lock;
+    struct list_head    lpi_list_head;
+    int                 lpi_list_count;
+};
+
+struct vgic_v2_cpu_if {
+    u32     vgic_hcr;
+    u32     vgic_vmcr;
+    u64     vgic_elrsr; /* Saved only */
+    u32     vgic_apr;
+    u32     vgic_lr[VGIC_V2_MAX_LRS];
+};
+
+struct vgic_v3_cpu_if {
+    u32     vgic_hcr;
+    u32     vgic_vmcr;
+    u32     vgic_sre;   /* Restored only, change ignored */
+    u32     vgic_elrsr; /* Saved only */
+    u32     vgic_ap0r[4];
+    u32     vgic_ap1r[4];
+    u64     vgic_lr[VGIC_V3_MAX_LRS];
+};
+
+struct vgic_cpu {
+    /* CPU vif control registers for world switch */
+    union
+    {
+        struct vgic_v2_cpu_if   vgic_v2;
+        struct vgic_v3_cpu_if   vgic_v3;
+    };
+
+    unsigned int used_lrs;
+    struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS];
+
+    spinlock_t ap_list_lock;    /* Protects the ap_list */
+
+    /*
+     * List of IRQs that this VCPU should consider because they are either
+     * Active or Pending (hence the name; AP list), or because they recently
+     * were one of the two and need to be migrated off this list to another
+     * VCPU.
+     */
+    struct list_head ap_list_head;
+
+    /*
+     * Members below are used with GICv3 emulation only and represent
+     * parts of the redistributor.
+     */
+    struct vgic_io_device   rd_iodev;
+    struct vgic_io_device   sgi_iodev;
+
+    /* Contains the attributes and gpa of the LPI pending tables. */
+    u64 pendbaser;
+
+    bool lpis_enabled;
+
+    /* Cache guest priority bits */
+    u32 num_pri_bits;
+
+    /* Cache guest interrupt ID bits */
+    u32 num_id_bits;
+};
+
+extern struct static_key_false vgic_v2_cpuif_trap;
+extern struct static_key_false vgic_v3_cpuif_trap;
+
+#define vgic_initialized(k) ((k)->arch.vgic.initialized)
+#define vgic_ready(k)       ((k)->arch.vgic.ready)
+#define vgic_valid_spi(k, i)    (((i) >= VGIC_NR_PRIVATE_IRQS) && \
+            ((i) < (k)->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS))
+
+bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
+bool vgic_v3_emulate(struct cpu_user_regs *regs, union hsr hsr);
+
+void vgic_clear_pending_irqs(struct vcpu *v);
+
+void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
+                      paddr_t vbase, void __iomem *hbase,
+                      uint32_t aliased_offset);
+struct rdist_region;
+void vgic_v3_setup_hw(paddr_t dbase, unsigned int nr_rdist_regions,
+                      const struct rdist_region *regions,
+                      unsigned int intid_bits);
+
+#endif /* __KVM_ARM_VGIC_H */
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 968ffb0c81..e8f2917140 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -198,7 +198,11 @@ struct arch_vcpu
     union gic_state_data gic;
     uint64_t lr_mask;
 
+#ifdef CONFIG_NEW_VGIC
+    struct vgic_cpu vgic_cpu;
+#else
     struct vgic_cpu vgic;
+#endif
 
     /* Timer registers  */
     uint32_t cntkctl;
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 85ad2aca79..96b99f5c85 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -18,6 +18,10 @@
 #ifndef __ASM_ARM_VGIC_H__
 #define __ASM_ARM_VGIC_H__
 
+#ifdef CONFIG_NEW_VGIC
+#include <asm/arm_vgic.h>
+#else
+
 #include <xen/bitops.h>
 #include <xen/radix-tree.h>
 #include <xen/rbtree.h>
@@ -313,6 +317,8 @@ void vgic_v3_setup_hw(paddr_t dbase,
                       unsigned int intid_bits);
 #endif
 
+#endif /* !CONFIG_NEW_VGIC */
+
 /*** Common VGIC functions used by Xen arch code ****/
 
 /*
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 21/49] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Andre Przywara]

The new VGIC implementation centers around a struct vgic_irq instance
per virtual IRQ.
Provide a function to retrieve the right instance for a given IRQ
number and (in case of private interrupts) the right VCPU.
This also includes the corresponding put function, which does nothing
for private interrupts and SPIs, but handles the ref-counting for LPIs.

This is based on Linux commit 64a959d66e47, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h |  32 ++++++++++++++
 2 files changed, 139 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic.c
 create mode 100644 xen/arch/arm/vgic/vgic.h

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
new file mode 100644
index 0000000000..3075091caa
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic.c
@@ -0,0 +1,107 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/bug.h>
+#include <xen/sched.h>
+
+#include <asm/arm_vgic.h>
+#include "vgic.h"
+
+/*
+ * Iterate over the VM's list of mapped LPIs to find the one with a
+ * matching interrupt ID and return a reference to the IRQ structure.
+ */
+static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+    struct vgic_irq *irq = NULL;
+
+    spin_lock(&dist->lpi_list_lock);
+
+    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
+    {
+        if ( irq->intid != intid )
+            continue;
+
+        /*
+         * This increases the refcount, the caller is expected to
+         * call vgic_put_irq() later once it's finished with the IRQ.
+         */
+        vgic_get_irq_kref(irq);
+        goto out_unlock;
+    }
+    irq = NULL;
+
+out_unlock:
+    spin_unlock(&dist->lpi_list_lock);
+
+    return irq;
+}
+
+/*
+ * This looks up the virtual interrupt ID to get the corresponding
+ * struct vgic_irq. It also increases the refcount, so any caller is expected
+ * to call vgic_put_irq() once it's finished with this IRQ.
+ */
+struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
+                  u32 intid)
+{
+    /* SGIs and PPIs */
+    if ( intid <= VGIC_MAX_PRIVATE )
+        return &vcpu->arch.vgic_cpu.private_irqs[intid];
+
+    /* SPIs */
+    if ( intid <= VGIC_MAX_SPI )
+        return &d->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS];
+
+    /* LPIs */
+    if ( intid >= VGIC_MIN_LPI )
+        return vgic_get_lpi(d, intid);
+
+    WARN();
+    return NULL;
+}
+
+void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+
+    if ( irq->intid < VGIC_MIN_LPI )
+        return;
+
+    spin_lock(&dist->lpi_list_lock);
+    if ( !atomic_dec_and_test(&irq->refcount) )
+    {
+        spin_unlock(&dist->lpi_list_lock);
+        return;
+    };
+
+    list_del(&irq->lpi_list);
+    dist->lpi_list_count--;
+    spin_unlock(&dist->lpi_list_lock);
+
+    xfree(irq);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
new file mode 100644
index 0000000000..7a15cfdd79
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic.h
@@ -0,0 +1,32 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __XEN_ARM_VGIC_NEW_H__
+#define __XEN_ARM_VGIC_NEW_H__
+
+struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
+                              u32 intid);
+void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
+
+static inline void vgic_get_irq_kref(struct vgic_irq *irq)
+{
+    if ( irq->intid < VGIC_MIN_LPI )
+        return;
+
+    atomic_inc(&irq->refcount);
+}
+
+#endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 22/49] ARM: new VGIC: Implement virtual IRQ injection

[Andre Przywara]

Provide a vgic_queue_irq_unlock() function which decides whether a
given IRQ needs to be queued to a VCPU's ap_list.
This should be called whenever an IRQ becomes pending or enabled,
either as a result of a hardware IRQ injection, from devices emulated by
Xen (like the architected timer) or from MMIO accesses to the distributor
emulation.
Also provides the necessary functions to allow to inject an IRQ to a guest.
Since this is the first code that starts using our locking mechanism,
we add some (hopefully) clear documentation of our locking strategy and
requirements along with this patch.

This is based on Linux commit 81eeb95ddbab, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 224 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h |  10 +++
 2 files changed, 234 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 3075091caa..f517df6d00 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -21,6 +21,32 @@
 #include <asm/arm_vgic.h>
 #include "vgic.h"
 
+/*
+ * Locking order is always:
+ * kvm->lock (mutex)
+ *   its->cmd_lock (mutex)
+ *     its->its_lock (mutex)
+ *       vgic_cpu->ap_list_lock
+ *         kvm->lpi_list_lock
+ *           vgic_irq->irq_lock
+ *
+ * If you need to take multiple locks, always take the upper lock first,
+ * then the lower ones, e.g. first take the its_lock, then the irq_lock.
+ * If you are already holding a lock and need to take a higher one, you
+ * have to drop the lower ranking lock first and re-aquire it after having
+ * taken the upper one.
+ *
+ * When taking more than one ap_list_lock at the same time, always take the
+ * lowest numbered VCPU's ap_list_lock first, so:
+ *   vcpuX->vcpu_id < vcpuY->vcpu_id:
+ *     spin_lock(vcpuX->arch.vgic_cpu.ap_list_lock);
+ *     spin_lock(vcpuY->arch.vgic_cpu.ap_list_lock);
+ *
+ * Since the VGIC must support injecting virtual interrupts from ISRs, we have
+ * to use the spin_lock_irqsave/spin_unlock_irqrestore versions of outer
+ * spinlocks for any lock that may be taken while injecting an interrupt.
+ */
+
 /*
  * Iterate over the VM's list of mapped LPIs to find the one with a
  * matching interrupt ID and return a reference to the IRQ structure.
@@ -97,6 +123,204 @@ void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
     xfree(irq);
 }
 
+/**
+ * vgic_target_oracle - compute the target vcpu for an irq
+ *
+ * @irq:    The irq to route. Must be already locked.
+ *
+ * Based on the current state of the interrupt (enabled, pending,
+ * active, vcpu and target_vcpu), compute the next vcpu this should be
+ * given to. Return NULL if this shouldn't be injected at all.
+ *
+ * Requires the IRQ lock to be held.
+ */
+static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
+{
+    ASSERT(spin_is_locked(&irq->irq_lock));
+
+    /* If the interrupt is active, it must stay on the current vcpu */
+    if ( irq->active )
+        return irq->vcpu ? : irq->target_vcpu;
+
+    /*
+     * If the IRQ is not active but enabled and pending, we should direct
+     * it to its configured target VCPU.
+     * If the distributor is disabled, pending interrupts shouldn't be
+     * forwarded.
+     */
+    if ( irq->enabled && irq_is_pending(irq) )
+    {
+        if ( unlikely(irq->target_vcpu &&
+                 !irq->target_vcpu->domain->arch.vgic.enabled) )
+            return NULL;
+
+        return irq->target_vcpu;
+    }
+
+    /* If neither active nor pending and enabled, then this IRQ should not
+     * be queued to any VCPU.
+     */
+    return NULL;
+}
+
+/*
+ * Only valid injection if changing level for level-triggered IRQs or for a
+ * rising edge.
+ */
+static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
+{
+    switch (irq->config)
+    {
+    case VGIC_CONFIG_LEVEL:
+        return irq->line_level != level;
+    case VGIC_CONFIG_EDGE:
+        return level;
+    }
+
+    return false;
+}
+
+/*
+ * Check whether an IRQ needs to (and can) be queued to a VCPU's ap list.
+ * Do the queuing if necessary, taking the right locks in the right order.
+ * Returns true when the IRQ was queued, false otherwise.
+ *
+ * Needs to be entered with the IRQ lock already held, but will return
+ * with all locks dropped.
+ */
+bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
+               unsigned long flags)
+{
+    struct vcpu *vcpu;
+    bool running;
+
+    ASSERT(spin_is_locked(&irq->irq_lock));
+
+retry:
+    vcpu = vgic_target_oracle(irq);
+    if ( irq->vcpu || !vcpu )
+    {
+        /*
+         * If this IRQ is already on a VCPU's ap_list, then it
+         * cannot be moved or modified and there is no more work for
+         * us to do.
+         *
+         * Otherwise, if the irq is not pending and enabled, it does
+         * not need to be inserted into an ap_list and there is also
+         * no more work for us to do.
+         */
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        /*
+         * We have to kick the VCPU here, because we could be
+         * queueing an edge-triggered interrupt for which we
+         * get no EOI maintenance interrupt. In that case,
+         * while the IRQ is already on the VCPU's AP list, the
+         * VCPU could have EOI'ed the original interrupt and
+         * won't see this one until it exits for some other
+         * reason.
+         */
+        if ( vcpu )
+            vcpu_unblock(vcpu);
+        return false;
+    }
+
+    /*
+     * We must unlock the irq lock to take the ap_list_lock where
+     * we are going to insert this new pending interrupt.
+     */
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+    /* someone can do stuff here, which we re-check below */
+
+    spin_lock_irqsave(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
+    spin_lock(&irq->irq_lock);
+
+    /*
+     * Did something change behind our backs?
+     *
+     * There are two cases:
+     * 1) The irq lost its pending state or was disabled behind our
+     *    backs and/or it was queued to another VCPU's ap_list.
+     * 2) Someone changed the affinity on this irq behind our
+     *    backs and we are now holding the wrong ap_list_lock.
+     *
+     * In both cases, drop the locks and retry.
+     */
+
+    if ( unlikely(irq->vcpu || vcpu != vgic_target_oracle(irq)) )
+    {
+        spin_unlock(&irq->irq_lock);
+        spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        goto retry;
+    }
+
+    /*
+     * Grab a reference to the irq to reflect the fact that it is
+     * now in the ap_list.
+     */
+    vgic_get_irq_kref(irq);
+    list_add_tail(&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head);
+    irq->vcpu = vcpu;
+
+    spin_unlock(&irq->irq_lock);
+    spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
+
+    running = vcpu->is_running;
+    vcpu_unblock(vcpu);
+    if ( running && vcpu != current )
+        smp_send_event_check_mask(cpumask_of(vcpu->processor));
+
+    return true;
+}
+
+/**
+ * vgic_inject_irq - Inject an IRQ from a device to the vgic
+ * @d:       The domain pointer
+ * @vcpu:    The vCPU for PPIs
+ * @intid:   The INTID to inject a new state to.
+ * @level:   Edge-triggered:  true:  to trigger the interrupt
+ *                false: to ignore the call
+ *       Level-sensitive  true:  raise the input signal
+ *                false: lower the input signal
+ *
+ * The VGIC is not concerned with devices being active-LOW or active-HIGH for
+ * level-sensitive interrupts.  You can think of the level parameter as 1
+ * being HIGH and 0 being LOW and all devices being active-HIGH.
+ */
+int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int intid,
+            bool level)
+{
+    struct vgic_irq *irq;
+    unsigned long flags;
+
+    irq = vgic_get_irq(d, vcpu, intid);
+    if ( !irq )
+        return -EINVAL;
+
+    spin_lock_irqsave(&irq->irq_lock, flags);
+
+    if ( !vgic_validate_injection(irq, level) )
+    {
+        /* Nothing to see here, move along... */
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(d, irq);
+        return 0;
+    }
+
+    if ( irq->config == VGIC_CONFIG_LEVEL )
+        irq->line_level = level;
+    else
+        irq->pending_latch = true;
+
+    vgic_queue_irq_unlock(d, irq, flags);
+    vgic_put_irq(d, irq);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 7a15cfdd79..5127739f0f 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -17,9 +17,19 @@
 #ifndef __XEN_ARM_VGIC_NEW_H__
 #define __XEN_ARM_VGIC_NEW_H__
 
+static inline bool irq_is_pending(struct vgic_irq *irq)
+{
+    if ( irq->config == VGIC_CONFIG_EDGE )
+        return irq->pending_latch;
+    else
+        return irq->pending_latch || irq->line_level;
+}
+
 struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
                               u32 intid);
 void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
+bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
+               unsigned long flags);
 
 static inline void vgic_get_irq_kref(struct vgic_irq *irq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 23/49] ARM: new VGIC: Add IRQ sorting

[Andre Przywara]

Adds the sorting function to cover the case where you have more IRQs
to consider than you have LRs. We consider their priorities.
This pulls in Linux' list_sort.c , which is a merge sort implementation
for linked lists.

This is based on Linux commit 8e4447457965, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c    |  59 +++++++++++++++
 xen/common/list_sort.c      | 170 ++++++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/list_sort.h |  11 +++
 3 files changed, 240 insertions(+)
 create mode 100644 xen/common/list_sort.c
 create mode 100644 xen/include/xen/list_sort.h

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index f517df6d00..a4efd1fd03 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -16,6 +16,7 @@
  */
 
 #include <asm/bug.h>
+#include <xen/list_sort.h>
 #include <xen/sched.h>
 
 #include <asm/arm_vgic.h>
@@ -163,6 +164,64 @@ static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
     return NULL;
 }
 
+/*
+ * The order of items in the ap_lists defines how we'll pack things in LRs as
+ * well, the first items in the list being the first things populated in the
+ * LRs.
+ *
+ * A hard rule is that active interrupts can never be pushed out of the LRs
+ * (and therefore take priority) since we cannot reliably trap on deactivation
+ * of IRQs and therefore they have to be present in the LRs.
+ *
+ * Otherwise things should be sorted by the priority field and the GIC
+ * hardware support will take care of preemption of priority groups etc.
+ *
+ * Return negative if "a" sorts before "b", 0 to preserve order, and positive
+ * to sort "b" before "a".
+ */
+static int vgic_irq_cmp(void *priv, struct list_head *a, struct list_head *b)
+{
+    struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list);
+    struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list);
+    bool penda, pendb;
+    int ret;
+
+    spin_lock(&irqa->irq_lock);
+    spin_lock(&irqb->irq_lock);
+
+    if ( irqa->active || irqb->active )
+    {
+        ret = (int)irqb->active - (int)irqa->active;
+        goto out;
+    }
+
+    penda = irqa->enabled && irq_is_pending(irqa);
+    pendb = irqb->enabled && irq_is_pending(irqb);
+
+    if ( !penda || !pendb )
+    {
+        ret = (int)pendb - (int)penda;
+        goto out;
+    }
+
+    /* Both pending and enabled, sort by priority */
+    ret = irqa->priority - irqb->priority;
+out:
+    spin_unlock(&irqb->irq_lock);
+    spin_unlock(&irqa->irq_lock);
+    return ret;
+}
+
+/* Must be called with the ap_list_lock held */
+static void vgic_sort_ap_list(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+
+    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
+
+    list_sort(NULL, &vgic_cpu->ap_list_head, vgic_irq_cmp);
+}
+
 /*
  * Only valid injection if changing level for level-triggered IRQs or for a
  * rising edge.
diff --git a/xen/common/list_sort.c b/xen/common/list_sort.c
new file mode 100644
index 0000000000..9c5cc58e43
--- /dev/null
+++ b/xen/common/list_sort.c
@@ -0,0 +1,170 @@
+/*
+ * list_sort.c: merge sort implementation for linked lists
+ * Copied from the Linux kernel (lib/list_sort.c)
+ * (without specific copyright notice there)
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+#include <xen/lib.h>
+#include <xen/list.h>
+
+#define MAX_LIST_LENGTH_BITS 20
+
+/*
+ * Returns a list organized in an intermediate format suited
+ * to chaining of merge() calls: null-terminated, no reserved or
+ * sentinel head node, "prev" links not maintained.
+ */
+static struct list_head *merge(void *priv,
+                               int (*cmp)(void *priv, struct list_head *a,
+                                          struct list_head *b),
+                               struct list_head *a, struct list_head *b)
+{
+    struct list_head head, *tail = &head;
+
+    while ( a && b )
+    {
+        /* if equal, take 'a' -- important for sort stability */
+        if ( (*cmp)(priv, a, b) <= 0 )
+        {
+            tail->next = a;
+            a = a->next;
+        }
+        else
+        {
+            tail->next = b;
+            b = b->next;
+        }
+        tail = tail->next;
+    }
+    tail->next = a?:b;
+    return head.next;
+}
+
+/*
+ * Combine final list merge with restoration of standard doubly-linked
+ * list structure.  This approach duplicates code from merge(), but
+ * runs faster than the tidier alternatives of either a separate final
+ * prev-link restoration pass, or maintaining the prev links
+ * throughout.
+ */
+static void merge_and_restore_back_links(void *priv,
+                                         int (*cmp)(void *priv,
+                                                    struct list_head *a,
+                                                    struct list_head *b),
+                                         struct list_head *head,
+                                         struct list_head *a,
+                                         struct list_head *b)
+{
+    struct list_head *tail = head;
+    u8 count = 0;
+
+    while ( a && b )
+    {
+        /* if equal, take 'a' -- important for sort stability */
+        if ( (*cmp)(priv, a, b) <= 0 )
+        {
+            tail->next = a;
+            a->prev = tail;
+            a = a->next;
+        }
+        else
+        {
+            tail->next = b;
+            b->prev = tail;
+            b = b->next;
+        }
+        tail = tail->next;
+    }
+    tail->next = a ? : b;
+
+    do
+    {
+        /*
+         * In worst cases this loop may run many iterations.
+         * Continue callbacks to the client even though no
+         * element comparison is needed, so the client's cmp()
+         * routine can invoke cond_resched() periodically.
+         */
+        if ( unlikely(!(++count)) )
+            (*cmp)(priv, tail->next, tail->next);
+
+        tail->next->prev = tail;
+        tail = tail->next;
+    } while ( tail->next );
+
+    tail->next = head;
+    head->prev = tail;
+}
+
+/**
+ * list_sort - sort a list
+ * @priv: private data, opaque to list_sort(), passed to @cmp
+ * @head: the list to sort
+ * @cmp: the elements comparison function
+ *
+ * This function implements "merge sort", which has O(nlog(n))
+ * complexity.
+ *
+ * The comparison function @cmp must return a negative value if @a
+ * should sort before @b, and a positive value if @a should sort after
+ * @b. If @a and @b are equivalent, and their original relative
+ * ordering is to be preserved, @cmp must return 0.
+ */
+void list_sort(void *priv, struct list_head *head,
+               int (*cmp)(void *priv, struct list_head *a, struct list_head *b))
+{
+    struct list_head *part[MAX_LIST_LENGTH_BITS+1]; /* sorted partial lists
+                        -- last slot is a sentinel */
+    int lev;  /* index into part[] */
+    int max_lev = 0;
+    struct list_head *list;
+
+    if ( list_empty(head) )
+        return;
+
+    memset(part, 0, sizeof(part));
+
+    head->prev->next = NULL;
+    list = head->next;
+
+    while ( list )
+    {
+        struct list_head *cur = list;
+        list = list->next;
+        cur->next = NULL;
+
+        for ( lev = 0; part[lev]; lev++ )
+        {
+            cur = merge(priv, cmp, part[lev], cur);
+            part[lev] = NULL;
+        }
+        if ( lev > max_lev )
+        {
+            if ( unlikely(lev >= ARRAY_SIZE(part)-1) )
+            {
+                dprintk(XENLOG_DEBUG, "list too long for efficiency\n");
+                lev--;
+            }
+            max_lev = lev;
+        }
+        part[lev] = cur;
+    }
+
+    for ( lev = 0; lev < max_lev; lev++ )
+        if ( part[lev] )
+            list = merge(priv, cmp, part[lev], list);
+
+    merge_and_restore_back_links(priv, cmp, head, part[max_lev], list);
+}
+EXPORT_SYMBOL(list_sort);
diff --git a/xen/include/xen/list_sort.h b/xen/include/xen/list_sort.h
new file mode 100644
index 0000000000..a60c589d4b
--- /dev/null
+++ b/xen/include/xen/list_sort.h
@@ -0,0 +1,11 @@
+#ifndef _LINUX_LIST_SORT_H
+#define _LINUX_LIST_SORT_H
+
+#include <xen/types.h>
+
+struct list_head;
+
+void list_sort(void *priv, struct list_head *head,
+               int (*cmp)(void *priv, struct list_head *a,
+                          struct list_head *b));
+#endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Andre Przywara]

Implement the framework for syncing IRQs between our emulation and the
list registers, which represent the guest's view of IRQs.
This is done in kvm_vgic_flush_hwstate and kvm_vgic_sync_hwstate, which
gets called on guest entry and exit.
The code talking to the actual GICv2/v3 hardware is added in the
following patches.

This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 246 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h |   2 +
 2 files changed, 248 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index a4efd1fd03..a1f77130d4 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -380,6 +380,252 @@ int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int intid,
     return 0;
 }
 
+/**
+ * vgic_prune_ap_list - Remove non-relevant interrupts from the list
+ *
+ * @vcpu: The VCPU pointer
+ *
+ * Go over the list of "interesting" interrupts, and prune those that we
+ * won't have to consider in the near future.
+ */
+static void vgic_prune_ap_list(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+    struct vgic_irq *irq, *tmp;
+    unsigned long flags;
+
+retry:
+    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
+
+    list_for_each_entry_safe( irq, tmp, &vgic_cpu->ap_list_head, ap_list )
+    {
+        struct vcpu *target_vcpu, *vcpuA, *vcpuB;
+
+        spin_lock(&irq->irq_lock);
+
+        BUG_ON(vcpu != irq->vcpu);
+
+        target_vcpu = vgic_target_oracle(irq);
+
+        if ( !target_vcpu )
+        {
+            /*
+             * We don't need to process this interrupt any
+             * further, move it off the list.
+             */
+            list_del(&irq->ap_list);
+            irq->vcpu = NULL;
+            spin_unlock(&irq->irq_lock);
+
+            /*
+             * This vgic_put_irq call matches the
+             * vgic_get_irq_kref in vgic_queue_irq_unlock,
+             * where we added the LPI to the ap_list. As
+             * we remove the irq from the list, we drop
+             * also drop the refcount.
+             */
+            vgic_put_irq(vcpu->domain, irq);
+            continue;
+        }
+
+        if ( target_vcpu == vcpu )
+        {
+            /* We're on the right CPU */
+            spin_unlock(&irq->irq_lock);
+            continue;
+        }
+
+        /* This interrupt looks like it has to be migrated. */
+
+        spin_unlock(&irq->irq_lock);
+        spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
+
+        /*
+         * Ensure locking order by always locking the smallest
+         * ID first.
+         */
+        if ( vcpu->vcpu_id < target_vcpu->vcpu_id )
+        {
+            vcpuA = vcpu;
+            vcpuB = target_vcpu;
+        }
+        else
+        {
+            vcpuA = target_vcpu;
+            vcpuB = vcpu;
+        }
+
+        spin_lock_irqsave(&vcpuA->arch.vgic_cpu.ap_list_lock, flags);
+        spin_lock(&vcpuB->arch.vgic_cpu.ap_list_lock);
+        spin_lock(&irq->irq_lock);
+
+        /*
+         * If the affinity has been preserved, move the
+         * interrupt around. Otherwise, it means things have
+         * changed while the interrupt was unlocked, and we
+         * need to replay this.
+         *
+         * In all cases, we cannot trust the list not to have
+         * changed, so we restart from the beginning.
+         */
+        if ( target_vcpu == vgic_target_oracle(irq) )
+        {
+            struct vgic_cpu *new_cpu = &target_vcpu->arch.vgic_cpu;
+
+            list_del(&irq->ap_list);
+            irq->vcpu = target_vcpu;
+            list_add_tail(&irq->ap_list, &new_cpu->ap_list_head);
+        }
+
+        spin_unlock(&irq->irq_lock);
+        spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
+        spin_unlock_irqrestore(&vcpuA->arch.vgic_cpu.ap_list_lock, flags);
+        goto retry;
+    }
+
+    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
+}
+
+static inline void vgic_fold_lr_state(struct vcpu *vcpu)
+{
+}
+
+/* Requires the irq_lock to be held. */
+static inline void vgic_populate_lr(struct vcpu *vcpu,
+                                    struct vgic_irq *irq, int lr)
+{
+    ASSERT(spin_is_locked(&irq->irq_lock));
+}
+
+static inline void vgic_clear_lr(struct vcpu *vcpu, int lr)
+{
+}
+
+static inline void vgic_set_underflow(struct vcpu *vcpu)
+{
+}
+
+/* Requires the ap_list_lock to be held. */
+static int compute_ap_list_depth(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+    struct vgic_irq *irq;
+    int count = 0;
+
+    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
+
+    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
+    {
+        spin_lock(&irq->irq_lock);
+        /* GICv2 SGIs can count for more than one... */
+        if ( vgic_irq_is_sgi(irq->intid) && irq->source )
+            count += hweight8(irq->source);
+        else
+            count++;
+        spin_unlock(&irq->irq_lock);
+    }
+    return count;
+}
+
+/* Requires the VCPU's ap_list_lock to be held. */
+static void vgic_flush_lr_state(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+    struct vgic_irq *irq;
+    int count = 0;
+
+    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
+
+    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
+        vgic_sort_ap_list(vcpu);
+
+    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
+    {
+        spin_lock(&irq->irq_lock);
+
+        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
+            goto next;
+
+        /*
+         * If we get an SGI with multiple sources, try to get
+         * them in all at once.
+         */
+        do
+        {
+            vgic_populate_lr(vcpu, irq, count++);
+        } while ( irq->source && count < gic_get_nr_lrs() );
+
+next:
+        spin_unlock(&irq->irq_lock);
+
+        if ( count == gic_get_nr_lrs() )
+        {
+            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
+                vgic_set_underflow(vcpu);
+            break;
+        }
+    }
+
+    vcpu->arch.vgic_cpu.used_lrs = count;
+
+    /* Nuke remaining LRs */
+    for ( ; count < gic_get_nr_lrs(); count++)
+        vgic_clear_lr(vcpu, count);
+}
+
+/*
+ * gic_clear_lrs() - Update the VGIC state from hardware after a guest's run.
+ * @vcpu: the VCPU.
+ *
+ * Sync back the hardware VGIC state after the guest has run, into our
+ * VGIC emulation structures, It reads the LRs and updates the respective
+ * struct vgic_irq, taking level/edge into account.
+ * This is the high level function which takes care of the conditions,
+ * also bails out early if there were no interrupts queued.
+ * Was: kvm_vgic_sync_hwstate()
+ */
+void gic_clear_lrs(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+
+    /* An empty ap_list_head implies used_lrs == 0 */
+    if ( list_empty(&vcpu->arch.vgic_cpu.ap_list_head) )
+        return;
+
+    if ( vgic_cpu->used_lrs )
+        vgic_fold_lr_state(vcpu);
+    vgic_prune_ap_list(vcpu);
+}
+
+/*
+ * gic_inject() - flush the emulation state into the hardware on guest entry
+ *
+ * Before we enter a guest, we have to translate the virtual GIC state of a
+ * VCPU into the GIC virtualization hardware registers, namely the LRs.
+ * This is the high level function which takes care about the conditions
+ * and the locking, also bails out early if there are no interrupts queued.
+ * Was: kvm_vgic_flush_hwstate()
+ */
+void gic_inject(void)
+{
+    /*
+     * If there are no virtual interrupts active or pending for this
+     * VCPU, then there is no work to do and we can bail out without
+     * taking any lock.  There is a potential race with someone injecting
+     * interrupts to the VCPU, but it is a benign race as the VCPU will
+     * either observe the new interrupt before or after doing this check,
+     * and introducing additional synchronization mechanism doesn't change
+     * this.
+     */
+    if ( list_empty(&current->arch.vgic_cpu.ap_list_head) )
+        return;
+
+    ASSERT(!local_irq_is_enabled());
+
+    spin_lock(&current->arch.vgic_cpu.ap_list_lock);
+    vgic_flush_lr_state(current);
+    spin_unlock(&current->arch.vgic_cpu.ap_list_lock);
+}
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 5127739f0f..47fc58b81e 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -17,6 +17,8 @@
 #ifndef __XEN_ARM_VGIC_NEW_H__
 #define __XEN_ARM_VGIC_NEW_H__
 
+#define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
+
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
     if ( irq->config == VGIC_CONFIG_EDGE )
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Processing maintenance interrupts and accessing the list registers
are dependent on the host's GIC version.
Introduce vgic-v2.c to contain GICv2 specific functions.
Implement the GICv2 specific code for syncing the emulation state
into the VGIC registers.
This also adds the hook to let Xen setup the host GIC addresses.

This is based on Linux commit 140b086dd197, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-v2.c | 261 ++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.c    |  20 ++++
 xen/arch/arm/vgic/vgic.h    |   8 ++
 3 files changed, 289 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-v2.c

diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
new file mode 100644
index 0000000000..10fc467ffa
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -0,0 +1,261 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/arm_vgic.h>
+#include <asm/bug.h>
+#include <asm/io.h>
+#include <xen/sched.h>
+#include <xen/sizes.h>
+
+#include "vgic.h"
+
+#define GICH_ELRSR0                     0x30
+#define GICH_ELRSR1                     0x34
+#define GICH_LR0                        0x100
+
+#define GICH_LR_VIRTUALID               (0x3ff << 0)
+#define GICH_LR_PHYSID_CPUID_SHIFT      (10)
+#define GICH_LR_PHYSID_CPUID            (0x3ff << GICH_LR_PHYSID_CPUID_SHIFT)
+#define GICH_LR_PRIORITY_SHIFT          23
+#define GICH_LR_STATE                   (3 << 28)
+#define GICH_LR_PENDING_BIT             (1 << 28)
+#define GICH_LR_ACTIVE_BIT              (1 << 29)
+#define GICH_LR_EOI                     (1 << 19)
+#define GICH_LR_HW                      (1 << 31)
+
+static struct {
+    bool enabled;
+    paddr_t dbase;          /* Distributor interface address */
+    paddr_t cbase;          /* CPU interface address & size */
+    paddr_t csize;
+    paddr_t vbase;          /* Virtual CPU interface address */
+    void __iomem *hbase;        /* Hypervisor control interface */
+
+    /* Offset to add to get an 8kB contiguous region if GIC is aliased */
+    uint32_t aliased_offset;
+} gic_v2_hw_data;
+
+void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
+              paddr_t vbase, void __iomem *hbase,
+              uint32_t aliased_offset)
+{
+    gic_v2_hw_data.enabled = true;
+    gic_v2_hw_data.dbase = dbase;
+    gic_v2_hw_data.cbase = cbase;
+    gic_v2_hw_data.csize = csize;
+    gic_v2_hw_data.vbase = vbase;
+    gic_v2_hw_data.hbase = hbase;
+    gic_v2_hw_data.aliased_offset = aliased_offset;
+}
+
+void vgic_v2_set_underflow(struct vcpu *vcpu)
+{
+    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
+}
+
+/*
+ * transfer the content of the LRs back into the corresponding ap_list:
+ * - active bit is transferred as is
+ * - pending bit is
+ *   - transferred as is in case of edge sensitive IRQs
+ *   - set to the line-level (resample time) for level sensitive IRQs
+ */
+void vgic_v2_fold_lr_state(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+    struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2;
+    int lr;
+    unsigned long flags;
+
+    cpuif->vgic_hcr &= ~GICH_HCR_UIE;
+
+    for ( lr = 0; lr < vgic_cpu->used_lrs; lr++ )
+    {
+        u32 val = cpuif->vgic_lr[lr];
+        u32 intid = val & GICH_LR_VIRTUALID;
+        struct vgic_irq *irq;
+
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        /* Always preserve the active bit */
+        irq->active = !!(val & GICH_LR_ACTIVE_BIT);
+
+        /* Edge is the only case where we preserve the pending bit */
+        if ( irq->config == VGIC_CONFIG_EDGE && (val & GICH_LR_PENDING_BIT) )
+        {
+            irq->pending_latch = true;
+
+            if ( vgic_irq_is_sgi(intid) )
+            {
+                u32 cpuid = val & GICH_LR_PHYSID_CPUID;
+
+                cpuid >>= GICH_LR_PHYSID_CPUID_SHIFT;
+                irq->source |= (1 << cpuid);
+            }
+        }
+
+        if ( irq->hw && irq->config == VGIC_CONFIG_LEVEL &&
+            (val & GICH_LR_PENDING_BIT) )
+        {
+            irq->line_level = gic_read_pending_state(irq->hwintid);
+
+            if ( !irq->line_level )
+                            gic_set_active_state(irq->hwintid, true);
+        }
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    vgic_cpu->used_lrs = 0;
+}
+
+/*
+ * Populates the particular LR with the state of a given IRQ:
+ * - for an edge sensitive IRQ the pending state is cleared in struct vgic_irq
+ * - for a level sensitive IRQ the pending state value is unchanged;
+ *   it is dictated directly by the input level
+ *
+ * If @irq describes an SGI with multiple sources, we choose the
+ * lowest-numbered source VCPU and clear that bit in the source bitmap.
+ *
+ * The irq_lock must be held by the caller.
+ */
+void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)
+{
+    u32 val = irq->intid;
+
+    if ( irq_is_pending(irq) )
+    {
+        val |= GICH_LR_PENDING_BIT;
+
+        if ( irq->config == VGIC_CONFIG_EDGE )
+            irq->pending_latch = false;
+
+        if ( vgic_irq_is_sgi(irq->intid) )
+        {
+            u32 src = ffs(irq->source);
+
+            BUG_ON(!src);
+            val |= (src - 1) << GICH_LR_PHYSID_CPUID_SHIFT;
+            irq->source &= ~(1 << (src - 1));
+            if ( irq->source )
+                irq->pending_latch = true;
+        }
+    }
+
+    if ( irq->active )
+        val |= GICH_LR_ACTIVE_BIT;
+
+    if ( irq->hw )
+    {
+        val |= GICH_LR_HW;
+        val |= irq->hwintid << GICH_LR_PHYSID_CPUID_SHIFT;
+        /*
+         * Never set pending+active on a HW interrupt, as the
+         * pending state is kept at the physical distributor
+         * level.
+         */
+        if ( irq->active && irq_is_pending(irq) )
+            val &= ~GICH_LR_PENDING_BIT;
+    }
+    else
+    {
+        if ( irq->config == VGIC_CONFIG_LEVEL )
+            val |= GICH_LR_EOI;
+    }
+
+    /*
+     * Level-triggered mapped IRQs are special because we only observe
+     * rising edges as input to the VGIC.  We therefore lower the line
+     * level here, so that we can take new virtual IRQs.  See
+     * vgic_v2_fold_lr_state for more info.
+     */
+    if ( irq->hw && irq->config == VGIC_CONFIG_LEVEL &&
+        (val & GICH_LR_PENDING_BIT) )
+        irq->line_level = false;
+
+    /* The GICv2 LR only holds five bits of priority. */
+    val |= (irq->priority >> 3) << GICH_LR_PRIORITY_SHIFT;
+
+    vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val;
+}
+
+void vgic_v2_clear_lr(struct vcpu *vcpu, int lr)
+{
+    vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = 0;
+}
+
+static void save_lrs(struct vcpu *vcpu, void __iomem *base)
+{
+    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+    u64 elrsr;
+    int i;
+
+    elrsr = readl_relaxed(base + GICH_ELRSR0);
+    if ( unlikely(used_lrs > 32) )
+        elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
+
+    for ( i = 0; i < used_lrs; i++ )
+    {
+        if ( elrsr & (1UL << i) )
+            cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
+        else
+            cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
+
+        writel_relaxed(0, base + GICH_LR0 + (i * 4));
+    }
+}
+
+void vgic_v2_save_state(struct vcpu *vcpu)
+{
+    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+
+    if ( used_lrs )
+    {
+        save_lrs(vcpu, gic_v2_hw_data.hbase);
+        writel_relaxed(0, gic_v2_hw_data.hbase + GICH_HCR);
+    }
+}
+
+void vgic_v2_restore_state(struct vcpu *vcpu)
+{
+    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
+    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
+    int i;
+
+    if ( used_lrs )
+    {
+        writel_relaxed(cpu_if->vgic_hcr,
+                       gic_v2_hw_data.hbase + GICH_HCR);
+        for ( i = 0; i < used_lrs; i++ )
+            writel_relaxed(cpu_if->vgic_lr[i],
+                           gic_v2_hw_data.hbase + GICH_LR0 + (i * 4));
+    }
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index a1f77130d4..f4f2a04a60 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -488,6 +488,7 @@ retry:
 
 static inline void vgic_fold_lr_state(struct vcpu *vcpu)
 {
+    vgic_v2_fold_lr_state(vcpu);
 }
 
 /* Requires the irq_lock to be held. */
@@ -495,14 +496,18 @@ static inline void vgic_populate_lr(struct vcpu *vcpu,
                                     struct vgic_irq *irq, int lr)
 {
     ASSERT(spin_is_locked(&irq->irq_lock));
+
+    vgic_v2_populate_lr(vcpu, irq, lr);
 }
 
 static inline void vgic_clear_lr(struct vcpu *vcpu, int lr)
 {
+    vgic_v2_clear_lr(vcpu, lr);
 }
 
 static inline void vgic_set_underflow(struct vcpu *vcpu)
 {
+    vgic_v2_set_underflow(vcpu);
 }
 
 /* Requires the ap_list_lock to be held. */
@@ -573,6 +578,11 @@ next:
         vgic_clear_lr(vcpu, count);
 }
 
+static inline void vgic_save_state(struct vcpu *vcpu)
+{
+    vgic_v2_save_state(vcpu);
+}
+
 /*
  * gic_clear_lrs() - Update the VGIC state from hardware after a guest's run.
  * @vcpu: the VCPU.
@@ -592,11 +602,18 @@ void gic_clear_lrs(struct vcpu *vcpu)
     if ( list_empty(&vcpu->arch.vgic_cpu.ap_list_head) )
         return;
 
+    vgic_save_state(vcpu);
+
     if ( vgic_cpu->used_lrs )
         vgic_fold_lr_state(vcpu);
     vgic_prune_ap_list(vcpu);
 }
 
+static inline void vgic_restore_state(struct vcpu *vcpu)
+{
+    vgic_v2_restore_state(vcpu);
+}
+
 /*
  * gic_inject() - flush the emulation state into the hardware on guest entry
  *
@@ -625,7 +642,10 @@ void gic_inject(void)
     spin_lock(&current->arch.vgic_cpu.ap_list_lock);
     vgic_flush_lr_state(current);
     spin_unlock(&current->arch.vgic_cpu.ap_list_lock);
+
+    vgic_restore_state(current);
 }
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 47fc58b81e..771ca6f046 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -41,4 +41,12 @@ static inline void vgic_get_irq_kref(struct vgic_irq *irq)
     atomic_inc(&irq->refcount);
 }
 
+void vgic_v2_fold_lr_state(struct vcpu *vcpu);
+void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
+void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
+void vgic_v2_set_underflow(struct vcpu *vcpu);
+
+void vgic_v2_save_state(struct vcpu *vcpu);
+void vgic_v2_restore_state(struct vcpu *vcpu);
+
 #endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Andre Przywara]

Tell Xen whether a particular VCPU has an IRQ that needs handling
in the guest. This is used to decide whether a VCPU is runnable.

This is based on Linux commit 90eee56c5f90, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index f4f2a04a60..9e7fb1edcb 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -646,6 +646,38 @@ void gic_inject(void)
     vgic_restore_state(current);
 }
 
+static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+    struct vgic_irq *irq;
+    bool pending = false;
+    unsigned long flags;
+
+    if ( !vcpu->domain->arch.vgic.enabled )
+        return false;
+
+    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
+
+    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
+    {
+        spin_lock(&irq->irq_lock);
+        pending = irq_is_pending(irq) && irq->enabled;
+        spin_unlock(&irq->irq_lock);
+
+        if ( pending )
+            break;
+    }
+
+    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
+
+    return pending;
+}
+
+int gic_events_need_delivery(void)
+{
+    return vgic_vcpu_pending_irq(current);
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 27/49] ARM: new VGIC: Add MMIO handling framework

[Andre Przywara]

Add an MMIO handling framework to the VGIC emulation:
Each register is described by its offset, size (or number of bits per
IRQ, if applicable) and the read/write handler functions. We provide
initialization macros to describe each GIC register later easily.

Separate dispatch functions for read and write accesses are connected
to Xen's MMIO handling framework and binary-search for the responsible
register handler based on the offset address within the region.

The register handler prototype are courtesy of Christoffer Dall.

This is based on Linux commit 4493b1c4866a, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio.c | 192 ++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h | 145 +++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h      |   4 +
 3 files changed, 341 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.h

diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
new file mode 100644
index 0000000000..3c70945466
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -0,0 +1,192 @@
+/*
+ * VGIC MMIO handling functions
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/bitops.h>
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <asm/arm_vgic.h>
+#include <asm/byteorder.h>
+
+#include "vgic.h"
+#include "vgic-mmio.h"
+
+unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len)
+{
+    return 0;
+}
+
+unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len)
+{
+    return -1UL;
+}
+
+void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
+            unsigned int len, unsigned long val)
+{
+    /* Ignore */
+}
+
+static int match_region(const void *key, const void *elt)
+{
+    const unsigned int offset = (unsigned long)key;
+    const struct vgic_register_region *region = elt;
+
+    if ( offset < region->reg_offset )
+        return -1;
+
+    if ( offset >= region->reg_offset + region->len )
+        return 1;
+
+    return 0;
+}
+
+const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+              int nr_regions, unsigned int offset)
+{
+    return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
+               sizeof(regions[0]), match_region);
+}
+
+static bool check_region(const struct domain *d,
+             const struct vgic_register_region *region,
+             paddr_t addr, int len)
+{
+    int flags, nr_irqs = d->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
+
+    switch (len)
+    {
+    case sizeof(u8):
+        flags = VGIC_ACCESS_8bit;
+        break;
+    case sizeof(u32):
+        flags = VGIC_ACCESS_32bit;
+        break;
+    case sizeof(u64):
+        flags = VGIC_ACCESS_64bit;
+        break;
+    default:
+        return false;
+    }
+
+    if ( (region->access_flags & flags) && IS_ALIGNED(addr, len) )
+    {
+        if ( !region->bits_per_irq )
+            return true;
+
+        /* Do we access a non-allocated IRQ? */
+        return VGIC_ADDR_TO_INTID(addr, region->bits_per_irq) < nr_irqs;
+    }
+
+    return false;
+}
+
+const struct vgic_register_region *
+vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
+             paddr_t addr, int len)
+{
+    const struct vgic_register_region *region;
+
+    region = vgic_find_mmio_region(iodev->regions, iodev->nr_regions,
+                       addr - iodev->base_addr);
+    if ( !region || !check_region(vcpu->domain, region, addr, len) )
+        return NULL;
+
+    return region;
+}
+
+static int dispatch_mmio_read(struct vcpu *vcpu, mmio_info_t *info,
+                  register_t *r, void *priv)
+{
+    struct vgic_io_device *iodev = priv;
+    const struct vgic_register_region *region;
+    unsigned long data = 0;
+    paddr_t addr = info->gpa;
+    int len = 1U << info->dabt.size;
+
+    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
+    if ( !region )
+    {
+        memset(r, 0, len);
+        return 0;
+    }
+
+    switch (iodev->iodev_type)
+    {
+    case IODEV_CPUIF:
+        data = region->read(vcpu, addr, len);
+        break;
+    case IODEV_DIST:
+        data = region->read(vcpu, addr, len);
+        break;
+    case IODEV_REDIST:
+        data = region->read(iodev->redist_vcpu, addr, len);
+        break;
+    case IODEV_ITS:
+        data = region->its_read(vcpu->domain, iodev->its, addr, len);
+        break;
+    }
+
+    memcpy(r, &data, len);
+
+    return 1;
+}
+
+static int dispatch_mmio_write(struct vcpu *vcpu, mmio_info_t *info,
+                   register_t r, void *priv)
+{
+    struct vgic_io_device *iodev = priv;
+    const struct vgic_register_region *region;
+    unsigned long data = r;
+    paddr_t addr = info->gpa;
+    int len = 1U << info->dabt.size;
+
+    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
+    if ( !region )
+        return 0;
+
+    switch (iodev->iodev_type)
+    {
+    case IODEV_CPUIF:
+        region->write(vcpu, addr, len, data);
+        break;
+    case IODEV_DIST:
+        region->write(vcpu, addr, len, data);
+        break;
+    case IODEV_REDIST:
+        region->write(iodev->redist_vcpu, addr, len, data);
+        break;
+    case IODEV_ITS:
+        region->its_write(vcpu->domain, iodev->its, addr, len, data);
+        break;
+    }
+
+    return 1;
+}
+
+struct mmio_handler_ops xen_io_gic_ops = {
+    .read = dispatch_mmio_read,
+    .write = dispatch_mmio_write,
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
new file mode 100644
index 0000000000..375b70561d
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -0,0 +1,145 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __KVM_ARM_VGIC_MMIO_H__
+#define __KVM_ARM_VGIC_MMIO_H__
+
+struct vgic_register_region {
+    unsigned int reg_offset;
+    unsigned int len;
+    unsigned int bits_per_irq;
+    unsigned int access_flags;
+    union
+    {
+        unsigned long (*read)(struct vcpu *vcpu, paddr_t addr,
+                              unsigned int len);
+        unsigned long (*its_read)(struct domain *d, struct vgic_its *its,
+                                  paddr_t addr, unsigned int len);
+    };
+    union
+    {
+        void (*write)(struct vcpu *vcpu, paddr_t addr,
+                      unsigned int len, unsigned long val);
+        void (*its_write)(struct domain *d, struct vgic_its *its,
+                          paddr_t addr, unsigned int len,
+                          unsigned long val);
+    };
+    unsigned long (*uaccess_read)(struct vcpu *vcpu, paddr_t addr,
+                                  unsigned int len);
+    union
+    {
+        void (*uaccess_write)(struct vcpu *vcpu, paddr_t addr,
+                              unsigned int len, unsigned long val);
+        int (*uaccess_its_write)(struct domain *d, struct vgic_its *its,
+                                 paddr_t addr, unsigned int len,
+                                 unsigned long val);
+    };
+};
+
+extern struct mmio_handler_ops xen_io_gic_ops;
+
+#define VGIC_ACCESS_8bit    1
+#define VGIC_ACCESS_32bit   2
+#define VGIC_ACCESS_64bit   4
+
+/*
+ * Generate a mask that covers the number of bytes required to address
+ * up to 1024 interrupts, each represented by <bits> bits. This assumes
+ * that <bits> is a power of two.
+ */
+#define VGIC_ADDR_IRQ_MASK(bits) (((bits) * 1024 / 8) - 1)
+
+/*
+ * (addr & mask) gives us the _byte_ offset for the INT ID.
+ * We multiply this by 8 the get the _bit_ offset, then divide this by
+ * the number of bits to learn the actual INT ID.
+ * But instead of a division (which requires a "long long div" implementation),
+ * we shift by the binary logarithm of <bits>.
+ * This assumes that <bits> is a power of two.
+ */
+#define VGIC_ADDR_TO_INTID(addr, bits)  (((addr) & VGIC_ADDR_IRQ_MASK(bits)) * \
+                                         8 >> LOG_2(bits))
+
+/*
+ * Some VGIC registers store per-IRQ information, with a different number
+ * of bits per IRQ. For those registers this macro is used.
+ * The _WITH_LENGTH version instantiates registers with a fixed length
+ * and is mutually exclusive with the _PER_IRQ version.
+ */
+#define REGISTER_DESC_WITH_BITS_PER_IRQ(off, rd, wr, ur, uw, bpi, acc)  \
+    {                           \
+        .reg_offset = off,      \
+        .bits_per_irq = bpi,    \
+        .len = bpi * 1024 / 8,  \
+        .access_flags = acc,    \
+        .read = rd,             \
+        .write = wr,            \
+        .uaccess_read = ur,     \
+        .uaccess_write = uw,    \
+    }
+
+#define REGISTER_DESC_WITH_LENGTH(off, rd, wr, length, acc)     \
+    {                           \
+        .reg_offset = off,      \
+        .bits_per_irq = 0,      \
+        .len = length,          \
+        .access_flags = acc,    \
+        .read = rd,             \
+        .write = wr,            \
+    }
+
+#define REGISTER_DESC_WITH_LENGTH_UACCESS(off, rd, wr, urd, uwr, length, acc) \
+    {                           \
+        .reg_offset = off,      \
+        .bits_per_irq = 0,      \
+        .len = length,          \
+        .access_flags = acc,    \
+        .read = rd,             \
+        .write = wr,            \
+        .uaccess_read = urd,    \
+        .uaccess_write = uwr,   \
+    }
+
+int kvm_vgic_register_mmio_region(struct domain *d, struct vcpu *vcpu,
+                                  struct vgic_register_region *reg_desc,
+                                  struct vgic_io_device *region,
+                                  int nr_irqs, bool offset_private);
+
+unsigned long vgic_data_mmio_bus_to_host(const void *val, unsigned int len);
+
+void vgic_data_host_to_mmio_bus(void *buf, unsigned int len,
+                                unsigned long data);
+
+unsigned long extract_bytes(u64 data, unsigned int offset,
+                            unsigned int num);
+
+u64 update_64bit_reg(u64 reg, unsigned int offset, unsigned int len,
+                     unsigned long val);
+
+unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
+                                 paddr_t addr, unsigned int len);
+
+unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
+                                 paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
+                        unsigned int len, unsigned long val);
+
+/* Find the proper register handler entry given a certain address offset */
+const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+                      int nr_regions, unsigned int offset);
+
+#endif
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 771ca6f046..426b34d0ce 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -27,6 +27,10 @@ static inline bool irq_is_pending(struct vgic_irq *irq)
         return irq->pending_latch || irq->line_level;
 }
 
+const struct vgic_register_region *
+vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
+             paddr_t addr, int len);
+
 struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
                               u32 intid);
 void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 28/49] ARM: new VGIC: Add GICv2 MMIO handling framework

[Andre Przywara]

Create vgic-mmio-v2.c to describe GICv2 emulation specific handlers
using the initializer macros provided by the VGIC MMIO framework.
Provide a function to register the GICv2 distributor registers to
the Xen MMIO framework.
The actual handler functions are still stubs in this patch.

This is based on Linux commit fb848db39661, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c | 83 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.c    | 26 +++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  2 +
 xen/arch/arm/vgic/vgic.h         |  2 +
 4 files changed, 113 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
new file mode 100644
index 0000000000..ee685a5a07
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -0,0 +1,83 @@
+/*
+ * VGICv2 MMIO handling functions
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/bitops.h>
+#include <xen/sched.h>
+#include <xen/sizes.h>
+#include <asm/arm_vgic.h>
+
+#include "vgic.h"
+#include "vgic-mmio.h"
+
+static const struct vgic_register_region vgic_v2_dist_registers[] = {
+    REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
+        vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 2,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+    REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+};
+
+unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev)
+{
+    dev->regions = vgic_v2_dist_registers;
+    dev->nr_regions = ARRAY_SIZE(vgic_v2_dist_registers);
+
+    return SZ_4K;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 3c70945466..59703a6909 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -182,6 +182,32 @@ struct mmio_handler_ops xen_io_gic_ops = {
     .write = dispatch_mmio_write,
 };
 
+int vgic_register_dist_iodev(struct domain *d, paddr_t dist_base_address,
+                 enum vgic_type type)
+{
+    struct vgic_io_device *io_device = &d->arch.vgic.dist_iodev;
+    int ret = 0;
+    unsigned int len;
+
+    switch (type)
+    {
+    case VGIC_V2:
+        len = vgic_v2_init_dist_iodev(io_device);
+        break;
+    default:
+        BUG_ON(1);
+    }
+
+    io_device->base_addr = dist_base_address;
+    io_device->iodev_type = IODEV_DIST;
+    io_device->redist_vcpu = NULL;
+
+    register_mmio_handler(d, &xen_io_gic_ops, dist_base_address, len,
+                  io_device);
+
+    return ret;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 375b70561d..10ac682296 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -137,6 +137,8 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
 void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
                         unsigned int len, unsigned long val);
 
+unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
+
 /* Find the proper register handler entry given a certain address offset */
 const struct vgic_register_region *
 vgic_find_mmio_region(const struct vgic_register_region *regions,
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 426b34d0ce..7747d3f3e0 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -49,6 +49,8 @@ void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
+int vgic_register_dist_iodev(struct domain *d, paddr_t dist_base_address,
+                 enum vgic_type);
 
 void vgic_v2_save_state(struct vcpu *vcpu);
 void vgic_v2_restore_state(struct vcpu *vcpu);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 29/49] ARM: new VGIC: Add CTLR, TYPER and IIDR handlers

[Andre Przywara]

Those three registers are v2 emulation specific, so their implementation
lives entirely in vgic-mmio-v2.c. Also they are handled in one function,
as their implementation is pretty simple.
When the guest enables the distributor, we kick all VCPUs to get
potentially pending interrupts serviced.

This is based on Linux commit 2b0cda878965, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c | 48 +++++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/vgic/vgic.c         | 15 +++++++++++++
 xen/arch/arm/vgic/vgic.h         |  4 ++++
 3 files changed, 66 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index ee685a5a07..0926b3243e 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -20,9 +20,55 @@
 #include "vgic.h"
 #include "vgic-mmio.h"
 
+static unsigned long vgic_mmio_read_v2_misc(struct vcpu *vcpu,
+                        paddr_t addr, unsigned int len)
+{
+    u32 value;
+
+    switch (addr & 0x0c)
+    {
+    case GICD_CTLR:
+        value = vcpu->domain->arch.vgic.enabled ? GICD_CTL_ENABLE : 0;
+        break;
+    case GICD_TYPER:
+        value = vcpu->domain->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
+        value = (value >> 5) - 1;
+        value |= (vcpu->domain->max_vcpus - 1) << 5;
+        break;
+    case GICD_IIDR:
+        value = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);
+        break;
+    default:
+        return 0;
+    }
+
+    return value;
+}
+
+static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len,
+                    unsigned long val)
+{
+    struct vgic_dist *dist = &vcpu->domain->arch.vgic;
+    bool was_enabled = dist->enabled;
+
+    switch (addr & 0x0c)
+    {
+    case GICD_CTLR:
+        dist->enabled = val & GICD_CTL_ENABLE;
+        if ( !was_enabled && dist->enabled )
+            vgic_kick_vcpus(vcpu->domain);
+        break;
+    case GICD_TYPER:
+    case GICD_IIDR:
+        /* Nothing to do */
+        return;
+    }
+}
+
 static const struct vgic_register_region vgic_v2_dist_registers[] = {
     REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
+        vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
         vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 9e7fb1edcb..dc5e011fa3 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -678,6 +678,21 @@ int gic_events_need_delivery(void)
     return vgic_vcpu_pending_irq(current);
 }
 
+void vgic_kick_vcpus(struct domain *d)
+{
+    struct vcpu *vcpu;
+
+    /*
+     * We've injected an interrupt, time to find out who deserves
+     * a good kick...
+     */
+    for_each_vcpu( d, vcpu )
+    {
+        if ( vgic_vcpu_pending_irq(vcpu) )
+            vcpu_unblock(vcpu);
+    }
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 7747d3f3e0..82fe902e26 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -17,6 +17,9 @@
 #ifndef __XEN_ARM_VGIC_NEW_H__
 #define __XEN_ARM_VGIC_NEW_H__
 
+#define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
+#define IMPLEMENTER_ARM     0x43b
+
 #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
@@ -36,6 +39,7 @@ struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
 void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
 bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
                unsigned long flags);
+void vgic_kick_vcpus(struct domain *d);
 
 static inline void vgic_get_irq_kref(struct vgic_irq *irq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Andre Przywara]

As the enable register handlers are shared between the v2 and v3
emulation, their implementation goes into vgic-mmio.c, to be easily
referenced from the v3 emulation as well later.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 114 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
 3 files changed, 127 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 0926b3243e..eca6840ff9 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -74,10 +74,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        vgic_mmio_read_enable, vgic_mmio_write_senable, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 59703a6909..3d9fa02a10 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -39,6 +39,120 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
     /* Ignore */
 }
 
+/*
+ * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the value
+ * of the enabled bit, so there is only one function for both here.
+ */
+unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    u32 value = 0;
+    int i;
+
+    /* Loop over all IRQs affected by this read */
+    for ( i = 0; i < len * 8; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq->enabled )
+            value |= (1U << i);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+static void vgic_handle_hardware_irq(irq_desc_t *desc, int irq_type,
+                     bool enable)
+{
+    unsigned long flags;
+
+//  irq_set_affinity(desc, cpumask_of(v_target->processor));
+    spin_lock_irqsave(&desc->lock, flags);
+    if ( enable )
+    {
+        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
+                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);
+        desc->handler->enable(desc);
+    }
+    else
+        desc->handler->disable(desc);
+    spin_unlock_irqrestore(&desc->lock, flags);
+}
+
+void vgic_mmio_write_senable(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    irq_desc_t *desc;
+    int i;
+    unsigned long flags;
+    enum vgic_irq_config config;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq;
+
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        irq->enabled = true;
+        if ( irq->hw )
+        {
+            /*
+             * The irq cannot be a PPI, we only support delivery
+             * of SPIs to guests.
+             */
+            ASSERT(irq->hwintid >= 32);
+
+            desc = irq_to_desc(irq->hwintid);
+            config = irq->config;
+        }
+        else
+            desc = NULL;
+        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
+
+        vgic_put_irq(vcpu->domain, irq);
+
+        if ( desc )
+            vgic_handle_hardware_irq(desc, config, true);
+    }
+}
+
+void vgic_mmio_write_cenable(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    int i;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq;
+        unsigned long flags;
+        irq_desc_t *desc;
+
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->enabled = false;
+
+        if ( irq->hw )
+            desc = irq_to_desc(irq->hwintid);
+        else
+            desc = NULL;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+
+        if ( desc )
+            vgic_handle_hardware_irq(desc, 0, false);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 10ac682296..9f34bd1aec 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -137,6 +137,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
 void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
                         unsigned int len, unsigned long val);
 
+unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_senable(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val);
+
+void vgic_mmio_write_cenable(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 /* Find the proper register handler entry given a certain address offset */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 31/49] ARM: new VGIC: Add PENDING registers handlers

[Andre Przywara]

The pending register handlers are shared between the v2 and v3
emulation, so their implementation goes into vgic-mmio.c, to be easily
referenced from the v3 emulation as well later.
For level triggered interrupts the real line level is unaffected by
this write, so we keep this state separate and combine it with the
device's level to get the actual pending state.

This is based on Linux commit 96b298000db4, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c |  4 +--
 xen/arch/arm/vgic/vgic-mmio.c    | 62 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    | 11 +++++++
 3 files changed, 75 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index eca6840ff9..ceb86900a0 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -80,10 +80,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        vgic_mmio_read_pending, vgic_mmio_write_spending, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        vgic_mmio_read_pending, vgic_mmio_write_cpending, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
         vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 3d9fa02a10..9a65e39d78 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -153,6 +153,68 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
     }
 }
 
+unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
+                     paddr_t addr, unsigned int len)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    u32 value = 0;
+    int i;
+
+    /* Loop over all IRQs affected by this read */
+    for ( i = 0; i < len * 8; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq_is_pending(irq) )
+            value |= (1U << i);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+void vgic_mmio_write_spending(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    int i;
+    unsigned long flags;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        irq->pending_latch = true;
+
+        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+void vgic_mmio_write_cpending(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    int i;
+    unsigned long flags;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->pending_latch = false;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 9f34bd1aec..209afbbb9a 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -148,6 +148,17 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
                  paddr_t addr, unsigned int len,
                  unsigned long val);
 
+unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
+                     paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_spending(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val);
+
+void vgic_mmio_write_cpending(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 /* Find the proper register handler entry given a certain address offset */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 32/49] ARM: new VGIC: Add ACTIVE registers handlers

[Andre Przywara]

The active register handlers are shared between the v2 and v3 emulation,
so their implementation goes into vgic-mmio.c, to be easily referenced
from the v3 emulation as well later.
Since activation/deactivation of an interrupt may happen entirely in the
guest without it ever exiting, we need some extra logic to properly track
the active state.
For clearing the active state, we would basically have to halt the guest
to make sure this is properly propagated into the respective VCPUs.
This is not yet implemented in Xen.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c |  4 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 94 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    | 11 +++++
 3 files changed, 107 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index ceb86900a0..eba24d9866 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -86,10 +86,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_pending, vgic_mmio_write_cpending, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        vgic_mmio_read_active, vgic_mmio_write_sactive, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
+        vgic_mmio_read_active, vgic_mmio_write_cactive, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 9a65e39d78..ac3aa03fbc 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -215,6 +215,100 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
     }
 }
 
+unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    u32 value = 0;
+    int i;
+
+    /* Loop over all IRQs affected by this read */
+    for ( i = 0; i < len * 8; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq->active )
+            value |= (1U << i);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+static void vgic_mmio_change_active(struct vcpu *vcpu, struct vgic_irq *irq,
+                    bool new_active_state)
+{
+}
+
+static void vgic_change_active_prepare(struct vcpu *vcpu, u32 intid)
+{
+}
+
+static void vgic_change_active_finish(struct vcpu *vcpu, u32 intid)
+{
+}
+
+static void __vgic_mmio_write_cactive(struct vcpu *vcpu,
+                      paddr_t addr, unsigned int len,
+                      unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    int i;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        vgic_mmio_change_active(vcpu, irq, false);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+void vgic_mmio_write_cactive(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+
+    spin_lock(&vcpu->domain->domain_lock);
+    vgic_change_active_prepare(vcpu, intid);
+
+    __vgic_mmio_write_cactive(vcpu, addr, len, val);
+
+    vgic_change_active_finish(vcpu, intid);
+    spin_unlock(&vcpu->domain->domain_lock);
+}
+
+static void __vgic_mmio_write_sactive(struct vcpu *vcpu,
+                      paddr_t addr, unsigned int len,
+                      unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+    int i;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        vgic_mmio_change_active(vcpu, irq, true);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+void vgic_mmio_write_sactive(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
+
+    spin_lock(&vcpu->domain->domain_lock);
+    vgic_change_active_prepare(vcpu, intid);
+
+    __vgic_mmio_write_sactive(vcpu, addr, len, val);
+
+    vgic_change_active_finish(vcpu, intid);
+    spin_unlock(&vcpu->domain->domain_lock);
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 209afbbb9a..39e854232e 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -159,6 +159,17 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
                   paddr_t addr, unsigned int len,
                   unsigned long val);
 
+unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_cactive(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val);
+
+void vgic_mmio_write_sactive(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 /* Find the proper register handler entry given a certain address offset */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 33/49] ARM: new VGIC: Add PRIORITY registers handlers

[Andre Przywara]

The priority register handlers are shared between the v2 and v3 emulation,
so their implementation goes into vgic-mmio.c, to be easily referenced
from the v3 emulation as well later.
There is a corner case when we change the priority of a pending
interrupt which we don't handle at the moment.

This is based on Linux commit dd238ec2b87b, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c |  4 ++--
 xen/arch/arm/vgic/vgic-mmio.c    | 47 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
 xen/arch/arm/vgic/vgic.h         |  2 ++
 4 files changed, 58 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index eba24d9866..0574ff9b16 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -92,8 +92,8 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_active, vgic_mmio_write_cactive, NULL, NULL, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
-        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+        vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
+        8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index ac3aa03fbc..14570d9d8e 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -309,6 +309,53 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
     spin_unlock(&vcpu->domain->domain_lock);
 }
 
+unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
+                      paddr_t addr, unsigned int len)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
+    int i;
+    u64 val = 0;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        val |= (u64)irq->priority << (i * 8);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return val;
+}
+
+/*
+ * We currently don't handle changing the priority of an interrupt that
+ * is already pending on a VCPU. If there is a need for this, we would
+ * need to make this VCPU exit and re-evaluate the priorities, potentially
+ * leading to this interrupt getting presented now to the guest (if it has
+ * been masked by the priority mask before).
+ */
+void vgic_mmio_write_priority(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
+    int i;
+    unsigned long flags;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        /* Narrow the priority range to what we actually support */
+        irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 39e854232e..30221096b9 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -170,6 +170,13 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
                  paddr_t addr, unsigned int len,
                  unsigned long val);
 
+unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
+                      paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_priority(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 /* Find the proper register handler entry given a certain address offset */
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 82fe902e26..b104f8e964 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -20,6 +20,8 @@
 #define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
 #define IMPLEMENTER_ARM     0x43b
 
+#define VGIC_PRI_BITS       5
+
 #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 34/49] ARM: new VGIC: Add CONFIG registers handlers

[Andre Przywara]

The config register handlers are shared between the v2 and v3 emulation,
so their implementation goes into vgic-mmio.c, to be easily referenced
from the v3 emulation as well later.

This is based on Linux commit 79717e4ac09c, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 54 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
 3 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 0574ff9b16..c0b88b347e 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -98,7 +98,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 2,
+        vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 14570d9d8e..626ce06986 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -356,6 +356,60 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
     }
 }
 
+unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 2);
+    u32 value = 0;
+    int i;
+
+    for ( i = 0; i < len * 4; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq->config == VGIC_CONFIG_EDGE )
+            value |= (2U << (i * 2));
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+void vgic_mmio_write_config(struct vcpu *vcpu,
+                paddr_t addr, unsigned int len,
+                unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 2);
+    int i;
+    unsigned long flags;
+
+    for ( i = 0; i < len * 4; i++ )
+    {
+        struct vgic_irq *irq;
+
+        /*
+         * The configuration cannot be changed for SGIs in general,
+         * for PPIs this is IMPLEMENTATION DEFINED. The arch timer
+         * code relies on PPIs being level triggered, so we also
+         * make them read-only here.
+         */
+        if ( intid + i < VGIC_NR_PRIVATE_IRQS )
+            continue;
+
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        if ( test_bit(i * 2 + 1, &val) )
+            irq->config = VGIC_CONFIG_EDGE;
+        else
+            irq->config = VGIC_CONFIG_LEVEL;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 30221096b9..b42ea1bd8a 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -177,6 +177,13 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
                   paddr_t addr, unsigned int len,
                   unsigned long val);
 
+unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_config(struct vcpu *vcpu,
+                paddr_t addr, unsigned int len,
+                unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 /* Find the proper register handler entry given a certain address offset */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 35/49] ARM: new VGIC: Add TARGET registers handlers

[Andre Przywara]

The target register handlers are v2 emulation specific, so their
implementation lives entirely in vgic-mmio-v2.c.
We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
set in the target mask instead of making it possibly pending on
multiple VCPUs.

This is based on Linux commit 2c234d6f1826, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c | 52 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 51 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index c0b88b347e..c59f2c1ba7 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -66,6 +66,56 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
     }
 }
 
+static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
+                       paddr_t addr, unsigned int len)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
+    int i;
+    u64 val = 0;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
+                            vcpu, intid + i);
+
+        val |= (u64)irq->targets << (i * 8);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return val;
+}
+
+static void vgic_mmio_write_target(struct vcpu *vcpu,
+                   paddr_t addr, unsigned int len,
+                   unsigned long val)
+{
+    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
+    u8 cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);
+    int i;
+    unsigned long flags;
+
+    /* GICD_ITARGETSR[0-7] are read-only */
+    if ( intid < VGIC_NR_PRIVATE_IRQS )
+        return;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
+                            NULL, intid + i);
+        int target;
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->targets = (val >> (i * 8)) & cpu_mask;
+        target = irq->targets ? (ffs(irq->targets) - 1) : 0;
+        irq->target_vcpu = vcpu->domain->vcpu[target];
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static const struct vgic_register_region vgic_v2_dist_registers[] = {
     REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
         vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
@@ -95,7 +145,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
         8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
+        vgic_mmio_read_target, vgic_mmio_write_target, NULL, NULL, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
         vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 36/49] ARM: new VGIC: Add SGIR register handler

[Andre Przywara]

Triggering an IPI via this register is v2 specific, so the
implementation lives entirely in vgic-mmio-v2.c.

This is based on Linux commit 55cc01fb9004, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c | 47 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 46 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index c59f2c1ba7..3f67b4659a 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -66,6 +66,51 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
     }
 }
 
+static void vgic_mmio_write_sgir(struct vcpu *source_vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val)
+{
+    struct domain *d = source_vcpu->domain;
+    int nr_vcpus = d->max_vcpus;
+    int intid = val & 0xf;
+    int targets = (val >> 16) & 0xff;
+    int mode = (val >> 24) & 0x03;
+    struct vcpu *vcpu;
+    unsigned long flags;
+
+    switch (mode)
+    {
+    case 0x0:       /* as specified by targets */
+        break;
+    case 0x1:
+        targets = (1U << nr_vcpus) - 1;         /* all, ... */
+        targets &= ~(1U << source_vcpu->vcpu_id);   /* but self */
+        break;
+    case 0x2:       /* this very vCPU only */
+        targets = (1U << source_vcpu->vcpu_id);
+        break;
+    case 0x3:       /* reserved */
+        return;
+    }
+
+    for_each_vcpu(d, vcpu)
+    {
+        struct vgic_irq *irq;
+
+        if ( !(targets & (1U << vcpu->vcpu_id)) )
+            continue;
+
+        irq = vgic_get_irq(d, vcpu, intid);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        irq->pending_latch = true;
+        irq->source |= 1U << source_vcpu->vcpu_id;
+
+        vgic_queue_irq_unlock(d, irq, flags);
+        vgic_put_irq(d, irq);
+    }
+}
+
 static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
                        paddr_t addr, unsigned int len)
 {
@@ -151,7 +196,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
+        vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 37/49] ARM: new VGIC: Add SGIPENDR register handlers

[Andre Przywara]

As this register is v2 specific, its implementation lives entirely
in vgic-mmio-v2.c.
This register allows setting the source mask of an IPI.

This is based on Linux commit ed40213ef9b0, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-mmio-v2.c | 77 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 75 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 3f67b4659a..c62307c3aa 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -161,6 +161,79 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
     }
 }
 
+static unsigned long vgic_mmio_read_sgipend(struct vcpu *vcpu,
+                        paddr_t addr, unsigned int len)
+{
+    u32 intid = addr & 0x0f;
+    int i;
+    u64 val = 0;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
+                            vcpu, intid + i);
+
+        val |= (u64)irq->source << (i * 8);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+    return val;
+}
+
+static void vgic_mmio_write_sgipendc(struct vcpu *vcpu,
+                     paddr_t addr, unsigned int len,
+                     unsigned long val)
+{
+    u32 intid = addr & 0x0f;
+    int i;
+    unsigned long flags;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
+                            vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->source &= ~((val >> (i * 8)) & 0xff);
+        if ( !irq->source )
+            irq->pending_latch = false;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+static void vgic_mmio_write_sgipends(struct vcpu *vcpu,
+                     paddr_t addr, unsigned int len,
+                     unsigned long val)
+{
+    u32 intid = addr & 0x0f;
+    int i;
+    unsigned long flags;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
+                            vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->source |= (val >> (i * 8)) & 0xff;
+
+        if ( irq->source )
+        {
+            irq->pending_latch = true;
+            vgic_queue_irq_unlock(vcpu->domain, irq, flags);
+        }
+        else
+        {
+            spin_unlock_irqrestore(&irq->irq_lock, flags);
+        }
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static const struct vgic_register_region vgic_v2_dist_registers[] = {
     REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
         vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
@@ -199,10 +272,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        vgic_mmio_read_sgipend, vgic_mmio_write_sgipendc, 16,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        vgic_mmio_read_sgipend, vgic_mmio_write_sgipends, 16,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
 };
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Andre Przywara]

The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
when a guest EOIs the virtual interrupt, it affects the state of that
corresponding interrupt on the hardware side at the same time.
Implement the interface that the Xen arch/core code expects to connect
the virtual and the physical world.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 63 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index dc5e011fa3..8d5260a7db 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
     }
 }
 
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq)
+{
+    struct irq_desc *desc = NULL;
+    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
+    unsigned long flags;
+
+    if ( !irq )
+        return NULL;
+
+    spin_lock_irqsave(&irq->irq_lock, flags);
+    if ( irq->hw )
+        desc = irq_to_desc(irq->hwintid);
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+    vgic_put_irq(d, irq);
+
+    return desc;
+}
+
+/*
+ * was:
+ *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
+ *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int virt_irq)
+ */
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
+            unsigned int virt_irq, struct irq_desc *desc,
+            bool connect)
+{
+    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
+    unsigned long flags;
+    int ret = 0;
+
+    if ( !irq )
+        return -EINVAL;
+
+    spin_lock_irqsave(&irq->irq_lock, flags);
+
+    if ( connect )                      /* assign a mapped IRQ */
+    {
+        /* The VIRQ should not be already enabled by the guest */
+        if ( !irq->hw && !irq->enabled )
+        {
+            irq->hw = true;
+            irq->hwintid = desc->irq;
+        }
+        else
+        {
+            ret = -EBUSY;
+        }
+    }
+    else                                /* remove a mapped IRQ */
+    {
+        irq->hw = false;
+        irq->hwintid = 0;
+    }
+
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+    vgic_put_irq(d, irq);
+
+    return ret;
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 39/49] ARM: new VGIC: Add event channel IRQ handling

[Andre Przywara]

The Xen core/arch code relies on two abstracted functions to inject an
event channel IRQ and to query its pending state.
Implement those to query the state of the new VGIC implementation.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 8d5260a7db..b62cda7d2f 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -693,6 +693,26 @@ void vgic_kick_vcpus(struct domain *d)
     }
 }
 
+void arch_evtchn_inject(struct vcpu *v)
+{
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
+}
+
+bool vgic_evtchn_irq_pending(struct vcpu *v)
+{
+    struct vgic_irq *irq;
+    bool pending;
+
+    /* Does not work for LPIs. */
+    ASSERT(!is_lpi(v->domain->arch.evtchn_irq));
+
+    irq = vgic_get_irq(v->domain, v, v->domain->arch.evtchn_irq);
+    pending = irq_is_pending(irq);
+    vgic_put_irq(v->domain, irq);
+
+    return pending;
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 40/49] ARM: new VGIC: Handle virtual IRQ allocation/reservation

[Andre Przywara]

To find an unused virtual IRQ number Xen uses a scheme to track used
virtual IRQs.
Implement this interface in the new VGIC to make the Xen core/arch code
happy.
This is actually somewhat VGIC agnostic, so is mostly a copy of the code
from the old VGIC. But it has to live in the VGIC files, so we can't
easily reuse the existing implementation.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index b62cda7d2f..3b475ed1a4 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -713,6 +713,50 @@ bool vgic_evtchn_irq_pending(struct vcpu *v)
     return pending;
 }
 
+bool vgic_reserve_virq(struct domain *d, unsigned int virq)
+{
+    if ( virq >= vgic_num_irqs(d) )
+        return false;
+
+    return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
+}
+
+int vgic_allocate_virq(struct domain *d, bool spi)
+{
+    int first, end;
+    unsigned int virq;
+
+    if ( !spi )
+    {
+        /* We only allocate PPIs. SGIs are all reserved */
+        first = 16;
+        end = 32;
+    }
+    else
+    {
+        first = 32;
+        end = vgic_num_irqs(d);
+    }
+
+    /*
+     * There is no spinlock to protect allocated_irqs, therefore
+     * test_and_set_bit may fail. If so retry it.
+     */
+    do
+    {
+        virq = find_next_zero_bit(d->arch.vgic.allocated_irqs, end, first);
+        if ( virq >= end )
+            return -1;
+    } while ( test_and_set_bit(virq, d->arch.vgic.allocated_irqs) );
+
+    return virq;
+}
+
+void vgic_free_virq(struct domain *d, unsigned int virq)
+{
+    clear_bit(virq, d->arch.vgic.allocated_irqs);
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 41/49] ARM: new VGIC: dump virtual IRQ info

[Andre Przywara]

When we dump guest state on the Xen console, we also print the state of
IRQs that are on a VCPU.
Add the code to dump the state of an IRQ handled by the new VGIC.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 3b475ed1a4..97ffdba5ad 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -757,6 +757,19 @@ void vgic_free_virq(struct domain *d, unsigned int virq)
     clear_bit(virq, d->arch.vgic.allocated_irqs);
 }
 
+void gic_dump_vgic_info(struct vcpu *v)
+{
+    struct vgic_cpu *vgic_cpu = &v->arch.vgic_cpu;
+    struct vgic_irq *irq;
+
+    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
+        printk("   on CPU: %s %s irq %u: %spending, %sactive, %senabled\n",
+               irq->hw ? "hardware" : "virtual",
+               irq->config == VGIC_CONFIG_LEVEL ? "level" : "edge",
+               irq->intid, irq_is_pending(irq) ? "" : "not ",
+               irq->active ? "" : "not ", irq->enabled ? "" : "not ");
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 42/49] ARM: new VGIC: provide system register emulation stub

[Andre Przywara]

The Xen arch code traps system registers writes from the guest and will
relay anything GIC related to the VGIC.
Since this affects only GICv3 (which we don't yet emulate), provide a
stub implementation of vgic_emulate() for now.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 97ffdba5ad..d91028bd43 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -790,6 +790,13 @@ struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
     return desc;
 }
 
+bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr)
+{
+    ASSERT(current->domain->arch.vgic.version == GIC_V3);
+
+    return false;
+}
+
 /*
  * was:
  *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 43/49] ARM: new VGIC: Add preliminary stub implementations

[Andre Przywara]

The Xen core code requires an interrupt controller emulation to implement
arch_move_irqs(), to move the affinity of an hardware mapped virtual IRQ
to another core. In the moment we don't implement this
physical-follow-virtual regime in our new VGIC, so just provide an empty
stub implementation to make the linker happy.
Similarily vgic_clear_pending_irqs() is required by the ARM code,
although it is suspected that it is actually not necessary. Go with a
stub for now.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index d91028bd43..77fa756329 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -770,6 +770,19 @@ void gic_dump_vgic_info(struct vcpu *v)
                irq->active ? "" : "not ", irq->enabled ? "" : "not ");
 }
 
+void vgic_clear_pending_irqs(struct vcpu *v)
+{
+    /*
+     * TODO: It is unclear whether we really need this, so we might instead
+     * remove it on the caller site.
+     */
+}
+
+void arch_move_irqs(struct vcpu *v)
+{
+    /* TODO: implement this (?) */
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 44/49] ARM: new VGIC: vgic-init: register VGIC

[Andre Przywara]

This patch implements the function which is called by Xen when it wants
to register the virtual GIC.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-init.c | 62 +++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h      |  3 +++
 2 files changed, 65 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-init.c

diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
new file mode 100644
index 0000000000..b5f1183a50
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-init.c
@@ -0,0 +1,62 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/sched.h>
+#include <asm/arm_vgic.h>
+
+#include "vgic.h"
+
+/* CREATION */
+
+/**
+ * domain_vgic_register: create a virtual GIC
+ * @d: domain pointer
+ * @mmio_count: pointer to add number of required MMIO regions
+ *
+ * was: kvm_vgic_create
+ */
+int domain_vgic_register(struct domain *d, int *mmio_count)
+{
+    switch ( d->arch.vgic.version )
+    {
+#ifdef CONFIG_HAS_GICV3
+    case GIC_V3:
+        d->arch.max_vcpus = VGIC_V3_MAX_CPUS;
+        break;
+#endif
+    case GIC_V2:
+        d->arch.max_vcpus = VGIC_V2_MAX_CPUS;
+        break;
+    }
+
+    if ( d->max_vcpus > d->arch.max_vcpus )
+        return -E2BIG;
+
+    d->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
+    d->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
+    d->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index b104f8e964..205ce10ffa 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -20,6 +20,9 @@
 #define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
 #define IMPLEMENTER_ARM     0x43b
 
+#define VGIC_ADDR_UNDEF     (-1)
+#define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
+
 #define VGIC_PRI_BITS       5
 
 #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 45/49] ARM: new VGIC: vgic-init: implement vgic_init

[Andre Przywara]

This patch allocates and initializes the data structures used to model
the vgic distributor and virtual cpu interfaces. At that stage the
number of IRQs and number of virtual CPUs is frozen.

This is based on Linux commit ad275b8bb1e6, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-init.c | 197 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 197 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
index b5f1183a50..0cd2dfc600 100644
--- a/xen/arch/arm/vgic/vgic-init.c
+++ b/xen/arch/arm/vgic/vgic-init.c
@@ -1,5 +1,6 @@
 /*
  * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
  *
  * This program is free software; you can redistribute it and/or modify
  * it under the terms of the GNU General Public License version 2 as
@@ -19,6 +20,77 @@
 
 #include "vgic.h"
 
+/*
+ * Initialization rules: there are multiple stages to the vgic
+ * initialization, both for the distributor and the CPU interfaces.  The basic
+ * idea is that even though the VGIC is not functional or not requested from
+ * user space, the critical path of the run loop can still call VGIC functions
+ * that just won't do anything, without them having to check additional
+ * initialization flags to ensure they don't look at uninitialized data
+ * structures.
+ *
+ * Distributor:
+ *
+ * - vgic_early_init(): initialization of static data that doesn't
+ *   depend on any sizing information or emulation type. No allocation
+ *   is allowed there.
+ *
+ * - vgic_init(): allocation and initialization of the generic data
+ *   structures that depend on sizing information (number of CPUs,
+ *   number of interrupts). Also initializes the vcpu specific data
+ *   structures. Can be executed lazily for GICv2.
+ *
+ * CPU Interface:
+ *
+ * - kvm_vgic_vcpu_early_init(): initialization of static data that
+ *   doesn't depend on any sizing information or emulation type. No
+ *   allocation is allowed there.
+ */
+
+/**
+ * vgic_vcpu_early_init() - Initialize static VGIC VCPU data structures
+ * @vcpu: The VCPU whose VGIC data structures whould be initialized
+ *
+ * Only do initialization, but do not actually enable the VGIC CPU interface
+ * yet.
+ */
+static void vgic_vcpu_early_init(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+    int i;
+
+    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
+    spin_lock_init(&vgic_cpu->ap_list_lock);
+
+    /*
+     * Enable and configure all SGIs to be edge-triggered and
+     * configure all PPIs as level-triggered.
+     */
+    for ( i = 0; i < VGIC_NR_PRIVATE_IRQS; i++ )
+    {
+        struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
+
+        INIT_LIST_HEAD(&irq->ap_list);
+        spin_lock_init(&irq->irq_lock);
+        irq->intid = i;
+        irq->vcpu = NULL;
+        irq->target_vcpu = vcpu;
+        irq->targets = 1U << vcpu->vcpu_id;
+        atomic_set(&irq->refcount, 0);
+        if ( vgic_irq_is_sgi(i) )
+        {
+            /* SGIs */
+            irq->enabled = 1;
+            irq->config = VGIC_CONFIG_EDGE;
+        }
+        else
+        {
+            /* PPIs */
+            irq->config = VGIC_CONFIG_LEVEL;
+        }
+    }
+}
+
 /* CREATION */
 
 /**
@@ -52,6 +124,131 @@ int domain_vgic_register(struct domain *d, int *mmio_count)
     return 0;
 }
 
+/* INIT/DESTROY */
+
+/**
+ * domain_vgic_init: initialize the dist data structures
+ * @d: domain pointer
+ * @nr_spis: number of SPIs
+ */
+int domain_vgic_init(struct domain *d, unsigned int nr_spis)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+    int i, ret;
+
+    /* Limit the number of virtual SPIs supported to (1020 - 32) = 988  */
+    if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
+        return -EINVAL;
+
+    dist->nr_spis = nr_spis;
+    dist->spis = xzalloc_array(struct vgic_irq, nr_spis);
+    if ( !dist->spis )
+        return  -ENOMEM;
+
+    /*
+     * In the following code we do not take the irq struct lock since
+     * no other action on irq structs can happen while the VGIC is
+     * not initialized yet:
+     * If someone wants to inject an interrupt or does a MMIO access, we
+     * require prior initialization in case of a virtual GICv3 or trigger
+     * initialization when using a virtual GICv2.
+     */
+    for ( i = 0; i < nr_spis; i++ )
+    {
+        struct vgic_irq *irq = &dist->spis[i];
+
+        irq->intid = i + VGIC_NR_PRIVATE_IRQS;
+        INIT_LIST_HEAD(&irq->ap_list);
+        spin_lock_init(&irq->irq_lock);
+        irq->vcpu = NULL;
+        irq->target_vcpu = NULL;
+        atomic_set(&irq->refcount, 0);
+        if ( dist->version == GIC_V2 )
+            irq->targets = 0;
+        else
+            irq->mpidr = 0;
+    }
+
+    INIT_LIST_HEAD(&dist->lpi_list_head);
+    spin_lock_init(&dist->lpi_list_lock);
+
+    if ( dist->version == GIC_V2 )
+        ret = vgic_v2_map_resources(d);
+    else
+        ret = -ENXIO;
+
+    if ( ret )
+        return ret;
+
+    /* allocated_irqs() is used by Xen to find available vIRQs */
+    d->arch.vgic.allocated_irqs =
+        xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
+    if ( !d->arch.vgic.allocated_irqs )
+        return -ENOMEM;
+
+    /* vIRQ0-15 (SGIs) are reserved */
+    for ( i = 0; i < NR_GIC_SGI; i++ )
+        set_bit(i, d->arch.vgic.allocated_irqs);
+
+    return 0;
+}
+
+/**
+ * vcpu_vgic_init() - Register VCPU-specific KVM iodevs
+ * was: kvm_vgic_vcpu_init()
+ * Xen: adding vgic_vx_enable() call
+ * @vcpu: pointer to the VCPU being created and initialized
+ */
+int vcpu_vgic_init(struct vcpu *vcpu)
+{
+    int ret = 0;
+
+    vgic_vcpu_early_init(vcpu);
+
+    if ( gic_hw_version() == GIC_V2 )
+        vgic_v2_enable(vcpu);
+    else
+        ret = -ENXIO;
+
+    return ret;
+}
+
+void domain_vgic_free(struct domain *d)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+        int i, ret;
+
+    for ( i = 0; i < dist->nr_spis; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(d, NULL, 32 + i);
+
+        if ( !irq->hw )
+            continue;
+
+        ret = release_guest_irq(d, irq->hwintid);
+        if ( ret )
+            dprintk(XENLOG_G_WARNING,
+                "d%u: Failed to release virq %u ret = %d\n",
+                d->domain_id, 32 + i, ret);
+    }
+
+    dist->ready = false;
+    dist->initialized = false;
+
+    xfree(dist->spis);
+    xfree(dist->allocated_irqs);
+    dist->nr_spis = 0;
+}
+
+int vcpu_vgic_free(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
+
+    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 46/49] ARM: new VGIC: vgic-init: implement map_resources

[Andre Przywara]

map_resources is the last initialization step needed before the first
VCPU is run. At that stage the code stores the MMIO base addresses used.
Also it registers the respective register frames with the MMIO framework.

This is based on Linux commit cbae53e663ea, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-v2.c | 65 +++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h    |  1 +
 2 files changed, 66 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
index 10fc467ffa..b5026bb050 100644
--- a/xen/arch/arm/vgic/vgic-v2.c
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -202,6 +202,71 @@ void vgic_v2_clear_lr(struct vcpu *vcpu, int lr)
     vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = 0;
 }
 
+int vgic_v2_map_resources(struct domain *d)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+    paddr_t cbase, csize;
+    paddr_t vbase;
+    int ret;
+
+    /*
+     * The hardware domain gets the hardware address.
+     * Guests get the virtual platform layout.
+     */
+    if ( is_hardware_domain(d) )
+    {
+        d->arch.vgic.vgic_dist_base = gic_v2_hw_data.dbase;
+        /*
+         * For the hardware domain, we always map the whole HW CPU
+         * interface region in order to match the device tree (the "reg"
+         * properties is copied as it is).
+         * Note that we assume the size of the CPU interface is always
+         * aligned to PAGE_SIZE.
+         */
+        cbase = gic_v2_hw_data.cbase;   /* was: dist->vgic_cpu_base */
+        csize = gic_v2_hw_data.csize;
+        vbase = gic_v2_hw_data.vbase; /* was: kvm_vgic_global_state.vcpu_base */
+    }
+    else
+    {
+        d->arch.vgic.vgic_dist_base = GUEST_GICD_BASE;
+        /*
+         * The CPU interface exposed to the guest is always 8kB. We may
+         * need to add an offset to the virtual CPU interface base
+         * address when in the GIC is aliased to get a 8kB contiguous
+         * region.
+         */
+        BUILD_BUG_ON(GUEST_GICC_SIZE != SZ_8K);
+        cbase = GUEST_GICC_BASE;
+        csize = GUEST_GICC_SIZE;
+        vbase = gic_v2_hw_data.vbase + gic_v2_hw_data.aliased_offset;
+    }
+
+
+    ret = vgic_register_dist_iodev(d, dist->vgic_dist_base, VGIC_V2);
+    if ( ret )
+    {
+        gdprintk(XENLOG_ERR, "Unable to register VGIC MMIO regions\n");
+        return ret;
+    }
+
+    /*
+     * Map the gic virtual cpu interface in the gic cpu interface
+     * region of the guest.
+     */
+    ret = map_mmio_regions(d, gaddr_to_gfn(cbase), csize / PAGE_SIZE,
+                   maddr_to_mfn(vbase));
+    if ( ret )
+    {
+        gdprintk(XENLOG_ERR, "Unable to remap VGIC CPU to VCPU\n");
+        return ret;
+    }
+
+    dist->ready = true;
+
+        return 0;
+}
+
 static void save_lrs(struct vcpu *vcpu, void __iomem *base)
 {
     struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 205ce10ffa..adb04f2f52 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -58,6 +58,7 @@ void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
+int vgic_v2_map_resources(struct domain *d);
 int vgic_register_dist_iodev(struct domain *d, paddr_t dist_base_address,
                  enum vgic_type);
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 47/49] ARM: new VGIC: Add vgic_v2_enable

[Andre Przywara]

Enable the VGIC operation by properly initialising the registers
in the hypervisor GIC interface.

This is based on Linux commit f7b6985cc3d0, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/vgic/vgic-v2.c | 13 +++++++++++++
 xen/arch/arm/vgic/vgic.h    |  1 +
 2 files changed, 14 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
index b5026bb050..16e9c8dbf0 100644
--- a/xen/arch/arm/vgic/vgic-v2.c
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -202,6 +202,19 @@ void vgic_v2_clear_lr(struct vcpu *vcpu, int lr)
     vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = 0;
 }
 
+void vgic_v2_enable(struct vcpu *vcpu)
+{
+    /*
+     * By forcing VMCR to zero, the GIC will restore the binary
+     * points to their reset values. Anything else resets to zero
+     * anyway.
+     */
+    vcpu->arch.vgic_cpu.vgic_v2.vgic_vmcr = 0;
+
+    /* Get the show on the road... */
+    vcpu->arch.vgic_cpu.vgic_v2.vgic_hcr = GICH_HCR_EN;
+}
+
 int vgic_v2_map_resources(struct domain *d)
 {
     struct vgic_dist *dist = &d->arch.vgic;
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index adb04f2f52..e28c002a9e 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -58,6 +58,7 @@ void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
+void vgic_v2_enable(struct vcpu *vcpu);
 int vgic_v2_map_resources(struct domain *d);
 int vgic_register_dist_iodev(struct domain *d, paddr_t dist_base_address,
                  enum vgic_type);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 48/49] ARM: allocate two pages for struct vcpu

[Andre Przywara]

At the moment we allocate exactly one page for struct vcpu on ARM, also
have a check in place to prevent it growing beyond 4KB.
As the struct includes the state of all 32 private (per-VCPU) interrupts,
we are at 3840 bytes on arm64 at the moment already. Growing the per-IRQ
VGIC structure even slightly makes the VCPU quickly exceed the 4K limit.
The new VGIC will need more space per virtual IRQ. I spent a few hours
trying to trim this down, but couldn't get it below 4KB, even with the
nasty hacks piling up to save some bytes here and there.
It turns out that beyond efficiency, maybe, there is no real technical
reason this struct has to fit in one page, so lifting the limit to two
pages seems like the most pragmatic solution.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/domain.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 87bd493924..4dd34393f1 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -502,10 +502,13 @@ void dump_pageframe_info(struct domain *d)
 struct vcpu *alloc_vcpu_struct(void)
 {
     struct vcpu *v;
-    BUILD_BUG_ON(sizeof(*v) > PAGE_SIZE);
-    v = alloc_xenheap_pages(0, 0);
-    if ( v != NULL )
+
+    BUILD_BUG_ON(sizeof(*v) > 2 * PAGE_SIZE);
+    v = alloc_xenheap_pages(1, 0);
+    if ( v != NULL ) {
         clear_page(v);
+        clear_page((void *)v + PAGE_SIZE);
+    }
     return v;
 }
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[RFC PATCH 49/49] ARM: VGIC: wire new VGIC(-v2) files into Xen build system

[Andre Przywara]

Now that we have both the old VGIC prepared to cope with a sibling and
the code for the new VGIC in place, lets add a Kconfig option to enable
the new code and wire it into the Xen build system.
This will add a compile time option to use either the "old" or the "new"
VGIC.
In the moment this is restricted to a vGIC-v2. To make the build system
happy, we provide a temporary dummy implementation of
vgic_v3_setup_hw() to allow building for now.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 xen/arch/arm/Kconfig     |  6 +++++-
 xen/arch/arm/Makefile    | 10 +++++++++-
 xen/arch/arm/vgic/vgic.c |  8 ++++++++
 xen/common/Makefile      |  1 +
 4 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 2782ee6589..aad19927ce 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -48,7 +48,11 @@ config HAS_GICV3
 config HAS_ITS
         bool
         prompt "GICv3 ITS MSI controller support" if EXPERT = "y"
-        depends on HAS_GICV3
+        depends on HAS_GICV3 && !NEW_VGIC
+
+config NEW_VGIC
+        bool
+        prompt "Use new VGIC implementation"
 
 config SBSA_VUART_CONSOLE
 	bool "Emulated SBSA UART console support"
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 41d7366527..2a3ec94a18 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -16,7 +16,6 @@ obj-y += domain_build.o
 obj-y += domctl.o
 obj-$(EARLY_PRINTK) += early_printk.o
 obj-y += gic.o
-obj-y += gic-vgic.o
 obj-y += gic-v2.o
 obj-$(CONFIG_HAS_GICV3) += gic-v3.o
 obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
@@ -47,10 +46,19 @@ obj-y += sysctl.o
 obj-y += time.o
 obj-y += traps.o
 obj-y += vcpreg.o
+ifeq ($(CONFIG_NEW_VGIC),y)
+obj-y += vgic/vgic.o
+obj-y += vgic/vgic-v2.o
+obj-y += vgic/vgic-mmio.o
+obj-y += vgic/vgic-mmio-v2.o
+obj-y += vgic/vgic-init.o
+else
+obj-y += gic-vgic.o
 obj-y += vgic.o
 obj-y += vgic-v2.o
 obj-$(CONFIG_HAS_GICV3) += vgic-v3.o
 obj-$(CONFIG_HAS_ITS) += vgic-v3-its.o
+endif
 obj-y += vm_event.o
 obj-y += vtimer.o
 obj-$(CONFIG_SBSA_VUART_CONSOLE) += vpl011.o
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 77fa756329..690ae892e0 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -853,6 +853,14 @@ int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
     return ret;
 }
 
+void vgic_v3_setup_hw(paddr_t dbase,
+                      unsigned int nr_rdist_regions,
+                      const struct rdist_region *regions,
+                      unsigned int intid_bits)
+{
+    /* Dummy implementation to allow building without actual vGICv3 support. */
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/common/Makefile b/xen/common/Makefile
index 3a349f478b..92a1d1fa58 100644
--- a/xen/common/Makefile
+++ b/xen/common/Makefile
@@ -19,6 +19,7 @@ obj-y += keyhandler.o
 obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_KEXEC) += kimage.o
 obj-y += lib.o
+obj-$(CONFIG_NEW_VGIC) += list_sort.o
 obj-$(CONFIG_LIVEPATCH) += livepatch.o livepatch_elf.o
 obj-y += lzo.o
 obj-$(CONFIG_HAS_MEM_ACCESS) += mem_access.o
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 00/49] New VGIC(-v2) implementation

[Andre Przywara]

Hi,

On 09/02/18 14:38, Andre Przywara wrote:
> tl;dr: More preparatory patches from patch 07, actual new VGIC starting
> at patch 20.
> =============
> 
> During development of the Dom0 ITS MSI support last year we realised
> that the existing GIC interrupt controller emulation has some shortcomings.
> After some tries to fix those in the existing code, it was agreed upon
> that the problems are fundamental and a new implementation based on the
> "new VGIC" in KVM is the best choice.
> This is the first drop of this new VGIC implementation.

Just in case you are *really* interested:
The code can also be found in the vgic-new/rfc branch here:
git://linux-arm.org/xen-ap.git
http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/vgic-new/rfc

Cheers,
Andre.

> It lives in the
> xen/arch/arm/vgic/ directory and is written to be a compile time option,
> so people can choose whether to use the new VGIC or the existing
> implementation. This is just for a transitional period, the old VGIC is
> expected to be removed after confidence in the new implementation has grown.
> 
> This series starts with some GICv3 redistributor cleanup, which I posted
> before. I need to incorporate the comments from the list, but for now I left
> those patches as it from the previous post.
> 
> Starting with patch 07 there are some more cleanups and preparations for
> the existing VGIC/GIC code. A big part of those patches are preparations to
> properly support level triggered interrupts. This is one of the biggest
> problems in the existing VGIC, which only correctly emulates edge triggered
> IRQs. This affects both arch code and some users like the timer and the
> event channel.
> 
> Starting with patch 20 we plumb in the new VGIC then. This is done in a
> new directory, with all the files actually not wired into the build system
> until the very last patch. The idea is to split the series into reviewable
> chunks without resorting to nasty hacks to keep bisectability.
> The code was forked from Linux' virt/kvm/arm/vgic/, as of 4.14-rc7, plus
> some recent changes to improve support for level triggered and hardware
> mapped interrupts, which is what we use heavily in Dom0. The code was
> heavily adapted to fit into Xen, starting with using the Xen coding style
> and using Xen structure and variable names (struct domain instead of
> struct kvm, for instance). Where interfacing functions were similar enough,
> they were changed over to the existing Xen name and prototypes (for instance
> kvm_vgic_create() was renamed to domain_vgic_register()). As far as possible
> the code layout and split was re-used from KVM, so patches in Linux should
> be relatively easy to port into Xen. Due to the mentioned changes this can
> not be done easily in an automatic way, but it should be not too complicated
> to extract the gist of the patch and re-apply this to our code base.
> 
> The actual VGIC code splits into several parts:
> - The core is the struct vgic_irq, which holds every information about a
> virtual IRQ, including a per-IRQ lock. Also there is on (ordered) per-VCPU
> list (ap_list), which links the interrupts to be considered by a VCPU.
> There are functions to deal with queuing and removing IRQs from those lists
> safely, obeying the locking order. (patches 20-23)
> - There are functions to push vIRQs on a VCPU list to the list registers,
> and handle their state changes. (patches 24-26)
> - The distributor MMIO emulation is using separate functions per register,
> also having read and write split. (patches 27-37)
> - There are functions to deal with Xen specialities. (patches 38-43)
> - The data structures and the wiring of the emulation into the hypervisor
>   and the guests are done in vgic-init.c. (patches 44-47)
> - Finally patch 49 enables the build of the new VGIC. This requires to
>   increase the size limit for struct vcpu in patch 48.
> 
> Please consider this series the first post that it is. It was briefly tested
> on a Juno and a Midway (with and without CONFIG_NEW_VGIC). It does not cover
> GICv3 yet, though supporting this should be relatively straightforward.
> In fact I have most of the code here already, but I wanted to wait for
> initial feedback before applying similar transformations to the *-v3.c files.
> Also this does not include ITS support, though the code itself is more ready
> for that than the old VGIC ever was.
> I did some semi-automatic translation of coding style, so I am sure there
> are quite some leftovers that slipped through, including identifiers with
> kvm_ in their name ;-)
> 
> I would appreciate if somehow could have a look at those patches, patch 20
> and following are surely the most interesting ones to have a look at.
> 
> Cheers,
> Andre
> 
> Andre Przywara (49):
>   tools: ARM: vGICv3: avoid inserting optional DT properties
>   ARM: vGICv3: drop GUEST_GICV3_RDIST_REGIONS symbol
>   ARM: GICv3: use hardware GICv3 redistributor regions for Dom0
>   ARM: GICv3: simplify GICv3 redistributor stride handling
>   ARM: vGICv3: always use architected redist stride
>   ARM: vGICv3: remove rdist_stride from VGIC structure
>   ARM: VGIC: move gic_remove_from_lr_pending() prototype
>   ARM: VGIC: move max_vcpus VGIC limit to struct arch_domain
>   ARM: VGIC: change to level-IRQ compatible IRQ injection interface
>   ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist
>   ARM: VGIC: reorder prototypes in vgic.h
>   ARM: VGIC: introduce gic_get_nr_lrs()
>   ARM: VGIC: Add hypervisor base address to vgic_v2_setup_hw()
>   ARM: VGIC: extend GIC CPU interface definitions
>   ARM: GIC: Allow tweaking the active state of an IRQ
>   ARM: GIC: allow reading pending state of a hardware IRQ
>   ARM: timer: Handle level triggered IRQs correctly
>   ARM: evtchn: Handle level triggered IRQs correctly
>   ARM: vPL011: Use the VGIC's level triggered IRQs handling if available
>   ARM: new VGIC: Add data structure definitions
>   ARM: new VGIC: Add acccessor to new struct vgic_irq instance
>   ARM: new VGIC: Implement virtual IRQ injection
>   ARM: new VGIC: Add IRQ sorting
>   ARM: new VGIC: Add IRQ sync/flush framework
>   ARM: new VGIC: Add GICv2 world switch backend
>   ARM: new VGIC: Implement vgic_vcpu_pending_irq
>   ARM: new VGIC: Add MMIO handling framework
>   ARM: new VGIC: Add GICv2 MMIO handling framework
>   ARM: new VGIC: Add CTLR, TYPER and IIDR handlers
>   ARM: new VGIC: Add ENABLE registers handlers
>   ARM: new VGIC: Add PENDING registers handlers
>   ARM: new VGIC: Add ACTIVE registers handlers
>   ARM: new VGIC: Add PRIORITY registers handlers
>   ARM: new VGIC: Add CONFIG registers handlers
>   ARM: new VGIC: Add TARGET registers handlers
>   ARM: new VGIC: Add SGIR register handler
>   ARM: new VGIC: Add SGIPENDR register handlers
>   ARM: new VGIC: handle hardware mapped IRQs
>   ARM: new VGIC: Add event channel IRQ handling
>   ARM: new VGIC: Handle virtual IRQ allocation/reservation
>   ARM: new VGIC: dump virtual IRQ info
>   ARM: new VGIC: provide system register emulation stub
>   ARM: new VGIC: Add preliminary stub implementations
>   ARM: new VGIC: vgic-init: register VGIC
>   ARM: new VGIC: vgic-init: implement vgic_init
>   ARM: new VGIC: vgic-init: implement map_resources
>   ARM: new VGIC: Add vgic_v2_enable
>   ARM: allocate two pages for struct vcpu
>   ARM: VGIC: wire new VGIC(-v2) files into Xen build system
> 
>  tools/libxl/libxl_arm.c           |   8 -
>  xen/arch/arm/Kconfig              |   6 +-
>  xen/arch/arm/Makefile             |  10 +-
>  xen/arch/arm/domain.c             |  25 +-
>  xen/arch/arm/gic-v2.c             |  20 +-
>  xen/arch/arm/gic-v3-lpi.c         |   2 +-
>  xen/arch/arm/gic-v3.c             |  72 ++--
>  xen/arch/arm/gic-vgic.c           |  10 +-
>  xen/arch/arm/gic.c                |  15 +
>  xen/arch/arm/irq.c                |   2 +-
>  xen/arch/arm/time.c               |  36 +-
>  xen/arch/arm/traps.c              |   2 +
>  xen/arch/arm/vgic-v2.c            |   3 +-
>  xen/arch/arm/vgic-v3.c            |  45 +-
>  xen/arch/arm/vgic.c               |  46 +-
>  xen/arch/arm/vgic/vgic-init.c     | 259 ++++++++++++
>  xen/arch/arm/vgic/vgic-mmio-v2.c  | 297 +++++++++++++
>  xen/arch/arm/vgic/vgic-mmio.c     | 589 ++++++++++++++++++++++++++
>  xen/arch/arm/vgic/vgic-mmio.h     | 194 +++++++++
>  xen/arch/arm/vgic/vgic-v2.c       | 339 +++++++++++++++
>  xen/arch/arm/vgic/vgic.c          | 871 ++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vgic/vgic.h          |  69 +++
>  xen/arch/arm/vpl011.c             |   6 +-
>  xen/arch/arm/vtimer.c             |   4 +-
>  xen/common/Makefile               |   1 +
>  xen/common/list_sort.c            | 170 ++++++++
>  xen/include/asm-arm/arm_vgic.h    | 269 ++++++++++++
>  xen/include/asm-arm/domain.h      |  92 +---
>  xen/include/asm-arm/event.h       |   1 +
>  xen/include/asm-arm/gic.h         |  31 +-
>  xen/include/asm-arm/gic_v3_defs.h |   5 +
>  xen/include/asm-arm/vgic.h        | 170 ++++++--
>  xen/include/public/arch-arm.h     |   3 -
>  xen/include/xen/list_sort.h       |  11 +
>  xen/include/xen/timer.h           |   2 +
>  35 files changed, 3456 insertions(+), 229 deletions(-)
>  create mode 100644 xen/arch/arm/vgic/vgic-init.c
>  create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c
>  create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
>  create mode 100644 xen/arch/arm/vgic/vgic-mmio.h
>  create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>  create mode 100644 xen/arch/arm/vgic/vgic.c
>  create mode 100644 xen/arch/arm/vgic/vgic.h
>  create mode 100644 xen/common/list_sort.c
>  create mode 100644 xen/include/asm-arm/arm_vgic.h
>  create mode 100644 xen/include/xen/list_sort.h
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 01/49] tools: ARM: vGICv3: avoid inserting optional DT properties

[Julien Grall]

Hi,

AFAICT, patch #1-#6 were sent separately and were reviewed. I will skip 
them for now, but please address the comments on the next version.

Cheers,

On 02/09/2018 02:38 PM, Andre Przywara wrote:
> When creating a GICv3 devicetree node, we currently insert the
> redistributor-stride and #redistributor-regions properties, with fixed
> values which are actually the architected ones. But those properties are
> optional and only needed to cover for broken platforms, where the values
> differ from the architected one. This will never be the case for the
> constructed DomU memory map.
> So we drop those properties altogether and provide a clean and architected
> GICv3 DT node for DomUs.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   tools/libxl/libxl_arm.c | 8 --------
>   1 file changed, 8 deletions(-)
> 
> diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
> index 3e46554301..b5bba3cd33 100644
> --- a/tools/libxl/libxl_arm.c
> +++ b/tools/libxl/libxl_arm.c
> @@ -524,14 +524,6 @@ static int make_gicv3_node(libxl__gc *gc, void *fdt)
>       res = fdt_property(fdt, "interrupt-controller", NULL, 0);
>       if (res) return res;
>   
> -    res = fdt_property_cell(fdt, "redistributor-stride",
> -                            GUEST_GICV3_RDIST_STRIDE);
> -    if (res) return res;
> -
> -    res = fdt_property_cell(fdt, "#redistributor-regions",
> -                            GUEST_GICV3_RDIST_REGIONS);
> -    if (res) return res;
> -
>       res = fdt_property_regs(gc, fdt, ROOT_ADDRESS_CELLS, ROOT_SIZE_CELLS,
>                               2,
>                               gicd_base, gicd_size,
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 07/49] ARM: VGIC: move gic_remove_from_lr_pending() prototype

[Julien Grall]

Hi,

On 02/09/2018 02:38 PM, Andre Przywara wrote:
> The prototype for gic_remove_from_lr_pending() is the last function in
> gic.h which references a VGIC data structure.
> Move it over to vgic.h, so that we can remove the inclusion of vgic.h
> from gic.h. We add it to asm/domain.h instead, where it is actually
> needed.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

> ---
>   xen/include/asm-arm/domain.h | 1 +
>   xen/include/asm-arm/gic.h    | 2 --
>   xen/include/asm-arm/vgic.h   | 1 +
>   3 files changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 3eda7196ff..1dd9683d25 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -8,6 +8,7 @@
>   #include <asm/vfp.h>
>   #include <asm/mmio.h>
>   #include <asm/gic.h>
> +#include <asm/vgic.h>
>   #include <public/hvm/params.h>
>   #include <xen/serial.h>
>   #include <xen/rbtree.h>
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 497f195bc1..1d382b0ade 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -156,7 +156,6 @@
>   #ifndef __ASSEMBLY__
>   #include <xen/device_tree.h>
>   #include <xen/irq.h>
> -#include <asm-arm/vgic.h>
>   
>   #define DT_COMPAT_GIC_CORTEX_A15 "arm,cortex-a15-gic"
>   
> @@ -245,7 +244,6 @@ extern void init_maintenance_interrupt(void);
>   extern void gic_raise_guest_irq(struct vcpu *v, unsigned int irq,
>           unsigned int priority);
>   extern void gic_raise_inflight_irq(struct vcpu *v, unsigned int virtual_irq);
> -extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
>   
>   /* Accept an interrupt from the GIC and dispatch its handler */
>   extern void gic_interrupt(struct cpu_user_regs *regs, int is_fiq);
> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index d61b54867b..d03298e12c 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -205,6 +205,7 @@ extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
>   extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
>   extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
>   extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
> +extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
>   extern void vgic_clear_pending_irqs(struct vcpu *v);
>   extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
>   extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 08/49] ARM: VGIC: move max_vcpus VGIC limit to struct arch_domain

[Julien Grall]

Hi,

On 02/09/2018 02:38 PM, Andre Przywara wrote:
> The VGIC model used for a domain (GICv2 or GICv3) determines the maximum
> number of VCPUs for that guest, as GICv2 can only handle 8 processors.
> In the moment we carry this per-VGIC-model limit in the vgic_ops,
> alongside the model specific functions. That makes some sense, but
> exposes some current VGIC implementation details to generic Xen code.
> Add a new arch specific field in our domain structure to hold this vcpu limit,
> and initialize it when we set the ops. This allows us to plug in the new
> VGIC later without also needing to carry some ops structure.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/domain.c        | 5 ++---
>   xen/arch/arm/vgic.c          | 3 ++-
>   xen/include/asm-arm/domain.h | 1 +
>   3 files changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index a010443bfd..9ad4cd0a6e 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -975,11 +975,10 @@ unsigned int domain_max_vcpus(const struct domain *d)
>        * allocation when the vgic_ops haven't been initialised yet,
>        * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
>        */
> -    if ( !d->arch.vgic.handler )
> +    if ( !d->arch.max_vcpus )
>           return MAX_VIRT_CPUS;
>       else
> -        return min_t(unsigned int, MAX_VIRT_CPUS,
> -                     d->arch.vgic.handler->max_vcpus);
> +        return d->arch.max_vcpus;
>   }
>   
>   /*
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 9921769b15..5f47aa84a9 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -166,7 +166,8 @@ int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>   
>   void register_vgic_ops(struct domain *d, const struct vgic_ops *ops)
>   {
> -   d->arch.vgic.handler = ops;
> +    d->arch.vgic.handler = ops;
> +    d->arch.max_vcpus = ops->max_vcpus;
>   }
>   
>   void domain_vgic_free(struct domain *d)
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 1dd9683d25..2fef32eaee 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -149,6 +149,7 @@ struct arch_domain
>   #ifdef CONFIG_SBSA_VUART_CONSOLE
>       struct vpl011 vpl011;
>   #endif
> +    unsigned int max_vcpus;

Now, you have max_vcpus defined in both arch_domain and domain. Which 
makes the code very confusing to read. This is becoming apparent in the 
check if (d->arch.max_vcpus > d->max_vcpus).

If you plan to ditch the ops, then I would prefer a check on the vGIC 
version. Even if it means carrying vGIC specific implementation in 
generic Xen code.

This would also avoid to grow the struct domain just for a "constant 
field" used mostly at guest creation.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 09/49] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Julien Grall]

Hi Andre,

On 09/02/18 14:38, Andre Przywara wrote:
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 5f47aa84a9..2fc6e19625 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -285,7 +285,7 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
>           vgic_remove_irq_from_queues(old, p);
>           irq_set_affinity(p->desc, cpumask_of(new->processor));
>           spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
> -        vgic_vcpu_inject_irq(new, irq);
> +        vgic_inject_irq(new->domain, new, irq, true);
>           return true;
>       }
>       /* if the IRQ is in a GICH_LR register, set GIC_IRQ_GUEST_MIGRATING
> @@ -444,7 +444,7 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
>                           sgir, target->list);
>                   continue;
>               }
> -            vgic_vcpu_inject_irq(d->vcpu[vcpuid], virq);
> +            vgic_inject_irq(d, d->vcpu[vcpuid], virq, true);
>           }
>           break;
>       case SGI_TARGET_OTHERS:
> @@ -453,12 +453,12 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
>           {
>               if ( i != current->vcpu_id && d->vcpu[i] != NULL &&
>                    is_vcpu_online(d->vcpu[i]) )
> -                vgic_vcpu_inject_irq(d->vcpu[i], virq);
> +                vgic_inject_irq(d, d->vcpu[i], virq, true);
>           }
>           break;
>       case SGI_TARGET_SELF:
>           perfc_incr(vgic_sgi_self);
> -        vgic_vcpu_inject_irq(d->vcpu[current->vcpu_id], virq);
> +        vgic_inject_irq(d, current, virq, true);
>           break;
>       default:
>           gprintk(XENLOG_WARNING,
> @@ -518,13 +518,29 @@ void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p)
>       gic_remove_from_lr_pending(v, p);
>   }
>   
> -void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
> +int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
> +                    bool level)

Looking at the code after the series has been applied, no one is caring 
about the return value of vgic_inject_irq. So what is the rationale 
behind changing the return type from void to int?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 10/49] ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist

[Julien Grall]

Hi Andre,

On 09/02/18 14:38, Andre Przywara wrote:
> Currently we describe the VGIC specific fields in an structure
> *embedded* in struct arch_domain and struct arch_vcpu. These members
> there are however related to the current VGIC implementation, and will
> be substantially different in the future.
> To allow coexistence of two implementations, move the definition of these
> embedded structures into vgic.h, and just use the opaque type in the arch
> specific structures.
> This allows easy switching between different implementations later.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
>   xen/include/asm-arm/domain.h | 85 +-----------------------------------------
>   xen/include/asm-arm/vgic.h   | 88 ++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 90 insertions(+), 83 deletions(-)
> 
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 2fef32eaee..968ffb0c81 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -74,57 +74,7 @@ struct arch_domain
>           uint64_t offset;
>       } virt_timer_base;
>   
> -    struct {
> -        /* Version of the vGIC */
> -        enum gic_version version;
> -        /* GIC HW version specific vGIC driver handler */
> -        const struct vgic_ops *handler;
> -        /*
> -         * Covers access to other members of this struct _except_ for
> -         * shared_irqs where each member contains its own locking.
> -         *
> -         * If both class of lock is required then this lock must be
> -         * taken first. If multiple rank locks are required (including
> -         * the per-vcpu private_irqs rank) then they must be taken in
> -         * rank order.
> -         */
> -        spinlock_t lock;
> -        uint32_t ctlr;
> -        int nr_spis; /* Number of SPIs */
> -        unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
> -        struct vgic_irq_rank *shared_irqs;
> -        /*
> -         * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> -         * struct arch_vcpu.
> -         */
> -        struct pending_irq *pending_irqs;
> -        /* Base address for guest GIC */
> -        paddr_t dbase; /* Distributor base address */
> -#ifdef CONFIG_HAS_GICV3
> -        /* GIC V3 addressing */
> -        /* List of contiguous occupied by the redistributors */
> -        struct vgic_rdist_region {
> -            paddr_t base;                   /* Base address */
> -            paddr_t size;                   /* Size */
> -            unsigned int first_cpu;         /* First CPU handled */
> -        } *rdist_regions;
> -        int nr_regions;                     /* Number of rdist regions */
> -        unsigned long int nr_lpis;
> -        uint64_t rdist_propbase;
> -        struct rb_root its_devices;         /* Devices mapped to an ITS */
> -        spinlock_t its_devices_lock;        /* Protects the its_devices tree */
> -        struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
> -        rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
> -        struct list_head vits_list;         /* List of virtual ITSes */
> -        unsigned int intid_bits;
> -        /*
> -         * TODO: if there are more bool's being added below, consider
> -         * a flags variable instead.
> -         */
> -        bool rdists_enabled;                /* Is any redistributor enabled? */
> -        bool has_its;
> -#endif
> -    } vgic;
> +    struct vgic_dist vgic;
>   
>       struct vuart {
>   #define VUART_BUF_SIZE 128
> @@ -248,38 +198,7 @@ struct arch_vcpu
>       union gic_state_data gic;
>       uint64_t lr_mask;
>   
> -    struct {
> -        /*
> -         * SGIs and PPIs are per-VCPU, SPIs are domain global and in
> -         * struct arch_domain.
> -         */
> -        struct pending_irq pending_irqs[32];
> -        struct vgic_irq_rank *private_irqs;
> -
> -        /* This list is ordered by IRQ priority and it is used to keep
> -         * track of the IRQs that the VGIC injected into the guest.
> -         * Depending on the availability of LR registers, the IRQs might
> -         * actually be in an LR, and therefore injected into the guest,
> -         * or queued in gic.lr_pending.
> -         * As soon as an IRQ is EOI'd by the guest and removed from the
> -         * corresponding LR it is also removed from this list. */
> -        struct list_head inflight_irqs;
> -        /* lr_pending is used to queue IRQs (struct pending_irq) that the
> -         * vgic tried to inject in the guest (calling gic_set_guest_irq) but
> -         * no LRs were available at the time.
> -         * As soon as an LR is freed we remove the first IRQ from this
> -         * list and write it to the LR register.
> -         * lr_pending is a subset of vgic.inflight_irqs. */
> -        struct list_head lr_pending;
> -        spinlock_t lock;
> -
> -        /* GICv3: redistributor base and flags for this vCPU */
> -        paddr_t rdist_base;
> -        uint64_t rdist_pendbase;
> -#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
> -#define VGIC_V3_LPIS_ENABLED    (1 << 1)
> -        uint8_t flags;
> -    } vgic;
> +    struct vgic_cpu vgic;
>   
>       /* Timer registers  */
>       uint32_t cntkctl;
> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index b75fdeb068..4e1c37f091 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -19,6 +19,9 @@
>   #define __ASM_ARM_VGIC_H__
>   
>   #include <xen/bitops.h>
> +#include <xen/radix-tree.h>
> +#include <xen/rbtree.h>
> +#include <asm/gic.h>
>   #include <asm/mmio.h>
>   #include <asm/vreg.h>
>   
> @@ -123,6 +126,91 @@ struct vgic_irq_rank {
>       uint8_t vcpu[32];
>   };
>   
> +struct vgic_dist {
> +    /* Version of the vGIC */
> +    enum gic_version version;
> +    /* GIC HW version specific vGIC driver handler */
> +    const struct vgic_ops *handler;
> +    /*
> +     * Covers access to other members of this struct _except_ for
> +     * shared_irqs where each member contains its own locking.
> +     *
> +     * If both class of lock is required then this lock must be
> +     * taken first. If multiple rank locks are required (including
> +     * the per-vcpu private_irqs rank) then they must be taken in
> +     * rank order.
> +     */
> +    spinlock_t lock;
> +    uint32_t ctlr;
> +    int nr_spis; /* Number of SPIs */
> +    unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
> +    struct vgic_irq_rank *shared_irqs;
> +    /*
> +     * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> +     * struct arch_vcpu.
> +     */
> +    struct pending_irq *pending_irqs;
> +    /* Base address for guest GIC */
> +    paddr_t dbase; /* Distributor base address */
> +#ifdef CONFIG_HAS_GICV3
> +    /* GIC V3 addressing */
> +    /* List of contiguous occupied by the redistributors */
> +    struct vgic_rdist_region {
> +        paddr_t base;                   /* Base address */
> +        paddr_t size;                   /* Size */
> +        unsigned int first_cpu;         /* First CPU handled */
> +    } *rdist_regions;
> +    int nr_regions;                     /* Number of rdist regions */
> +    unsigned long int nr_lpis;
> +    uint64_t rdist_propbase;
> +    struct rb_root its_devices;         /* Devices mapped to an ITS */
> +    spinlock_t its_devices_lock;        /* Protects the its_devices tree */
> +    struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
> +    rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
> +    struct list_head vits_list;         /* List of virtual ITSes */
> +    unsigned int intid_bits;
> +    /*
> +     * TODO: if there are more bool's being added below, consider
> +     * a flags variable instead.
> +     */
> +    bool rdists_enabled;                /* Is any redistributor enabled? */
> +    bool has_its;
> +#endif
> +};
> +
> +struct vgic_cpu {
> +    /*
> +     * SGIs and PPIs are per-VCPU, SPIs are domain global and in
> +     * struct arch_domain.
> +     */
> +    struct pending_irq pending_irqs[32];
> +    struct vgic_irq_rank *private_irqs;
> +
> +    /* This list is ordered by IRQ priority and it is used to keep
> +     * track of the IRQs that the VGIC injected into the guest.
> +     * Depending on the availability of LR registers, the IRQs might
> +     * actually be in an LR, and therefore injected into the guest,
> +     * or queued in gic.lr_pending.
> +     * As soon as an IRQ is EOI'd by the guest and removed from the
> +     * corresponding LR it is also removed from this list. */
> +    struct list_head inflight_irqs;
> +    /* lr_pending is used to queue IRQs (struct pending_irq) that the
> +     * vgic tried to inject in the guest (calling gic_set_guest_irq) but
> +     * no LRs were available at the time.
> +     * As soon as an LR is freed we remove the first IRQ from this
> +     * list and write it to the LR register.
> +     * lr_pending is a subset of vgic.inflight_irqs. */
> +    struct list_head lr_pending;
> +    spinlock_t lock;
> +
> +    /* GICv3: redistributor base and flags for this vCPU */
> +    paddr_t rdist_base;
> +    uint64_t rdist_pendbase;
> +#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
> +#define VGIC_V3_LPIS_ENABLED    (1 << 1)
> +    uint8_t flags;
> +};
> +
>   struct sgi_target {
>       uint8_t aff1;
>       uint16_t list;
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 00/49] New VGIC(-v2) implementation

[Julien Grall]

On 09/02/18 15:06, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 09/02/18 14:38, Andre Przywara wrote:
>> tl;dr: More preparatory patches from patch 07, actual new VGIC starting
>> at patch 20.
>> =============
>>
>> During development of the Dom0 ITS MSI support last year we realised
>> that the existing GIC interrupt controller emulation has some shortcomings.
>> After some tries to fix those in the existing code, it was agreed upon
>> that the problems are fundamental and a new implementation based on the
>> "new VGIC" in KVM is the best choice.
>> This is the first drop of this new VGIC implementation.
> 
> Just in case you are *really* interested:
> The code can also be found in the vgic-new/rfc branch here:
> git://linux-arm.org/xen-ap.git
> http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/vgic-new/rfc

This branch seems to contain more than this series. Anything we should 
care in it?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 00/49] New VGIC(-v2) implementation

[Andre Przywara]

Hi,

On 12/02/18 11:48, Julien Grall wrote:
> 
> 
> On 09/02/18 15:06, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 09/02/18 14:38, Andre Przywara wrote:
>>> tl;dr: More preparatory patches from patch 07, actual new VGIC starting
>>> at patch 20.
>>> =============
>>>
>>> During development of the Dom0 ITS MSI support last year we realised
>>> that the existing GIC interrupt controller emulation has some
>>> shortcomings.
>>> After some tries to fix those in the existing code, it was agreed upon
>>> that the problems are fundamental and a new implementation based on the
>>> "new VGIC" in KVM is the best choice.
>>> This is the first drop of this new VGIC implementation.
>>
>> Just in case you are *really* interested:
>> The code can also be found in the vgic-new/rfc branch here:
>> git://linux-arm.org/xen-ap.git
>> http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/vgic-new/rfc
>>
> 
> This branch seems to contain more than this series. Anything we should
> care in it?

Oh dear, this was my development branch. Thanks for the heads up! I just
trimmed it to match the series I posted.
Please, re-fetch it now and then go wash your hands in case you looked
at any of the top patches ;-)

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 11/49] ARM: VGIC: reorder prototypes in vgic.h

[Julien Grall]

Hi Andre,

On 09/02/18 14:38, Andre Przywara wrote:
>   /*
> - * Allocate a guest VIRQ
> - *  - spi == 0 => allocate a PPI. It will be the same on every vCPU
> - *  - spi == 1 => allocate an SPI
> + * In the moment vgic_num_irqs() just covers SPIs and the private IRQs,
> + * as it's mostly used for allocating the pending_irq and irq_desc array,
> + * in which LPIs don't participate.
>    */
> -extern int vgic_allocate_virq(struct domain *d, bool spi);
> +#define vgic_num_irqs(d)        ((d)->arch.vgic.nr_spis + 32)
>   
> +/*
> + * Allocate a guest VIRQ
> + *  - is_spi == 0 => allocate a PPI. It will be the same on every vCPU
> + *  - is_spi == 1 => allocate an SPI
> + */
> +extern int vgic_allocate_virq(struct domain *d, bool is_spi);
> +/* Reserve a specific guest vIRQ */
> +extern bool vgic_reserve_virq(struct domain *d, unsigned int virq);
> +extern void vgic_free_virq(struct domain *d, unsigned int virq);

newline here please.

Otherwise the split looks good:

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

>   static inline int vgic_allocate_ppi(struct domain *d)
>   {
>       return vgic_allocate_virq(d, false /* ppi */);
> @@ -340,18 +340,21 @@ static inline int vgic_allocate_spi(struct domain *d)
>       return vgic_allocate_virq(d, true /* spi */);
>   }
>   
> -extern void vgic_free_virq(struct domain *d, unsigned int virq);
> +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
> +                                      unsigned int virq);
> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
> +                        struct irq_desc *desc, bool connect);
>   
> -void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
> -                      paddr_t vbase, uint32_t aliased_offset);
> +bool vgic_evtchn_irq_pending(struct vcpu *v);
>   
> -#ifdef CONFIG_HAS_GICV3
> -struct rdist_region;
> -void vgic_v3_setup_hw(paddr_t dbase,
> -                      unsigned int nr_rdist_regions,
> -                      const struct rdist_region *regions,
> -                      unsigned int intid_bits);
> -#endif
> +int domain_vgic_register(struct domain *d, int *mmio_count);
> +int domain_vgic_init(struct domain *d, unsigned int nr_spis);
> +void domain_vgic_free(struct domain *d);
> +int vcpu_vgic_init(struct vcpu *vcpu);
> +int vcpu_vgic_free(struct vcpu *vcpu);
> +
> +int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
> +                    bool level);
>   
>   #endif /* __ASM_ARM_VGIC_H__ */
>   
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 12/49] ARM: VGIC: introduce gic_get_nr_lrs()

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> So far the number of list registers (LRs) a GIC implements is only
> needed in the hardware facing side of the VGIC code (gic-vgic.c).
> The new VGIC will need this information in more and multiple places, so
> export a function that returns the number.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/gic-vgic.c   | 10 +++++-----
>   xen/arch/arm/gic.c        |  5 +++++
>   xen/include/asm-arm/gic.h |  1 +
>   3 files changed, 11 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
> index d273863556..c92626e4ee 100644
> --- a/xen/arch/arm/gic-vgic.c
> +++ b/xen/arch/arm/gic-vgic.c
> @@ -25,7 +25,7 @@
>   #include <asm/gic.h>
>   #include <asm/vgic.h>
>   
> -#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->info->nr_lrs) - 1))
> +#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_get_nr_lrs()) - 1))
>   
>   #undef GIC_DEBUG
>   
> @@ -110,7 +110,7 @@ static unsigned int gic_find_unused_lr(struct vcpu *v,
>                                          struct pending_irq *p,
>                                          unsigned int lr)
>   {
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>       unsigned long *lr_mask = (unsigned long *) &this_cpu(lr_mask);
>       struct gic_lr lr_val;
>   
> @@ -137,7 +137,7 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq,
>           unsigned int priority)
>   {
>       int i;
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>       struct pending_irq *p = irq_to_pending(v, virtual_irq);
>   
>       ASSERT(spin_is_locked(&v->arch.vgic.lock));
> @@ -251,7 +251,7 @@ void gic_clear_lrs(struct vcpu *v)
>   {
>       int i = 0;
>       unsigned long flags;
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>   
>       /* The idle domain has no LRs to be cleared. Since gic_restore_state
>        * doesn't write any LR registers for the idle domain they could be
> @@ -278,7 +278,7 @@ static void gic_restore_pending_irqs(struct vcpu *v)
>       struct pending_irq *p, *t, *p_r;
>       struct list_head *inflight_r;
>       unsigned long flags;
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>       int lrs = nr_lrs;
>   
>       spin_lock_irqsave(&v->arch.vgic.lock, flags);
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 968e46fabb..89873c1df4 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -47,6 +47,11 @@ void register_gic_ops(const struct gic_hw_operations *ops)
>       gic_hw_ops = ops;
>   }
>   
> +int gic_get_nr_lrs(void)

unsigned int here please.

Also, given that gic_hw_ops is exported in gic.h, it would make sense to 
make that helper static inline in gic.h.

> +{
> +    return gic_hw_ops->info->nr_lrs;
> +}
> +
>   static void clear_cpu_lr_mask(void)
>   {
>       this_cpu(lr_mask) = 0ULL;
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 1d382b0ade..c1f027d703 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -222,6 +222,7 @@ enum gic_version {
>   DECLARE_PER_CPU(uint64_t, lr_mask);
>   
>   extern enum gic_version gic_hw_version(void);
> +extern int gic_get_nr_lrs(void);
>   
>   /* Program the IRQ type into the GIC */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 09/49] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Andre Przywara]

Hi,

On 12/02/18 11:15, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:38, Andre Przywara wrote:
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 5f47aa84a9..2fc6e19625 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -285,7 +285,7 @@ bool vgic_migrate_irq(struct vcpu *old, struct
>> vcpu *new, unsigned int irq)
>>           vgic_remove_irq_from_queues(old, p);
>>           irq_set_affinity(p->desc, cpumask_of(new->processor));
>>           spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
>> -        vgic_vcpu_inject_irq(new, irq);
>> +        vgic_inject_irq(new->domain, new, irq, true);
>>           return true;
>>       }
>>       /* if the IRQ is in a GICH_LR register, set GIC_IRQ_GUEST_MIGRATING
>> @@ -444,7 +444,7 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir,
>> enum gic_sgi_mode irqmode,
>>                           sgir, target->list);
>>                   continue;
>>               }
>> -            vgic_vcpu_inject_irq(d->vcpu[vcpuid], virq);
>> +            vgic_inject_irq(d, d->vcpu[vcpuid], virq, true);
>>           }
>>           break;
>>       case SGI_TARGET_OTHERS:
>> @@ -453,12 +453,12 @@ bool vgic_to_sgi(struct vcpu *v, register_t
>> sgir, enum gic_sgi_mode irqmode,
>>           {
>>               if ( i != current->vcpu_id && d->vcpu[i] != NULL &&
>>                    is_vcpu_online(d->vcpu[i]) )
>> -                vgic_vcpu_inject_irq(d->vcpu[i], virq);
>> +                vgic_inject_irq(d, d->vcpu[i], virq, true);
>>           }
>>           break;
>>       case SGI_TARGET_SELF:
>>           perfc_incr(vgic_sgi_self);
>> -        vgic_vcpu_inject_irq(d->vcpu[current->vcpu_id], virq);
>> +        vgic_inject_irq(d, current, virq, true);
>>           break;
>>       default:
>>           gprintk(XENLOG_WARNING,
>> @@ -518,13 +518,29 @@ void vgic_remove_irq_from_queues(struct vcpu *v,
>> struct pending_irq *p)
>>       gic_remove_from_lr_pending(v, p);
>>   }
>>   -void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>> +int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
>> +                    bool level)
> 
> Looking at the code after the series has been applied, no one is caring
> about the return value of vgic_inject_irq. So what is the rationale
> behind changing the return type from void to int?

The KVM version returns an error value, in particular when:
- the VGIC has not been initialized yet
- we can't determine the VCPU for a private interrupt
- the interrupt ID is invalid (SPI beyond limit, not mapped LPI)
In the moment it's not very useful for Xen: the first two conditions
don't really happen, consequently I removed those checks. But the third
check may become interesting once we get LPIs. Also since Xen currently
uses a void prototype for injection, *this* patch *now* doesn't exploit
the newly gained possibility of properly handling errors. From briefly
checking all the users, all of them seem to be in void functions, so
indeed an error return is not very useful.
The reasons I kept it in was to allow introduction of checks later. I
think having a function returning an error where some users ignore this
is better than the other way round.

So of course I can easily make this void, but I wonder what we do in
those cases where the SPI is not valid, for instance? Shall we print
some (rate-limited) warning?

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 13/49] ARM: VGIC: Add hypervisor base address to vgic_v2_setup_hw()

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The new VGIC will need to know the hypervisor base address at some
> point, which is private to the hardware facing part of the VGIC so far.
> Add a parameter to vgic_v2_setup_hw() to pass this address on, so a VGIC
> implementation can make use of it.
> The current VGIC ignores this new parameter.
> 
> TODO: add proper value for GICv2 on GICv3 emulation!
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/gic-v2.c      | 3 ++-
>   xen/arch/arm/gic-v3.c      | 3 ++-
>   xen/arch/arm/vgic-v2.c     | 3 ++-
>   xen/include/asm-arm/vgic.h | 3 ++-
>   4 files changed, 8 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 2b271ba322..7a18abecfa 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -1207,7 +1207,8 @@ static int __init gicv2_init(void)
>       if ( !gicv2.map_hbase )
>           panic("GICv2: Failed to ioremap for GIC Virtual interface\n");
>   
> -    vgic_v2_setup_hw(dbase, cbase, csize, vbase, aliased_offset);
> +    vgic_v2_setup_hw(dbase, cbase, csize, vbase, gicv2.map_hbase,
> +                     aliased_offset);
>   
>       /* Global settings: interrupt distributor */
>       spin_lock_init(&gicv2.lock);
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index ea14ab4028..08d4703687 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -1238,7 +1238,8 @@ static void __init gicv3_init_v2(void)
>       printk("GICv3 compatible with GICv2 cbase %#"PRIpaddr" vbase %#"PRIpaddr"\n",
>              cbase, vbase);
>   
> -    vgic_v2_setup_hw(dbase, cbase, csize, vbase, 0);
> +    /* TODO: provide the proper HBASE address! */

Well, on GICv3 the hypervisor interface will be configured using system 
registers. So I am not sure how your new interface is going to work with 
GICv3.

But IHMO, this is breaking the spirit of this interface. The goal is to 
tell "we can support a virtual GICv2". How the LRs (or anything touching 
the hardware) should be done using gic_hw_ops.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 09/49] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Julien Grall]

On 12/02/18 11:59, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 12/02/18 11:15, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:38, Andre Przywara wrote:
>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>> index 5f47aa84a9..2fc6e19625 100644
>>> --- a/xen/arch/arm/vgic.c
>>> +++ b/xen/arch/arm/vgic.c
>>> @@ -285,7 +285,7 @@ bool vgic_migrate_irq(struct vcpu *old, struct
>>> vcpu *new, unsigned int irq)
>>>            vgic_remove_irq_from_queues(old, p);
>>>            irq_set_affinity(p->desc, cpumask_of(new->processor));
>>>            spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
>>> -        vgic_vcpu_inject_irq(new, irq);
>>> +        vgic_inject_irq(new->domain, new, irq, true);
>>>            return true;
>>>        }
>>>        /* if the IRQ is in a GICH_LR register, set GIC_IRQ_GUEST_MIGRATING
>>> @@ -444,7 +444,7 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir,
>>> enum gic_sgi_mode irqmode,
>>>                            sgir, target->list);
>>>                    continue;
>>>                }
>>> -            vgic_vcpu_inject_irq(d->vcpu[vcpuid], virq);
>>> +            vgic_inject_irq(d, d->vcpu[vcpuid], virq, true);
>>>            }
>>>            break;
>>>        case SGI_TARGET_OTHERS:
>>> @@ -453,12 +453,12 @@ bool vgic_to_sgi(struct vcpu *v, register_t
>>> sgir, enum gic_sgi_mode irqmode,
>>>            {
>>>                if ( i != current->vcpu_id && d->vcpu[i] != NULL &&
>>>                     is_vcpu_online(d->vcpu[i]) )
>>> -                vgic_vcpu_inject_irq(d->vcpu[i], virq);
>>> +                vgic_inject_irq(d, d->vcpu[i], virq, true);
>>>            }
>>>            break;
>>>        case SGI_TARGET_SELF:
>>>            perfc_incr(vgic_sgi_self);
>>> -        vgic_vcpu_inject_irq(d->vcpu[current->vcpu_id], virq);
>>> +        vgic_inject_irq(d, current, virq, true);
>>>            break;
>>>        default:
>>>            gprintk(XENLOG_WARNING,
>>> @@ -518,13 +518,29 @@ void vgic_remove_irq_from_queues(struct vcpu *v,
>>> struct pending_irq *p)
>>>        gic_remove_from_lr_pending(v, p);
>>>    }
>>>    -void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>>> +int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
>>> +                    bool level)
>>
>> Looking at the code after the series has been applied, no one is caring
>> about the return value of vgic_inject_irq. So what is the rationale
>> behind changing the return type from void to int?
> 
> The KVM version returns an error value, in particular when:
> - the VGIC has not been initialized yet
> - we can't determine the VCPU for a private interrupt
> - the interrupt ID is invalid (SPI beyond limit, not mapped LPI)
> In the moment it's not very useful for Xen: the first two conditions
> don't really happen, consequently I removed those checks. But the third
> check may become interesting once we get LPIs. Also since Xen currently
> uses a void prototype for injection, *this* patch *now* doesn't exploit
> the newly gained possibility of properly handling errors. From briefly
> checking all the users, all of them seem to be in void functions, so
> indeed an error return is not very useful.
> The reasons I kept it in was to allow introduction of checks later. I
> think having a function returning an error where some users ignore this
> is better than the other way round.

I don't think it is much better. This is a way to expose yet another 
security issue because the return is not correctly checked (see XSA-246 
for instance). Any return value should be checked or have a comment 
explaining why it is fine.

> 
> So of course I can easily make this void, but I wonder what we do in
> those cases where the SPI is not valid, for instance? Shall we print
> some (rate-limited) warning?

I can understand why KVM needs such interface as the interrupt 
controller may be emulated QEMU. But I can't see why a SPI would not be 
valid in Xen context (except programming error). So could give an example?

What would you expect the caller to do on error? Except printing an 
error message?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 14/49] ARM: VGIC: extend GIC CPU interface definitions

[Julien Grall]

Hi,

This patch seem to modify the GICv2 CPU interface definitions. If so, 
please make it clear in the commit message/title.

On 09/02/18 14:39, Andre Przywara wrote:
> The new VGIC will shortly use more bits of the GICC_CTLR register, so
> add the respective definitions from the manual.
> Also add a missing definition for GICV_PMR_PRIORITY_MASK.
You also add GICC_ABPR here.

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/gic-v2.c     |  2 +-
>   xen/include/asm-arm/gic.h | 18 ++++++++++++++++--
>   2 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 7a18abecfa..2e35892881 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -358,7 +358,7 @@ static void gicv2_cpu_init(void)
>       /* Finest granularity of priority */
>       writel_gicc(0x0, GICC_BPR);
>       /* Turn on delivery */
> -    writel_gicc(GICC_CTL_ENABLE|GICC_CTL_EOI, GICC_CTLR);
> +    writel_gicc(GICC_CTL_ENABLE0|GICC_CTL_EOI, GICC_CTLR);
>   }
>   
>   static void gicv2_cpu_disable(void)
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index c1f027d703..c4c68c7770 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -77,6 +77,7 @@
>   #define GICC_EOIR       (0x0010)
>   #define GICC_RPR        (0x0014)
>   #define GICC_HPPIR      (0x0018)
> +#define GICC_ABPR       (0x001c)
>   #define GICC_APR        (0x00D0)
>   #define GICC_NSAPR      (0x00E0)
>   #define GICC_IIDR       (0x00FC)
> @@ -102,8 +103,18 @@
>   #define GICD_TYPE_SEC   0x400
>   #define GICD_TYPER_DVIS (1U << 18)
>   
> -#define GICC_CTL_ENABLE 0x1
> -#define GICC_CTL_EOI    (0x1 << 9)
> +#define GICC_CTL_ENABLE0_SHIFT  0
> +#define GICC_CTL_ENABLE0        (1U << GICC_CTL_ENABLE0_SHIFT)

I guess GICCC_CTLR_ENABLE is renamed to GICC_CTL_ENABLE0 to match the 
spec. If so, please mention it in the commit message.

> +#define GICC_CTL_ENABLE1_SHIFT  1
> +#define GICC_CTL_ENABLE1        (1U << GICC_CTL_ENABLE1)
> +#define GICC_CTL_AC_SHIFT       2
> +#define GICC_CTL_AC             (1U << GICC_CTL_AC_SHIFT)
> +#define GICC_CTL_FIQEN_SHIFT    3
> +#define GICC_CTL_FIQEN          (1U << GICC_CTL_FIQEN_SHIFT)
> +#define GICC_CTL_CBPR_SHIFT     4
> +#define GICC_CTL_CBPR           (1U << GICC_CTL_CBPR_SHIFT)
> +#define GICC_CTL_EOI_SHIFT      9
> +#define GICC_CTL_EOI            (1U << GICC_CTL_EOI_SHIFT)
>   
>   #define GICC_IA_IRQ       0x03ff
>   #define GICC_IA_CPU_MASK  0x1c00
> @@ -127,6 +138,9 @@
>   #define GICH_MISR_VGRP1E  (1 << 6)
>   #define GICH_MISR_VGRP1D  (1 << 7)
>   
> +#define GICV_PMR_PRIORITY_SHIFT		3
> +#define GICV_PMR_PRIORITY_MASK		(0x1f << GICV_PMR_PRIORITY_SHIFT)
> +
>   /*
>    * The minimum GICC_BPR is required to be in the range 0-3. We set
>    * GICC_BPR to 0 but we must expect that it might be 3. This means we
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 15/49] ARM: GIC: Allow tweaking the active state of an IRQ

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> When playing around with hardware mapped, level triggered virtual IRQs,
> there is the need to explicitly set the active state of an interrupt at
> some point in time.
> To prepare the GIC for that, we introduce a set_active_state() function
> to let the VGIC manipulate the state of an associated hardware IRQ.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/gic-v2.c     |  9 +++++++++
>   xen/arch/arm/gic-v3.c     | 16 ++++++++++++++++
>   xen/arch/arm/gic.c        |  5 +++++
>   xen/include/asm-arm/gic.h |  5 +++++
>   4 files changed, 35 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 2e35892881..5339f69fbc 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -235,6 +235,14 @@ static unsigned int gicv2_read_irq(void)
>       return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>   }
>   
> +static void gicv2_set_active_state(int irq, bool active)

I would much prefer to have an irq_desc in parameter. This is matching 
the other interface and you could update the flags such as 
_IRQ_INPROGRESS which you don't do at the moment.

Also, who is preventing two CPUs to clear the active bit at the same time?

> +{
> +    if (active)
> +        writel_gicd(1U << (irq % 32), GICD_ISACTIVER + (irq / 32) * 4);
> +    else
> +        writel_gicd(1U << (irq % 32), GICD_ICACTIVER + (irq / 32) * 4);

You will have a few places in the code usually similar construct. It 
would make sense to introduce a helper poke as we have in the GICv3 code.

> +}
> +
>   static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int type)
>   {
>       uint32_t cfg, actual, edgebit;
> @@ -1241,6 +1249,7 @@ const static struct gic_hw_operations gicv2_ops = {
>       .eoi_irq             = gicv2_eoi_irq,
>       .deactivate_irq      = gicv2_dir_irq,
>       .read_irq            = gicv2_read_irq,
> +    .set_active_state    = gicv2_set_active_state,
>       .set_irq_type        = gicv2_set_irq_type,
>       .set_irq_priority    = gicv2_set_irq_priority,
>       .send_SGI            = gicv2_send_SGI,
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 08d4703687..595eaef43a 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -475,6 +475,21 @@ static unsigned int gicv3_read_irq(void)
>       return irq;
>   }
>   
> +static void gicv3_set_active_state(int irq, bool active)
> +{
> +    void __iomem *base;
> +
> +    if ( irq >= NR_GIC_LOCAL_IRQS)
> +        base = GICD + (irq / 32) * 4;
> +    else
> +        base = GICD_RDIST_SGI_BASE;
> +
> +    if ( active )
> +        writel(1U << (irq % 32), base + GICD_ISACTIVER);
> +    else
> +        writel(1U << (irq % 32), base + GICD_ICACTIVER);

Shouldn't you wait until RWP bits is cleared here?

> +}

Why don't you use the function poke?

> +
>   static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
>   {
>        uint64_t mpidr = cpu_logical_map(cpu);
> @@ -1722,6 +1737,7 @@ static const struct gic_hw_operations gicv3_ops = {
>       .eoi_irq             = gicv3_eoi_irq,
>       .deactivate_irq      = gicv3_dir_irq,
>       .read_irq            = gicv3_read_irq,
> +    .set_active_state    = gicv3_set_active_state,
>       .set_irq_type        = gicv3_set_irq_type,
>       .set_irq_priority    = gicv3_set_irq_priority,
>       .send_SGI            = gicv3_send_sgi,
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 89873c1df4..dfc2108c4d 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -92,6 +92,11 @@ void gic_restore_state(struct vcpu *v)
>       isb();
>   }
>   
> +void gic_set_active_state(int irq, bool state)
> +{
> +    gic_hw_ops->set_active_state(irq, state);
> +}
> +
>   /* desc->irq needs to be disabled before calling this function */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type)
>   {
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index c4c68c7770..d330860580 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -238,6 +238,9 @@ DECLARE_PER_CPU(uint64_t, lr_mask);
>   extern enum gic_version gic_hw_version(void);
>   extern int gic_get_nr_lrs(void);
>   
> +/* Force the state of an IRQ to active. */
> +void gic_set_active_state(int irq, bool state);
> +
>   /* Program the IRQ type into the GIC */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
>   
> @@ -347,6 +350,8 @@ struct gic_hw_operations {
>       void (*deactivate_irq)(struct irq_desc *irqd);
>       /* Read IRQ id and Ack */
>       unsigned int (*read_irq)(void);
> +    /* Force the state of an IRQ to active */
> +    void (*set_active_state)(int irq, bool state);
>       /* Set IRQ type */
>       void (*set_irq_type)(struct irq_desc *desc, unsigned int type);
>       /* Set IRQ priority */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 16/49] ARM: GIC: allow reading pending state of a hardware IRQ

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> To synchronize level triggered interrupts which are mapped into a guest,
> we need to update the virtual line level at certain points in time.
> For a hardware mapped interrupt the GIC is the only place where we can
> easily access this information.
> Implement a gic_hw_operations member to return the pending state of a
> particular interrupt. Due to hardware limitations this only works for
> private interrupts of the current CPU, so there is not CPU field in the
> prototype.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/gic-v2.c     |  6 ++++++
>   xen/arch/arm/gic-v3.c     | 13 +++++++++++++
>   xen/arch/arm/gic.c        |  5 +++++
>   xen/include/asm-arm/gic.h |  5 +++++
>   4 files changed, 29 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 5339f69fbc..30081640ac 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -514,6 +514,11 @@ static unsigned int gicv2_read_apr(int apr_reg)
>      return readl_gich(GICH_APR);
>   }
>   
> +bool gicv2_read_pending_state(int irq)

static

Also, I would like to see the irq turned into irq_desc to match the 
other interface.

> +{
> +    return readl_gicd(GICD_ISPENDR + (irq / 32) * 4) & (1U << (irq % 32));

See my remark in the previous patch. You might want to introduce an 
helper peek.

> +}
> +
>   static void gicv2_irq_enable(struct irq_desc *desc)
>   {
>       unsigned long flags;
> @@ -1261,6 +1266,7 @@ const static struct gic_hw_operations gicv2_ops = {
>       .write_lr            = gicv2_write_lr,
>       .read_vmcr_priority  = gicv2_read_vmcr_priority,
>       .read_apr            = gicv2_read_apr,
> +    .read_pending_state  = gicv2_read_pending_state,
>       .make_hwdom_dt_node  = gicv2_make_hwdom_dt_node,
>       .make_hwdom_madt     = gicv2_make_hwdom_madt,
>       .get_hwdom_extra_madt_size = gicv2_get_hwdom_extra_madt_size,
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 595eaef43a..2cbfeb8e03 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -1081,6 +1081,18 @@ static unsigned int gicv3_read_apr(int apr_reg)
>       }
>   }
>   
> +static bool gicv3_read_pending_state(int irq)
> +{
> +    void __iomem *base;
> +
> +    if ( irq >= NR_GIC_LOCAL_IRQS)
> +        base = GICD + (irq / 32) * 4;
> +    else
> +        base = GICD_RDIST_SGI_BASE;
> +
> +    return readl(base + GICD_ISPENDR) & (1U << (irq % 32));

Looking at the GICv3 code we don't have a peek helper. It is probably 
worth to add one.

> +}
> +
>   static void gicv3_irq_enable(struct irq_desc *desc)
>   {
>       unsigned long flags;
> @@ -1749,6 +1761,7 @@ static const struct gic_hw_operations gicv3_ops = {
>       .write_lr            = gicv3_write_lr,
>       .read_vmcr_priority  = gicv3_read_vmcr_priority,
>       .read_apr            = gicv3_read_apr,
> +    .read_pending_state  = gicv3_read_pending_state,
>       .secondary_init      = gicv3_secondary_cpu_init,
>       .make_hwdom_dt_node  = gicv3_make_hwdom_dt_node,
>       .make_hwdom_madt     = gicv3_make_hwdom_madt,
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index dfc2108c4d..ce9ab2367e 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -116,6 +116,11 @@ static void gic_set_irq_priority(struct irq_desc *desc, unsigned int priority)
>       gic_hw_ops->set_irq_priority(desc, priority);
>   }
>   
> +bool gic_read_pending_state(int irq)
> +{
> +    return gic_hw_ops->read_pending_state(irq);
> +}
> +
>   /* Program the GIC to route an interrupt to the host (i.e. Xen)
>    * - needs to be called with desc.lock held
>    */
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index d330860580..d7fd18fd47 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -244,6 +244,9 @@ void gic_set_active_state(int irq, bool state);
>   /* Program the IRQ type into the GIC */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
>   
> +/* Read the pending state of an interrupt from the distributor. */
> +bool gic_read_pending_state(int irq);
> +
>   /* Program the GIC to route an interrupt */
>   extern void gic_route_irq_to_xen(struct irq_desc *desc, unsigned int priority);
>   extern int gic_route_irq_to_guest(struct domain *, unsigned int virq,
> @@ -376,6 +379,8 @@ struct gic_hw_operations {
>       unsigned int (*read_vmcr_priority)(void);
>       /* Read APRn register */
>       unsigned int (*read_apr)(int apr_reg);
> +    /* Query the pending state of an interrupt at the distributor level. */
> +    bool (*read_pending_state)(int irq);
>       /* Secondary CPU init */
>       int (*secondary_init)(void);
>       /* Create GIC node for the hardware domain */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 09/49] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Andre Przywara]

Hi,

On 12/02/18 12:19, Julien Grall wrote:
> 
> 
> On 12/02/18 11:59, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 12/02/18 11:15, Julien Grall wrote:
>>> Hi Andre,
>>>
>>> On 09/02/18 14:38, Andre Przywara wrote:
>>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>>> index 5f47aa84a9..2fc6e19625 100644
>>>> --- a/xen/arch/arm/vgic.c
>>>> +++ b/xen/arch/arm/vgic.c
>>>> @@ -285,7 +285,7 @@ bool vgic_migrate_irq(struct vcpu *old, struct
>>>> vcpu *new, unsigned int irq)
>>>>            vgic_remove_irq_from_queues(old, p);
>>>>            irq_set_affinity(p->desc, cpumask_of(new->processor));
>>>>            spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
>>>> -        vgic_vcpu_inject_irq(new, irq);
>>>> +        vgic_inject_irq(new->domain, new, irq, true);
>>>>            return true;
>>>>        }
>>>>        /* if the IRQ is in a GICH_LR register, set
>>>> GIC_IRQ_GUEST_MIGRATING
>>>> @@ -444,7 +444,7 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir,
>>>> enum gic_sgi_mode irqmode,
>>>>                            sgir, target->list);
>>>>                    continue;
>>>>                }
>>>> -            vgic_vcpu_inject_irq(d->vcpu[vcpuid], virq);
>>>> +            vgic_inject_irq(d, d->vcpu[vcpuid], virq, true);
>>>>            }
>>>>            break;
>>>>        case SGI_TARGET_OTHERS:
>>>> @@ -453,12 +453,12 @@ bool vgic_to_sgi(struct vcpu *v, register_t
>>>> sgir, enum gic_sgi_mode irqmode,
>>>>            {
>>>>                if ( i != current->vcpu_id && d->vcpu[i] != NULL &&
>>>>                     is_vcpu_online(d->vcpu[i]) )
>>>> -                vgic_vcpu_inject_irq(d->vcpu[i], virq);
>>>> +                vgic_inject_irq(d, d->vcpu[i], virq, true);
>>>>            }
>>>>            break;
>>>>        case SGI_TARGET_SELF:
>>>>            perfc_incr(vgic_sgi_self);
>>>> -        vgic_vcpu_inject_irq(d->vcpu[current->vcpu_id], virq);
>>>> +        vgic_inject_irq(d, current, virq, true);
>>>>            break;
>>>>        default:
>>>>            gprintk(XENLOG_WARNING,
>>>> @@ -518,13 +518,29 @@ void vgic_remove_irq_from_queues(struct vcpu *v,
>>>> struct pending_irq *p)
>>>>        gic_remove_from_lr_pending(v, p);
>>>>    }
>>>>    -void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>>>> +int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int
>>>> virq,
>>>> +                    bool level)
>>>
>>> Looking at the code after the series has been applied, no one is caring
>>> about the return value of vgic_inject_irq. So what is the rationale
>>> behind changing the return type from void to int?
>>
>> The KVM version returns an error value, in particular when:
>> - the VGIC has not been initialized yet
>> - we can't determine the VCPU for a private interrupt
>> - the interrupt ID is invalid (SPI beyond limit, not mapped LPI)
>> In the moment it's not very useful for Xen: the first two conditions
>> don't really happen, consequently I removed those checks. But the third
>> check may become interesting once we get LPIs. Also since Xen currently
>> uses a void prototype for injection, *this* patch *now* doesn't exploit
>> the newly gained possibility of properly handling errors. From briefly
>> checking all the users, all of them seem to be in void functions, so
>> indeed an error return is not very useful.
>> The reasons I kept it in was to allow introduction of checks later. I
>> think having a function returning an error where some users ignore this
>> is better than the other way round.
> 
> I don't think it is much better. This is a way to expose yet another
> security issue because the return is not correctly checked (see XSA-246
> for instance). Any return value should be checked or have a comment
> explaining why it is fine.
> 
>>
>> So of course I can easily make this void, but I wonder what we do in
>> those cases where the SPI is not valid, for instance? Shall we print
>> some (rate-limited) warning?
> 
> I can understand why KVM needs such interface as the interrupt
> controller may be emulated QEMU.

It is also that interrupts from emulated devices are injected from QEMU
in userland, and the ioctl used for that can and will return an error.
So we can propagate this condition to the device. If and what the
devices does with that information, is another question, though, but out
of scope for KVM.

> But I can't see why a SPI would not be
> valid in Xen context (except programming error). So could give an example?

In KVM the KVM_IRQ_LINE ioctl allows to inject an arbitrary number, so
checking this and returning an error is natural and mandatory.

> What would you expect the caller to do on error? Except printing an
> error message?

I don't know either. Comparing this to hardware, an IRQ is usually
fire-and-forget (separating the interrupt line from the interrupt state
here), so a device doesn't really handle the case when an IRQ does not
make it through (it can't know easily anyway). However the whole state
machine might get busted in the process (if no one lowers the line, for
instance).
So looking at this printing a message looks like the best choice.

I checked all users of vgic_inject_irq(), at the moment all IRQ numbers
passed in look safe: they are either hardcoded (timer, evtchn) or
validated before (hardware IRQs, when they are tied to a virtual IRQ).
So indeed we *should* never see an invalid IRQ number, at the moment.
I need to check how this changes with the ITS, though.

So we could change the prototype (back) to void, but print some error
message if the vgic_get_irq() call fails within vgic_inject_irq().

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 17/49] ARM: timer: Handle level triggered IRQs correctly

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The ARM Generic Timer uses a level-sensitive interrupt semantic. We
> easily catch when the line goes high, as this triggers the hardware IRQ.
> However we have to sync the state of the interrupt condition at certain
> points to catch when the line goes low and we can remove the vtimer vIRQ
> from the vGIC (and the LR).
> The VGIC in Xen so far only implemented edge triggered vIRQs, really, so
> we need to add new functionality to re-sample the interrupt state.

You might want to make a summary of the discussion we had with Marc Z. 
today here. This would help the other to understand why sample the 
interrupt state is necessary :).

Also do we need to do that for the emulated physical timer?

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/time.c     | 34 ++++++++++++++++++++++++++++++++++
>   xen/arch/arm/traps.c    |  1 +
>   xen/include/xen/timer.h |  2 ++
>   3 files changed, 37 insertions(+)
> 
> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
> index c11fcfeadd..98ebb4305d 100644
> --- a/xen/arch/arm/time.c
> +++ b/xen/arch/arm/time.c
> @@ -263,6 +263,40 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
>       vgic_inject_irq(current->domain, current, current->arch.virt_timer.irq, true);
>   }
>   
> +/**

One * is enough.

> + * vtimer_sync() - update the state of the virtual timer after a guest run
> + * @vcpu: The VCPU to sync the arch timer state
> + *
> + * After returning from a guest, update the state of the virtual interrupt
> + * line, to model the level triggered interrupt correctly.
> + * If the guest has handled a timer interrupt, the virtual interrupt line
> + * needs to be lowered explicitly. vgic_inject_irq() takes care of that.
> + */
> +void vtimer_sync(struct vcpu *vcpu)
> +{
> +    struct vtimer *vtimer = &vcpu->arch.virt_timer;
> +    bool level;
> +
> +    vtimer->ctl = READ_SYSREG32(CNTV_CTL_EL0);
> +    vtimer->cval = READ_SYSREG64(CNTV_CVAL_EL0);

Why do you need to save cval?

> +
> +    /*
> +     * Technically we should mask with 0x7 here, to catch if the timer
> +     * interrupt is masked. However Xen always masks the timer upon entering
> +     * the hypervisor, leaving it up to the guest to un-mask it.
> +     * So we would always read a "low" level, despite the condition being
> +     * actually "high". Igoring the mask bit solves this (for now).

s/Igoring/Ignoring/

> +     * Another possible check would be to compare the value of CNTVCT_EL0
> +     * against vtimer->cval and derive the interrupt state from that.
> +     *
> +     * TODO: The proper fix for this is to make vtimer vIRQ hardware mapped,
> +     * but this requires reworking the arch timer to implement this.

That something we should look at it once the vGIC is done :).

> +     */
> +    level = (vtimer->ctl & 0x5) == (CNTx_CTL_ENABLE | CNTx_CTL_PENDING);

Can you please use the proper define rather than plain value?

> +
> +    vgic_inject_irq(vcpu->domain, vcpu, vtimer->irq, level);
> +}
> +
>   /*
>    * Arch timer interrupt really ought to be level triggered, since the
>    * design of the timer/comparator mechanism is based around that
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 1cba7e584d..2d770a14a5 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2024,6 +2024,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
>           if ( current->arch.hcr_el2 & HCR_VA )
>               current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
>   

You need to sample the virtual timer before clearing the LRs, right? If 
so, you likely want to add a comment here to avoid reshuffling the code.

> +        vtimer_sync(current);

I am a bit worry about re-sampling the virtual interrupt state at every 
traps. It might be worth thinking to do the re-sample when syncing the 
LRs (as you do for HW level interrupt in patch #25). Probably once we 
get the new vGIC merged.

>           gic_clear_lrs(current);
>       }
>   }
> diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
> index 4513260b0d..eddbbf3903 100644
> --- a/xen/include/xen/timer.h
> +++ b/xen/include/xen/timer.h
> @@ -94,6 +94,8 @@ DECLARE_PER_CPU(s_time_t, timer_deadline);
>   /* Arch-defined function to reprogram timer hardware for new deadline. */
>   int reprogram_timer(s_time_t timeout);
>   
> +void vtimer_sync(struct vcpu *vcpu);
> +
>   /* Calculate the aligned first tick time for a given periodic timer. */
>   s_time_t align_timer(s_time_t firsttick, uint64_t period);
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 20/49] ARM: new VGIC: Add data structure definitions

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> Add a new header file for the new and improved GIC implementation.
> The big change is that we now have a struct vgic_irq per IRQ instead
> of spreading all the information over various bitmaps in the ranks.
> 
> We include this new header conditionally from within the old header
> file for the time being to avoid touching all the users.
> 
> This is based on Linux commit b18b57787f5e, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/include/asm-arm/arm_vgic.h | 269 +++++++++++++++++++++++++++++++++++++++++
>   xen/include/asm-arm/domain.h   |   4 +
>   xen/include/asm-arm/vgic.h     |   6 +
>   3 files changed, 279 insertions(+)
>   create mode 100644 xen/include/asm-arm/arm_vgic.h
> 
> diff --git a/xen/include/asm-arm/arm_vgic.h b/xen/include/asm-arm/arm_vgic.h
> new file mode 100644
> index 0000000000..865e9ee5bc
> --- /dev/null
> +++ b/xen/include/asm-arm/arm_vgic.h

arm_vgic.h is a confusing name. Can we name it vgic-new.h?

> @@ -0,0 +1,269 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#ifndef __KVM_ARM_VGIC_H
> +#define __KVM_ARM_VGIC_H

Please modify the guards based on the file name.

> +
> +#include <asm/atomic.h>
> +#include <asm/mmio.h>
> +#include <xen/spinlock.h>
> +#include <xen/list.h>
> +
> +#define VGIC_V3_MAX_CPUS        255
> +#define VGIC_V2_MAX_CPUS        8
> +#define VGIC_NR_IRQS_LEGACY     256
> +#define VGIC_NR_SGIS            16
> +#define VGIC_NR_PPIS            16
> +#define VGIC_NR_PRIVATE_IRQS    (VGIC_NR_SGIS + VGIC_NR_PPIS)
> +#define VGIC_MAX_PRIVATE        (VGIC_NR_PRIVATE_IRQS - 1)
> +#define VGIC_MAX_SPI            1019
> +#define VGIC_MAX_RESERVED       1023
> +#define VGIC_MIN_LPI            8192
> +
> +#define irq_is_ppi(irq) ((irq) >= VGIC_NR_SGIS && (irq) < VGIC_NR_PRIVATE_IRQS)
> +#define irq_is_spi(irq) ((irq) >= VGIC_NR_PRIVATE_IRQS && \
> +                         (irq) <= VGIC_MAX_SPI)
> +
> +enum vgic_type {
> +    VGIC_V2,        /* Good ol' GICv2 */
> +    VGIC_V3,        /* New fancy GICv3 */
> +};
> +
> +#define VGIC_V2_MAX_LRS         (1 << 6)
> +#define VGIC_V3_MAX_LRS         16
> +#define VGIC_V3_LR_INDEX(lr)    (VGIC_V3_MAX_LRS - 1 - lr)
> +
> +enum vgic_irq_config {
> +    VGIC_CONFIG_EDGE = 0,

I don't think it is necessary to set 0 here as IIRC an enum always start 
at 0 if not specified.

> +    VGIC_CONFIG_LEVEL
> +};
> +
> +struct vgic_irq {
> +    spinlock_t irq_lock;        /* Protects the content of the struct */
> +    struct list_head lpi_list;  /* Used to link all LPIs together */
> +    struct list_head ap_list;
> +
> +    struct vcpu *vcpu;          /*
> +                                 * SGIs and PPIs: The VCPU
> +                                 * SPIs and LPIs: The VCPU whose ap_list
> +                                 * this is queued on.
> +                                 */
> +
> +    struct vcpu *target_vcpu;   /*
> +                                 * The VCPU that this interrupt should
> +                                 * be sent to, as a result of the
> +                                 * targets reg (v2) or the affinity reg (v3).
> +                                 */
> +
> +    u32 intid;                  /* Guest visible INTID */

As we are using Xen coding style, please replace all u* with uint_*.

> +    bool line_level;            /* Level only */
> +    bool pending_latch;         /*
> +                                 * The pending latch state used to
> +                                 * calculate the pending state for both
> +                                 * level and edge triggered IRQs.
> +                                 */
> +    bool active;                /* not used for LPIs */
> +    bool enabled;
> +    bool hw;                    /* Tied to HW IRQ */
> +    atomic_t refcount;          /* Used for LPIs */
> +    u32 hwintid;                /* HW INTID number */
> +    union
> +    {
> +        u8 targets;             /* GICv2 target VCPUs mask */
> +        u32 mpidr;              /* GICv3 target VCPU */
> +    };
> +    u8 source;                  /* GICv2 SGIs only */
> +    u8 priority;
> +    enum vgic_irq_config config;    /* Level or edge */
> +};
> +
> +struct vgic_register_region;

Do we really need the forward declaration here?

> +struct vgic_its;
> +
> +enum iodev_type {
> +    IODEV_CPUIF,

I don't think this one is necessary.

> +    IODEV_DIST,
> +    IODEV_REDIST,
> +    IODEV_ITS
> +};
> +
> +struct vgic_io_device {
> +    paddr_t base_addr;
> +    union
> +    {
> +        struct vcpu *redist_vcpu;
> +        struct vgic_its *its;
> +    };
> +    const struct vgic_register_region *regions;
> +    enum iodev_type iodev_type;
> +    int nr_regions;

unsigned int please.

> +};
> +
> +struct vgic_its {
> +    /* The base address of the ITS control register frame */
> +    paddr_t     vgic_its_base;
> +
> +    bool        enabled;
> +    struct vgic_io_device       iodev;
> +
> +    /* These registers correspond to GITS_BASER{0,1} */
> +    u64         baser_device_table;
> +    u64         baser_coll_table;
> +
> +    /* Protects the command queue */
> +    spinlock_t  cmd_lock;
> +    u64         cbaser;
> +    u32         creadr;
> +    u32         cwriter;
> +
> +    /* migration ABI revision in use */
> +    u32         abi_rev;
> +
> +    /* Protects the device and collection lists */
> +    spinlock_t  its_lock;
> +    struct list_head            device_list;
> +    struct list_head            collection_list;
> +};
> +
> +struct vgic_dist {
> +    bool                ready;
> +    bool                initialized;
> +
> +    /* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
> +    u32                 version;
> +
> +    /* Do injected MSIs require an additional device ID? */
> +    bool                msis_require_devid;
> +
> +    int                 nr_spis;

Ditto.

> +
> +    /* base addresses in guest physical address space: */
> +    paddr_t             vgic_dist_base;     /* distributor */
> +    union
> +    {
> +        /* either a GICv2 CPU interface */
> +        paddr_t         vgic_cpu_base;
> +        /* or a number of GICv3 redistributor regions */
> +        struct
> +        {
> +            paddr_t     vgic_redist_base;
> +            paddr_t     vgic_redist_free_offset;
> +        };
> +    };
> +
> +    /* distributor enabled */
> +    bool                enabled;
> +
> +    struct vgic_irq     *spis;
> +    unsigned long       *allocated_irqs; /* bitmap of IRQs allocated */
> +
> +    struct vgic_io_device   dist_iodev;
> +
> +    bool                has_its;
> +
> +    /*
> +     * Contains the attributes and gpa of the LPI configuration table.
> +     * Since we report GICR_TYPER.CommonLPIAff as 0b00, we can share
> +     * one address across all redistributors.
> +     * GICv3 spec: 6.1.2 "LPI Configuration tables"
> +     */
> +    u64                 propbaser;
> +
> +    /* Protects the lpi_list and the count value below. */
> +    spinlock_t          lpi_list_lock;
> +    struct list_head    lpi_list_head;
> +    int                 lpi_list_count;

Ditto.

> +};
> +
> +struct vgic_v2_cpu_if {
> +    u32     vgic_hcr;
> +    u32     vgic_vmcr;
> +    u64     vgic_elrsr; /* Saved only */
> +    u32     vgic_apr;
> +    u32     vgic_lr[VGIC_V2_MAX_LRS];
> +};
> +
> +struct vgic_v3_cpu_if {
> +    u32     vgic_hcr;
> +    u32     vgic_vmcr;
> +    u32     vgic_sre;   /* Restored only, change ignored */
> +    u32     vgic_elrsr; /* Saved only */
> +    u32     vgic_ap0r[4];
> +    u32     vgic_ap1r[4];
> +    u64     vgic_lr[VGIC_V3_MAX_LRS];
> +};
> +
> +struct vgic_cpu {
> +    /* CPU vif control registers for world switch */
> +    union
> +    {
> +        struct vgic_v2_cpu_if   vgic_v2;
> +        struct vgic_v3_cpu_if   vgic_v3;
> +    };
> +
> +    unsigned int used_lrs;
> +    struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS];
> +
> +    spinlock_t ap_list_lock;    /* Protects the ap_list */
> +
> +    /*
> +     * List of IRQs that this VCPU should consider because they are either
> +     * Active or Pending (hence the name; AP list), or because they recently
> +     * were one of the two and need to be migrated off this list to another
> +     * VCPU.
> +     */
> +    struct list_head ap_list_head;
> +
> +    /*
> +     * Members below are used with GICv3 emulation only and represent
> +     * parts of the redistributor.
> +     */
> +    struct vgic_io_device   rd_iodev;
> +    struct vgic_io_device   sgi_iodev;
> +
> +    /* Contains the attributes and gpa of the LPI pending tables. */
> +    u64 pendbaser;
> +
> +    bool lpis_enabled;
> +
> +    /* Cache guest priority bits */
> +    u32 num_pri_bits;
> +
> +    /* Cache guest interrupt ID bits */
> +    u32 num_id_bits;
> +};
> +
> +extern struct static_key_false vgic_v2_cpuif_trap;
> +extern struct static_key_false vgic_v3_cpuif_trap;

static_key_false does not exist on Xen.

> +
> +#define vgic_initialized(k) ((k)->arch.vgic.initialized)
> +#define vgic_ready(k)       ((k)->arch.vgic.ready)
> +#define vgic_valid_spi(k, i)    (((i) >= VGIC_NR_PRIVATE_IRQS) && \
> +            ((i) < (k)->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS))
> +
> +bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
> +bool vgic_v3_emulate(struct cpu_user_regs *regs, union hsr hsr);

While I am ok with the structure been defined up front, I would rather 
prefer to see the prototype defined with the actual implementation. This 
is much easier to see if you miss a prototype. And technically, this 
patch only add "data structure definitions".

> +
> +void vgic_clear_pending_irqs(struct vcpu *v);
> +
> +void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
> +                      paddr_t vbase, void __iomem *hbase,
> +                      uint32_t aliased_offset);
> +struct rdist_region;
> +void vgic_v3_setup_hw(paddr_t dbase, unsigned int nr_rdist_regions,
> +                      const struct rdist_region *regions,
> +                      unsigned int intid_bits);

Please don't re-define prototypes existing between the two implementation.

> +
> +#endif /* __KVM_ARM_VGIC_H */

Please fix the name here.

> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 968ffb0c81..e8f2917140 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -198,7 +198,11 @@ struct arch_vcpu
>       union gic_state_data gic;
>       uint64_t lr_mask;
>   
> +#ifdef CONFIG_NEW_VGIC
> +    struct vgic_cpu vgic_cpu;

Hmmm. Is there any point of the naming? You are in arch_vcpu so likely 
it will describe the vGIC CPU interface.

If you real want to do the renaming, then do it for the current vGIC as 
well.

> +#else
>       struct vgic_cpu vgic;
> +#endif
>   
>       /* Timer registers  */
>       uint32_t cntkctl;
> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index 85ad2aca79..96b99f5c85 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -18,6 +18,10 @@
>   #ifndef __ASM_ARM_VGIC_H__
>   #define __ASM_ARM_VGIC_H__
>   
> +#ifdef CONFIG_NEW_VGIC
> +#include <asm/arm_vgic.h>
> +#else
> +
>   #include <xen/bitops.h>
>   #include <xen/radix-tree.h>
>   #include <xen/rbtree.h>
> @@ -313,6 +317,8 @@ void vgic_v3_setup_hw(paddr_t dbase,
>                         unsigned int intid_bits);
>   #endif
>   
> +#endif /* !CONFIG_NEW_VGIC */
> +
>   /*** Common VGIC functions used by Xen arch code ****/
>   
>   /*
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 21/49] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The new VGIC implementation centers around a struct vgic_irq instance
> per virtual IRQ.
> Provide a function to retrieve the right instance for a given IRQ
> number and (in case of private interrupts) the right VCPU.
> This also includes the corresponding put function, which does nothing
> for private interrupts and SPIs, but handles the ref-counting for LPIs.
> 
> This is based on Linux commit 64a959d66e47, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 107 +++++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h |  32 ++++++++++++++
>   2 files changed, 139 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic.c
>   create mode 100644 xen/arch/arm/vgic/vgic.h
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> new file mode 100644
> index 0000000000..3075091caa
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -0,0 +1,107 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/bug.h>
> +#include <xen/sched.h>
> +
> +#include <asm/arm_vgic.h>
> +#include "vgic.h"

Please order the include alphabetically.

> +
> +/*
> + * Iterate over the VM's list of mapped LPIs to find the one with a
> + * matching interrupt ID and return a reference to the IRQ structure.
> + */
> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +    struct vgic_irq *irq = NULL;
> +
> +    spin_lock(&dist->lpi_list_lock);
> +
> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )

I think it would be worth thinking of a different data structure here. 
The number of LPIs can be quite high for the hardware domain.

> +    {
> +        if ( irq->intid != intid )
> +            continue;
> +
> +        /*
> +         * This increases the refcount, the caller is expected to
> +         * call vgic_put_irq() later once it's finished with the IRQ.
> +         */
> +        vgic_get_irq_kref(irq);
> +        goto out_unlock;
> +    }
> +    irq = NULL;
> +
> +out_unlock:
> +    spin_unlock(&dist->lpi_list_lock);
> +
> +    return irq;
> +}
> +
> +/*
> + * This looks up the virtual interrupt ID to get the corresponding
> + * struct vgic_irq. It also increases the refcount, so any caller is expected
> + * to call vgic_put_irq() once it's finished with this IRQ.
> + */
> +struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
> +                  u32 intid)

Indentation.

> +{
> +    /* SGIs and PPIs */
> +    if ( intid <= VGIC_MAX_PRIVATE )
> +        return &vcpu->arch.vgic_cpu.private_irqs[intid];
> +
> +    /* SPIs */
> +    if ( intid <= VGIC_MAX_SPI )
> +        return &d->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS];
> +
> +    /* LPIs */
> +    if ( intid >= VGIC_MIN_LPI )
> +        return vgic_get_lpi(d, intid);
> +
> +    WARN();

Newline here please.

I would turn into an ASSERT_UNREACHABLE() so it is only happening in 
debug build and avoid to worry about a guest exploiting that :).

> +    return NULL;
> +}
> +
> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +
> +    if ( irq->intid < VGIC_MIN_LPI )
> +        return;
> +
> +    spin_lock(&dist->lpi_list_lock);
> +    if ( !atomic_dec_and_test(&irq->refcount) )
> +    {
> +        spin_unlock(&dist->lpi_list_lock);
> +        return;
> +    };
> +
> +    list_del(&irq->lpi_list);

I would add

ASSERT(lpi_list_count >= 1);

But it is a bit hard to know whether this code is valid given you don't 
have any implementation of ITS so far.

> +    dist->lpi_list_count--;
> +    spin_unlock(&dist->lpi_list_lock);
> +
> +    xfree(irq);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> new file mode 100644
> index 0000000000..7a15cfdd79
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic.h

To be honest, I am not a big fan of headers defined in the code bits. So 
I would need a reason for that to be there and not in the include you 
defined in the previous patch.

> @@ -0,0 +1,32 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#ifndef __XEN_ARM_VGIC_NEW_H__
> +#define __XEN_ARM_VGIC_NEW_H__

This does not match the filename/path.

> +
> +struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
> +                              u32 intid);
> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
> +
> +static inline void vgic_get_irq_kref(struct vgic_irq *irq)
> +{
> +    if ( irq->intid < VGIC_MIN_LPI )
> +        return;
> +
> +    atomic_inc(&irq->refcount);
> +}
> +
> +#endif

Missing emacs magic.

> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 15/49] ARM: GIC: Allow tweaking the active state of an IRQ

[Andre Przywara]

Hi,

On 12/02/18 13:55, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> When playing around with hardware mapped, level triggered virtual IRQs,
>> there is the need to explicitly set the active state of an interrupt at
>> some point in time.
>> To prepare the GIC for that, we introduce a set_active_state() function
>> to let the VGIC manipulate the state of an associated hardware IRQ.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/gic-v2.c     |  9 +++++++++
>>   xen/arch/arm/gic-v3.c     | 16 ++++++++++++++++
>>   xen/arch/arm/gic.c        |  5 +++++
>>   xen/include/asm-arm/gic.h |  5 +++++
>>   4 files changed, 35 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>> index 2e35892881..5339f69fbc 100644
>> --- a/xen/arch/arm/gic-v2.c
>> +++ b/xen/arch/arm/gic-v2.c
>> @@ -235,6 +235,14 @@ static unsigned int gicv2_read_irq(void)
>>       return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>>   }
>>   +static void gicv2_set_active_state(int irq, bool active)
> 
> I would much prefer to have an irq_desc in parameter. This is matching
> the other interface 

... and that's why I had it just like this in my first version. However
this proved to be nasty because I now need to get this irq_desc pointer
first, as the caller doesn't have it already. Since all we have and need
is the actual hardware IRQ number, I found it more straight-forward to
just use that number directly instead of going via the pointer and back
(h/w intid => irq_desc => irq).

> and you could update the flags such as
> _IRQ_INPROGRESS which you don't do at the moment.

Mmh, interesting point. I guess I should also clear this bit in the new
VGIC. At least once I wrapped my head around what this flag is
*actually* for (in conjunction with _IRQ_GUEST).
Anyway I guess this bit would still be set in our case.

> Also, who is preventing two CPUs to clear the active bit at the same time?

A certain hardware IRQ is assigned to one virtual IRQ on one VCPU at one
time only. Besides, GICD_ICACTIVERn has wired NAND semantics, so that's
naturally race free (as it was designed to be).
Unless I miss something here (happy to be pointed to an example where it
causes problems).

>> +{
>> +    if (active)
>> +        writel_gicd(1U << (irq % 32), GICD_ISACTIVER + (irq / 32) * 4);
>> +    else
>> +        writel_gicd(1U << (irq % 32), GICD_ICACTIVER + (irq / 32) * 4);
> 
> You will have a few places in the code usually similar construct. It
> would make sense to introduce a helper poke as we have in the GICv3 code.

Sure.

>> +}
>> +
>>   static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int
>> type)
>>   {
>>       uint32_t cfg, actual, edgebit;
>> @@ -1241,6 +1249,7 @@ const static struct gic_hw_operations gicv2_ops = {
>>       .eoi_irq             = gicv2_eoi_irq,
>>       .deactivate_irq      = gicv2_dir_irq,
>>       .read_irq            = gicv2_read_irq,
>> +    .set_active_state    = gicv2_set_active_state,
>>       .set_irq_type        = gicv2_set_irq_type,
>>       .set_irq_priority    = gicv2_set_irq_priority,
>>       .send_SGI            = gicv2_send_SGI,
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index 08d4703687..595eaef43a 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -475,6 +475,21 @@ static unsigned int gicv3_read_irq(void)
>>       return irq;
>>   }
>>   +static void gicv3_set_active_state(int irq, bool active)
>> +{
>> +    void __iomem *base;
>> +
>> +    if ( irq >= NR_GIC_LOCAL_IRQS)
>> +        base = GICD + (irq / 32) * 4;
>> +    else
>> +        base = GICD_RDIST_SGI_BASE;
>> +
>> +    if ( active )
>> +        writel(1U << (irq % 32), base + GICD_ISACTIVER);
>> +    else
>> +        writel(1U << (irq % 32), base + GICD_ICACTIVER);
> 
> Shouldn't you wait until RWP bits is cleared here?

I don't see why. I think this action has some posted semantics anyway,
so no need for any synchronisation. And also RWP does not track
I[SC]ACTIVER, only ICENABLER and some CTLR bits (ARM IHI 0069D, 8.9.4:
RWP[31]).

> 
>> +}
> 
> Why don't you use the function poke?

Ah, I didn't see this. But then this now does this quite costly RWP
dance now. We could add a check in there to only do this if we change
the affected registers or pass an explicit "bool wait_for_rwp" in there.

Thanks for staying awake on this ;-)

Cheers,
Andre.

>> +
>>   static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
>>   {
>>        uint64_t mpidr = cpu_logical_map(cpu);
>> @@ -1722,6 +1737,7 @@ static const struct gic_hw_operations gicv3_ops = {
>>       .eoi_irq             = gicv3_eoi_irq,
>>       .deactivate_irq      = gicv3_dir_irq,
>>       .read_irq            = gicv3_read_irq,
>> +    .set_active_state    = gicv3_set_active_state,
>>       .set_irq_type        = gicv3_set_irq_type,
>>       .set_irq_priority    = gicv3_set_irq_priority,
>>       .send_SGI            = gicv3_send_sgi,
>> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
>> index 89873c1df4..dfc2108c4d 100644
>> --- a/xen/arch/arm/gic.c
>> +++ b/xen/arch/arm/gic.c
>> @@ -92,6 +92,11 @@ void gic_restore_state(struct vcpu *v)
>>       isb();
>>   }
>>   +void gic_set_active_state(int irq, bool state)
>> +{
>> +    gic_hw_ops->set_active_state(irq, state);
>> +}
>> +
>>   /* desc->irq needs to be disabled before calling this function */
>>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type)
>>   {
>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>> index c4c68c7770..d330860580 100644
>> --- a/xen/include/asm-arm/gic.h
>> +++ b/xen/include/asm-arm/gic.h
>> @@ -238,6 +238,9 @@ DECLARE_PER_CPU(uint64_t, lr_mask);
>>   extern enum gic_version gic_hw_version(void);
>>   extern int gic_get_nr_lrs(void);
>>   +/* Force the state of an IRQ to active. */
>> +void gic_set_active_state(int irq, bool state);
>> +
>>   /* Program the IRQ type into the GIC */
>>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
>>   @@ -347,6 +350,8 @@ struct gic_hw_operations {
>>       void (*deactivate_irq)(struct irq_desc *irqd);
>>       /* Read IRQ id and Ack */
>>       unsigned int (*read_irq)(void);
>> +    /* Force the state of an IRQ to active */
>> +    void (*set_active_state)(int irq, bool state);
>>       /* Set IRQ type */
>>       void (*set_irq_type)(struct irq_desc *desc, unsigned int type);
>>       /* Set IRQ priority */
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 17/49] ARM: timer: Handle level triggered IRQs correctly

[Andre Przywara]

Hi,

On 12/02/18 15:19, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The ARM Generic Timer uses a level-sensitive interrupt semantic. We
>> easily catch when the line goes high, as this triggers the hardware IRQ.
>> However we have to sync the state of the interrupt condition at certain
>> points to catch when the line goes low and we can remove the vtimer vIRQ
>> from the vGIC (and the LR).
>> The VGIC in Xen so far only implemented edge triggered vIRQs, really, so
>> we need to add new functionality to re-sample the interrupt state.
> 
> You might want to make a summary of the discussion we had with Marc Z.
> today here. This would help the other to understand why sample the
> interrupt state is necessary :).

Yes, I just saw that I somehow missed copying the elaborate comment from
Christoffer. Fixed now, indeed without this background it's next to
impossible to understand this ;-)

> Also do we need to do that for the emulated physical timer?

Mmh, good question. I believe this whole timer story needs a good think
again.

>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/time.c     | 34 ++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/traps.c    |  1 +
>>   xen/include/xen/timer.h |  2 ++
>>   3 files changed, 37 insertions(+)
>>
>> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
>> index c11fcfeadd..98ebb4305d 100644
>> --- a/xen/arch/arm/time.c
>> +++ b/xen/arch/arm/time.c
>> @@ -263,6 +263,40 @@ static void vtimer_interrupt(int irq, void
>> *dev_id, struct cpu_user_regs *regs)
>>       vgic_inject_irq(current->domain, current,
>> current->arch.virt_timer.irq, true);
>>   }
>>   +/**
> 
> One * is enough.

That's kernel-doc comment style:
https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#writing-kernel-doc-comments

That allows tools to scan the tree and extract function documentation in
an automated way. A bit like markdown: still perfectly readable by
humans, but parse-able by scripts as well.

I was hoping that it wouldn't hurt to have this in Xen as well, as I
copied this already in other parts of this code.

>> + * vtimer_sync() - update the state of the virtual timer after a
>> guest run
>> + * @vcpu: The VCPU to sync the arch timer state
>> + *
>> + * After returning from a guest, update the state of the virtual
>> interrupt
>> + * line, to model the level triggered interrupt correctly.
>> + * If the guest has handled a timer interrupt, the virtual interrupt
>> line
>> + * needs to be lowered explicitly. vgic_inject_irq() takes care of that.
>> + */
>> +void vtimer_sync(struct vcpu *vcpu)
>> +{
>> +    struct vtimer *vtimer = &vcpu->arch.virt_timer;
>> +    bool level;
>> +
>> +    vtimer->ctl = READ_SYSREG32(CNTV_CTL_EL0);
>> +    vtimer->cval = READ_SYSREG64(CNTV_CVAL_EL0);
> 
> Why do you need to save cval?

Originally I was copying KVM code which checked the actual IRQ condition
(by comparing the counter with CVAL). So this might be a leftover from
there. Need to check whether we actually need an up-to-date value of this.

>> +
>> +    /*
>> +     * Technically we should mask with 0x7 here, to catch if the timer
>> +     * interrupt is masked. However Xen always masks the timer upon
>> entering
>> +     * the hypervisor, leaving it up to the guest to un-mask it.
>> +     * So we would always read a "low" level, despite the condition
>> being
>> +     * actually "high". Igoring the mask bit solves this (for now).
> 
> s/Igoring/Ignoring/
> 
>> +     * Another possible check would be to compare the value of
>> CNTVCT_EL0
>> +     * against vtimer->cval and derive the interrupt state from that.
>> +     *
>> +     * TODO: The proper fix for this is to make vtimer vIRQ hardware
>> mapped,
>> +     * but this requires reworking the arch timer to implement this.
> 
> That something we should look at it once the vGIC is done :).

Indeed, looking forward to it (well, somewhat ... ) ;-)

>> +     */
>> +    level = (vtimer->ctl & 0x5) == (CNTx_CTL_ENABLE | CNTx_CTL_PENDING);
> 
> Can you please use the proper define rather than plain value?

Ah, right, that was a leftover from experimentation.
Also a test to see if reviewers are really reading this ;-)

>> +
>> +    vgic_inject_irq(vcpu->domain, vcpu, vtimer->irq, level);
>> +}
>> +
>>   /*
>>    * Arch timer interrupt really ought to be level triggered, since the
>>    * design of the timer/comparator mechanism is based around that
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index 1cba7e584d..2d770a14a5 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -2024,6 +2024,7 @@ static void enter_hypervisor_head(struct
>> cpu_user_regs *regs)
>>           if ( current->arch.hcr_el2 & HCR_VA )
>>               current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
>>   
> 
> You need to sample the virtual timer before clearing the LRs, right? If
> so, you likely want to add a comment here to avoid reshuffling the code.

Yes, good point.

>> +        vtimer_sync(current);
> 
> I am a bit worry about re-sampling the virtual interrupt state at every
> traps. It might be worth thinking to do the re-sample when syncing the
> LRs (as you do for HW level interrupt in patch #25). Probably once we
> get the new vGIC merged.

I share your concerns, but didn't dare to optimise this yet. But indeed
it is something worth to think about.

Cheers,
Andre.

>>           gic_clear_lrs(current);
>>       }
>>   }
>> diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
>> index 4513260b0d..eddbbf3903 100644
>> --- a/xen/include/xen/timer.h
>> +++ b/xen/include/xen/timer.h
>> @@ -94,6 +94,8 @@ DECLARE_PER_CPU(s_time_t, timer_deadline);
>>   /* Arch-defined function to reprogram timer hardware for new
>> deadline. */
>>   int reprogram_timer(s_time_t timeout);
>>   +void vtimer_sync(struct vcpu *vcpu);
>> +
>>   /* Calculate the aligned first tick time for a given periodic timer. */
>>   s_time_t align_timer(s_time_t firsttick, uint64_t period);
>>  
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 22/49] ARM: new VGIC: Implement virtual IRQ injection

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> Provide a vgic_queue_irq_unlock() function which decides whether a
> given IRQ needs to be queued to a VCPU's ap_list.
> This should be called whenever an IRQ becomes pending or enabled,
> either as a result of a hardware IRQ injection, from devices emulated by
> Xen (like the architected timer) or from MMIO accesses to the distributor
> emulation.
> Also provides the necessary functions to allow to inject an IRQ to a guest.
> Since this is the first code that starts using our locking mechanism,
> we add some (hopefully) clear documentation of our locking strategy and
> requirements along with this patch.
> 
> This is based on Linux commit 81eeb95ddbab, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 224 +++++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h |  10 +++
>   2 files changed, 234 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 3075091caa..f517df6d00 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -21,6 +21,32 @@
>   #include <asm/arm_vgic.h>
>   #include "vgic.h"
>   
> +/*
> + * Locking order is always:
> + * kvm->lock (mutex)

You probably want to update the locking order to match Xen one. In that 
case, I am not sure if we need to take the domain lock in the code?


> + *   its->cmd_lock (mutex)
> + *     its->its_lock (mutex)

> + *       vgic_cpu->ap_list_lock
> + *         kvm->lpi_list_lock
> + *           vgic_irq->irq_lock
> + *
> + * If you need to take multiple locks, always take the upper lock first,
> + * then the lower ones, e.g. first take the its_lock, then the irq_lock.
> + * If you are already holding a lock and need to take a higher one, you
> + * have to drop the lower ranking lock first and re-aquire it after having

s/re-aquite/acquire/

> + * taken the upper one.
> + *
> + * When taking more than one ap_list_lock at the same time, always take the
> + * lowest numbered VCPU's ap_list_lock first, so:
> + *   vcpuX->vcpu_id < vcpuY->vcpu_id:
> + *     spin_lock(vcpuX->arch.vgic_cpu.ap_list_lock);
> + *     spin_lock(vcpuY->arch.vgic_cpu.ap_list_lock);
> + *
> + * Since the VGIC must support injecting virtual interrupts from ISRs, we have
> + * to use the spin_lock_irqsave/spin_unlock_irqrestore versions of outer
> + * spinlocks for any lock that may be taken while injecting an interrupt.

It is quite nice to see the locking explained in the file and in general 
a lot of explanation within the code :).

> + */
> +
>   /*
>    * Iterate over the VM's list of mapped LPIs to find the one with a
>    * matching interrupt ID and return a reference to the IRQ structure.
> @@ -97,6 +123,204 @@ void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
>       xfree(irq);
>   }
>   
> +/**
> + * vgic_target_oracle - compute the target vcpu for an irq
> + *
> + * @irq:    The irq to route. Must be already locked.
> + *
> + * Based on the current state of the interrupt (enabled, pending,
> + * active, vcpu and target_vcpu), compute the next vcpu this should be
> + * given to. Return NULL if this shouldn't be injected at all.
> + *
> + * Requires the IRQ lock to be held.
> + */
> +static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
> +{
> +    ASSERT(spin_is_locked(&irq->irq_lock));
> +
> +    /* If the interrupt is active, it must stay on the current vcpu */
> +    if ( irq->active )
> +        return irq->vcpu ? : irq->target_vcpu;
I am not sure to understand why you check whether irq->vcpu is NULL. If 
the interrupt is active, then irq->vcpu should be NULL. Did I miss anything?

> +
> +    /*
> +     * If the IRQ is not active but enabled and pending, we should direct
> +     * it to its configured target VCPU.
> +     * If the distributor is disabled, pending interrupts shouldn't be
> +     * forwarded.
> +     */
> +    if ( irq->enabled && irq_is_pending(irq) )
> +    {
> +        if ( unlikely(irq->target_vcpu &&
> +                 !irq->target_vcpu->domain->arch.vgic.enabled) )

The indentation looks wrong here.

> +            return NULL;
> +
> +        return irq->target_vcpu;
> +    }
> +
> +    /* If neither active nor pending and enabled, then this IRQ should not

Comment style:

/*
  * ...

> +     * be queued to any VCPU.
> +     */
> +    return NULL;
> +}
> +
> +/*
> + * Only valid injection if changing level for level-triggered IRQs or for a
> + * rising edge.
> + */
> +static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
> +{
> +    switch (irq->config)

switch ( ... )

> +    {
> +    case VGIC_CONFIG_LEVEL:
> +        return irq->line_level != level;
> +    case VGIC_CONFIG_EDGE:
> +        return level;
> +    }
> +

I would add an ASSERT_UNREACHABLE().

> +    return false;
> +}
> +
> +/*
> + * Check whether an IRQ needs to (and can) be queued to a VCPU's ap list.
> + * Do the queuing if necessary, taking the right locks in the right order.
> + * Returns true when the IRQ was queued, false otherwise.
> + *
> + * Needs to be entered with the IRQ lock already held, but will return
> + * with all locks dropped.
> + */
> +bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
> +               unsigned long flags)

Indentation. Also same remark as from vgic_inject_irq. No-one seems to 
care about the return (even in KVM :)).

> +{
> +    struct vcpu *vcpu;
> +    bool running;
> +
> +    ASSERT(spin_is_locked(&irq->irq_lock));
> +
> +retry:
> +    vcpu = vgic_target_oracle(irq);
> +    if ( irq->vcpu || !vcpu )
> +    {
> +        /*
> +         * If this IRQ is already on a VCPU's ap_list, then it
> +         * cannot be moved or modified and there is no more work for
> +         * us to do.
> +         *
> +         * Otherwise, if the irq is not pending and enabled, it does
> +         * not need to be inserted into an ap_list and there is also
> +         * no more work for us to do.
> +         */
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        /*
> +         * We have to kick the VCPU here, because we could be
> +         * queueing an edge-triggered interrupt for which we
> +         * get no EOI maintenance interrupt. In that case,
> +         * while the IRQ is already on the VCPU's AP list, the
> +         * VCPU could have EOI'ed the original interrupt and
> +         * won't see this one until it exits for some other
> +         * reason.
> +         */
> +        if ( vcpu )
> +            vcpu_unblock(vcpu);

vcpu_unblock will only "unblock" a vCPU that is blocked. It won't notify 
a running vCPU. So you want to have something similar to:

vcpu_unblock(vcpu);
if ( running && vcpu != current )
   smp_send_event_check_mask(...);

It is probably worth to introduce an helper for that.

> +        return false;
> +    }
> +
> +    /*
> +     * We must unlock the irq lock to take the ap_list_lock where
> +     * we are going to insert this new pending interrupt.
> +     */
> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +    /* someone can do stuff here, which we re-check below */
> +
> +    spin_lock_irqsave(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
> +    spin_lock(&irq->irq_lock);
> +
> +    /*
> +     * Did something change behind our backs?
> +     *
> +     * There are two cases:
> +     * 1) The irq lost its pending state or was disabled behind our
> +     *    backs and/or it was queued to another VCPU's ap_list.
> +     * 2) Someone changed the affinity on this irq behind our
> +     *    backs and we are now holding the wrong ap_list_lock.
> +     *
> +     * In both cases, drop the locks and retry.
> +     */
> +
> +    if ( unlikely(irq->vcpu || vcpu != vgic_target_oracle(irq)) )
> +    {
> +        spin_unlock(&irq->irq_lock);
> +        spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        goto retry;
> +    }
> +
> +    /*
> +     * Grab a reference to the irq to reflect the fact that it is
> +     * now in the ap_list.
> +     */
> +    vgic_get_irq_kref(irq);
> +    list_add_tail(&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head);

I was expecting the list to be sorted here. But you seem to do it only 
in vgic_flush_lr_state() which is quite interesting.

I can foresee quite a few issues with this choice on Xen:
	1) You compute the size of ap list in vgic_flush_lr_state() and take 
lock on every IRQ one by one. A guest could be nasty and make that list 
quite big by make IRQs pending but never "active" them (i.e read IAR).
	2) This might be an issue while checking whether you need to deliver an 
interrupt (vgic_vcpu_pending_irq) because the list is not sorted.

> +    irq->vcpu = vcpu;
> +
> +    spin_unlock(&irq->irq_lock);
> +    spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
> +
> +    running = vcpu->is_running;
> +    vcpu_unblock(vcpu);
> +    if ( running && vcpu != current )
> +        smp_send_event_check_mask(cpumask_of(vcpu->processor));
> +
> +    return true;
> +}
> +
> +/**
> + * vgic_inject_irq - Inject an IRQ from a device to the vgic
> + * @d:       The domain pointer
> + * @vcpu:    The vCPU for PPIs
> + * @intid:   The INTID to inject a new state to.
> + * @level:   Edge-triggered:  true:  to trigger the interrupt
> + *                false: to ignore the call
> + *       Level-sensitive  true:  raise the input signal
> + *                false: lower the input signal
> + *
> + * The VGIC is not concerned with devices being active-LOW or active-HIGH for
> + * level-sensitive interrupts.  You can think of the level parameter as 1
> + * being HIGH and 0 being LOW and all devices being active-HIGH.
> + */
> +int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int intid,
> +            bool level)

Indentation.

> +{
> +    struct vgic_irq *irq;
> +    unsigned long flags;
> +
> +    irq = vgic_get_irq(d, vcpu, intid);
> +    if ( !irq )
> +        return -EINVAL;
> +
> +    spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +    if ( !vgic_validate_injection(irq, level) )
> +    {
> +        /* Nothing to see here, move along... */
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(d, irq);
> +        return 0;
> +    }
> +
> +    if ( irq->config == VGIC_CONFIG_LEVEL )
> +        irq->line_level = level;
> +    else
> +        irq->pending_latch = true;
> +
> +    vgic_queue_irq_unlock(d, irq, flags);
> +    vgic_put_irq(d, irq);
> +
> +    return 0;
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 7a15cfdd79..5127739f0f 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -17,9 +17,19 @@
>   #ifndef __XEN_ARM_VGIC_NEW_H__
>   #define __XEN_ARM_VGIC_NEW_H__
>   
> +static inline bool irq_is_pending(struct vgic_irq *irq)
> +{
> +    if ( irq->config == VGIC_CONFIG_EDGE )
> +        return irq->pending_latch;
> +    else
> +        return irq->pending_latch || irq->line_level;
> +}
> +
>   struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>                                 u32 intid);
>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
> +bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
> +               unsigned long flags);
>   
>   static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>   {
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 21/49] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Andre Przywara]

Hi,

On 12/02/18 17:42, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The new VGIC implementation centers around a struct vgic_irq instance
>> per virtual IRQ.
>> Provide a function to retrieve the right instance for a given IRQ
>> number and (in case of private interrupts) the right VCPU.
>> This also includes the corresponding put function, which does nothing
>> for private interrupts and SPIs, but handles the ref-counting for LPIs.
>>
>> This is based on Linux commit 64a959d66e47, written by Christoffer Dall.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 107
>> +++++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.h |  32 ++++++++++++++
>>   2 files changed, 139 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic.c
>>   create mode 100644 xen/arch/arm/vgic/vgic.h
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> new file mode 100644
>> index 0000000000..3075091caa
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -0,0 +1,107 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/bug.h>
>> +#include <xen/sched.h>
>> +
>> +#include <asm/arm_vgic.h>
>> +#include "vgic.h"
> 
> Please order the include alphabetically.

Sure.

>> +
>> +/*
>> + * Iterate over the VM's list of mapped LPIs to find the one with a
>> + * matching interrupt ID and return a reference to the IRQ structure.
>> + */
>> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
>> +{
>> +    struct vgic_dist *dist = &d->arch.vgic;
>> +    struct vgic_irq *irq = NULL;
>> +
>> +    spin_lock(&dist->lpi_list_lock);
>> +
>> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
> 
> I think it would be worth thinking of a different data structure here.
> The number of LPIs can be quite high for the hardware domain.

Probably true. I just didn't want to waste time on this yet, as we don't
have LPIs at the moment. Having a list has the big advantage of being
easy to understand and to implement, so I consider this an optimization
that we can have later.

>> +    {
>> +        if ( irq->intid != intid )
>> +            continue;
>> +
>> +        /*
>> +         * This increases the refcount, the caller is expected to
>> +         * call vgic_put_irq() later once it's finished with the IRQ.
>> +         */
>> +        vgic_get_irq_kref(irq);
>> +        goto out_unlock;
>> +    }
>> +    irq = NULL;
>> +
>> +out_unlock:
>> +    spin_unlock(&dist->lpi_list_lock);
>> +
>> +    return irq;
>> +}
>> +
>> +/*
>> + * This looks up the virtual interrupt ID to get the corresponding
>> + * struct vgic_irq. It also increases the refcount, so any caller is
>> expected
>> + * to call vgic_put_irq() once it's finished with this IRQ.
>> + */
>> +struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>> +                  u32 intid)
> 
> Indentation.
> 
>> +{
>> +    /* SGIs and PPIs */
>> +    if ( intid <= VGIC_MAX_PRIVATE )
>> +        return &vcpu->arch.vgic_cpu.private_irqs[intid];
>> +
>> +    /* SPIs */
>> +    if ( intid <= VGIC_MAX_SPI )
>> +        return &d->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS];
>> +
>> +    /* LPIs */
>> +    if ( intid >= VGIC_MIN_LPI )
>> +        return vgic_get_lpi(d, intid);
>> +
>> +    WARN();
> 
> Newline here please.
> 
> I would turn into an ASSERT_UNREACHABLE() so it is only happening in
> debug build and avoid to worry about a guest exploiting that :).

Good point - and easy to fix :-D

>> +    return NULL;
>> +}
>> +
>> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
>> +{
>> +    struct vgic_dist *dist = &d->arch.vgic;
>> +
>> +    if ( irq->intid < VGIC_MIN_LPI )
>> +        return;
>> +
>> +    spin_lock(&dist->lpi_list_lock);
>> +    if ( !atomic_dec_and_test(&irq->refcount) )
>> +    {
>> +        spin_unlock(&dist->lpi_list_lock);
>> +        return;
>> +    };
>> +
>> +    list_del(&irq->lpi_list);
> 
> I would add
> 
> ASSERT(lpi_list_count >= 1);
> 
> But it is a bit hard to know whether this code is valid given you don't
> have any implementation of ITS so far.

Is it? You should not need the actual ITS code to validate this
function. In fact there are only very few users in vgic-its.c.
The main point here is that you have textbook ref-counting: *Every* time
you take a pointer to an IRQ (vgic_get_irq), you have to tell the code
when you are done with it (vgic_put_irq).
So we decided to have this ref-counting done properly even though it's
pointless for SPIs, PPIs and SGIs, as it makes the code very clear to
read and verify.
Mostly you have get and put in one function, but sometimes there is more
time between them: for instance if an interrupt goes to the ap_list. We
"get" it, add it to the list, then return. When the guest has actually
handled this interrupt, we remove it from the list and only then "put"
it again.

>> +    dist->lpi_list_count--;
>> +    spin_unlock(&dist->lpi_list_lock);
>> +
>> +    xfree(irq);
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> new file mode 100644
>> index 0000000000..7a15cfdd79
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic.h
> 
> To be honest, I am not a big fan of headers defined in the code bits. So
> I would need a reason for that to be there and not in the include you
> defined in the previous patch.

What is the problem with that?
The rationale here is to gather all definitions and prototypes that are
actually VGIC *internal*. No code outside of the actual VGIC
(xen/arch/arm/vgic/) should be concerned with it, and so I consider this
good style to keep this header file local. This makes it very clear that
no generic or arch code should ever tinker with anything defined in it.

Think about it like we could actually glue all those files in this new
directory together into one glorious new-vgic.c. Then we would not need
this header. But it's terrible to read and review, so we have this nice
split-up into vgic-mmio.c and vgic.c, for instance. And now we need this
header file to link those files together, to allow the MMIO emulation to
manipulate the state of an interrupt and queue it to a VCPU, for instance.

It's totally possible that there are definitions and prototypes in here
which don't belong here. TBH I didn't review this file here very
carefully for what we still need and what not, so I am happy to take
advice on what's wrong here.

Cheers,
Andre.


> 
>> @@ -0,0 +1,32 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +#ifndef __XEN_ARM_VGIC_NEW_H__
>> +#define __XEN_ARM_VGIC_NEW_H__
> 
> This does not match the filename/path.
> 
>> +
>> +struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>> +                              u32 intid);
>> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
>> +
>> +static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>> +{
>> +    if ( irq->intid < VGIC_MIN_LPI )
>> +        return;
>> +
>> +    atomic_inc(&irq->refcount);
>> +}
>> +
>> +#endif
> 
> Missing emacs magic.
> 
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 09/49] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Julien Grall]

Hi,

On 12/02/18 14:24, Andre Przywara wrote:
>> What would you expect the caller to do on error? Except printing an
>> error message?
> 
> I don't know either. Comparing this to hardware, an IRQ is usually
> fire-and-forget (separating the interrupt line from the interrupt state
> here), so a device doesn't really handle the case when an IRQ does not
> make it through (it can't know easily anyway). However the whole state
> machine might get busted in the process (if no one lowers the line, for
> instance).
> So looking at this printing a message looks like the best choice.
> 
> I checked all users of vgic_inject_irq(), at the moment all IRQ numbers
> passed in look safe: they are either hardcoded (timer, evtchn) or
> validated before (hardware IRQs, when they are tied to a virtual IRQ).
> So indeed we *should* never see an invalid IRQ number, at the moment.
> I need to check how this changes with the ITS, though.
> 
> So we could change the prototype (back) to void, but print some error
> message if the vgic_get_irq() call fails within vgic_inject_irq().

Sounds good to me. Make sure to have those message using the log level 
guest debug message. I might even be tempt to use dgprintk(...) here so 
they get dropped in non-debug build.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 15/49] ARM: GIC: Allow tweaking the active state of an IRQ

[Julien Grall]

On 12/02/18 17:53, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 12/02/18 13:55, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> When playing around with hardware mapped, level triggered virtual IRQs,
>>> there is the need to explicitly set the active state of an interrupt at
>>> some point in time.
>>> To prepare the GIC for that, we introduce a set_active_state() function
>>> to let the VGIC manipulate the state of an associated hardware IRQ.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/gic-v2.c     |  9 +++++++++
>>>    xen/arch/arm/gic-v3.c     | 16 ++++++++++++++++
>>>    xen/arch/arm/gic.c        |  5 +++++
>>>    xen/include/asm-arm/gic.h |  5 +++++
>>>    4 files changed, 35 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>>> index 2e35892881..5339f69fbc 100644
>>> --- a/xen/arch/arm/gic-v2.c
>>> +++ b/xen/arch/arm/gic-v2.c
>>> @@ -235,6 +235,14 @@ static unsigned int gicv2_read_irq(void)
>>>        return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>>>    }
>>>    +static void gicv2_set_active_state(int irq, bool active)
>>
>> I would much prefer to have an irq_desc in parameter. This is matching
>> the other interface
> 
> ... and that's why I had it just like this in my first version. However
> this proved to be nasty because I now need to get this irq_desc pointer
> first, as the caller doesn't have it already. Since all we have and need
> is the actual hardware IRQ number, I found it more straight-forward to
> just use that number directly instead of going via the pointer and back
> (h/w intid => irq_desc => irq).
> 
>> and you could update the flags such as
>> _IRQ_INPROGRESS which you don't do at the moment.
> 
> Mmh, interesting point. I guess I should also clear this bit in the new
> VGIC. At least once I wrapped my head around what this flag is
> *actually* for (in conjunction with _IRQ_GUEST).
> Anyway I guess this bit would still be set in our case.

For IRQ routed to the guest, the flag is used to know whether you need 
to EOI the interrupt on domain destruction.

In general, I would like to keep desc->status in sync for the guest IRQ. 
This is useful for debugging and potentially some ratelimit on interrupt 
(I am thinking for ITS).

> 
>> Also, who is preventing two CPUs to clear the active bit at the same time?
> 
> A certain hardware IRQ is assigned to one virtual IRQ on one VCPU at one
> time only. Besides, GICD_ICACTIVERn has wired NAND semantics, so that's
> naturally race free (as it was designed to be).
> Unless I miss something here (happy to be pointed to an example where it
> causes problems).

You could potentially have a race between ICACTIVER an ISACTIVER. This 
is very similar to the enable/disable part. This matters a lot when 
updating desc->status.

>>> +}
>>> +
>>>    static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int
>>> type)
>>>    {
>>>        uint32_t cfg, actual, edgebit;
>>> @@ -1241,6 +1249,7 @@ const static struct gic_hw_operations gicv2_ops = {
>>>        .eoi_irq             = gicv2_eoi_irq,
>>>        .deactivate_irq      = gicv2_dir_irq,
>>>        .read_irq            = gicv2_read_irq,
>>> +    .set_active_state    = gicv2_set_active_state,
>>>        .set_irq_type        = gicv2_set_irq_type,
>>>        .set_irq_priority    = gicv2_set_irq_priority,
>>>        .send_SGI            = gicv2_send_SGI,
>>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>>> index 08d4703687..595eaef43a 100644
>>> --- a/xen/arch/arm/gic-v3.c
>>> +++ b/xen/arch/arm/gic-v3.c
>>> @@ -475,6 +475,21 @@ static unsigned int gicv3_read_irq(void)
>>>        return irq;
>>>    }
>>>    +static void gicv3_set_active_state(int irq, bool active)
>>> +{
>>> +    void __iomem *base;
>>> +
>>> +    if ( irq >= NR_GIC_LOCAL_IRQS)
>>> +        base = GICD + (irq / 32) * 4;
>>> +    else
>>> +        base = GICD_RDIST_SGI_BASE;
>>> +
>>> +    if ( active )
>>> +        writel(1U << (irq % 32), base + GICD_ISACTIVER);
>>> +    else
>>> +        writel(1U << (irq % 32), base + GICD_ICACTIVER);
>>
>> Shouldn't you wait until RWP bits is cleared here?
> 
> I don't see why. I think this action has some posted semantics anyway,
> so no need for any synchronisation. And also RWP does not track
> I[SC]ACTIVER, only ICENABLER and some CTLR bits (ARM IHI 0069D, 8.9.4:
> RWP[31]).
> 
>>
>>> +}
>>
>> Why don't you use the function poke?
> 
> Ah, I didn't see this. But then this now does this quite costly RWP
> dance now. We could add a check in there to only do this if we change
> the affected registers or pass an explicit "bool wait_for_rwp" in there.

I guess this would be useful even for the current code. If I understand 
correctly the RWP semantics, it should not be necessary to wait when 
write to ISENABLER also.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 17/49] ARM: timer: Handle level triggered IRQs correctly

[Julien Grall]

On 12/02/18 18:23, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 12/02/18 15:19, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> The ARM Generic Timer uses a level-sensitive interrupt semantic. We
>>> easily catch when the line goes high, as this triggers the hardware IRQ.
>>> However we have to sync the state of the interrupt condition at certain
>>> points to catch when the line goes low and we can remove the vtimer vIRQ
>>> from the vGIC (and the LR).
>>> The VGIC in Xen so far only implemented edge triggered vIRQs, really, so
>>> we need to add new functionality to re-sample the interrupt state.
>>
>> You might want to make a summary of the discussion we had with Marc Z.
>> today here. This would help the other to understand why sample the
>> interrupt state is necessary :).
> 
> Yes, I just saw that I somehow missed copying the elaborate comment from
> Christoffer. Fixed now, indeed without this background it's next to
> impossible to understand this ;-)
> 
>> Also do we need to do that for the emulated physical timer?
> 
> Mmh, good question. I believe this whole timer story needs a good think
> again.
> 
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/time.c     | 34 ++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/traps.c    |  1 +
>>>    xen/include/xen/timer.h |  2 ++
>>>    3 files changed, 37 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
>>> index c11fcfeadd..98ebb4305d 100644
>>> --- a/xen/arch/arm/time.c
>>> +++ b/xen/arch/arm/time.c
>>> @@ -263,6 +263,40 @@ static void vtimer_interrupt(int irq, void
>>> *dev_id, struct cpu_user_regs *regs)
>>>        vgic_inject_irq(current->domain, current,
>>> current->arch.virt_timer.irq, true);
>>>    }
>>>    +/**
>>
>> One * is enough.
> 
> That's kernel-doc comment style:
> https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#writing-kernel-doc-comments
> 
> That allows tools to scan the tree and extract function documentation in
> an automated way. A bit like markdown: still perfectly readable by
> humans, but parse-able by scripts as well.
> 
> I was hoping that it wouldn't hurt to have this in Xen as well, as I
> copied this already in other parts of this code.

I was blindly following the CODING_STYLE requirements :). But let's keep 
/** if it helps script.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 23/49] ARM: new VGIC: Add IRQ sorting

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> Adds the sorting function to cover the case where you have more IRQs
> to consider than you have LRs. We consider their priorities.
> This pulls in Linux' list_sort.c , which is a merge sort implementation
> for linked lists.
> 
> This is based on Linux commit 8e4447457965, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c    |  59 +++++++++++++++
>   xen/common/list_sort.c      | 170 ++++++++++++++++++++++++++++++++++++++++++++
>   xen/include/xen/list_sort.h |  11 +++

You need to CC "THE REST" maintainers for this code. It would also make 
sense to have a separate patch for adding list_sort.c

>   3 files changed, 240 insertions(+)
>   create mode 100644 xen/common/list_sort.c
>   create mode 100644 xen/include/xen/list_sort.h
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index f517df6d00..a4efd1fd03 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -16,6 +16,7 @@
>    */
>   
>   #include <asm/bug.h>
> +#include <xen/list_sort.h>
>   #include <xen/sched.h>
>   
>   #include <asm/arm_vgic.h>
> @@ -163,6 +164,64 @@ static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
>       return NULL;
>   }
>   
> +/*
> + * The order of items in the ap_lists defines how we'll pack things in LRs as
> + * well, the first items in the list being the first things populated in the
> + * LRs.
> + *
> + * A hard rule is that active interrupts can never be pushed out of the LRs
> + * (and therefore take priority) since we cannot reliably trap on deactivation
> + * of IRQs and therefore they have to be present in the LRs.
> + *
> + * Otherwise things should be sorted by the priority field and the GIC
> + * hardware support will take care of preemption of priority groups etc.
> + *
> + * Return negative if "a" sorts before "b", 0 to preserve order, and positive
> + * to sort "b" before "a".

Finally a good explanation of the return value of a sort function :). I 
always get confused what the return is supposed to be.

> + */
> +static int vgic_irq_cmp(void *priv, struct list_head *a, struct list_head *b)
> +{
> +    struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list);
> +    struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list);
> +    bool penda, pendb;
> +    int ret;
> +
> +    spin_lock(&irqa->irq_lock);
> +    spin_lock(&irqb->irq_lock);

I guess the locking order does not matter here because this is the only 
place where two IRQs lock have to be taken?

Also, this will be done with irq disabled right? In that case, may I ask 
for an ASSERT(!local_irq_is_enabled())? Or maybe in vgic_sort_ap_list.

> +
> +    if ( irqa->active || irqb->active )
> +    {
> +        ret = (int)irqb->active - (int)irqa->active;
> +        goto out;
> +    }
> +
> +    penda = irqa->enabled && irq_is_pending(irqa);
> +    pendb = irqb->enabled && irq_is_pending(irqb);
> +
> +    if ( !penda || !pendb )
> +    {
> +        ret = (int)pendb - (int)penda;
> +        goto out;
> +    }
> +
> +    /* Both pending and enabled, sort by priority */
> +    ret = irqa->priority - irqb->priority;
> +out:
> +    spin_unlock(&irqb->irq_lock);
> +    spin_unlock(&irqa->irq_lock);
> +    return ret;
> +}
> +
> +/* Must be called with the ap_list_lock held */
> +static void vgic_sort_ap_list(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +
> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
> +
> +    list_sort(NULL, &vgic_cpu->ap_list_head, vgic_irq_cmp);
> +}
> +
>   /*
>    * Only valid injection if changing level for level-triggered IRQs or for a
>    * rising edge.
> diff --git a/xen/common/list_sort.c b/xen/common/list_sort.c
> new file mode 100644
> index 0000000000..9c5cc58e43
> --- /dev/null
> +++ b/xen/common/list_sort.c
> @@ -0,0 +1,170 @@
> +/*
> + * list_sort.c: merge sort implementation for linked lists
> + * Copied from the Linux kernel (lib/list_sort.c)
> + * (without specific copyright notice there)

I can see you moved from Linux to Xen coding style. Is there any other 
changes made?

> + *
> + * This program is free software; you can redistribute it and/or modify it
> + * under the terms and conditions of the GNU General Public License,
> + * version 2, as published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope it will be useful, but WITHOUT
> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
> + * more details.
> + *
> + * You should have received a copy of the GNU General Public License along with
> + * this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +#include <xen/lib.h>
> +#include <xen/list.h>
> +
> +#define MAX_LIST_LENGTH_BITS 20
> +
> +/*
> + * Returns a list organized in an intermediate format suited
> + * to chaining of merge() calls: null-terminated, no reserved or
> + * sentinel head node, "prev" links not maintained.
> + */
> +static struct list_head *merge(void *priv,
> +                               int (*cmp)(void *priv, struct list_head *a,
> +                                          struct list_head *b),
> +                               struct list_head *a, struct list_head *b)
> +{
> +    struct list_head head, *tail = &head;
> +
> +    while ( a && b )
> +    {
> +        /* if equal, take 'a' -- important for sort stability */
> +        if ( (*cmp)(priv, a, b) <= 0 )
> +        {
> +            tail->next = a;
> +            a = a->next;
> +        }
> +        else
> +        {
> +            tail->next = b;
> +            b = b->next;
> +        }
> +        tail = tail->next;
> +    }
> +    tail->next = a?:b;
> +    return head.next;
> +}
> +
> +/*
> + * Combine final list merge with restoration of standard doubly-linked
> + * list structure.  This approach duplicates code from merge(), but
> + * runs faster than the tidier alternatives of either a separate final
> + * prev-link restoration pass, or maintaining the prev links
> + * throughout.
> + */
> +static void merge_and_restore_back_links(void *priv,
> +                                         int (*cmp)(void *priv,
> +                                                    struct list_head *a,
> +                                                    struct list_head *b),
> +                                         struct list_head *head,
> +                                         struct list_head *a,
> +                                         struct list_head *b)
> +{
> +    struct list_head *tail = head;
> +    u8 count = 0;
> +
> +    while ( a && b )
> +    {
> +        /* if equal, take 'a' -- important for sort stability */
> +        if ( (*cmp)(priv, a, b) <= 0 )
> +        {
> +            tail->next = a;
> +            a->prev = tail;
> +            a = a->next;
> +        }
> +        else
> +        {
> +            tail->next = b;
> +            b->prev = tail;
> +            b = b->next;
> +        }
> +        tail = tail->next;
> +    }
> +    tail->next = a ? : b;
> +
> +    do
> +    {
> +        /*
> +         * In worst cases this loop may run many iterations.
> +         * Continue callbacks to the client even though no
> +         * element comparison is needed, so the client's cmp()
> +         * routine can invoke cond_resched() periodically.
> +         */
> +        if ( unlikely(!(++count)) )
> +            (*cmp)(priv, tail->next, tail->next);
> +
> +        tail->next->prev = tail;
> +        tail = tail->next;
> +    } while ( tail->next );
> +
> +    tail->next = head;
> +    head->prev = tail;
> +}
> +
> +/**
> + * list_sort - sort a list
> + * @priv: private data, opaque to list_sort(), passed to @cmp
> + * @head: the list to sort
> + * @cmp: the elements comparison function
> + *
> + * This function implements "merge sort", which has O(nlog(n))
> + * complexity.
> + *
> + * The comparison function @cmp must return a negative value if @a
> + * should sort before @b, and a positive value if @a should sort after
> + * @b. If @a and @b are equivalent, and their original relative
> + * ordering is to be preserved, @cmp must return 0.
> + */
> +void list_sort(void *priv, struct list_head *head,
> +               int (*cmp)(void *priv, struct list_head *a, struct list_head *b))
> +{
> +    struct list_head *part[MAX_LIST_LENGTH_BITS+1]; /* sorted partial lists
> +                        -- last slot is a sentinel */
> +    int lev;  /* index into part[] */
> +    int max_lev = 0;
> +    struct list_head *list;
> +
> +    if ( list_empty(head) )
> +        return;
> +
> +    memset(part, 0, sizeof(part));
> +
> +    head->prev->next = NULL;
> +    list = head->next;
> +
> +    while ( list )
> +    {
> +        struct list_head *cur = list;
> +        list = list->next;
> +        cur->next = NULL;
> +
> +        for ( lev = 0; part[lev]; lev++ )
> +        {
> +            cur = merge(priv, cmp, part[lev], cur);
> +            part[lev] = NULL;
> +        }
> +        if ( lev > max_lev )
> +        {
> +            if ( unlikely(lev >= ARRAY_SIZE(part)-1) )
> +            {
> +                dprintk(XENLOG_DEBUG, "list too long for efficiency\n");
> +                lev--;
> +            }
> +            max_lev = lev;
> +        }
> +        part[lev] = cur;
> +    }
> +
> +    for ( lev = 0; lev < max_lev; lev++ )
> +        if ( part[lev] )
> +            list = merge(priv, cmp, part[lev], list);
> +
> +    merge_and_restore_back_links(priv, cmp, head, part[max_lev], list);
> +}
> +EXPORT_SYMBOL(list_sort);
> diff --git a/xen/include/xen/list_sort.h b/xen/include/xen/list_sort.h
> new file mode 100644
> index 0000000000..a60c589d4b
> --- /dev/null
> +++ b/xen/include/xen/list_sort.h
> @@ -0,0 +1,11 @@
> +#ifndef _LINUX_LIST_SORT_H
> +#define _LINUX_LIST_SORT_H
> +
> +#include <xen/types.h>
> +
> +struct list_head;
> +
> +void list_sort(void *priv, struct list_head *head,
> +               int (*cmp)(void *priv, struct list_head *a,
> +                          struct list_head *b));
> +#endif
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> Implement the framework for syncing IRQs between our emulation and the
> list registers, which represent the guest's view of IRQs.
> This is done in kvm_vgic_flush_hwstate and kvm_vgic_sync_hwstate, which

You probably want to update the names here.

> gets called on guest entry and exit.
> The code talking to the actual GICv2/v3 hardware is added in the
> following patches.
> 
> This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 246 +++++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h |   2 +
>   2 files changed, 248 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index a4efd1fd03..a1f77130d4 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -380,6 +380,252 @@ int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int intid,
>       return 0;
>   }
>   
> +/**
> + * vgic_prune_ap_list - Remove non-relevant interrupts from the list
> + *
> + * @vcpu: The VCPU pointer
> + *
> + * Go over the list of "interesting" interrupts, and prune those that we
> + * won't have to consider in the near future.
> + */
> +static void vgic_prune_ap_list(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    struct vgic_irq *irq, *tmp;
> +    unsigned long flags;
> +
> +retry:
> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
> +
> +    list_for_each_entry_safe( irq, tmp, &vgic_cpu->ap_list_head, ap_list )

See my comment on patch #22, this is where I am worry about going 
through the list every time we enter to the hypervisor from the guest.

> +    {
> +        struct vcpu *target_vcpu, *vcpuA, *vcpuB;
> +
> +        spin_lock(&irq->irq_lock);
> +
> +        BUG_ON(vcpu != irq->vcpu);
> +
> +        target_vcpu = vgic_target_oracle(irq);
> +
> +        if ( !target_vcpu )
> +        {
> +            /*
> +             * We don't need to process this interrupt any
> +             * further, move it off the list.
> +             */
> +            list_del(&irq->ap_list);
> +            irq->vcpu = NULL;
> +            spin_unlock(&irq->irq_lock);
> +
> +            /*
> +             * This vgic_put_irq call matches the
> +             * vgic_get_irq_kref in vgic_queue_irq_unlock,
> +             * where we added the LPI to the ap_list. As
> +             * we remove the irq from the list, we drop
> +             * also drop the refcount.
> +             */
> +            vgic_put_irq(vcpu->domain, irq);
> +            continue;
> +        }
> +
> +        if ( target_vcpu == vcpu )
> +        {
> +            /* We're on the right CPU */
> +            spin_unlock(&irq->irq_lock);
> +            continue;
> +        }
> +
> +        /* This interrupt looks like it has to be migrated. */
> +
> +        spin_unlock(&irq->irq_lock);
> +        spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
> +
> +        /*
> +         * Ensure locking order by always locking the smallest
> +         * ID first.
> +         */
> +        if ( vcpu->vcpu_id < target_vcpu->vcpu_id )
> +        {
> +            vcpuA = vcpu;
> +            vcpuB = target_vcpu;
> +        }
> +        else
> +        {
> +            vcpuA = target_vcpu;
> +            vcpuB = vcpu;
> +        }
> +
> +        spin_lock_irqsave(&vcpuA->arch.vgic_cpu.ap_list_lock, flags);
> +        spin_lock(&vcpuB->arch.vgic_cpu.ap_list_lock);
> +        spin_lock(&irq->irq_lock);
> +
> +        /*
> +         * If the affinity has been preserved, move the
> +         * interrupt around. Otherwise, it means things have
> +         * changed while the interrupt was unlocked, and we
> +         * need to replay this.
> +         *
> +         * In all cases, we cannot trust the list not to have
> +         * changed, so we restart from the beginning.
> +         */
> +        if ( target_vcpu == vgic_target_oracle(irq) )
> +        {
> +            struct vgic_cpu *new_cpu = &target_vcpu->arch.vgic_cpu;
> +
> +            list_del(&irq->ap_list);
> +            irq->vcpu = target_vcpu;
> +            list_add_tail(&irq->ap_list, &new_cpu->ap_list_head);
> +        }
> +
> +        spin_unlock(&irq->irq_lock);
> +        spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
> +        spin_unlock_irqrestore(&vcpuA->arch.vgic_cpu.ap_list_lock, flags);
> +        goto retry;
> +    }
> +
> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
> +}
> +
> +static inline void vgic_fold_lr_state(struct vcpu *vcpu)
> +{
> +}
> +
> +/* Requires the irq_lock to be held. */
> +static inline void vgic_populate_lr(struct vcpu *vcpu,
> +                                    struct vgic_irq *irq, int lr)
> +{
> +    ASSERT(spin_is_locked(&irq->irq_lock));
> +}
> +
> +static inline void vgic_clear_lr(struct vcpu *vcpu, int lr)
> +{
> +}
> +
> +static inline void vgic_set_underflow(struct vcpu *vcpu)
> +{
> +}
> +
> +/* Requires the ap_list_lock to be held. */
> +static int compute_ap_list_depth(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    struct vgic_irq *irq;
> +    int count = 0;
> +
> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
> +
> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)

Here another example.

> +    {
> +        spin_lock(&irq->irq_lock);
> +        /* GICv2 SGIs can count for more than one... */
> +        if ( vgic_irq_is_sgi(irq->intid) && irq->source )
> +            count += hweight8(irq->source);
> +        else
> +            count++;
> +        spin_unlock(&irq->irq_lock);
> +    }
> +    return count;
> +}
> +
> +/* Requires the VCPU's ap_list_lock to be held. */
> +static void vgic_flush_lr_state(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    struct vgic_irq *irq;
> +    int count = 0;
> +
> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
> +
> +    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
> +        vgic_sort_ap_list(vcpu);
> +
> +    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
> +    {
> +        spin_lock(&irq->irq_lock);
> +
> +        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
> +            goto next;
> +
> +        /*
> +         * If we get an SGI with multiple sources, try to get
> +         * them in all at once.
> +         */
> +        do
> +        {
> +            vgic_populate_lr(vcpu, irq, count++);
> +        } while ( irq->source && count < gic_get_nr_lrs() );
> +
> +next:
> +        spin_unlock(&irq->irq_lock);
> +
> +        if ( count == gic_get_nr_lrs() )
> +        {
> +            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
> +                vgic_set_underflow(vcpu);
> +            break;
> +        }
> +    }
> +
> +    vcpu->arch.vgic_cpu.used_lrs = count;
> +
> +    /* Nuke remaining LRs */
> +    for ( ; count < gic_get_nr_lrs(); count++)
> +        vgic_clear_lr(vcpu, count);
> +}
> +
> +/*
> + * gic_clear_lrs() - Update the VGIC state from hardware after a guest's run.
> + * @vcpu: the VCPU.
> + *
> + * Sync back the hardware VGIC state after the guest has run, into our
> + * VGIC emulation structures, It reads the LRs and updates the respective
> + * struct vgic_irq, taking level/edge into account.
> + * This is the high level function which takes care of the conditions,
> + * also bails out early if there were no interrupts queued.
> + * Was: kvm_vgic_sync_hwstate()
> + */
> +void gic_clear_lrs(struct vcpu *vcpu)

I think I would prefer if we stick with the KVM name.

> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +
> +    /* An empty ap_list_head implies used_lrs == 0 */
> +    if ( list_empty(&vcpu->arch.vgic_cpu.ap_list_head) )
> +        return;
> +
> +    if ( vgic_cpu->used_lrs )
> +        vgic_fold_lr_state(vcpu);
> +    vgic_prune_ap_list(vcpu);
> +}
> +
> +/*
> + * gic_inject() - flush the emulation state into the hardware on guest entry
> + *
> + * Before we enter a guest, we have to translate the virtual GIC state of a
> + * VCPU into the GIC virtualization hardware registers, namely the LRs.
> + * This is the high level function which takes care about the conditions
> + * and the locking, also bails out early if there are no interrupts queued.
> + * Was: kvm_vgic_flush_hwstate()

Ditto.

> + */
> +void gic_inject(void)
> +{
> +    /*
> +     * If there are no virtual interrupts active or pending for this
> +     * VCPU, then there is no work to do and we can bail out without
> +     * taking any lock.  There is a potential race with someone injecting
> +     * interrupts to the VCPU, but it is a benign race as the VCPU will
> +     * either observe the new interrupt before or after doing this check,
> +     * and introducing additional synchronization mechanism doesn't change
> +     * this.
> +     */
> +    if ( list_empty(&current->arch.vgic_cpu.ap_list_head) )
> +        return;
> +
> +    ASSERT(!local_irq_is_enabled());
> +
> +    spin_lock(&current->arch.vgic_cpu.ap_list_lock);
> +    vgic_flush_lr_state(current);
> +    spin_unlock(&current->arch.vgic_cpu.ap_list_lock);
> +}
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 5127739f0f..47fc58b81e 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -17,6 +17,8 @@
>   #ifndef __XEN_ARM_VGIC_NEW_H__
>   #define __XEN_ARM_VGIC_NEW_H__
>   
> +#define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
> +
>   static inline bool irq_is_pending(struct vgic_irq *irq)
>   {
>       if ( irq->config == VGIC_CONFIG_EDGE )
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> +/* Requires the VCPU's ap_list_lock to be held. */
> +static void vgic_flush_lr_state(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    struct vgic_irq *irq;
> +    int count = 0;
> +
> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
> +
> +    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
> +        vgic_sort_ap_list(vcpu);
> +
> +    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
> +    {
> +        spin_lock(&irq->irq_lock);
> +
> +        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
> +            goto next;
> +
> +        /*
> +         * If we get an SGI with multiple sources, try to get
> +         * them in all at once.
> +         */
> +        do
> +        {
> +            vgic_populate_lr(vcpu, irq, count++);
> +        } while ( irq->source && count < gic_get_nr_lrs() );
> +
> +next:
> +        spin_unlock(&irq->irq_lock);
> +
> +        if ( count == gic_get_nr_lrs() )
> +        {
> +            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
> +                vgic_set_underflow(vcpu);
> +            break;
> +        }
> +    }
> +
> +    vcpu->arch.vgic_cpu.used_lrs = count;
> +
> +    /* Nuke remaining LRs */
> +    for ( ; count < gic_get_nr_lrs(); count++)
> +        vgic_clear_lr(vcpu, count);

Why do you need to nuke the LRs here, don't you always zero them when 
clearing it?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> Processing maintenance interrupts and accessing the list registers
> are dependent on the host's GIC version.
> Introduce vgic-v2.c to contain GICv2 specific functions.
> Implement the GICv2 specific code for syncing the emulation state
> into the VGIC registers.
> This also adds the hook to let Xen setup the host GIC addresses.
> 
> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-v2.c | 261 ++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.c    |  20 ++++
>   xen/arch/arm/vgic/vgic.h    |   8 ++
>   3 files changed, 289 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic-v2.c
> 
> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
> new file mode 100644
> index 0000000000..10fc467ffa
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-v2.c
> @@ -0,0 +1,261 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/arm_vgic.h>
> +#include <asm/bug.h>
> +#include <asm/io.h>
> +#include <xen/sched.h>
> +#include <xen/sizes.h>
> +
> +#include "vgic.h"
> +
> +#define GICH_ELRSR0                     0x30
> +#define GICH_ELRSR1                     0x34
> +#define GICH_LR0                        0x100
> +
> +#define GICH_LR_VIRTUALID               (0x3ff << 0)
> +#define GICH_LR_PHYSID_CPUID_SHIFT      (10)
> +#define GICH_LR_PHYSID_CPUID            (0x3ff << GICH_LR_PHYSID_CPUID_SHIFT)
> +#define GICH_LR_PRIORITY_SHIFT          23
> +#define GICH_LR_STATE                   (3 << 28)
> +#define GICH_LR_PENDING_BIT             (1 << 28)
> +#define GICH_LR_ACTIVE_BIT              (1 << 29)
> +#define GICH_LR_EOI                     (1 << 19)
> +#define GICH_LR_HW                      (1 << 31)

Can we define them in either in gic.h or a new header gic-v2.h?

> +
> +static struct {
> +    bool enabled;
> +    paddr_t dbase;          /* Distributor interface address */
> +    paddr_t cbase;          /* CPU interface address & size */
> +    paddr_t csize;
> +    paddr_t vbase;          /* Virtual CPU interface address */
> +    void __iomem *hbase;        /* Hypervisor control interface */
> +
> +    /* Offset to add to get an 8kB contiguous region if GIC is aliased */
> +    uint32_t aliased_offset;
> +} gic_v2_hw_data;
> +
> +void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
> +              paddr_t vbase, void __iomem *hbase,
> +              uint32_t aliased_offset)
> +{
> +    gic_v2_hw_data.enabled = true;
> +    gic_v2_hw_data.dbase = dbase;
> +    gic_v2_hw_data.cbase = cbase;
> +    gic_v2_hw_data.csize = csize;
> +    gic_v2_hw_data.vbase = vbase;
> +    gic_v2_hw_data.hbase = hbase;
> +    gic_v2_hw_data.aliased_offset = aliased_offset;
> +}
> +
> +void vgic_v2_set_underflow(struct vcpu *vcpu)
> +{
> +    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
> +}
> +
> +/*
> + * transfer the content of the LRs back into the corresponding ap_list:
> + * - active bit is transferred as is
> + * - pending bit is
> + *   - transferred as is in case of edge sensitive IRQs
> + *   - set to the line-level (resample time) for level sensitive IRQs
> + */
> +void vgic_v2_fold_lr_state(struct vcpu *vcpu)

I am wondering how much we could share this code with vgic_v3_fold_lr_state.

> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2;
> +    int lr;

unsigned please.

> +    unsigned long flags;
> +
> +    cpuif->vgic_hcr &= ~GICH_HCR_UIE;
> +
> +    for ( lr = 0; lr < vgic_cpu->used_lrs; lr++ )
> +    {
> +        u32 val = cpuif->vgic_lr[lr];
> +        u32 intid = val & GICH_LR_VIRTUALID;
> +        struct vgic_irq *irq;
> +
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        /* Always preserve the active bit */
> +        irq->active = !!(val & GICH_LR_ACTIVE_BIT);
> +
> +        /* Edge is the only case where we preserve the pending bit */
> +        if ( irq->config == VGIC_CONFIG_EDGE && (val & GICH_LR_PENDING_BIT) )
> +        {
> +            irq->pending_latch = true;
> +
> +            if ( vgic_irq_is_sgi(intid) )
> +            {
> +                u32 cpuid = val & GICH_LR_PHYSID_CPUID;
> +
> +                cpuid >>= GICH_LR_PHYSID_CPUID_SHIFT;
> +                irq->source |= (1 << cpuid);
> +            }
> +        }
> +

May I ask to keep the big comments from KVM around? It looks quite 
useful to have it.

> +        if ( irq->hw && irq->config == VGIC_CONFIG_LEVEL &&

You probably want to have the helper vgic_irq_is_mapped_level(...) as in 
KVM.

> +            (val & GICH_LR_PENDING_BIT) )
> +        {
> +            irq->line_level = gic_read_pending_state(irq->hwintid);
> +
> +            if ( !irq->line_level )
> +                            gic_set_active_state(irq->hwintid, true);
> +        }
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    vgic_cpu->used_lrs = 0;
> +}
> +
> +/*
> + * Populates the particular LR with the state of a given IRQ:
> + * - for an edge sensitive IRQ the pending state is cleared in struct vgic_irq
> + * - for a level sensitive IRQ the pending state value is unchanged;
> + *   it is dictated directly by the input level
> + *
> + * If @irq describes an SGI with multiple sources, we choose the
> + * lowest-numbered source VCPU and clear that bit in the source bitmap.
> + *
> + * The irq_lock must be held by the caller.
> + */
> +void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)

I am wondering how much we could share this code with vgic_v3_populate_lr.

> +{
> +    u32 val = irq->intid;
> +
> +    if ( irq_is_pending(irq) )
> +    {
> +        val |= GICH_LR_PENDING_BIT;
> +
> +        if ( irq->config == VGIC_CONFIG_EDGE )
> +            irq->pending_latch = false;
> +
> +        if ( vgic_irq_is_sgi(irq->intid) )
> +        {
> +            u32 src = ffs(irq->source);
> +
> +            BUG_ON(!src);
> +            val |= (src - 1) << GICH_LR_PHYSID_CPUID_SHIFT;
> +            irq->source &= ~(1 << (src - 1));
> +            if ( irq->source )
> +                irq->pending_latch = true;
> +        }
> +    }
> +
> +    if ( irq->active )
> +        val |= GICH_LR_ACTIVE_BIT;
> +
> +    if ( irq->hw )
> +    {
> +        val |= GICH_LR_HW;
> +        val |= irq->hwintid << GICH_LR_PHYSID_CPUID_SHIFT;
> +        /*
> +         * Never set pending+active on a HW interrupt, as the
> +         * pending state is kept at the physical distributor
> +         * level.
> +         */
> +        if ( irq->active && irq_is_pending(irq) )
> +            val &= ~GICH_LR_PENDING_BIT;
> +    }
> +    else
> +    {
> +        if ( irq->config == VGIC_CONFIG_LEVEL )
> +            val |= GICH_LR_EOI;
> +    }
> +
> +    /*
> +     * Level-triggered mapped IRQs are special because we only observe
> +     * rising edges as input to the VGIC.  We therefore lower the line
> +     * level here, so that we can take new virtual IRQs.  See
> +     * vgic_v2_fold_lr_state for more info.
> +     */
> +    if ( irq->hw && irq->config == VGIC_CONFIG_LEVEL &&

Same remark for the helper.

> +        (val & GICH_LR_PENDING_BIT) )
> +        irq->line_level = false;
> +
> +    /* The GICv2 LR only holds five bits of priority. */
> +    val |= (irq->priority >> 3) << GICH_LR_PRIORITY_SHIFT;
> +
> +    vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val;
> +}
> +
> +void vgic_v2_clear_lr(struct vcpu *vcpu, int lr)
> +{
> +    vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = 0;
> +}
> +
> +static void save_lrs(struct vcpu *vcpu, void __iomem *base)
> +{
> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
> +    u64 elrsr;
> +    int i;
> +
> +    elrsr = readl_relaxed(base + GICH_ELRSR0);
> +    if ( unlikely(used_lrs > 32) )
> +        elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
> +
> +    for ( i = 0; i < used_lrs; i++ )
> +    {
> +        if ( elrsr & (1UL << i) )
> +            cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
> +        else
> +            cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i * 4));
> +
> +        writel_relaxed(0, base + GICH_LR0 + (i * 4));
> +    }
> +}
> +
> +void vgic_v2_save_state(struct vcpu *vcpu)
> +{
> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
> +
> +    if ( used_lrs )
> +    {
> +        save_lrs(vcpu, gic_v2_hw_data.hbase);
> +        writel_relaxed(0, gic_v2_hw_data.hbase + GICH_HCR);
> +    }
> +}

I am not entirely convinced that have a separate function to save the 
LRs is necessary. This could be done in fold_lr_state().

> +
> +void vgic_v2_restore_state(struct vcpu *vcpu)
> +{
> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
> +    int i;
> +
> +    if ( used_lrs )
> +    {
> +        writel_relaxed(cpu_if->vgic_hcr,
> +                       gic_v2_hw_data.hbase + GICH_HCR);
> +        for ( i = 0; i < used_lrs; i++ )
> +            writel_relaxed(cpu_if->vgic_lr[i],
> +                           gic_v2_hw_data.hbase + GICH_LR0 + (i * 4));
> +    }

Same here but with populate_lr_state(). This would make the code easier 
to follow and also avoid a lot ifery in the vgic.c code.

> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index a1f77130d4..f4f2a04a60 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -488,6 +488,7 @@ retry:
>   
>   static inline void vgic_fold_lr_state(struct vcpu *vcpu)
>   {
> +    vgic_v2_fold_lr_state(vcpu);
>   }
>   
>   /* Requires the irq_lock to be held. */
> @@ -495,14 +496,18 @@ static inline void vgic_populate_lr(struct vcpu *vcpu,
>                                       struct vgic_irq *irq, int lr)
>   {
>       ASSERT(spin_is_locked(&irq->irq_lock));
> +
> +    vgic_v2_populate_lr(vcpu, irq, lr);
>   }
>   
>   static inline void vgic_clear_lr(struct vcpu *vcpu, int lr)
>   {
> +    vgic_v2_clear_lr(vcpu, lr);
>   }
>   
>   static inline void vgic_set_underflow(struct vcpu *vcpu)
>   {
> +    vgic_v2_set_underflow(vcpu);
>   }
>   
>   /* Requires the ap_list_lock to be held. */
> @@ -573,6 +578,11 @@ next:
>           vgic_clear_lr(vcpu, count);
>   }
>   
> +static inline void vgic_save_state(struct vcpu *vcpu)
> +{
> +    vgic_v2_save_state(vcpu);
> +}
> +
>   /*
>    * gic_clear_lrs() - Update the VGIC state from hardware after a guest's run.
>    * @vcpu: the VCPU.
> @@ -592,11 +602,18 @@ void gic_clear_lrs(struct vcpu *vcpu)
>       if ( list_empty(&vcpu->arch.vgic_cpu.ap_list_head) )
>           return;
>   
> +    vgic_save_state(vcpu);
> +
>       if ( vgic_cpu->used_lrs )
>           vgic_fold_lr_state(vcpu);
>       vgic_prune_ap_list(vcpu);
>   }
>   
> +static inline void vgic_restore_state(struct vcpu *vcpu)
> +{
> +    vgic_v2_restore_state(vcpu);
> +}
> +
>   /*
>    * gic_inject() - flush the emulation state into the hardware on guest entry
>    *
> @@ -625,7 +642,10 @@ void gic_inject(void)
>       spin_lock(&current->arch.vgic_cpu.ap_list_lock);
>       vgic_flush_lr_state(current);
>       spin_unlock(&current->arch.vgic_cpu.ap_list_lock);
> +
> +    vgic_restore_state(current);
>   }
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 47fc58b81e..771ca6f046 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -41,4 +41,12 @@ static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>       atomic_inc(&irq->refcount);
>   }
>   
> +void vgic_v2_fold_lr_state(struct vcpu *vcpu);
> +void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
> +void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
> +void vgic_v2_set_underflow(struct vcpu *vcpu);
> +
> +void vgic_v2_save_state(struct vcpu *vcpu);
> +void vgic_v2_restore_state(struct vcpu *vcpu);
> +
>   #endif
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Andre Przywara]

Hi,

On 13/02/18 14:31, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> +/* Requires the VCPU's ap_list_lock to be held. */
>> +static void vgic_flush_lr_state(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_irq *irq;
>> +    int count = 0;
>> +
>> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
>> +
>> +    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
>> +        vgic_sort_ap_list(vcpu);
>> +
>> +    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
>> +    {
>> +        spin_lock(&irq->irq_lock);
>> +
>> +        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
>> +            goto next;
>> +
>> +        /*
>> +         * If we get an SGI with multiple sources, try to get
>> +         * them in all at once.
>> +         */
>> +        do
>> +        {
>> +            vgic_populate_lr(vcpu, irq, count++);
>> +        } while ( irq->source && count < gic_get_nr_lrs() );
>> +
>> +next:
>> +        spin_unlock(&irq->irq_lock);
>> +
>> +        if ( count == gic_get_nr_lrs() )
>> +        {
>> +            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
>> +                vgic_set_underflow(vcpu);
>> +            break;
>> +        }
>> +    }
>> +
>> +    vcpu->arch.vgic_cpu.used_lrs = count;
>> +
>> +    /* Nuke remaining LRs */
>> +    for ( ; count < gic_get_nr_lrs(); count++)
>> +        vgic_clear_lr(vcpu, count);
> 
> Why do you need to nuke the LRs here, don't you always zero them when
> clearing it?

We nuke our internal LR copies in here.
It might be interesting to see if we can get rid of those in Xen,
because we can always write to the LRs directly. But this is an
optimization I am not too keen on addressing too early, because this
deviates from the KVM VGIC architecture.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 23/49] ARM: new VGIC: Add IRQ sorting

[Andre Przywara]

Hi,

Christoffer, Eric, Marc,
a question about locking order between multiple IRQs below. Could you
have a brief look, please?

On 13/02/18 12:30, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Adds the sorting function to cover the case where you have more IRQs
>> to consider than you have LRs. We consider their priorities.
>> This pulls in Linux' list_sort.c , which is a merge sort implementation
>> for linked lists.
>>
>> This is based on Linux commit 8e4447457965, written by Christoffer Dall.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c    |  59 +++++++++++++++
>>   xen/common/list_sort.c      | 170
>> ++++++++++++++++++++++++++++++++++++++++++++
>>   xen/include/xen/list_sort.h |  11 +++
> 
> You need to CC "THE REST" maintainers for this code. It would also make
> sense to have a separate patch for adding list_sort.c

Yeah, will do.

>>   3 files changed, 240 insertions(+)
>>   create mode 100644 xen/common/list_sort.c
>>   create mode 100644 xen/include/xen/list_sort.h
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index f517df6d00..a4efd1fd03 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -16,6 +16,7 @@
>>    */
>>     #include <asm/bug.h>
>> +#include <xen/list_sort.h>
>>   #include <xen/sched.h>
>>     #include <asm/arm_vgic.h>
>> @@ -163,6 +164,64 @@ static struct vcpu *vgic_target_oracle(struct
>> vgic_irq *irq)
>>       return NULL;
>>   }
>>   +/*
>> + * The order of items in the ap_lists defines how we'll pack things
>> in LRs as
>> + * well, the first items in the list being the first things populated
>> in the
>> + * LRs.
>> + *
>> + * A hard rule is that active interrupts can never be pushed out of
>> the LRs
>> + * (and therefore take priority) since we cannot reliably trap on
>> deactivation
>> + * of IRQs and therefore they have to be present in the LRs.
>> + *
>> + * Otherwise things should be sorted by the priority field and the GIC
>> + * hardware support will take care of preemption of priority groups etc.
>> + *
>> + * Return negative if "a" sorts before "b", 0 to preserve order, and
>> positive
>> + * to sort "b" before "a".
> 
> Finally a good explanation of the return value of a sort function :). I
> always get confused what the return is supposed to be.
> 
>> + */
>> +static int vgic_irq_cmp(void *priv, struct list_head *a, struct
>> list_head *b)
>> +{
>> +    struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list);
>> +    struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list);
>> +    bool penda, pendb;
>> +    int ret;
>> +
>> +    spin_lock(&irqa->irq_lock);
>> +    spin_lock(&irqb->irq_lock);
> 
> I guess the locking order does not matter here because this is the only
> place where two IRQs lock have to be taken?

Mmh, good question. I guess indeed in practice this will not be a problem:
- As you mentioned this should be the only(?) place where we take
multiple IRQ locks, but that sounds fragile.
- A certain IRQ should only be on one VCPU list at a given point in
time. So there would be no race with two instances of this compare
function trying to lock the same IRQ.

But that sounds a bit dodgy to rely on. It should be relatively straight
forward to fix this with a simple comparison, shouldn't it?
CC:ing Christoffer, Marc and Eric here to see if we should add this (in
KVM as well).

> Also, this will be done with irq disabled right? In that case, may I ask
> for an ASSERT(!local_irq_is_enabled())? Or maybe in vgic_sort_ap_list.

OK.

>> +
>> +    if ( irqa->active || irqb->active )
>> +    {
>> +        ret = (int)irqb->active - (int)irqa->active;
>> +        goto out;
>> +    }
>> +
>> +    penda = irqa->enabled && irq_is_pending(irqa);
>> +    pendb = irqb->enabled && irq_is_pending(irqb);
>> +
>> +    if ( !penda || !pendb )
>> +    {
>> +        ret = (int)pendb - (int)penda;
>> +        goto out;
>> +    }
>> +
>> +    /* Both pending and enabled, sort by priority */
>> +    ret = irqa->priority - irqb->priority;
>> +out:
>> +    spin_unlock(&irqb->irq_lock);
>> +    spin_unlock(&irqa->irq_lock);
>> +    return ret;
>> +}
>> +
>> +/* Must be called with the ap_list_lock held */
>> +static void vgic_sort_ap_list(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +
>> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
>> +
>> +    list_sort(NULL, &vgic_cpu->ap_list_head, vgic_irq_cmp);
>> +}
>> +
>>   /*
>>    * Only valid injection if changing level for level-triggered IRQs
>> or for a
>>    * rising edge.
>> diff --git a/xen/common/list_sort.c b/xen/common/list_sort.c
>> new file mode 100644
>> index 0000000000..9c5cc58e43
>> --- /dev/null
>> +++ b/xen/common/list_sort.c
>> @@ -0,0 +1,170 @@
>> +/*
>> + * list_sort.c: merge sort implementation for linked lists
>> + * Copied from the Linux kernel (lib/list_sort.c)
>> + * (without specific copyright notice there)
> 
> I can see you moved from Linux to Xen coding style. Is there any other
> changes made?

Just the list of include files, but I didn't touch any actual code.
Will mention this in the commit message for this separate patch.

Cheers,
Andre.

> 
>> + *
>> + * This program is free software; you can redistribute it and/or
>> modify it
>> + * under the terms and conditions of the GNU General Public License,
>> + * version 2, as published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope it will be useful, but
>> WITHOUT
>> + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
>> + * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public
>> License for
>> + * more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> along with
>> + * this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +#include <xen/lib.h>
>> +#include <xen/list.h>
>> +
>> +#define MAX_LIST_LENGTH_BITS 20
>> +
>> +/*
>> + * Returns a list organized in an intermediate format suited
>> + * to chaining of merge() calls: null-terminated, no reserved or
>> + * sentinel head node, "prev" links not maintained.
>> + */
>> +static struct list_head *merge(void *priv,
>> +                               int (*cmp)(void *priv, struct
>> list_head *a,
>> +                                          struct list_head *b),
>> +                               struct list_head *a, struct list_head *b)
>> +{
>> +    struct list_head head, *tail = &head;
>> +
>> +    while ( a && b )
>> +    {
>> +        /* if equal, take 'a' -- important for sort stability */
>> +        if ( (*cmp)(priv, a, b) <= 0 )
>> +        {
>> +            tail->next = a;
>> +            a = a->next;
>> +        }
>> +        else
>> +        {
>> +            tail->next = b;
>> +            b = b->next;
>> +        }
>> +        tail = tail->next;
>> +    }
>> +    tail->next = a?:b;
>> +    return head.next;
>> +}
>> +
>> +/*
>> + * Combine final list merge with restoration of standard doubly-linked
>> + * list structure.  This approach duplicates code from merge(), but
>> + * runs faster than the tidier alternatives of either a separate final
>> + * prev-link restoration pass, or maintaining the prev links
>> + * throughout.
>> + */
>> +static void merge_and_restore_back_links(void *priv,
>> +                                         int (*cmp)(void *priv,
>> +                                                    struct list_head *a,
>> +                                                    struct list_head
>> *b),
>> +                                         struct list_head *head,
>> +                                         struct list_head *a,
>> +                                         struct list_head *b)
>> +{
>> +    struct list_head *tail = head;
>> +    u8 count = 0;
>> +
>> +    while ( a && b )
>> +    {
>> +        /* if equal, take 'a' -- important for sort stability */
>> +        if ( (*cmp)(priv, a, b) <= 0 )
>> +        {
>> +            tail->next = a;
>> +            a->prev = tail;
>> +            a = a->next;
>> +        }
>> +        else
>> +        {
>> +            tail->next = b;
>> +            b->prev = tail;
>> +            b = b->next;
>> +        }
>> +        tail = tail->next;
>> +    }
>> +    tail->next = a ? : b;
>> +
>> +    do
>> +    {
>> +        /*
>> +         * In worst cases this loop may run many iterations.
>> +         * Continue callbacks to the client even though no
>> +         * element comparison is needed, so the client's cmp()
>> +         * routine can invoke cond_resched() periodically.
>> +         */
>> +        if ( unlikely(!(++count)) )
>> +            (*cmp)(priv, tail->next, tail->next);
>> +
>> +        tail->next->prev = tail;
>> +        tail = tail->next;
>> +    } while ( tail->next );
>> +
>> +    tail->next = head;
>> +    head->prev = tail;
>> +}
>> +
>> +/**
>> + * list_sort - sort a list
>> + * @priv: private data, opaque to list_sort(), passed to @cmp
>> + * @head: the list to sort
>> + * @cmp: the elements comparison function
>> + *
>> + * This function implements "merge sort", which has O(nlog(n))
>> + * complexity.
>> + *
>> + * The comparison function @cmp must return a negative value if @a
>> + * should sort before @b, and a positive value if @a should sort after
>> + * @b. If @a and @b are equivalent, and their original relative
>> + * ordering is to be preserved, @cmp must return 0.
>> + */
>> +void list_sort(void *priv, struct list_head *head,
>> +               int (*cmp)(void *priv, struct list_head *a, struct
>> list_head *b))
>> +{
>> +    struct list_head *part[MAX_LIST_LENGTH_BITS+1]; /* sorted partial
>> lists
>> +                        -- last slot is a sentinel */
>> +    int lev;  /* index into part[] */
>> +    int max_lev = 0;
>> +    struct list_head *list;
>> +
>> +    if ( list_empty(head) )
>> +        return;
>> +
>> +    memset(part, 0, sizeof(part));
>> +
>> +    head->prev->next = NULL;
>> +    list = head->next;
>> +
>> +    while ( list )
>> +    {
>> +        struct list_head *cur = list;
>> +        list = list->next;
>> +        cur->next = NULL;
>> +
>> +        for ( lev = 0; part[lev]; lev++ )
>> +        {
>> +            cur = merge(priv, cmp, part[lev], cur);
>> +            part[lev] = NULL;
>> +        }
>> +        if ( lev > max_lev )
>> +        {
>> +            if ( unlikely(lev >= ARRAY_SIZE(part)-1) )
>> +            {
>> +                dprintk(XENLOG_DEBUG, "list too long for efficiency\n");
>> +                lev--;
>> +            }
>> +            max_lev = lev;
>> +        }
>> +        part[lev] = cur;
>> +    }
>> +
>> +    for ( lev = 0; lev < max_lev; lev++ )
>> +        if ( part[lev] )
>> +            list = merge(priv, cmp, part[lev], list);
>> +
>> +    merge_and_restore_back_links(priv, cmp, head, part[max_lev], list);
>> +}
>> +EXPORT_SYMBOL(list_sort);
>> diff --git a/xen/include/xen/list_sort.h b/xen/include/xen/list_sort.h
>> new file mode 100644
>> index 0000000000..a60c589d4b
>> --- /dev/null
>> +++ b/xen/include/xen/list_sort.h
>> @@ -0,0 +1,11 @@
>> +#ifndef _LINUX_LIST_SORT_H
>> +#define _LINUX_LIST_SORT_H
>> +
>> +#include <xen/types.h>
>> +
>> +struct list_head;
>> +
>> +void list_sort(void *priv, struct list_head *head,
>> +               int (*cmp)(void *priv, struct list_head *a,
>> +                          struct list_head *b));
>> +#endif
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 23/49] ARM: new VGIC: Add IRQ sorting

[Julien Grall]

On 13/02/18 14:56, Andre Przywara wrote:
>>> diff --git a/xen/common/list_sort.c b/xen/common/list_sort.c
>>> new file mode 100644
>>> index 0000000000..9c5cc58e43
>>> --- /dev/null
>>> +++ b/xen/common/list_sort.c
>>> @@ -0,0 +1,170 @@
>>> +/*
>>> + * list_sort.c: merge sort implementation for linked lists
>>> + * Copied from the Linux kernel (lib/list_sort.c)
>>> + * (without specific copyright notice there)
>>
>> I can see you moved from Linux to Xen coding style. Is there any other
>> changes made?
> 
> Just the list of include files, but I didn't touch any actual code.
> Will mention this in the commit message for this separate patch.

Can you keep the list coding style in that case please?

Thank you.

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 15/49] ARM: GIC: Allow tweaking the active state of an IRQ

[Andre Przywara]

Hi,

On 13/02/18 12:02, Julien Grall wrote:
> On 12/02/18 17:53, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 12/02/18 13:55, Julien Grall wrote:
>>> Hi Andre,
>>>
>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>> When playing around with hardware mapped, level triggered virtual IRQs,
>>>> there is the need to explicitly set the active state of an interrupt at
>>>> some point in time.
>>>> To prepare the GIC for that, we introduce a set_active_state() function
>>>> to let the VGIC manipulate the state of an associated hardware IRQ.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>> ---
>>>>    xen/arch/arm/gic-v2.c     |  9 +++++++++
>>>>    xen/arch/arm/gic-v3.c     | 16 ++++++++++++++++
>>>>    xen/arch/arm/gic.c        |  5 +++++
>>>>    xen/include/asm-arm/gic.h |  5 +++++
>>>>    4 files changed, 35 insertions(+)
>>>>
>>>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>>>> index 2e35892881..5339f69fbc 100644
>>>> --- a/xen/arch/arm/gic-v2.c
>>>> +++ b/xen/arch/arm/gic-v2.c
>>>> @@ -235,6 +235,14 @@ static unsigned int gicv2_read_irq(void)
>>>>        return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>>>>    }
>>>>    +static void gicv2_set_active_state(int irq, bool active)
>>>
>>> I would much prefer to have an irq_desc in parameter. This is matching
>>> the other interface
>>
>> ... and that's why I had it just like this in my first version. However
>> this proved to be nasty because I now need to get this irq_desc pointer
>> first, as the caller doesn't have it already. Since all we have and need
>> is the actual hardware IRQ number, I found it more straight-forward to
>> just use that number directly instead of going via the pointer and back
>> (h/w intid => irq_desc => irq).
>>
>>> and you could update the flags such as
>>> _IRQ_INPROGRESS which you don't do at the moment.
>>
>> Mmh, interesting point. I guess I should also clear this bit in the new
>> VGIC. At least once I wrapped my head around what this flag is
>> *actually* for (in conjunction with _IRQ_GUEST).
>> Anyway I guess this bit would still be set in our case.
> 
> For IRQ routed to the guest, the flag is used to know whether you need
> to EOI the interrupt on domain destruction.

Yeah, I found that. In general I am a bit suspicious of replicating and
tracking the hardware IRQ state in software.

> In general, I would like to keep desc->status in sync for the guest IRQ.
> This is useful for debugging and potentially some ratelimit on interrupt
> (I am thinking for ITS).
> 
>>
>>> Also, who is preventing two CPUs to clear the active bit at the same
>>> time?
>>
>> A certain hardware IRQ is assigned to one virtual IRQ on one VCPU at one
>> time only. Besides, GICD_ICACTIVERn has wired NAND semantics, so that's
>> naturally race free (as it was designed to be).
>> Unless I miss something here (happy to be pointed to an example where it
>> causes problems).
> 
> You could potentially have a race between ICACTIVER an ISACTIVER.

I don't see why this would be a problem:
Either you activate the IRQ or you deactivate it. The
wired-OR/wired-NAND semantics makes sure this never gets inconsistent on
the hardware side. If you issue two conflicting requests at the same
time, that's a benign race, which you either don't care about or handle
via locking in the code which triggers these requests.

Besides, we only do one direction in the code at the moment anyway.
And this should be *clearing* the active state, and not setting it,
which is a bug I discovered yesterday.

> is very similar to the enable/disable part. This matters a lot when
> updating desc->status.

Which is one reason why I am suspicious of this whole state replication.
But the desc lock should take care of this in general, no?

>>>> +}
>>>> +
>>>>    static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int
>>>> type)
>>>>    {
>>>>        uint32_t cfg, actual, edgebit;
>>>> @@ -1241,6 +1249,7 @@ const static struct gic_hw_operations
>>>> gicv2_ops = {
>>>>        .eoi_irq             = gicv2_eoi_irq,
>>>>        .deactivate_irq      = gicv2_dir_irq,
>>>>        .read_irq            = gicv2_read_irq,
>>>> +    .set_active_state    = gicv2_set_active_state,
>>>>        .set_irq_type        = gicv2_set_irq_type,
>>>>        .set_irq_priority    = gicv2_set_irq_priority,
>>>>        .send_SGI            = gicv2_send_SGI,
>>>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>>>> index 08d4703687..595eaef43a 100644
>>>> --- a/xen/arch/arm/gic-v3.c
>>>> +++ b/xen/arch/arm/gic-v3.c
>>>> @@ -475,6 +475,21 @@ static unsigned int gicv3_read_irq(void)
>>>>        return irq;
>>>>    }
>>>>    +static void gicv3_set_active_state(int irq, bool active)
>>>> +{
>>>> +    void __iomem *base;
>>>> +
>>>> +    if ( irq >= NR_GIC_LOCAL_IRQS)
>>>> +        base = GICD + (irq / 32) * 4;
>>>> +    else
>>>> +        base = GICD_RDIST_SGI_BASE;
>>>> +
>>>> +    if ( active )
>>>> +        writel(1U << (irq % 32), base + GICD_ISACTIVER);
>>>> +    else
>>>> +        writel(1U << (irq % 32), base + GICD_ICACTIVER);
>>>
>>> Shouldn't you wait until RWP bits is cleared here?
>>
>> I don't see why. I think this action has some posted semantics anyway,
>> so no need for any synchronisation. And also RWP does not track
>> I[SC]ACTIVER, only ICENABLER and some CTLR bits (ARM IHI 0069D, 8.9.4:
>> RWP[31]).
>>
>>>
>>>> +}
>>>
>>> Why don't you use the function poke?
>>
>> Ah, I didn't see this. But then this now does this quite costly RWP
>> dance now. We could add a check in there to only do this if we change
>> the affected registers or pass an explicit "bool wait_for_rwp" in there.
> 
> I guess this would be useful even for the current code. If I understand
> correctly the RWP semantics, it should not be necessary to wait when
> write to ISENABLER also.

Exactly. I changed poke_irq() to do:
    if ( offset == GICD_ICENABLER )
        gicv3_wait_for_rwp(irqd->irq);

Does that sound acceptable?

I also just added poke_irq()/peek_irq() to gic-v2.c as well.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Julien Grall]

On 13/02/18 14:56, Andre Przywara wrote:
> Hi,
> 
> On 13/02/18 14:31, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> +/* Requires the VCPU's ap_list_lock to be held. */
>>> +static void vgic_flush_lr_state(struct vcpu *vcpu)
>>> +{
>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>> +    struct vgic_irq *irq;
>>> +    int count = 0;
>>> +
>>> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
>>> +
>>> +    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
>>> +        vgic_sort_ap_list(vcpu);
>>> +
>>> +    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
>>> +    {
>>> +        spin_lock(&irq->irq_lock);
>>> +
>>> +        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
>>> +            goto next;
>>> +
>>> +        /*
>>> +         * If we get an SGI with multiple sources, try to get
>>> +         * them in all at once.
>>> +         */
>>> +        do
>>> +        {
>>> +            vgic_populate_lr(vcpu, irq, count++);
>>> +        } while ( irq->source && count < gic_get_nr_lrs() );
>>> +
>>> +next:
>>> +        spin_unlock(&irq->irq_lock);
>>> +
>>> +        if ( count == gic_get_nr_lrs() )
>>> +        {
>>> +            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
>>> +                vgic_set_underflow(vcpu);
>>> +            break;
>>> +        }
>>> +    }
>>> +
>>> +    vcpu->arch.vgic_cpu.used_lrs = count;
>>> +
>>> +    /* Nuke remaining LRs */
>>> +    for ( ; count < gic_get_nr_lrs(); count++)
>>> +        vgic_clear_lr(vcpu, count);
>>
>> Why do you need to nuke the LRs here, don't you always zero them when
>> clearing it?
> 
> We nuke our internal LR copies in here.
> It might be interesting to see if we can get rid of those in Xen,
> because we can always write to the LRs directly. But this is an
> optimization I am not too keen on addressing too early, because this
> deviates from the KVM VGIC architecture.

Oh, I thought you were writing back in the hardware when clearing. My 
mistake.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Andre Przywara]

Hi,

On 13/02/18 12:41, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Implement the framework for syncing IRQs between our emulation and the
>> list registers, which represent the guest's view of IRQs.
>> This is done in kvm_vgic_flush_hwstate and kvm_vgic_sync_hwstate, which
> 
> You probably want to update the names here.

Sure.

>> gets called on guest entry and exit.
>> The code talking to the actual GICv2/v3 hardware is added in the
>> following patches.
>>
>> This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 246
>> +++++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.h |   2 +
>>   2 files changed, 248 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index a4efd1fd03..a1f77130d4 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -380,6 +380,252 @@ int vgic_inject_irq(struct domain *d, struct
>> vcpu *vcpu, unsigned int intid,
>>       return 0;
>>   }
>>   +/**
>> + * vgic_prune_ap_list - Remove non-relevant interrupts from the list
>> + *
>> + * @vcpu: The VCPU pointer
>> + *
>> + * Go over the list of "interesting" interrupts, and prune those that we
>> + * won't have to consider in the near future.
>> + */
>> +static void vgic_prune_ap_list(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_irq *irq, *tmp;
>> +    unsigned long flags;
>> +
>> +retry:
>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>> +
>> +    list_for_each_entry_safe( irq, tmp, &vgic_cpu->ap_list_head,
>> ap_list )
> 
> See my comment on patch #22, this is where I am worry about going
> through the list every time we enter to the hypervisor from the guest.

I am not sure we can avoid this here, as this function is crucial to the
VGIC state machine.
We might later look into if we can avoid iterating through the whole
list or if we can shortcut some interrupts somehow, but I really would
be careful tinkering with this function too much.

>> +    {
>> +        struct vcpu *target_vcpu, *vcpuA, *vcpuB;
>> +
>> +        spin_lock(&irq->irq_lock);
>> +
>> +        BUG_ON(vcpu != irq->vcpu);
>> +
>> +        target_vcpu = vgic_target_oracle(irq);
>> +
>> +        if ( !target_vcpu )
>> +        {
>> +            /*
>> +             * We don't need to process this interrupt any
>> +             * further, move it off the list.
>> +             */
>> +            list_del(&irq->ap_list);
>> +            irq->vcpu = NULL;
>> +            spin_unlock(&irq->irq_lock);
>> +
>> +            /*
>> +             * This vgic_put_irq call matches the
>> +             * vgic_get_irq_kref in vgic_queue_irq_unlock,
>> +             * where we added the LPI to the ap_list. As
>> +             * we remove the irq from the list, we drop
>> +             * also drop the refcount.
>> +             */
>> +            vgic_put_irq(vcpu->domain, irq);
>> +            continue;
>> +        }
>> +
>> +        if ( target_vcpu == vcpu )
>> +        {
>> +            /* We're on the right CPU */
>> +            spin_unlock(&irq->irq_lock);
>> +            continue;
>> +        }
>> +
>> +        /* This interrupt looks like it has to be migrated. */
>> +
>> +        spin_unlock(&irq->irq_lock);
>> +        spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>> +
>> +        /*
>> +         * Ensure locking order by always locking the smallest
>> +         * ID first.
>> +         */
>> +        if ( vcpu->vcpu_id < target_vcpu->vcpu_id )
>> +        {
>> +            vcpuA = vcpu;
>> +            vcpuB = target_vcpu;
>> +        }
>> +        else
>> +        {
>> +            vcpuA = target_vcpu;
>> +            vcpuB = vcpu;
>> +        }
>> +
>> +        spin_lock_irqsave(&vcpuA->arch.vgic_cpu.ap_list_lock, flags);
>> +        spin_lock(&vcpuB->arch.vgic_cpu.ap_list_lock);
>> +        spin_lock(&irq->irq_lock);
>> +
>> +        /*
>> +         * If the affinity has been preserved, move the
>> +         * interrupt around. Otherwise, it means things have
>> +         * changed while the interrupt was unlocked, and we
>> +         * need to replay this.
>> +         *
>> +         * In all cases, we cannot trust the list not to have
>> +         * changed, so we restart from the beginning.
>> +         */
>> +        if ( target_vcpu == vgic_target_oracle(irq) )
>> +        {
>> +            struct vgic_cpu *new_cpu = &target_vcpu->arch.vgic_cpu;
>> +
>> +            list_del(&irq->ap_list);
>> +            irq->vcpu = target_vcpu;
>> +            list_add_tail(&irq->ap_list, &new_cpu->ap_list_head);
>> +        }
>> +
>> +        spin_unlock(&irq->irq_lock);
>> +        spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
>> +        spin_unlock_irqrestore(&vcpuA->arch.vgic_cpu.ap_list_lock,
>> flags);
>> +        goto retry;
>> +    }
>> +
>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>> +}
>> +
>> +static inline void vgic_fold_lr_state(struct vcpu *vcpu)
>> +{
>> +}
>> +
>> +/* Requires the irq_lock to be held. */
>> +static inline void vgic_populate_lr(struct vcpu *vcpu,
>> +                                    struct vgic_irq *irq, int lr)
>> +{
>> +    ASSERT(spin_is_locked(&irq->irq_lock));
>> +}
>> +
>> +static inline void vgic_clear_lr(struct vcpu *vcpu, int lr)
>> +{
>> +}
>> +
>> +static inline void vgic_set_underflow(struct vcpu *vcpu)
>> +{
>> +}
>> +
>> +/* Requires the ap_list_lock to be held. */
>> +static int compute_ap_list_depth(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_irq *irq;
>> +    int count = 0;
>> +
>> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
>> +
>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
> 
> Here another example.

This can be short cut, indeed. The only place we call this function is
where we compare it against the number of LRs below.
So we don't need to know the actual number of elements, but could bail
out once we reached the number of LRs.
This function here would then return a bool, comparing internally
against the number of LRs already.

>> +    {
>> +        spin_lock(&irq->irq_lock);
>> +        /* GICv2 SGIs can count for more than one... */
>> +        if ( vgic_irq_is_sgi(irq->intid) && irq->source )
>> +            count += hweight8(irq->source);
>> +        else
>> +            count++;
>> +        spin_unlock(&irq->irq_lock);
>> +    }
>> +    return count;
>> +}
>> +
>> +/* Requires the VCPU's ap_list_lock to be held. */
>> +static void vgic_flush_lr_state(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_irq *irq;
>> +    int count = 0;
>> +
>> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
>> +
>> +    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
>> +        vgic_sort_ap_list(vcpu);
>> +
>> +    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
>> +    {
>> +        spin_lock(&irq->irq_lock);
>> +
>> +        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
>> +            goto next;
>> +
>> +        /*
>> +         * If we get an SGI with multiple sources, try to get
>> +         * them in all at once.
>> +         */
>> +        do
>> +        {
>> +            vgic_populate_lr(vcpu, irq, count++);
>> +        } while ( irq->source && count < gic_get_nr_lrs() );
>> +
>> +next:
>> +        spin_unlock(&irq->irq_lock);
>> +
>> +        if ( count == gic_get_nr_lrs() )
>> +        {
>> +            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
>> +                vgic_set_underflow(vcpu);
>> +            break;
>> +        }
>> +    }
>> +
>> +    vcpu->arch.vgic_cpu.used_lrs = count;
>> +
>> +    /* Nuke remaining LRs */
>> +    for ( ; count < gic_get_nr_lrs(); count++)
>> +        vgic_clear_lr(vcpu, count);
>> +}
>> +
>> +/*
>> + * gic_clear_lrs() - Update the VGIC state from hardware after a
>> guest's run.
>> + * @vcpu: the VCPU.
>> + *
>> + * Sync back the hardware VGIC state after the guest has run, into our
>> + * VGIC emulation structures, It reads the LRs and updates the
>> respective
>> + * struct vgic_irq, taking level/edge into account.
>> + * This is the high level function which takes care of the conditions,
>> + * also bails out early if there were no interrupts queued.
>> + * Was: kvm_vgic_sync_hwstate()
>> + */
>> +void gic_clear_lrs(struct vcpu *vcpu)
> 
> I think I would prefer if we stick with the KVM name.

Yes, please! ;-)
I found those names always confusing, especially gic_inject(). To be
honest I never know which is which in the KVM case as well (sync vs.
flush), so I was wondering if we call both the entry and the exit
handler "sync", but denote the direction, like:
vgic_sync_from_lrs() and vgic_sync_to_lrs().

Cheers,
Andre.

>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +
>> +    /* An empty ap_list_head implies used_lrs == 0 */
>> +    if ( list_empty(&vcpu->arch.vgic_cpu.ap_list_head) )
>> +        return;
>> +
>> +    if ( vgic_cpu->used_lrs )
>> +        vgic_fold_lr_state(vcpu);
>> +    vgic_prune_ap_list(vcpu);
>> +}
>> +
>> +/*
>> + * gic_inject() - flush the emulation state into the hardware on
>> guest entry
>> + *
>> + * Before we enter a guest, we have to translate the virtual GIC
>> state of a
>> + * VCPU into the GIC virtualization hardware registers, namely the LRs.
>> + * This is the high level function which takes care about the conditions
>> + * and the locking, also bails out early if there are no interrupts
>> queued.
>> + * Was: kvm_vgic_flush_hwstate()
> 
> Ditto.
> 
>> + */
>> +void gic_inject(void)
>> +{
>> +    /*
>> +     * If there are no virtual interrupts active or pending for this
>> +     * VCPU, then there is no work to do and we can bail out without
>> +     * taking any lock.  There is a potential race with someone
>> injecting
>> +     * interrupts to the VCPU, but it is a benign race as the VCPU will
>> +     * either observe the new interrupt before or after doing this
>> check,
>> +     * and introducing additional synchronization mechanism doesn't
>> change
>> +     * this.
>> +     */
>> +    if ( list_empty(&current->arch.vgic_cpu.ap_list_head) )
>> +        return;
>> +
>> +    ASSERT(!local_irq_is_enabled());
>> +
>> +    spin_lock(&current->arch.vgic_cpu.ap_list_lock);
>> +    vgic_flush_lr_state(current);
>> +    spin_unlock(&current->arch.vgic_cpu.ap_list_lock);
>> +}
>>   /*
>>    * Local variables:
>>    * mode: C
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 5127739f0f..47fc58b81e 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -17,6 +17,8 @@
>>   #ifndef __XEN_ARM_VGIC_NEW_H__
>>   #define __XEN_ARM_VGIC_NEW_H__
>>   +#define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>> +
>>   static inline bool irq_is_pending(struct vgic_irq *irq)
>>   {
>>       if ( irq->config == VGIC_CONFIG_EDGE )
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 23/49] ARM: new VGIC: Add IRQ sorting

[Christoffer Dall]

On Tue, Feb 13, 2018 at 3:56 PM, Andre Przywara
<andre.przywara@linaro.org> wrote:
> Hi,
>
> Christoffer, Eric, Marc,
> a question about locking order between multiple IRQs below. Could you
> have a brief look, please?
>
> On 13/02/18 12:30, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> Adds the sorting function to cover the case where you have more IRQs
>>> to consider than you have LRs. We consider their priorities.
>>> This pulls in Linux' list_sort.c , which is a merge sort implementation
>>> for linked lists.
>>>
>>> This is based on Linux commit 8e4447457965, written by Christoffer Dall.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>   xen/arch/arm/vgic/vgic.c    |  59 +++++++++++++++
>>>   xen/common/list_sort.c      | 170
>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>   xen/include/xen/list_sort.h |  11 +++
>>
>> You need to CC "THE REST" maintainers for this code. It would also make
>> sense to have a separate patch for adding list_sort.c
>
> Yeah, will do.
>
>>>   3 files changed, 240 insertions(+)
>>>   create mode 100644 xen/common/list_sort.c
>>>   create mode 100644 xen/include/xen/list_sort.h
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> index f517df6d00..a4efd1fd03 100644
>>> --- a/xen/arch/arm/vgic/vgic.c
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -16,6 +16,7 @@
>>>    */
>>>     #include <asm/bug.h>
>>> +#include <xen/list_sort.h>
>>>   #include <xen/sched.h>
>>>     #include <asm/arm_vgic.h>
>>> @@ -163,6 +164,64 @@ static struct vcpu *vgic_target_oracle(struct
>>> vgic_irq *irq)
>>>       return NULL;
>>>   }
>>>   +/*
>>> + * The order of items in the ap_lists defines how we'll pack things
>>> in LRs as
>>> + * well, the first items in the list being the first things populated
>>> in the
>>> + * LRs.
>>> + *
>>> + * A hard rule is that active interrupts can never be pushed out of
>>> the LRs
>>> + * (and therefore take priority) since we cannot reliably trap on
>>> deactivation
>>> + * of IRQs and therefore they have to be present in the LRs.
>>> + *
>>> + * Otherwise things should be sorted by the priority field and the GIC
>>> + * hardware support will take care of preemption of priority groups etc.
>>> + *
>>> + * Return negative if "a" sorts before "b", 0 to preserve order, and
>>> positive
>>> + * to sort "b" before "a".
>>
>> Finally a good explanation of the return value of a sort function :). I
>> always get confused what the return is supposed to be.
>>
>>> + */
>>> +static int vgic_irq_cmp(void *priv, struct list_head *a, struct
>>> list_head *b)
>>> +{
>>> +    struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list);
>>> +    struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list);
>>> +    bool penda, pendb;
>>> +    int ret;
>>> +
>>> +    spin_lock(&irqa->irq_lock);
>>> +    spin_lock(&irqb->irq_lock);
>>
>> I guess the locking order does not matter here because this is the only
>> place where two IRQs lock have to be taken?
>
> Mmh, good question. I guess indeed in practice this will not be a problem:
> - As you mentioned this should be the only(?) place where we take
> multiple IRQ locks, but that sounds fragile.
> - A certain IRQ should only be on one VCPU list at a given point in
> time. So there would be no race with two instances of this compare
> function trying to lock the same IRQ.
>
> But that sounds a bit dodgy to rely on. It should be relatively straight
> forward to fix this with a simple comparison, shouldn't it?
> CC:ing Christoffer, Marc and Eric here to see if we should add this (in
> KVM as well).
>

The only concern about holding two locks at the same time is the risk
of another thread attempting to hold a number of locks at the same
time in a different order, leading to a deadlock (either directly or
via a circular dependency).

As you point out, the only place where we take two irq locks at the
same time is in vgic_irq_cmp().  Now, the concern can be reduced to
calling this function more than once in parallel, operating on the
same set of struct irqs.

An IRQ can only be on a single AP list at any time, and we call
vgic_irq_cmp() from exactly one place in the KVM code, which holds the
ap_list_lock, and our locking order defines that the ap_list_lock must
be taken before irq locks.  This means that vgic_irq_cmp() can only
execute in parallel on different AP lists and therefore not operate on
the same set of struct irqs.

There is no need to change anything in the implementation.

Thanks,
-Christoffer

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> Tell Xen whether a particular VCPU has an IRQ that needs handling
> in the guest. This is used to decide whether a VCPU is runnable.
> 
> This is based on Linux commit 90eee56c5f90, written by Eric Auger.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 32 ++++++++++++++++++++++++++++++++
>   1 file changed, 32 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index f4f2a04a60..9e7fb1edcb 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -646,6 +646,38 @@ void gic_inject(void)
>       vgic_restore_state(current);
>   }
>   
> +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    struct vgic_irq *irq;
> +    bool pending = false;
> +    unsigned long flags;
> +
> +    if ( !vcpu->domain->arch.vgic.enabled )
> +        return false;
> +
> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
> +
> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
> +    {
> +        spin_lock(&irq->irq_lock);
> +        pending = irq_is_pending(irq) && irq->enabled;
> +        spin_unlock(&irq->irq_lock);
> +
> +        if ( pending )
> +            break;
> +    }
> +
> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
> +
> +    return pending;
> +}
> +
> +int gic_events_need_delivery(void)

You probably want to rename that function or just expose 
vgic_vcpu_pending_irq().

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Julien Grall]

On 13/02/18 16:35, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Tell Xen whether a particular VCPU has an IRQ that needs handling
>> in the guest. This is used to decide whether a VCPU is runnable.

I forgot to mention one thing. This is not the main usage of this 
function in Xen. That function will mostly be used to check whether we 
need to preempt an hypercall to run interrupt.

Please update the commit message accordingly.

>>
>> This is based on Linux commit 90eee56c5f90, written by Eric Auger.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 32 ++++++++++++++++++++++++++++++++
>>   1 file changed, 32 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index f4f2a04a60..9e7fb1edcb 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>       vgic_restore_state(current);
>>   }
>> +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_irq *irq;
>> +    bool pending = false;
>> +    unsigned long flags;
>> +
>> +    if ( !vcpu->domain->arch.vgic.enabled )
>> +        return false;
>> +
>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>> +
>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>> +    {
>> +        spin_lock(&irq->irq_lock);
>> +        pending = irq_is_pending(irq) && irq->enabled;
>> +        spin_unlock(&irq->irq_lock);
>> +
>> +        if ( pending )
>> +            break;
>> +    }
>> +
>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>> +
>> +    return pending;
>> +}
>> +
>> +int gic_events_need_delivery(void)
> 
> You probably want to rename that function or just expose 
> vgic_vcpu_pending_irq().
> 
> Cheers,
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 27/49] ARM: new VGIC: Add MMIO handling framework

[Julien Grall]

Hi Andre,7

On 09/02/18 14:39, Andre Przywara wrote:
> Add an MMIO handling framework to the VGIC emulation:
> Each register is described by its offset, size (or number of bits per
> IRQ, if applicable) and the read/write handler functions. We provide
> initialization macros to describe each GIC register later easily.
> 
> Separate dispatch functions for read and write accesses are connected
> to Xen's MMIO handling framework and binary-search for the responsible
> register handler based on the offset address within the region.
> 
> The register handler prototype are courtesy of Christoffer Dall.
> 
> This is based on Linux commit 4493b1c4866a, written by Marc Zyngier.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio.c | 192 ++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h | 145 +++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h      |   4 +
>   3 files changed, 341 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
>   create mode 100644 xen/arch/arm/vgic/vgic-mmio.h
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> new file mode 100644
> index 0000000000..3c70945466
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -0,0 +1,192 @@
> +/*
> + * VGIC MMIO handling functions
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/bitops.h>
> +#include <xen/lib.h>
> +#include <xen/sched.h>
> +#include <asm/arm_vgic.h>
> +#include <asm/byteorder.h>
> +
> +#include "vgic.h"
> +#include "vgic-mmio.h"
> +
> +unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len)

Indentation.

> +{
> +    return 0;
> +}
> +
> +unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len)

Indentation.

> +{
> +    return -1UL;
> +}
> +
> +void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
> +            unsigned int len, unsigned long val)

Indentation.

> +{
> +    /* Ignore */
> +}
> +
> +static int match_region(const void *key, const void *elt)
> +{
> +    const unsigned int offset = (unsigned long)key;
> +    const struct vgic_register_region *region = elt;
> +
> +    if ( offset < region->reg_offset )
> +        return -1;
> +
> +    if ( offset >= region->reg_offset + region->len )
> +        return 1;
> +
> +    return 0;
> +}
> +
> +const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,

Any reason to export this?

> +              int nr_regions, unsigned int offset)

Indentation.

> +{
> +    return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
> +               sizeof(regions[0]), match_region);
> +}
> +
> +static bool check_region(const struct domain *d,
> +             const struct vgic_register_region *region,
> +             paddr_t addr, int len)

Indentation.

> +{
> +    int flags, nr_irqs = d->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
> + > +    switch (len)

switch ( ... )

> +    {
> +    case sizeof(u8):

s/u8/uint8_t/ here an below.

> +        flags = VGIC_ACCESS_8bit;
> +        break;
> +    case sizeof(u32):
> +        flags = VGIC_ACCESS_32bit;
> +        break;
> +    case sizeof(u64):
> +        flags = VGIC_ACCESS_64bit;
> +        break;
> +    default:
> +        return false;
> +    }
> +
> +    if ( (region->access_flags & flags) && IS_ALIGNED(addr, len) )
> +    {
> +        if ( !region->bits_per_irq )
> +            return true;
> +
> +        /* Do we access a non-allocated IRQ? */
> +        return VGIC_ADDR_TO_INTID(addr, region->bits_per_irq) < nr_irqs;
> +    }
> +
> +    return false;
> +}
> +
> +const struct vgic_register_region *
> +vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,


Any reason to export this?

> +             paddr_t addr, int len)

Indentation and unsigned int please.

> +{
> +    const struct vgic_register_region *region;
> +
> +    region = vgic_find_mmio_region(iodev->regions, iodev->nr_regions,
> +                       addr - iodev->base_addr);
> +    if ( !region || !check_region(vcpu->domain, region, addr, len) )
> +        return NULL;
> +
> +    return region;
> +}
> +
> +static int dispatch_mmio_read(struct vcpu *vcpu, mmio_info_t *info,
> +                  register_t *r, void *priv)

Indentation.

> +{
> +    struct vgic_io_device *iodev = priv;
> +    const struct vgic_register_region *region;
> +    unsigned long data = 0;
> +    paddr_t addr = info->gpa;
> +    int len = 1U << info->dabt.size;
> +
> +    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
> +    if ( !region )
> +    {
> +        memset(r, 0, len);
> +        return 0;
> +    }
> +
> +    switch (iodev->iodev_type)
> +    {
> +    case IODEV_CPUIF:
> +        data = region->read(vcpu, addr, len);
> +        break;
> +    case IODEV_DIST:
> +        data = region->read(vcpu, addr, len);
> +        break;
> +    case IODEV_REDIST:
> +        data = region->read(iodev->redist_vcpu, addr, len);
> +        break;
> +    case IODEV_ITS:
> +        data = region->its_read(vcpu->domain, iodev->its, addr, len);
> +        break;
> +    }
> +
> +    memcpy(r, &data, len);
> +
> +    return 1;
> +}
> +
> +static int dispatch_mmio_write(struct vcpu *vcpu, mmio_info_t *info,
> +                   register_t r, void *priv)
> +{
> +    struct vgic_io_device *iodev = priv;
> +    const struct vgic_register_region *region;
> +    unsigned long data = r;
> +    paddr_t addr = info->gpa;
> +    int len = 1U << info->dabt.size;
> +
> +    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
> +    if ( !region )
> +        return 0;
> +
> +    switch (iodev->iodev_type)
> +    {
> +    case IODEV_CPUIF:
> +        region->write(vcpu, addr, len, data);
> +        break;
> +    case IODEV_DIST:
> +        region->write(vcpu, addr, len, data);
> +        break;
> +    case IODEV_REDIST:
> +        region->write(iodev->redist_vcpu, addr, len, data);
> +        break;
> +    case IODEV_ITS:
> +        region->its_write(vcpu->domain, iodev->its, addr, len, data);
> +        break;
> +    }
> +
> +    return 1;
> +}
> +
> +struct mmio_handler_ops xen_io_gic_ops = {

I would rename to vgic_io_ops.

> +    .read = dispatch_mmio_read,
> +    .write = dispatch_mmio_write,
> +};
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> new file mode 100644
> index 0000000000..375b70561d
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -0,0 +1,145 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#ifndef __KVM_ARM_VGIC_MMIO_H__
> +#define __KVM_ARM_VGIC_MMIO_H__

Please use update the guard.

> +
> +struct vgic_register_region {
> +    unsigned int reg_offset;
> +    unsigned int len;
> +    unsigned int bits_per_irq;
> +    unsigned int access_flags;
> +    union
> +    {
> +        unsigned long (*read)(struct vcpu *vcpu, paddr_t addr,
> +                              unsigned int len);
> +        unsigned long (*its_read)(struct domain *d, struct vgic_its *its,
> +                                  paddr_t addr, unsigned int len);
> +    };
> +    union
> +    {
> +        void (*write)(struct vcpu *vcpu, paddr_t addr,
> +                      unsigned int len, unsigned long val);
> +        void (*its_write)(struct domain *d, struct vgic_its *its,
> +                          paddr_t addr, unsigned int len,
> +                          unsigned long val);
> +    };
> +    unsigned long (*uaccess_read)(struct vcpu *vcpu, paddr_t addr,
> +                                  unsigned int len);
> +    union
> +    {
> +        void (*uaccess_write)(struct vcpu *vcpu, paddr_t addr,
> +                              unsigned int len, unsigned long val);
> +        int (*uaccess_its_write)(struct domain *d, struct vgic_its *its,
> +                                 paddr_t addr, unsigned int len,
> +                                 unsigned long val);
> +    };

I don't think uaccess helpers makes sense for Xen.

> +};
> +
> +extern struct mmio_handler_ops xen_io_gic_ops;
> +
> +#define VGIC_ACCESS_8bit    1
> +#define VGIC_ACCESS_32bit   2
> +#define VGIC_ACCESS_64bit   4
> +
> +/*
> + * Generate a mask that covers the number of bytes required to address
> + * up to 1024 interrupts, each represented by <bits> bits. This assumes
> + * that <bits> is a power of two.
> + */
> +#define VGIC_ADDR_IRQ_MASK(bits) (((bits) * 1024 / 8) - 1)
> +
> +/*
> + * (addr & mask) gives us the _byte_ offset for the INT ID.
> + * We multiply this by 8 the get the _bit_ offset, then divide this by
> + * the number of bits to learn the actual INT ID.
> + * But instead of a division (which requires a "long long div" implementation),
> + * we shift by the binary logarithm of <bits>.
> + * This assumes that <bits> is a power of two.
> + */
> +#define VGIC_ADDR_TO_INTID(addr, bits)  (((addr) & VGIC_ADDR_IRQ_MASK(bits)) * \
> +                                         8 >> LOG_2(bits))

We are going to switch to ilog2 (see Sameer's patch "xen/bitops: Rename 
LOG_2 to ilog2").

> +
> +/*
> + * Some VGIC registers store per-IRQ information, with a different number
> + * of bits per IRQ. For those registers this macro is used.
> + * The _WITH_LENGTH version instantiates registers with a fixed length
> + * and is mutually exclusive with the _PER_IRQ version.
> + */
> +#define REGISTER_DESC_WITH_BITS_PER_IRQ(off, rd, wr, ur, uw, bpi, acc)  \
> +    {                           \
> +        .reg_offset = off,      \
> +        .bits_per_irq = bpi,    \
> +        .len = bpi * 1024 / 8,  \
> +        .access_flags = acc,    \
> +        .read = rd,             \
> +        .write = wr,            \
> +        .uaccess_read = ur,     \
> +        .uaccess_write = uw,    \
> +    }
> +
> +#define REGISTER_DESC_WITH_LENGTH(off, rd, wr, length, acc)     \
> +    {                           \
> +        .reg_offset = off,      \
> +        .bits_per_irq = 0,      \
> +        .len = length,          \
> +        .access_flags = acc,    \
> +        .read = rd,             \
> +        .write = wr,            \
> +    }
> +
> +#define REGISTER_DESC_WITH_LENGTH_UACCESS(off, rd, wr, urd, uwr, length, acc) \
> +    {                           \
> +        .reg_offset = off,      \
> +        .bits_per_irq = 0,      \
> +        .len = length,          \
> +        .access_flags = acc,    \
> +        .read = rd,             \
> +        .write = wr,            \
> +        .uaccess_read = urd,    \
> +        .uaccess_write = uwr,   \
> +    }
> +
> +int kvm_vgic_register_mmio_region(struct domain *d, struct vcpu *vcpu,
> +                                  struct vgic_register_region *reg_desc,
> +                                  struct vgic_io_device *region,
> +                                  int nr_irqs, bool offset_private);

You want to do some clean-up in the prototype below. Only the one used 
in the patch should be added. The other should either move in there 
corresponding patch or dropped if not used.

> +
> +unsigned long vgic_data_mmio_bus_to_host(const void *val, unsigned int len);
> +
> +void vgic_data_host_to_mmio_bus(void *buf, unsigned int len,
> +                                unsigned long data);
> +
> +unsigned long extract_bytes(u64 data, unsigned int offset,
> +                            unsigned int num);
> +
> +u64 update_64bit_reg(u64 reg, unsigned int offset, unsigned int len,
> +                     unsigned long val);
> +
> +unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
> +                                 paddr_t addr, unsigned int len);
> +
> +unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
> +                                 paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
> +                        unsigned int len, unsigned long val);
> +
> +/* Find the proper register handler entry given a certain address offset */
> +const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> +                      int nr_regions, unsigned int offset);
> +
> +#endif
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 771ca6f046..426b34d0ce 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -27,6 +27,10 @@ static inline bool irq_is_pending(struct vgic_irq *irq)
>           return irq->pending_latch || irq->line_level;
>   }
>   
> +const struct vgic_register_region *
> +vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
> +             paddr_t addr, int len);
> +

Why this one is added in vgic.h and not kept in vgic-mmio.h?

>   struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>                                 u32 intid);
>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 27/49] ARM: new VGIC: Add MMIO handling framework

[Andre Przywara]

Hi,

On 13/02/18 16:52, Julien Grall wrote:
> Hi Andre,7
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Add an MMIO handling framework to the VGIC emulation:
>> Each register is described by its offset, size (or number of bits per
>> IRQ, if applicable) and the read/write handler functions. We provide
>> initialization macros to describe each GIC register later easily.
>>
>> Separate dispatch functions for read and write accesses are connected
>> to Xen's MMIO handling framework and binary-search for the responsible
>> register handler based on the offset address within the region.
>>
>> The register handler prototype are courtesy of Christoffer Dall.
>>
>> This is based on Linux commit 4493b1c4866a, written by Marc Zyngier.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio.c | 192
>> ++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h | 145 +++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.h      |   4 +
>>   3 files changed, 341 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
>>   create mode 100644 xen/arch/arm/vgic/vgic-mmio.h
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> new file mode 100644
>> index 0000000000..3c70945466
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -0,0 +1,192 @@
>> +/*
>> + * VGIC MMIO handling functions
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <xen/bitops.h>
>> +#include <xen/lib.h>
>> +#include <xen/sched.h>
>> +#include <asm/arm_vgic.h>
>> +#include <asm/byteorder.h>
>> +
>> +#include "vgic.h"
>> +#include "vgic-mmio.h"
>> +
>> +unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len)
> 
> Indentation.
> 
>> +{
>> +    return 0;
>> +}
>> +
>> +unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len)
> 
> Indentation.
> 
>> +{
>> +    return -1UL;
>> +}
>> +
>> +void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>> +            unsigned int len, unsigned long val)
> 
> Indentation.
> 
>> +{
>> +    /* Ignore */
>> +}
>> +
>> +static int match_region(const void *key, const void *elt)
>> +{
>> +    const unsigned int offset = (unsigned long)key;
>> +    const struct vgic_register_region *region = elt;
>> +
>> +    if ( offset < region->reg_offset )
>> +        return -1;
>> +
>> +    if ( offset >= region->reg_offset + region->len )
>> +        return 1;
>> +
>> +    return 0;
>> +}
>> +
>> +const struct vgic_register_region *
>> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> 
> Any reason to export this?

Good catch, this is needed in KVM to do the user space access, where we
re-use these functions to call into the MMIO handlers.
So I can make them static and then loose the prototype down below as well.

> 
>> +              int nr_regions, unsigned int offset)
> 
> Indentation.
> 
>> +{
>> +    return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
>> +               sizeof(regions[0]), match_region);
>> +}
>> +
>> +static bool check_region(const struct domain *d,
>> +             const struct vgic_register_region *region,
>> +             paddr_t addr, int len)
> 
> Indentation.
> 
>> +{
>> +    int flags, nr_irqs = d->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
>> + > +    switch (len)
> 
> switch ( ... )
> 
>> +    {
>> +    case sizeof(u8):
> 
> s/u8/uint8_t/ here an below.
> 
>> +        flags = VGIC_ACCESS_8bit;
>> +        break;
>> +    case sizeof(u32):
>> +        flags = VGIC_ACCESS_32bit;
>> +        break;
>> +    case sizeof(u64):
>> +        flags = VGIC_ACCESS_64bit;
>> +        break;
>> +    default:
>> +        return false;
>> +    }
>> +
>> +    if ( (region->access_flags & flags) && IS_ALIGNED(addr, len) )
>> +    {
>> +        if ( !region->bits_per_irq )
>> +            return true;
>> +
>> +        /* Do we access a non-allocated IRQ? */
>> +        return VGIC_ADDR_TO_INTID(addr, region->bits_per_irq) < nr_irqs;
>> +    }
>> +
>> +    return false;
>> +}
>> +
>> +const struct vgic_register_region *
>> +vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
> 
> 
> Any reason to export this?
> 
>> +             paddr_t addr, int len)
> 
> Indentation and unsigned int please.
> 
>> +{
>> +    const struct vgic_register_region *region;
>> +
>> +    region = vgic_find_mmio_region(iodev->regions, iodev->nr_regions,
>> +                       addr - iodev->base_addr);
>> +    if ( !region || !check_region(vcpu->domain, region, addr, len) )
>> +        return NULL;
>> +
>> +    return region;
>> +}
>> +
>> +static int dispatch_mmio_read(struct vcpu *vcpu, mmio_info_t *info,
>> +                  register_t *r, void *priv)
> 
> Indentation.
> 
>> +{
>> +    struct vgic_io_device *iodev = priv;
>> +    const struct vgic_register_region *region;
>> +    unsigned long data = 0;
>> +    paddr_t addr = info->gpa;
>> +    int len = 1U << info->dabt.size;
>> +
>> +    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
>> +    if ( !region )
>> +    {
>> +        memset(r, 0, len);
>> +        return 0;
>> +    }
>> +
>> +    switch (iodev->iodev_type)
>> +    {
>> +    case IODEV_CPUIF:
>> +        data = region->read(vcpu, addr, len);
>> +        break;
>> +    case IODEV_DIST:
>> +        data = region->read(vcpu, addr, len);
>> +        break;
>> +    case IODEV_REDIST:
>> +        data = region->read(iodev->redist_vcpu, addr, len);
>> +        break;
>> +    case IODEV_ITS:
>> +        data = region->its_read(vcpu->domain, iodev->its, addr, len);
>> +        break;
>> +    }
>> +
>> +    memcpy(r, &data, len);
>> +
>> +    return 1;
>> +}
>> +
>> +static int dispatch_mmio_write(struct vcpu *vcpu, mmio_info_t *info,
>> +                   register_t r, void *priv)
>> +{
>> +    struct vgic_io_device *iodev = priv;
>> +    const struct vgic_register_region *region;
>> +    unsigned long data = r;
>> +    paddr_t addr = info->gpa;
>> +    int len = 1U << info->dabt.size;
>> +
>> +    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
>> +    if ( !region )
>> +        return 0;
>> +
>> +    switch (iodev->iodev_type)
>> +    {
>> +    case IODEV_CPUIF:
>> +        region->write(vcpu, addr, len, data);
>> +        break;
>> +    case IODEV_DIST:
>> +        region->write(vcpu, addr, len, data);
>> +        break;
>> +    case IODEV_REDIST:
>> +        region->write(iodev->redist_vcpu, addr, len, data);
>> +        break;
>> +    case IODEV_ITS:
>> +        region->its_write(vcpu->domain, iodev->its, addr, len, data);
>> +        break;
>> +    }
>> +
>> +    return 1;
>> +}
>> +
>> +struct mmio_handler_ops xen_io_gic_ops = {
> 
> I would rename to vgic_io_ops.
> 
>> +    .read = dispatch_mmio_read,
>> +    .write = dispatch_mmio_write,
>> +};
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> new file mode 100644
>> index 0000000000..375b70561d
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -0,0 +1,145 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +#ifndef __KVM_ARM_VGIC_MMIO_H__
>> +#define __KVM_ARM_VGIC_MMIO_H__
> 
> Please use update the guard.
> 
>> +
>> +struct vgic_register_region {
>> +    unsigned int reg_offset;
>> +    unsigned int len;
>> +    unsigned int bits_per_irq;
>> +    unsigned int access_flags;
>> +    union
>> +    {
>> +        unsigned long (*read)(struct vcpu *vcpu, paddr_t addr,
>> +                              unsigned int len);
>> +        unsigned long (*its_read)(struct domain *d, struct vgic_its
>> *its,
>> +                                  paddr_t addr, unsigned int len);
>> +    };
>> +    union
>> +    {
>> +        void (*write)(struct vcpu *vcpu, paddr_t addr,
>> +                      unsigned int len, unsigned long val);
>> +        void (*its_write)(struct domain *d, struct vgic_its *its,
>> +                          paddr_t addr, unsigned int len,
>> +                          unsigned long val);
>> +    };
>> +    unsigned long (*uaccess_read)(struct vcpu *vcpu, paddr_t addr,
>> +                                  unsigned int len);
>> +    union
>> +    {
>> +        void (*uaccess_write)(struct vcpu *vcpu, paddr_t addr,
>> +                              unsigned int len, unsigned long val);
>> +        int (*uaccess_its_write)(struct domain *d, struct vgic_its *its,
>> +                                 paddr_t addr, unsigned int len,
>> +                                 unsigned long val);
>> +    };
> 
> I don't think uaccess helpers makes sense for Xen.

True, I was unsure about whether to keep them. I have the gut feeling we
need it later when we want to suspend/resume the VGIC, so removing
everything and then simplifying the code afterwards might bite us in the
future.
So as long as it doesn't really hurt, I am tempted to keep that code in,
which also keeps it closer the the KVM implementation.

But if you insist: deleting this is easy ;-)

>> +};
>> +
>> +extern struct mmio_handler_ops xen_io_gic_ops;
>> +
>> +#define VGIC_ACCESS_8bit    1
>> +#define VGIC_ACCESS_32bit   2
>> +#define VGIC_ACCESS_64bit   4
>> +
>> +/*
>> + * Generate a mask that covers the number of bytes required to address
>> + * up to 1024 interrupts, each represented by <bits> bits. This assumes
>> + * that <bits> is a power of two.
>> + */
>> +#define VGIC_ADDR_IRQ_MASK(bits) (((bits) * 1024 / 8) - 1)
>> +
>> +/*
>> + * (addr & mask) gives us the _byte_ offset for the INT ID.
>> + * We multiply this by 8 the get the _bit_ offset, then divide this by
>> + * the number of bits to learn the actual INT ID.
>> + * But instead of a division (which requires a "long long div"
>> implementation),
>> + * we shift by the binary logarithm of <bits>.
>> + * This assumes that <bits> is a power of two.
>> + */
>> +#define VGIC_ADDR_TO_INTID(addr, bits)  (((addr) &
>> VGIC_ADDR_IRQ_MASK(bits)) * \
>> +                                         8 >> LOG_2(bits))
> 
> We are going to switch to ilog2 (see Sameer's patch "xen/bitops: Rename
> LOG_2 to ilog2").

\o/

> 
>> +
>> +/*
>> + * Some VGIC registers store per-IRQ information, with a different
>> number
>> + * of bits per IRQ. For those registers this macro is used.
>> + * The _WITH_LENGTH version instantiates registers with a fixed length
>> + * and is mutually exclusive with the _PER_IRQ version.
>> + */
>> +#define REGISTER_DESC_WITH_BITS_PER_IRQ(off, rd, wr, ur, uw, bpi,
>> acc)  \
>> +    {                           \
>> +        .reg_offset = off,      \
>> +        .bits_per_irq = bpi,    \
>> +        .len = bpi * 1024 / 8,  \
>> +        .access_flags = acc,    \
>> +        .read = rd,             \
>> +        .write = wr,            \
>> +        .uaccess_read = ur,     \
>> +        .uaccess_write = uw,    \
>> +    }
>> +
>> +#define REGISTER_DESC_WITH_LENGTH(off, rd, wr, length, acc)     \
>> +    {                           \
>> +        .reg_offset = off,      \
>> +        .bits_per_irq = 0,      \
>> +        .len = length,          \
>> +        .access_flags = acc,    \
>> +        .read = rd,             \
>> +        .write = wr,            \
>> +    }
>> +
>> +#define REGISTER_DESC_WITH_LENGTH_UACCESS(off, rd, wr, urd, uwr,
>> length, acc) \
>> +    {                           \
>> +        .reg_offset = off,      \
>> +        .bits_per_irq = 0,      \
>> +        .len = length,          \
>> +        .access_flags = acc,    \
>> +        .read = rd,             \
>> +        .write = wr,            \
>> +        .uaccess_read = urd,    \
>> +        .uaccess_write = uwr,   \
>> +    }
>> +
>> +int kvm_vgic_register_mmio_region(struct domain *d, struct vcpu *vcpu,
>> +                                  struct vgic_register_region *reg_desc,
>> +                                  struct vgic_io_device *region,
>> +                                  int nr_irqs, bool offset_private);
> 
> You want to do some clean-up in the prototype below. Only the one used
> in the patch should be added. The other should either move in there
> corresponding patch or dropped if not used.

Thanks for the heads up. In general I didn't spend much time on the
prototypes in header files, so there might indeed by some leftovers from
KVM.

Consider the rest fixed.

Cheers,
Andre.

>> +
>> +unsigned long vgic_data_mmio_bus_to_host(const void *val, unsigned
>> int len);
>> +
>> +void vgic_data_host_to_mmio_bus(void *buf, unsigned int len,
>> +                                unsigned long data);
>> +
>> +unsigned long extract_bytes(u64 data, unsigned int offset,
>> +                            unsigned int num);
>> +
>> +u64 update_64bit_reg(u64 reg, unsigned int offset, unsigned int len,
>> +                     unsigned long val);
>> +
>> +unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
>> +                                 paddr_t addr, unsigned int len);
>> +
>> +unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>> +                                 paddr_t addr, unsigned int len);
>> +
>> +void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>> +                        unsigned int len, unsigned long val);
>> +
>> +/* Find the proper register handler entry given a certain address
>> offset */
>> +const struct vgic_register_region *
>> +vgic_find_mmio_region(const struct vgic_register_region *regions,
>> +                      int nr_regions, unsigned int offset);
>> +
>> +#endif
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 771ca6f046..426b34d0ce 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -27,6 +27,10 @@ static inline bool irq_is_pending(struct vgic_irq
>> *irq)
>>           return irq->pending_latch || irq->line_level;
>>   }
>>   +const struct vgic_register_region *
>> +vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
>> +             paddr_t addr, int len);
>> +
> 
> Why this one is added in vgic.h and not kept in vgic-mmio.h?
> 
>>   struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>>                                 u32 intid);
>>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 15/49] ARM: GIC: Allow tweaking the active state of an IRQ

[Julien Grall]

On 13/02/18 15:01, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 13/02/18 12:02, Julien Grall wrote:
>> On 12/02/18 17:53, Andre Przywara wrote:
>>> Hi,
>>
>> Hi Andre,
>>
>>> On 12/02/18 13:55, Julien Grall wrote:
>>>> Hi Andre,
>>>>
>>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>>> When playing around with hardware mapped, level triggered virtual IRQs,
>>>>> there is the need to explicitly set the active state of an interrupt at
>>>>> some point in time.
>>>>> To prepare the GIC for that, we introduce a set_active_state() function
>>>>> to let the VGIC manipulate the state of an associated hardware IRQ.
>>>>>
>>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>>> ---
>>>>>     xen/arch/arm/gic-v2.c     |  9 +++++++++
>>>>>     xen/arch/arm/gic-v3.c     | 16 ++++++++++++++++
>>>>>     xen/arch/arm/gic.c        |  5 +++++
>>>>>     xen/include/asm-arm/gic.h |  5 +++++
>>>>>     4 files changed, 35 insertions(+)
>>>>>
>>>>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>>>>> index 2e35892881..5339f69fbc 100644
>>>>> --- a/xen/arch/arm/gic-v2.c
>>>>> +++ b/xen/arch/arm/gic-v2.c
>>>>> @@ -235,6 +235,14 @@ static unsigned int gicv2_read_irq(void)
>>>>>         return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>>>>>     }
>>>>>     +static void gicv2_set_active_state(int irq, bool active)
>>>>
>>>> I would much prefer to have an irq_desc in parameter. This is matching
>>>> the other interface
>>>
>>> ... and that's why I had it just like this in my first version. However
>>> this proved to be nasty because I now need to get this irq_desc pointer
>>> first, as the caller doesn't have it already. Since all we have and need
>>> is the actual hardware IRQ number, I found it more straight-forward to
>>> just use that number directly instead of going via the pointer and back
>>> (h/w intid => irq_desc => irq).
>>>
>>>> and you could update the flags such as
>>>> _IRQ_INPROGRESS which you don't do at the moment.
>>>
>>> Mmh, interesting point. I guess I should also clear this bit in the new
>>> VGIC. At least once I wrapped my head around what this flag is
>>> *actually* for (in conjunction with _IRQ_GUEST).
>>> Anyway I guess this bit would still be set in our case.
>>
>> For IRQ routed to the guest, the flag is used to know whether you need
>> to EOI the interrupt on domain destruction.
> 
> Yeah, I found that. In general I am a bit suspicious of replicating and
> tracking the hardware IRQ state in software.

This is how Xen has been designed and I am pretty sure Linux does that 
same. This makes easier if you want in the future to share interrupts 
between multiple domains (think legacy PCI interrupt) and even abstract 
the hardware from the virtual GIC.

> 
>> In general, I would like to keep desc->status in sync for the guest IRQ.
>> This is useful for debugging and potentially some ratelimit on interrupt
>> (I am thinking for ITS).
>>
>>>
>>>> Also, who is preventing two CPUs to clear the active bit at the same
>>>> time?
>>>
>>> A certain hardware IRQ is assigned to one virtual IRQ on one VCPU at one
>>> time only. Besides, GICD_ICACTIVERn has wired NAND semantics, so that's
>>> naturally race free (as it was designed to be).
>>> Unless I miss something here (happy to be pointed to an example where it
>>> causes problems).
>>
>> You could potentially have a race between ICACTIVER an ISACTIVER.
> 
> I don't see why this would be a problem:
> Either you activate the IRQ or you deactivate it. The
> wired-OR/wired-NAND semantics makes sure this never gets inconsistent on
> the hardware side. If you issue two conflicting requests at the same
> time, that's a benign race, which you either don't care about or handle
> via locking in the code which triggers these requests.
> 
> Besides, we only do one direction in the code at the moment anyway.
> And this should be *clearing* the active state, and not setting it,
> which is a bug I discovered yesterday.

If the code do handle only one direction, then you should add an ASSERT 
to check the state value rather than making believe the code is safe. It 
is only safe for hardware interrupt assigned to a guest (thanks to 
locking). It would not be for Xen.

> 
>> is very similar to the enable/disable part. This matters a lot when
>> updating desc->status.
> 
> Which is one reason why I am suspicious of this whole state replication.
> But the desc lock should take care of this in general, no?

Yes if you take it.

> 
>>>>> +}
>>>>> +
>>>>>     static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int
>>>>> type)
>>>>>     {
>>>>>         uint32_t cfg, actual, edgebit;
>>>>> @@ -1241,6 +1249,7 @@ const static struct gic_hw_operations
>>>>> gicv2_ops = {
>>>>>         .eoi_irq             = gicv2_eoi_irq,
>>>>>         .deactivate_irq      = gicv2_dir_irq,
>>>>>         .read_irq            = gicv2_read_irq,
>>>>> +    .set_active_state    = gicv2_set_active_state,
>>>>>         .set_irq_type        = gicv2_set_irq_type,
>>>>>         .set_irq_priority    = gicv2_set_irq_priority,
>>>>>         .send_SGI            = gicv2_send_SGI,
>>>>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>>>>> index 08d4703687..595eaef43a 100644
>>>>> --- a/xen/arch/arm/gic-v3.c
>>>>> +++ b/xen/arch/arm/gic-v3.c
>>>>> @@ -475,6 +475,21 @@ static unsigned int gicv3_read_irq(void)
>>>>>         return irq;
>>>>>     }
>>>>>     +static void gicv3_set_active_state(int irq, bool active)
>>>>> +{
>>>>> +    void __iomem *base;
>>>>> +
>>>>> +    if ( irq >= NR_GIC_LOCAL_IRQS)
>>>>> +        base = GICD + (irq / 32) * 4;
>>>>> +    else
>>>>> +        base = GICD_RDIST_SGI_BASE;
>>>>> +
>>>>> +    if ( active )
>>>>> +        writel(1U << (irq % 32), base + GICD_ISACTIVER);
>>>>> +    else
>>>>> +        writel(1U << (irq % 32), base + GICD_ICACTIVER);
>>>>
>>>> Shouldn't you wait until RWP bits is cleared here?
>>>
>>> I don't see why. I think this action has some posted semantics anyway,
>>> so no need for any synchronisation. And also RWP does not track
>>> I[SC]ACTIVER, only ICENABLER and some CTLR bits (ARM IHI 0069D, 8.9.4:
>>> RWP[31]).
>>>
>>>>
>>>>> +}
>>>>
>>>> Why don't you use the function poke?
>>>
>>> Ah, I didn't see this. But then this now does this quite costly RWP
>>> dance now. We could add a check in there to only do this if we change
>>> the affected registers or pass an explicit "bool wait_for_rwp" in there.
>>
>> I guess this would be useful even for the current code. If I understand
>> correctly the RWP semantics, it should not be necessary to wait when
>> write to ISENABLER also.
> 
> Exactly. I changed poke_irq() to do:
>      if ( offset == GICD_ICENABLER )
>          gicv3_wait_for_rwp(irqd->irq);
> 
> Does that sound acceptable?

I would prefer the prototype to be updated with another parameter that 
will tell whether we want to wait or not.

> 
> I also just added poke_irq()/peek_irq() to gic-v2.c as well.
> 
> Cheers,
> Andre.
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 21/49] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Julien Grall]

On 13/02/18 11:18, Andre Przywara wrote:
> Hi,

Hi Andre,

> 
> On 12/02/18 17:42, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> The new VGIC implementation centers around a struct vgic_irq instance
>>> per virtual IRQ.
>>> Provide a function to retrieve the right instance for a given IRQ
>>> number and (in case of private interrupts) the right VCPU.
>>> This also includes the corresponding put function, which does nothing
>>> for private interrupts and SPIs, but handles the ref-counting for LPIs.
>>>
>>> This is based on Linux commit 64a959d66e47, written by Christoffer Dall.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic.c | 107
>>> +++++++++++++++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/vgic/vgic.h |  32 ++++++++++++++
>>>    2 files changed, 139 insertions(+)
>>>    create mode 100644 xen/arch/arm/vgic/vgic.c
>>>    create mode 100644 xen/arch/arm/vgic/vgic.h
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> new file mode 100644
>>> index 0000000000..3075091caa
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -0,0 +1,107 @@
>>> +/*
>>> + * Copyright (C) 2015, 2016 ARM Ltd.
>>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <asm/bug.h>
>>> +#include <xen/sched.h>
>>> +
>>> +#include <asm/arm_vgic.h>
>>> +#include "vgic.h"
>>
>> Please order the include alphabetically.
> 
> Sure.
> 
>>> +
>>> +/*
>>> + * Iterate over the VM's list of mapped LPIs to find the one with a
>>> + * matching interrupt ID and return a reference to the IRQ structure.
>>> + */
>>> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
>>> +{
>>> +    struct vgic_dist *dist = &d->arch.vgic;
>>> +    struct vgic_irq *irq = NULL;
>>> +
>>> +    spin_lock(&dist->lpi_list_lock);
>>> +
>>> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
>>
>> I think it would be worth thinking of a different data structure here.
>> The number of LPIs can be quite high for the hardware domain.
> 
> Probably true. I just didn't want to waste time on this yet, as we don't
> have LPIs at the moment. Having a list has the big advantage of being
> easy to understand and to implement, so I consider this an optimization
> that we can have later.

I am not a big fan of adding code that will never be used as it is and 
just too slow. That's going to have an impact on platform such as Thunder-X.

>>> +    return NULL;
>>> +}
>>> +
>>> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
>>> +{
>>> +    struct vgic_dist *dist = &d->arch.vgic;
>>> +
>>> +    if ( irq->intid < VGIC_MIN_LPI )
>>> +        return;
>>> +
>>> +    spin_lock(&dist->lpi_list_lock);
>>> +    if ( !atomic_dec_and_test(&irq->refcount) )
>>> +    {
>>> +        spin_unlock(&dist->lpi_list_lock);
>>> +        return;
>>> +    };
>>> +
>>> +    list_del(&irq->lpi_list);
>>
>> I would add
>>
>> ASSERT(lpi_list_count >= 1);
>>
>> But it is a bit hard to know whether this code is valid given you don't
>> have any implementation of ITS so far.
> 
> Is it? You should not need the actual ITS code to validate this
> function. In fact there are only very few users in vgic-its.c.
> The main point here is that you have textbook ref-counting: *Every* time
> you take a pointer to an IRQ (vgic_get_irq), you have to tell the code
> when you are done with it (vgic_put_irq).
> So we decided to have this ref-counting done properly even though it's
> pointless for SPIs, PPIs and SGIs, as it makes the code very clear to
> read and verify.
> Mostly you have get and put in one function, but sometimes there is more
> time between them: for instance if an interrupt goes to the ap_list. We
> "get" it, add it to the list, then return. When the guest has actually
> handled this interrupt, we remove it from the list and only then "put"
> it again.

When I read only this series, I can't tell why refcounting is necessary 
for LPIs only. This is neither said in the commit message nor in the 
cover letter. Note that I know the answer, I doubt the other may know it.

So yes, the code is trivial. But without the full logic/explanation, it 
is hard to tell how the ITS is going to fit in and whether what you do 
know looks sensible.

> 
>>> +    dist->lpi_list_count--;
>>> +    spin_unlock(&dist->lpi_list_lock);
>>> +
>>> +    xfree(irq);
>>> +}
>>> +
>>> +/*
>>> + * Local variables:
>>> + * mode: C
>>> + * c-file-style: "BSD"
>>> + * c-basic-offset: 4
>>> + * indent-tabs-mode: nil
>>> + * End:
>>> + */
>>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>>> new file mode 100644
>>> index 0000000000..7a15cfdd79
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vgic/vgic.h
>>
>> To be honest, I am not a big fan of headers defined in the code bits. So
>> I would need a reason for that to be there and not in the include you
>> defined in the previous patch.
> 
> What is the problem with that?
> The rationale here is to gather all definitions and prototypes that are
> actually VGIC *internal*. No code outside of the actual VGIC
> (xen/arch/arm/vgic/) should be concerned with it, and so I consider this
> good style to keep this header file local. This makes it very clear that
> no generic or arch code should ever tinker with anything defined in it.
> 
> Think about it like we could actually glue all those files in this new
> directory together into one glorious new-vgic.c. Then we would not need
> this header. But it's terrible to read and review, so we have this nice
> split-up into vgic-mmio.c and vgic.c, for instance. And now we need this
> header file to link those files together, to allow the MMIO emulation to
> manipulate the state of an interrupt and queue it to a VCPU, for instance.
> 
> It's totally possible that there are definitions and prototypes in here
> which don't belong here. TBH I didn't review this file here very
> carefully for what we still need and what not, so I am happy to take
> advice on what's wrong here.

Fair enough.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 24/49] ARM: new VGIC: Add IRQ sync/flush framework

[Julien Grall]

On 13/02/18 15:40, Andre Przywara wrote:
> Hi,
> 
> On 13/02/18 12:41, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> gets called on guest entry and exit.
>>> The code talking to the actual GICv2/v3 hardware is added in the
>>> following patches.
>>>
>>> This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic.c | 246
>>> +++++++++++++++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/vgic/vgic.h |   2 +
>>>    2 files changed, 248 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> index a4efd1fd03..a1f77130d4 100644
>>> --- a/xen/arch/arm/vgic/vgic.c
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -380,6 +380,252 @@ int vgic_inject_irq(struct domain *d, struct
>>> vcpu *vcpu, unsigned int intid,
>>>        return 0;
>>>    }
>>>    +/**
>>> + * vgic_prune_ap_list - Remove non-relevant interrupts from the list
>>> + *
>>> + * @vcpu: The VCPU pointer
>>> + *
>>> + * Go over the list of "interesting" interrupts, and prune those that we
>>> + * won't have to consider in the near future.
>>> + */
>>> +static void vgic_prune_ap_list(struct vcpu *vcpu)
>>> +{
>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>> +    struct vgic_irq *irq, *tmp;
>>> +    unsigned long flags;
>>> +
>>> +retry:
>>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>>> +
>>> +    list_for_each_entry_safe( irq, tmp, &vgic_cpu->ap_list_head,
>>> ap_list )
>>
>> See my comment on patch #22, this is where I am worry about going
>> through the list every time we enter to the hypervisor from the guest.
> 
> I am not sure we can avoid this here, as this function is crucial to the
> VGIC state machine.
> We might later look into if we can avoid iterating through the whole
> list or if we can shortcut some interrupts somehow, but I really would
> be careful tinkering with this function too much.

My biggest concern is how big the list can be. If the list is quite 
small, then this function is ok to call when entering to the hypervisor. 
If you tell me it can be bigger, then I am quite reluctant to see this 
code in Xen. Obviously this could be solved afterwards but clearly 
before we make this a default option in Xen.

To be clear, I am asking how much a guest can control the size of the list.

[...]

>>> +/* Requires the ap_list_lock to be held. */
>>> +static int compute_ap_list_depth(struct vcpu *vcpu)
>>> +{
>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>> +    struct vgic_irq *irq;
>>> +    int count = 0;
>>> +
>>> +    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
>>> +
>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>
>> Here another example.
> 
> This can be short cut, indeed. The only place we call this function is
> where we compare it against the number of LRs below.
> So we don't need to know the actual number of elements, but could bail
> out once we reached the number of LRs.
> This function here would then return a bool, comparing internally
> against the number of LRs already.

This would only solve one part of the problem. Then you go sorting the 
list if you have more vIRQ in the list than the number of LRs.

See above for my thoughts on the list.

>>> + * gic_clear_lrs() - Update the VGIC state from hardware after a
>>> guest's run.
>>> + * @vcpu: the VCPU.
>>> + *
>>> + * Sync back the hardware VGIC state after the guest has run, into our
>>> + * VGIC emulation structures, It reads the LRs and updates the
>>> respective
>>> + * struct vgic_irq, taking level/edge into account.
>>> + * This is the high level function which takes care of the conditions,
>>> + * also bails out early if there were no interrupts queued.
>>> + * Was: kvm_vgic_sync_hwstate()
>>> + */
>>> +void gic_clear_lrs(struct vcpu *vcpu)
>>
>> I think I would prefer if we stick with the KVM name.
> 
> Yes, please! ;-)
> I found those names always confusing, especially gic_inject(). To be
> honest I never know which is which in the KVM case as well (sync vs.
> flush), so I was wondering if we call both the entry and the exit
> handler "sync", but denote the direction, like:
> vgic_sync_from_lrs() and vgic_sync_to_lrs().

+1 on the suggested naming :).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 27/49] ARM: new VGIC: Add MMIO handling framework

[Julien Grall]

Hi Andre,

On 13/02/18 18:17, Andre Przywara wrote:
> On 13/02/18 16:52, Julien Grall wrote:
>>> +struct vgic_register_region {
>>> +    unsigned int reg_offset;
>>> +    unsigned int len;
>>> +    unsigned int bits_per_irq;
>>> +    unsigned int access_flags;
>>> +    union
>>> +    {
>>> +        unsigned long (*read)(struct vcpu *vcpu, paddr_t addr,
>>> +                              unsigned int len);
>>> +        unsigned long (*its_read)(struct domain *d, struct vgic_its
>>> *its,
>>> +                                  paddr_t addr, unsigned int len);
>>> +    };
>>> +    union
>>> +    {
>>> +        void (*write)(struct vcpu *vcpu, paddr_t addr,
>>> +                      unsigned int len, unsigned long val);
>>> +        void (*its_write)(struct domain *d, struct vgic_its *its,
>>> +                          paddr_t addr, unsigned int len,
>>> +                          unsigned long val);
>>> +    };
>>> +    unsigned long (*uaccess_read)(struct vcpu *vcpu, paddr_t addr,
>>> +                                  unsigned int len);
>>> +    union
>>> +    {
>>> +        void (*uaccess_write)(struct vcpu *vcpu, paddr_t addr,
>>> +                              unsigned int len, unsigned long val);
>>> +        int (*uaccess_its_write)(struct domain *d, struct vgic_its *its,
>>> +                                 paddr_t addr, unsigned int len,
>>> +                                 unsigned long val);
>>> +    };
>>
>> I don't think uaccess helpers makes sense for Xen.
> 
> True, I was unsure about whether to keep them. I have the gut feeling we
> need it later when we want to suspend/resume the VGIC, so removing
> everything and then simplifying the code afterwards might bite us in the
> future.
> So as long as it doesn't really hurt, I am tempted to keep that code in,
> which also keeps it closer the the KVM implementation.

I don't want to see code that is going to potentially rot. If we really 
need it, we can add it afterwards.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 28/49] ARM: new VGIC: Add GICv2 MMIO handling framework

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> Create vgic-mmio-v2.c to describe GICv2 emulation specific handlers
> using the initializer macros provided by the VGIC MMIO framework.
> Provide a function to register the GICv2 distributor registers to
> the Xen MMIO framework.
> The actual handler functions are still stubs in this patch.
> 
> This is based on Linux commit fb848db39661, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 83 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.c    | 26 +++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  2 +
>   xen/arch/arm/vgic/vgic.h         |  2 +
>   4 files changed, 113 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> new file mode 100644
> index 0000000000..ee685a5a07
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -0,0 +1,83 @@
> +/*
> + * VGICv2 MMIO handling functions
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/bitops.h>
> +#include <xen/sched.h>
> +#include <xen/sizes.h>
> +#include <asm/arm_vgic.h>
> +
> +#include "vgic.h"
> +#include "vgic-mmio.h"
> +
> +static const struct vgic_register_region vgic_v2_dist_registers[] = {
> +    REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
> +        vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 2,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
> +        VGIC_ACCESS_32bit),
> +    REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
> +    REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
> +};
> +
> +unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev)
> +{
> +    dev->regions = vgic_v2_dist_registers;
> +    dev->nr_regions = ARRAY_SIZE(vgic_v2_dist_registers);
> +
> +    return SZ_4K;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 3c70945466..59703a6909 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -182,6 +182,32 @@ struct mmio_handler_ops xen_io_gic_ops = {
>       .write = dispatch_mmio_write,
>   };
>   
> +int vgic_register_dist_iodev(struct domain *d, paddr_t dist_base_address,

I would rather prefer to use gfn_t over paddr_t. The former deal with 
frame only is safer to use.

> +                 enum vgic_type type)
> +{
> +    struct vgic_io_device *io_device = &d->arch.vgic.dist_iodev;
> +    int ret = 0;

This variable is pointless. You never set it after so always return 0.

> +    unsigned int len;
> +
> +    switch (type)

switch ( ... )

> +    {
> +    case VGIC_V2:
> +        len = vgic_v2_init_dist_iodev(io_device);
> +        break;
> +    default:
> +        BUG_ON(1);

Please use BUG() here. But have you checked we will never reach here 
with the wrong vgic_type?

> +    }
> +
> +    io_device->base_addr = dist_base_address;

Also base_addr & co would only contain frame.

> +    io_device->iodev_type = IODEV_DIST;
> +    io_device->redist_vcpu = NULL;
> +
> +    register_mmio_handler(d, &xen_io_gic_ops, dist_base_address, len,
> +                  io_device);
> +
> +    return ret;
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 375b70561d..10ac682296 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -137,6 +137,8 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>                           unsigned int len, unsigned long val);
>   
> +unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
> +
>   /* Find the proper register handler entry given a certain address offset */
>   const struct vgic_register_region *
>   vgic_find_mmio_region(const struct vgic_register_region *regions,
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 426b34d0ce..7747d3f3e0 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -49,6 +49,8 @@ void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
>   void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
>   void vgic_v2_set_underflow(struct vcpu *vcpu);
> +int vgic_register_dist_iodev(struct domain *d, paddr_t dist_base_address,
> +                 enum vgic_type);
>   
>   void vgic_v2_save_state(struct vcpu *vcpu);
>   void vgic_v2_restore_state(struct vcpu *vcpu);
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 29/49] ARM: new VGIC: Add CTLR, TYPER and IIDR handlers

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> Those three registers are v2 emulation specific, so their implementation
> lives entirely in vgic-mmio-v2.c. Also they are handled in one function,
> as their implementation is pretty simple.
> When the guest enables the distributor, we kick all VCPUs to get
> potentially pending interrupts serviced.
> 
> This is based on Linux commit 2b0cda878965, written by Marc Zyngier.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 48 +++++++++++++++++++++++++++++++++++++++-
>   xen/arch/arm/vgic/vgic.c         | 15 +++++++++++++
>   xen/arch/arm/vgic/vgic.h         |  4 ++++
>   3 files changed, 66 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index ee685a5a07..0926b3243e 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -20,9 +20,55 @@
>   #include "vgic.h"
>   #include "vgic-mmio.h"
>   
> +static unsigned long vgic_mmio_read_v2_misc(struct vcpu *vcpu,
> +                        paddr_t addr, unsigned int len)

Indentation.

> +{
> +    u32 value;

Please use uint32_t.

> +
> +    switch (addr & 0x0c)

Coding style switch ( ... ). But I do admit the mask is actually quite 
confusing. You rely on the caller only call it with the right reg here.

It is probably worth to have a comment explaining it.

> +    {
> +    case GICD_CTLR:
> +        value = vcpu->domain->arch.vgic.enabled ? GICD_CTL_ENABLE : 0;
> +        break;
> +    case GICD_TYPER:
> +        value = vcpu->domain->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
> +        value = (value >> 5) - 1;
> +        value |= (vcpu->domain->max_vcpus - 1) << 5;
> +        break;
> +    case GICD_IIDR:
> +        value = (PRODUCT_ID_KVM << 24) | (IMPLEMENTER_ARM << 0);

We probably want to use _ID_XEN here.

> +        break;
> +    default:
> +        return 0;
> +    }
> +
> +    return value;
> +}
> +
> +static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len,
> +                    unsigned long val)

Indentation.

> +{
> +    struct vgic_dist *dist = &vcpu->domain->arch.vgic;
> +    bool was_enabled = dist->enabled;
> +
> +    switch (addr & 0x0c)

Ditto for coding style switch and the mask.

> +    {
> +    case GICD_CTLR:
> +        dist->enabled = val & GICD_CTL_ENABLE;
> +        if ( !was_enabled && dist->enabled )
> +            vgic_kick_vcpus(vcpu->domain);

On Xen, this code is definitely not atomic if you have multiple callers.

> +        break;
> +    case GICD_TYPER:
> +    case GICD_IIDR:
> +        /* Nothing to do */
> +        return;
> +    }
> +}
> +
>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
> +        vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
>           vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 9e7fb1edcb..dc5e011fa3 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -678,6 +678,21 @@ int gic_events_need_delivery(void)
>       return vgic_vcpu_pending_irq(current);
>   }
>   
> +void vgic_kick_vcpus(struct domain *d)
> +{
> +    struct vcpu *vcpu;
> +
> +    /*
> +     * We've injected an interrupt, time to find out who deserves
> +     * a good kick...
> +     */
> +    for_each_vcpu( d, vcpu )
> +    {
> +        if ( vgic_vcpu_pending_irq(vcpu) )
> +            vcpu_unblock(vcpu);

Unblock will not notify a vCPU running. You want also want to send SGI 
if the CPU is running.

> +    }
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 7747d3f3e0..82fe902e26 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -17,6 +17,9 @@
>   #ifndef __XEN_ARM_VGIC_NEW_H__
>   #define __XEN_ARM_VGIC_NEW_H__
>   
> +#define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
> +#define IMPLEMENTER_ARM     0x43b
> +
>   #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>   
>   static inline bool irq_is_pending(struct vgic_irq *irq)
> @@ -36,6 +39,7 @@ struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
>   bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>                  unsigned long flags);
> +void vgic_kick_vcpus(struct domain *d);
>   
>   static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>   {
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> As the enable register handlers are shared between the v2 and v3
> emulation, their implementation goes into vgic-mmio.c, to be easily
> referenced from the v3 emulation as well later.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 114 +++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
>   3 files changed, 127 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 0926b3243e..eca6840ff9 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -74,10 +74,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        vgic_mmio_read_enable, vgic_mmio_write_senable, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 59703a6909..3d9fa02a10 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -39,6 +39,120 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>       /* Ignore */
>   }
>   
> +/*
> + * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the value
> + * of the enabled bit, so there is only one function for both here.
> + */
> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len)

Indentation.

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t here please.

> +    u32 value = 0;

Same here.

> +    int i;
> +
> +    /* Loop over all IRQs affected by this read */
> +    for ( i = 0; i < len * 8; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq->enabled )
> +            value |= (1U << i);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +static void vgic_handle_hardware_irq(irq_desc_t *desc, int irq_type,

Looking below irq_type should a enum vgic_irq_config and not an int.

> +                     bool enable)

Indentation.

> +{
> +    unsigned long flags;
> +
> +//  irq_set_affinity(desc, cpumask_of(v_target->processor));

Why is that commented?

> +    spin_lock_irqsave(&desc->lock, flags);
> +    if ( enable )
> +    {
> +        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
> +                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);

Indentation and I would prefer a helper to convert between the vgic 
value and the IRQ_TYPE. This would make the code easier to read.

Also, this code does not replicate correctly the current vGIC. 
gic_set_irq_type is only allowed to be used when 
irq_set_type_by_domain(d) returns true. If you consider this change 
valid, then I would like to know why.

> +        desc->handler->enable(desc);
> +    }
> +    else
> +        desc->handler->disable(desc);
> +    spin_unlock_irqrestore(&desc->lock, flags);
> +}
> +
> +void vgic_mmio_write_senable(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val)

Indentation.

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t.

> +    irq_desc_t *desc;
> +    int i;
> +    unsigned long flags;
> +    enum vgic_irq_config config;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq;
> +
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        irq->enabled = true;
> +        if ( irq->hw )
> +        {
> +            /*
> +             * The irq cannot be a PPI, we only support delivery
> +             * of SPIs to guests.
> +             */
> +            ASSERT(irq->hwintid >= 32);
> +
> +            desc = irq_to_desc(irq->hwintid);

What is the rationale behind storing hwintid rather than the irq_desc 
directly?

> +            config = irq->config;
> +        }
> +        else
> +            desc = NULL;
> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +
> +        if ( desc )
> +            vgic_handle_hardware_irq(desc, config, true);

This is slightly strange. You handle the hardware IRQ outside the 
virtual IRQ lock. It means that the hardware IRQ may end up enabled but 
the virtual IRQ disabled.

> +    }
> +}
> +
> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    int i;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq;
> +        unsigned long flags;
> +        irq_desc_t *desc;
> +
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->enabled = false;
> +
> +        if ( irq->hw )
> +            desc = irq_to_desc(irq->hwintid);
> +        else
> +            desc = NULL;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +
> +        if ( desc )
> +            vgic_handle_hardware_irq(desc, 0, false);

Same remark here.

> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 10ac682296..9f34bd1aec 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -137,6 +137,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>                           unsigned int len, unsigned long val);
>   
> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len);

Indentation.

> +
> +void vgic_mmio_write_senable(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val);

Ditto.

> +
> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val);

Ditto.

> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   /* Find the proper register handler entry given a certain address offset */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 31/49] ARM: new VGIC: Add PENDING registers handlers

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The pending register handlers are shared between the v2 and v3
> emulation, so their implementation goes into vgic-mmio.c, to be easily
> referenced from the v3 emulation as well later.
> For level triggered interrupts the real line level is unaffected by
> this write, so we keep this state separate and combine it with the
> device's level to get the actual pending state.
> 
> This is based on Linux commit 96b298000db4, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c |  4 +--
>   xen/arch/arm/vgic/vgic-mmio.c    | 62 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    | 11 +++++++
>   3 files changed, 75 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index eca6840ff9..ceb86900a0 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -80,10 +80,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        vgic_mmio_read_pending, vgic_mmio_write_spending, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        vgic_mmio_read_pending, vgic_mmio_write_cpending, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 3d9fa02a10..9a65e39d78 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -153,6 +153,68 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
>       }
>   }
>   
> +unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
> +                     paddr_t addr, unsigned int len)

Indentation.

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t

> +    u32 value = 0;

uint32_t

> +    int i;

unsigned int

> +
> +    /* Loop over all IRQs affected by this read */
> +    for ( i = 0; i < len * 8; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq_is_pending(irq) )
> +            value |= (1U << i);

Don't you need to propagate the value to the hardware for virtual 
interrupt mapped to physical interrupt?

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +void vgic_mmio_write_spending(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t

> +    int i;

unsigned int

> +    unsigned long flags;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        irq->pending_latch = true;

Ditto.

> +
> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +void vgic_mmio_write_cpending(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    int i;
> +    unsigned long flags;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->pending_latch = false;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 9f34bd1aec..209afbbb9a 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -148,6 +148,17 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
>                    paddr_t addr, unsigned int len,
>                    unsigned long val);
>   
> +unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
> +                     paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_spending(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val);
> +
> +void vgic_mmio_write_cpending(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   /* Find the proper register handler entry given a certain address offset */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 32/49] ARM: new VGIC: Add ACTIVE registers handlers

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The active register handlers are shared between the v2 and v3 emulation,
> so their implementation goes into vgic-mmio.c, to be easily referenced
> from the v3 emulation as well later.
> Since activation/deactivation of an interrupt may happen entirely in the
> guest without it ever exiting, we need some extra logic to properly track
> the active state.
> For clearing the active state, we would basically have to halt the guest
> to make sure this is properly propagated into the respective VCPUs.
> This is not yet implemented in Xen.

I don't mind if it is not implemented in the first series. But I think 
we need to make it clearer in the code.

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c |  4 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 94 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    | 11 +++++
>   3 files changed, 107 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index ceb86900a0..eba24d9866 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -86,10 +86,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_pending, vgic_mmio_write_cpending, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        vgic_mmio_read_active, vgic_mmio_write_sactive, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
> +        vgic_mmio_read_active, vgic_mmio_write_cactive, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 9a65e39d78..ac3aa03fbc 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -215,6 +215,100 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
>       }
>   }
>   
> +unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    u32 value = 0;
> +    int i;
> +
> +    /* Loop over all IRQs affected by this read */
> +    for ( i = 0; i < len * 8; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq->active )
> +            value |= (1U << i);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +static void vgic_mmio_change_active(struct vcpu *vcpu, struct vgic_irq *irq,
> +                    bool new_active_state)
> +{

Can we add a gdprintk here? So if we don't implement it in the first 
series, we still keep the current warning?

> +}
> +
> +static void vgic_change_active_prepare(struct vcpu *vcpu, u32 intid)
> +{
> +}
> +
> +static void vgic_change_active_finish(struct vcpu *vcpu, u32 intid)
> +{
> +}
> +
> +static void __vgic_mmio_write_cactive(struct vcpu *vcpu,
> +                      paddr_t addr, unsigned int len,
> +                      unsigned long val)

Indentation.

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t

> +    int i;

unsigned int

> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);

Newline here please.

> +        vgic_mmio_change_active(vcpu, irq, false);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val)

Indentation

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t

> +
> +    spin_lock(&vcpu->domain->domain_lock);

If you want to use the domain lock, then please use domain_lock(..) and 
domain_unlock(..) macros. But I am not sure whether this is the right 
lock here and how long it will be taken.

> +    vgic_change_active_prepare(vcpu, intid);
> +
> +    __vgic_mmio_write_cactive(vcpu, addr, len, val);
> +
> +    vgic_change_active_finish(vcpu, intid);
> +    spin_unlock(&vcpu->domain->domain_lock);
> +}
> +
> +static void __vgic_mmio_write_sactive(struct vcpu *vcpu,
> +                      paddr_t addr, unsigned int len,
> +                      unsigned long val)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t

> +    int i;

unsigned int.

> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +        vgic_mmio_change_active(vcpu, irq, true);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +void vgic_mmio_write_sactive(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);

uint32_t

> +
> +    spin_lock(&vcpu->domain->domain_lock);

See my remark on the domain_lock above.

> +    vgic_change_active_prepare(vcpu, intid);
> +
> +    __vgic_mmio_write_sactive(vcpu, addr, len, val);
> +
> +    vgic_change_active_finish(vcpu, intid);
> +    spin_unlock(&vcpu->domain->domain_lock);
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 209afbbb9a..39e854232e 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -159,6 +159,17 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
>                     paddr_t addr, unsigned int len,
>                     unsigned long val);
>   
> +unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len);

Indentation and belows

> +
> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val);
> +
> +void vgic_mmio_write_sactive(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   /* Find the proper register handler entry given a certain address offset */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 33/49] ARM: new VGIC: Add PRIORITY registers handlers

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The priority register handlers are shared between the v2 and v3 emulation,
> so their implementation goes into vgic-mmio.c, to be easily referenced
> from the v3 emulation as well later.
> There is a corner case when we change the priority of a pending
> interrupt which we don't handle at the moment.
> 
> This is based on Linux commit dd238ec2b87b, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c |  4 ++--
>   xen/arch/arm/vgic/vgic-mmio.c    | 47 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
>   xen/arch/arm/vgic/vgic.h         |  2 ++
>   4 files changed, 58 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index eba24d9866..0574ff9b16 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -92,8 +92,8 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_active, vgic_mmio_write_cactive, NULL, NULL, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
> -        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
> +        vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
> +        8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index ac3aa03fbc..14570d9d8e 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -309,6 +309,53 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>       spin_unlock(&vcpu->domain->domain_lock);
>   }
>   
> +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
> +                      paddr_t addr, unsigned int len)

Indentation.

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);

uin32_t

> +    int i;

unsigned int

> +    u64 val = 0;

uint64_t

> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        val |= (u64)irq->priority << (i * 8);

uint64_t

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return val;
> +}
> +
> +/*
> + * We currently don't handle changing the priority of an interrupt that
> + * is already pending on a VCPU. If there is a need for this, we would
> + * need to make this VCPU exit and re-evaluate the priorities, potentially
> + * leading to this interrupt getting presented now to the guest (if it has
> + * been masked by the priority mask before).
> + */
> +void vgic_mmio_write_priority(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val)

Indentation

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);

uint32_t

> +    int i;

unsigned int

> +    unsigned long flags;
> +
> +    for ( i = 0; i < len; i++ )
> +    {

I believe this code will be follow the atomicity when 2 vCPUs (A and B) 
write to this register. The result should be either the value of write A 
  or write B. But not a mix of the both.

> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        /* Narrow the priority range to what we actually support */
> +        irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 39e854232e..30221096b9 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -170,6 +170,13 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>                    paddr_t addr, unsigned int len,
>                    unsigned long val);
>   
> +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
> +                      paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_priority(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   /* Find the proper register handler entry given a certain address offset */
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 82fe902e26..b104f8e964 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -20,6 +20,8 @@
>   #define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
>   #define IMPLEMENTER_ARM     0x43b
>   
> +#define VGIC_PRI_BITS       5
> +
>   #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>   
>   static inline bool irq_is_pending(struct vgic_irq *irq)
> 

Cheers,


-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 34/49] ARM: new VGIC: Add CONFIG registers handlers

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The config register handlers are shared between the v2 and v3 emulation,
> so their implementation goes into vgic-mmio.c, to be easily referenced
> from the v3 emulation as well later.
> 
> This is based on Linux commit 79717e4ac09c, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 54 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
>   3 files changed, 62 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 0574ff9b16..c0b88b347e 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -98,7 +98,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 2,
> +        vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 14570d9d8e..626ce06986 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -356,6 +356,60 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
>       }
>   }
>   
> +unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len)

Indentation.

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 2);

uint32_t

> +    u32 value = 0;

uint32_t.

> +    int i;

unsigned int.

> +
> +    for ( i = 0; i < len * 4; i++ )
> +    {

Same question as priority regarding the atomicity of ICFGR.

> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq->config == VGIC_CONFIG_EDGE )
> +            value |= (2U << (i * 2));
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +void vgic_mmio_write_config(struct vcpu *vcpu,
> +                paddr_t addr, unsigned int len,
> +                unsigned long val)

Indentation

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 2);

uint32_t

> +    int i;

unsigned int

> +    unsigned long flags;
> +
> +    for ( i = 0; i < len * 4; i++ )
> +    {
> +        struct vgic_irq *irq;
> +
> +        /*
> +         * The configuration cannot be changed for SGIs in general,
> +         * for PPIs this is IMPLEMENTATION DEFINED. The arch timer
> +         * code relies on PPIs being level triggered, so we also
> +         * make them read-only here.
> +         */
> +        if ( intid + i < VGIC_NR_PRIVATE_IRQS )
> +            continue;
> +
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        if ( test_bit(i * 2 + 1, &val) )
> +            irq->config = VGIC_CONFIG_EDGE;
> +        else
> +            irq->config = VGIC_CONFIG_LEVEL;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 30221096b9..b42ea1bd8a 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -177,6 +177,13 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
>                     paddr_t addr, unsigned int len,
>                     unsigned long val);
>   
> +unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len);

Indenation here and below.

> +
> +void vgic_mmio_write_config(struct vcpu *vcpu,
> +                paddr_t addr, unsigned int len,
> +                unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   /* Find the proper register handler entry given a certain address offset */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 35/49] ARM: new VGIC: Add TARGET registers handlers

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> The target register handlers are v2 emulation specific, so their
> implementation lives entirely in vgic-mmio-v2.c.
> We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
> set in the target mask instead of making it possibly pending on
> multiple VCPUs.
> 
> This is based on Linux commit 2c234d6f1826, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 52 +++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 51 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index c0b88b347e..c59f2c1ba7 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -66,6 +66,56 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>       }
>   }
>   
> +static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
> +                       paddr_t addr, unsigned int len)

Indentation

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);

uint32_t

> +    int i;

unsigned int

> +    u64 val = 0;


Why 64-bit? IIRC, the target register is 32-bit.


> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            vcpu, intid + i);

Indenation.

> +
> +        val |= (u64)irq->targets << (i * 8);

See above regarding u64.

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return val;
> +}
> +
> +static void vgic_mmio_write_target(struct vcpu *vcpu,
> +                   paddr_t addr, unsigned int len,
> +                   unsigned long val)

Indentation

> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);

uint32_t

> +    u8 cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);

uint32_t

> +    int i;

unsigned int

> +    unsigned long flags;
> +
> +    /* GICD_ITARGETSR[0-7] are read-only */
> +    if ( intid < VGIC_NR_PRIVATE_IRQS )
> +        return;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            NULL, intid + i);
> +        int target;

unsigned int

> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->targets = (val >> (i * 8)) & cpu_mask;
> +        target = irq->targets ? (ffs(irq->targets) - 1) : 0;

Here you will route the IRQ to vCPU 0 if the mask is invalid. Is it 
intended? Do you have a pointer in the spec?

> +        irq->target_vcpu = vcpu->domain->vcpu[target];
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
> @@ -95,7 +145,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
>           8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
> +        vgic_mmio_read_target, vgic_mmio_write_target, NULL, NULL, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
>           vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 36/49] ARM: new VGIC: Add SGIR register handler

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> Triggering an IPI via this register is v2 specific, so the
> implementation lives entirely in vgic-mmio-v2.c.
> 
> This is based on Linux commit 55cc01fb9004, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 47 +++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 46 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index c59f2c1ba7..3f67b4659a 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -66,6 +66,51 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>       }
>   }
>   
> +static void vgic_mmio_write_sgir(struct vcpu *source_vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val)
> +{
> +    struct domain *d = source_vcpu->domain;
> +    int nr_vcpus = d->max_vcpus;
> +    int intid = val & 0xf;
> +    int targets = (val >> 16) & 0xff; > +    int mode = (val >> 24) & 0x03;

Please use unsigned for value that does not required to be signed. Also, 
please use define however hardcoded value whenever it is possible./

> +    struct vcpu *vcpu;
> +    unsigned long flags;
> +
> +    switch (mode)
> +    {
> +    case 0x0:       /* as specified by targets */
> +        break;
> +    case 0x1:
> +        targets = (1U << nr_vcpus) - 1;         /* all, ... */
> +        targets &= ~(1U << source_vcpu->vcpu_id);   /* but self */

Please keep the both comment indented the same way.

> +        break;
> +    case 0x2:       /* this very vCPU only */
> +        targets = (1U << source_vcpu->vcpu_id);
> +        break;
> +    case 0x3:       /* reserved */
> +        return;
> +    }
> +
> +    for_each_vcpu(d, vcpu)

It would make more sense to iterate on bit set in targets. This would 
avoid to go through all the vCPUs most of the time.

> +    {
> +        struct vgic_irq *irq;
> +
> +        if ( !(targets & (1U << vcpu->vcpu_id)) )
> +            continue;
> +
> +        irq = vgic_get_irq(d, vcpu, intid);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        irq->pending_latch = true;
> +        irq->source |= 1U << source_vcpu->vcpu_id;
> +
> +        vgic_queue_irq_unlock(d, irq, flags);
> +        vgic_put_irq(d, irq);
> +    }
> +}
> +
>   static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
>                          paddr_t addr, unsigned int len)
>   {
> @@ -151,7 +196,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
> +        vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 37/49] ARM: new VGIC: Add SGIPENDR register handlers

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> As this register is v2 specific, its implementation lives entirely
> in vgic-mmio-v2.c.
> This register allows setting the source mask of an IPI.
> 
> This is based on Linux commit ed40213ef9b0, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 77 ++++++++++++++++++++++++++++++++++++++--
>   1 file changed, 75 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 3f67b4659a..c62307c3aa 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -161,6 +161,79 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
>       }
>   }
>   
> +static unsigned long vgic_mmio_read_sgipend(struct vcpu *vcpu,
> +                        paddr_t addr, unsigned int len)

Indentation

> +{
> +    u32 intid = addr & 0x0f;

uint32_t also why not using VGIC_ADDR_TO_INTID here?

> +    int i;

unsigned int.

> +    u64 val = 0;

Why 64-bit?

> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            vcpu, intid + i);

Indentation.

> +
> +        val |= (u64)irq->source << (i * 8);

See above for u64.

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +    return val;
> +}
> +
> +static void vgic_mmio_write_sgipendc(struct vcpu *vcpu,
> +                     paddr_t addr, unsigned int len,
> +                     unsigned long val)

Same remarks as above for this function.

> +{ > +    u32 intid = addr & 0x0f;> +    int i;
> +    unsigned long flags;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->source &= ~((val >> (i * 8)) & 0xff);
> +        if ( !irq->source )
> +            irq->pending_latch = false;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +static void vgic_mmio_write_sgipends(struct vcpu *vcpu,
> +                     paddr_t addr, unsigned int len,
> +                     unsigned long val)

Ditto.

> +{
> +    u32 intid = addr & 0x0f;
> +    int i;
> +    unsigned long flags;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->source |= (val >> (i * 8)) & 0xff;
> +
> +        if ( irq->source )
> +        {
> +            irq->pending_latch = true;
> +            vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> +        }
> +        else
> +        {
> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        }
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
> @@ -199,10 +272,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> +        vgic_mmio_read_sgipend, vgic_mmio_write_sgipendc, 16,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> +        vgic_mmio_read_sgipend, vgic_mmio_write_sgipends, 16,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>   };
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
> when a guest EOIs the virtual interrupt, it affects the state of that
> corresponding interrupt on the hardware side at the same time.
> Implement the interface that the Xen arch/core code expects to connect
> the virtual and the physical world.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 63 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index dc5e011fa3..8d5260a7db 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>       }
>   }
>   
> +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
> +                                      unsigned int virq)
> +{
> +    struct irq_desc *desc = NULL;
> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
> +    unsigned long flags;
> +
> +    if ( !irq )
> +        return NULL;
> +
> +    spin_lock_irqsave(&irq->irq_lock, flags);
> +    if ( irq->hw )
> +        desc = irq_to_desc(irq->hwintid);

This is not going to work well for PPIs. We should consider to add at 
least an ASSERT(...) in the code to prevent bad use of it.

> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +    vgic_put_irq(d, irq);
> +
> +    return desc;
> +}
> +
> +/*
> + * was:
> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int virt_irq)
> + */
> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
> +            unsigned int virt_irq, struct irq_desc *desc,
> +            bool connect)

Indentation.

> +{
> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
> +    unsigned long flags;
> +    int ret = 0;
> +
> +    if ( !irq )
> +        return -EINVAL;
> +
> +    spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +    if ( connect )                      /* assign a mapped IRQ */
> +    {
> +        /* The VIRQ should not be already enabled by the guest */
> +        if ( !irq->hw && !irq->enabled )
> +        {
> +            irq->hw = true;
> +            irq->hwintid = desc->irq;
> +        }
> +        else
> +        {
> +            ret = -EBUSY;
> +        }

I know that it should not matter for SPIs today. But aren't you meant to 
get a reference on that interrupt if you connect it?

> +    }
> +    else                                /* remove a mapped IRQ */
> +    {
> +        irq->hw = false;
> +        irq->hwintid = 0;

Here you blindly remove the interrupt without been sure it was the 
correct physical one. We should have a check like in the current vGIC 
version.

> +    }
> +
> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
> +    vgic_put_irq(d, irq);
> +
> +    return ret;
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 28/49] ARM: new VGIC: Add GICv2 MMIO handling framework

[Andre Przywara]

Hi,

On 16/02/18 15:39, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Create vgic-mmio-v2.c to describe GICv2 emulation specific handlers
>> using the initializer macros provided by the VGIC MMIO framework.
>> Provide a function to register the GICv2 distributor registers to
>> the Xen MMIO framework.
>> The actual handler functions are still stubs in this patch.
>>
>> This is based on Linux commit fb848db39661, written by Andre Przywara.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c | 83
>> ++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.c    | 26 +++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  2 +
>>   xen/arch/arm/vgic/vgic.h         |  2 +
>>   4 files changed, 113 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> new file mode 100644
>> index 0000000000..ee685a5a07
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -0,0 +1,83 @@
>> +/*
>> + * VGICv2 MMIO handling functions
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <xen/bitops.h>
>> +#include <xen/sched.h>
>> +#include <xen/sizes.h>
>> +#include <asm/arm_vgic.h>
>> +
>> +#include "vgic.h"
>> +#include "vgic-mmio.h"
>> +
>> +static const struct vgic_register_region vgic_v2_dist_registers[] = {
>> +    REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
>> +        vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>> +    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 2,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
>> +        VGIC_ACCESS_32bit),
>> +    REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
>> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>> +    REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
>> +        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
>> +        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>> +};
>> +
>> +unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev)
>> +{
>> +    dev->regions = vgic_v2_dist_registers;
>> +    dev->nr_regions = ARRAY_SIZE(vgic_v2_dist_registers);
>> +
>> +    return SZ_4K;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index 3c70945466..59703a6909 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -182,6 +182,32 @@ struct mmio_handler_ops xen_io_gic_ops = {
>>       .write = dispatch_mmio_write,
>>   };
>>   +int vgic_register_dist_iodev(struct domain *d, paddr_t
>> dist_base_address,
> 
> I would rather prefer to use gfn_t over paddr_t. The former deal with
> frame only is safer to use.

OK.

>> +                 enum vgic_type type)
>> +{
>> +    struct vgic_io_device *io_device = &d->arch.vgic.dist_iodev;
>> +    int ret = 0;
> 
> This variable is pointless. You never set it after so always return 0.

True. The KVM version of register_mmio_handler() returns an error value,
which is held in this variable (since it is called within a lock). Will
remove it.

>> +    unsigned int len;
>> +
>> +    switch (type)
> 
> switch ( ... )
> 
>> +    {
>> +    case VGIC_V2:
>> +        len = vgic_v2_init_dist_iodev(io_device);
>> +        break;
>> +    default:
>> +        BUG_ON(1);
> 
> Please use BUG() here. But have you checked we will never reach here
> with the wrong vgic_type?

This is just a placeholder for now, the proper vGICv3 function will plug
in later. Actually I don't expect this line to be ever part of running code.
Even if, at the moment no one sets the type to anything other than
VGIC_V2, so this should be safe.

Cheers,
Andre.

>> +    }
>> +
>> +    io_device->base_addr = dist_base_address;
> 
> Also base_addr & co would only contain frame.
> 
>> +    io_device->iodev_type = IODEV_DIST;
>> +    io_device->redist_vcpu = NULL;
>> +
>> +    register_mmio_handler(d, &xen_io_gic_ops, dist_base_address, len,
>> +                  io_device);
>> +
>> +    return ret;
>> +}
>> +
>>   /*
>>    * Local variables:
>>    * mode: C
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 375b70561d..10ac682296 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -137,6 +137,8 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>>                           unsigned int len, unsigned long val);
>>   +unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>> +
>>   /* Find the proper register handler entry given a certain address
>> offset */
>>   const struct vgic_register_region *
>>   vgic_find_mmio_region(const struct vgic_register_region *regions,
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 426b34d0ce..7747d3f3e0 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -49,6 +49,8 @@ void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq,
>> int lr);
>>   void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
>>   void vgic_v2_set_underflow(struct vcpu *vcpu);
>> +int vgic_register_dist_iodev(struct domain *d, paddr_t
>> dist_base_address,
>> +                 enum vgic_type);
>>     void vgic_v2_save_state(struct vcpu *vcpu);
>>   void vgic_v2_restore_state(struct vcpu *vcpu);
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 41/49] ARM: new VGIC: dump virtual IRQ info

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> When we dump guest state on the Xen console, we also print the state of
> IRQs that are on a VCPU.
> Add the code to dump the state of an IRQ handled by the new VGIC.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
>   1 file changed, 13 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 3b475ed1a4..97ffdba5ad 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -757,6 +757,19 @@ void vgic_free_virq(struct domain *d, unsigned int virq)
>       clear_bit(virq, d->arch.vgic.allocated_irqs);
>   }
>   
> +void gic_dump_vgic_info(struct vcpu *v)
> +{
> +    struct vgic_cpu *vgic_cpu = &v->arch.vgic_cpu;
> +    struct vgic_irq *irq;
> +
> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)

I don't think you can assume that the vCPU is not running somewhere 
else. So likely you want to take the lock while dumping the info.

> +        printk("   on CPU: %s %s irq %u: %spending, %sactive, %senabled\n",

I am not sure the value of "on CPU".

> +               irq->hw ? "hardware" : "virtual",
> +               irq->config == VGIC_CONFIG_LEVEL ? "level" : "edge",
> +               irq->intid, irq_is_pending(irq) ? "" : "not ",
> +               irq->active ? "" : "not ", irq->enabled ? "" : "not ");
> +}
> +
>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>                                         unsigned int virq)
>   {
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 35/49] ARM: new VGIC: Add TARGET registers handlers

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> The target register handlers are v2 emulation specific, so their
> implementation lives entirely in vgic-mmio-v2.c.
> We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
> set in the target mask instead of making it possibly pending on
> multiple VCPUs.
> 
> This is based on Linux commit 2c234d6f1826, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 52 +++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 51 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index c0b88b347e..c59f2c1ba7 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -66,6 +66,56 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>       }
>   }
>   
> +static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
> +                       paddr_t addr, unsigned int len)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    int i;
> +    u64 val = 0;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            vcpu, intid + i);
> +
> +        val |= (u64)irq->targets << (i * 8);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return val;
> +}
> +
> +static void vgic_mmio_write_target(struct vcpu *vcpu,
> +                   paddr_t addr, unsigned int len,
> +                   unsigned long val)
> +{
> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    u8 cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);
> +    int i;
> +    unsigned long flags;
> +
> +    /* GICD_ITARGETSR[0-7] are read-only */
> +    if ( intid < VGIC_NR_PRIVATE_IRQS )
> +        return;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
> +                            NULL, intid + i);
> +        int target;
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->targets = (val >> (i * 8)) & cpu_mask;
> +        target = irq->targets ? (ffs(irq->targets) - 1) : 0;
> +        irq->target_vcpu = vcpu->domain->vcpu[target];

You should modify the affinity of the physical interrupt here. So it 
will fire on the pCPU where the new vCPU run.

> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
> @@ -95,7 +145,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
>           8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
> +        vgic_mmio_read_target, vgic_mmio_write_target, NULL, NULL, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
>           vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 43/49] ARM: new VGIC: Add preliminary stub implementations

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> The Xen core code requires an interrupt controller emulation to implement
> arch_move_irqs(), to move the affinity of an hardware mapped virtual IRQ
> to another core. In the moment we don't implement this
> physical-follow-virtual regime in our new VGIC, so just provide an empty
> stub implementation to make the linker happy.

physical-follow-virtual is a must feature for the new vGIC. This has 
shown better interrupt latency.

> Similarily vgic_clear_pending_irqs() is required by the ARM code,
> although it is suspected that it is actually not necessary. Go with a
> stub for now.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
>   1 file changed, 13 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index d91028bd43..77fa756329 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -770,6 +770,19 @@ void gic_dump_vgic_info(struct vcpu *v)
>                  irq->active ? "" : "not ", irq->enabled ? "" : "not ");
>   }
>   
> +void vgic_clear_pending_irqs(struct vcpu *v)
> +{
> +    /*
> +     * TODO: It is unclear whether we really need this, so we might instead
> +     * remove it on the caller site.
> +     */

I remember some issue with the current vGIC when not removing pending 
interrupts on PSCI CPU ON. But that was a while ago. I will have another 
try and see if we can drop it.

> +}
> +
> +void arch_move_irqs(struct vcpu *v)
> +{
> +    /* TODO: implement this (?) */

Here you would need to go through the interrupt and modify the affinity 
of the physical IRQs routed to that vCPU.

> +}
> +
>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>                                         unsigned int virq)
>   {
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 44/49] ARM: new VGIC: vgic-init: register VGIC

[Julien Grall]

Hi Andre,

On 09/02/18 14:39, Andre Przywara wrote:
> This patch implements the function which is called by Xen when it wants
> to register the virtual GIC.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-init.c | 62 +++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h      |  3 +++
>   2 files changed, 65 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic-init.c
> 
> diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
> new file mode 100644
> index 0000000000..b5f1183a50
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-init.c
> @@ -0,0 +1,62 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/sched.h>
> +#include <asm/arm_vgic.h>
> +
> +#include "vgic.h"
> +
> +/* CREATION */
> +
> +/**
> + * domain_vgic_register: create a virtual GIC
> + * @d: domain pointer
> + * @mmio_count: pointer to add number of required MMIO regions
> + *
> + * was: kvm_vgic_create
> + */
> +int domain_vgic_register(struct domain *d, int *mmio_count)

mmio_count should be set to the number of I/O region you will register.

> +{
> +    switch ( d->arch.vgic.version )
> +    {
> +#ifdef CONFIG_HAS_GICV3
> +    case GIC_V3:
> +        d->arch.max_vcpus = VGIC_V3_MAX_CPUS;
> +        break;
> +#endif
> +    case GIC_V2:
> +        d->arch.max_vcpus = VGIC_V2_MAX_CPUS;
> +        break;
> +    }
> +
> +    if ( d->max_vcpus > d->arch.max_vcpus )
> +        return -E2BIG;
> +
> +    d->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
> +    d->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
> +    d->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;

Is there any reason to store an address rather than a frame? The latter 
would add a be more safety.

> +
> +    return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index b104f8e964..205ce10ffa 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -20,6 +20,9 @@
>   #define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
>   #define IMPLEMENTER_ARM     0x43b
>   
> +#define VGIC_ADDR_UNDEF     (-1)

Please use INVALID_PADDR here.

> +#define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
> +
>   #define VGIC_PRI_BITS       5
>   
>   #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Andre Przywara]

Hi,

On 16/02/18 16:57, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> As the enable register handlers are shared between the v2 and v3
>> emulation, their implementation goes into vgic-mmio.c, to be easily
>> referenced from the v3 emulation as well later.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>>   xen/arch/arm/vgic/vgic-mmio.c    | 114
>> +++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
>>   3 files changed, 127 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index 0926b3243e..eca6840ff9 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -74,10 +74,10 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        vgic_mmio_read_enable, vgic_mmio_write_senable, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index 59703a6909..3d9fa02a10 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -39,6 +39,120 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t
>> addr,
>>       /* Ignore */
>>   }
>>   +/*
>> + * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the
>> value
>> + * of the enabled bit, so there is only one function for both here.
>> + */
>> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
>> +                    paddr_t addr, unsigned int len)
> 
> Indentation.
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> 
> uint32_t here please.
> 
>> +    u32 value = 0;
> 
> Same here.
> 
>> +    int i;
>> +
>> +    /* Loop over all IRQs affected by this read */
>> +    for ( i = 0; i < len * 8; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        if ( irq->enabled )
>> +            value |= (1U << i);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return value;
>> +}
>> +
>> +static void vgic_handle_hardware_irq(irq_desc_t *desc, int irq_type,
> 
> Looking below irq_type should a enum vgic_irq_config and not an int.
> 
>> +                     bool enable)
> 
> Indentation.
> 
>> +{
>> +    unsigned long flags;
>> +
>> +//  irq_set_affinity(desc, cpumask_of(v_target->processor));
> 
> Why is that commented?

That should indeed be a TODO:, actually.
As we already discussed, KVM does not implement this
hardware-follows-virtual affinity. This line is just copied from the old
VGIC, to remind me to address this. But I need to check the locking
order here first and if there are any other side effects of changing the
hardware affinity in this particular context.

>> +    spin_lock_irqsave(&desc->lock, flags);
>> +    if ( enable )
>> +    {
>> +        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
>> +                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);
> 
> Indentation and I would prefer a helper to convert between the vgic
> value and the IRQ_TYPE. This would make the code easier to read.
> 
> Also, this code does not replicate correctly the current vGIC.
> gic_set_irq_type is only allowed to be used when
> irq_set_type_by_domain(d) returns true. If you consider this change
> valid, then I would like to know why.

So what is/was the rationale for not allowing IRQ type changes for
non-privileged guests? If you allow to pass through an hardware IRQ to a
guest (which is the case this function handles), then I don't see why a
guest would not be allowed to change the configuration? It seems rather
odd, I guess it's up to the guest to know which type of IRQ this is?

>> +        desc->handler->enable(desc);
>> +    }
>> +    else
>> +        desc->handler->disable(desc);
>> +    spin_unlock_irqrestore(&desc->lock, flags);
>> +}
>> +
>> +void vgic_mmio_write_senable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val)
> 
> Indentation.
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> 
> uint32_t.
> 
>> +    irq_desc_t *desc;
>> +    int i;
>> +    unsigned long flags;
>> +    enum vgic_irq_config config;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq;
>> +
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +        irq->enabled = true;
>> +        if ( irq->hw )
>> +        {
>> +            /*
>> +             * The irq cannot be a PPI, we only support delivery
>> +             * of SPIs to guests.
>> +             */
>> +            ASSERT(irq->hwintid >= 32);
>> +
>> +            desc = irq_to_desc(irq->hwintid);
> 
> What is the rationale behind storing hwintid rather than the irq_desc
> directly?
> 
>> +            config = irq->config;
>> +        }
>> +        else
>> +            desc = NULL;
>> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +
>> +        if ( desc )
>> +            vgic_handle_hardware_irq(desc, config, true);
> 
> This is slightly strange. You handle the hardware IRQ outside the
> virtual IRQ lock. It means that the hardware IRQ may end up enabled but
> the virtual IRQ disabled.

Yeah, good catch. This can't be easily called before dropping the IRQ
lock, as this would violate the locking order.
But I can try to re-take the IRQ lock after having acquired the desc
lock, then use the enabled (and config) value from struct vgic_irq directly.

Cheers,
Andre.

> 
>> +    }
>> +}
>> +
>> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val)
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    int i;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq;
>> +        unsigned long flags;
>> +        irq_desc_t *desc;
>> +
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->enabled = false;
>> +
>> +        if ( irq->hw )
>> +            desc = irq_to_desc(irq->hwintid);
>> +        else
>> +            desc = NULL;
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +
>> +        if ( desc )
>> +            vgic_handle_hardware_irq(desc, 0, false);
> 
> Same remark here.
> 
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 10ac682296..9f34bd1aec 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -137,6 +137,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>>                           unsigned int len, unsigned long val);
>>   +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
>> +                    paddr_t addr, unsigned int len);
> 
> Indentation.
> 
>> +
>> +void vgic_mmio_write_senable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val);
> 
> Ditto.
> 
>> +
>> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val);
> 
> Ditto.
> 
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     /* Find the proper register handler entry given a certain address
>> offset */
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 45/49] ARM: new VGIC: vgic-init: implement vgic_init

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> This patch allocates and initializes the data structures used to model
> the vgic distributor and virtual cpu interfaces. At that stage the
> number of IRQs and number of virtual CPUs is frozen.
> 
> This is based on Linux commit ad275b8bb1e6, written by Eric Auger.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/vgic/vgic-init.c | 197 ++++++++++++++++++++++++++++++++++++++++++

This file is exporting a lot of function, all the corresponding 
prototype should be declared within this patch as well.

Cheers,

>   1 file changed, 197 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
> index b5f1183a50..0cd2dfc600 100644
> --- a/xen/arch/arm/vgic/vgic-init.c
> +++ b/xen/arch/arm/vgic/vgic-init.c
> @@ -1,5 +1,6 @@
>   /*
>    * Copyright (C) 2015, 2016 ARM Ltd.
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>    *
>    * This program is free software; you can redistribute it and/or modify
>    * it under the terms of the GNU General Public License version 2 as
> @@ -19,6 +20,77 @@
>   
>   #include "vgic.h"
>   
> +/*
> + * Initialization rules: there are multiple stages to the vgic
> + * initialization, both for the distributor and the CPU interfaces.  The basic
> + * idea is that even though the VGIC is not functional or not requested from
> + * user space, the critical path of the run loop can still call VGIC functions
> + * that just won't do anything, without them having to check additional
> + * initialization flags to ensure they don't look at uninitialized data
> + * structures.
> + *
> + * Distributor:
> + *
> + * - vgic_early_init(): initialization of static data that doesn't
> + *   depend on any sizing information or emulation type. No allocation
> + *   is allowed there.
> + *
> + * - vgic_init(): allocation and initialization of the generic data
> + *   structures that depend on sizing information (number of CPUs,
> + *   number of interrupts). Also initializes the vcpu specific data
> + *   structures. Can be executed lazily for GICv2.
> + *
> + * CPU Interface:
> + *
> + * - kvm_vgic_vcpu_early_init(): initialization of static data that
> + *   doesn't depend on any sizing information or emulation type. No
> + *   allocation is allowed there.
> + */
> +
> +/**
> + * vgic_vcpu_early_init() - Initialize static VGIC VCPU data structures
> + * @vcpu: The VCPU whose VGIC data structures whould be initialized
> + *
> + * Only do initialization, but do not actually enable the VGIC CPU interface
> + * yet.
> + */
> +static void vgic_vcpu_early_init(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +    int i;
> +
> +    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
> +    spin_lock_init(&vgic_cpu->ap_list_lock);
> +
> +    /*
> +     * Enable and configure all SGIs to be edge-triggered and
> +     * configure all PPIs as level-triggered.
> +     */
> +    for ( i = 0; i < VGIC_NR_PRIVATE_IRQS; i++ )
> +    {
> +        struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
> +
> +        INIT_LIST_HEAD(&irq->ap_list);
> +        spin_lock_init(&irq->irq_lock);
> +        irq->intid = i;
> +        irq->vcpu = NULL;
> +        irq->target_vcpu = vcpu;
> +        irq->targets = 1U << vcpu->vcpu_id;
> +        atomic_set(&irq->refcount, 0);
> +        if ( vgic_irq_is_sgi(i) )
> +        {
> +            /* SGIs */
> +            irq->enabled = 1;
> +            irq->config = VGIC_CONFIG_EDGE;
> +        }
> +        else
> +        {
> +            /* PPIs */
> +            irq->config = VGIC_CONFIG_LEVEL;
> +        }
> +    }
> +}
> +
>   /* CREATION */
>   
>   /**
> @@ -52,6 +124,131 @@ int domain_vgic_register(struct domain *d, int *mmio_count)
>       return 0;
>   }
>   
> +/* INIT/DESTROY */
> +
> +/**
> + * domain_vgic_init: initialize the dist data structures
> + * @d: domain pointer
> + * @nr_spis: number of SPIs
> + */
> +int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +    int i, ret;
> +
> +    /* Limit the number of virtual SPIs supported to (1020 - 32) = 988  */
> +    if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
> +        return -EINVAL;
> +
> +    dist->nr_spis = nr_spis;
> +    dist->spis = xzalloc_array(struct vgic_irq, nr_spis);
> +    if ( !dist->spis )
> +        return  -ENOMEM;
> +
> +    /*
> +     * In the following code we do not take the irq struct lock since
> +     * no other action on irq structs can happen while the VGIC is
> +     * not initialized yet:
> +     * If someone wants to inject an interrupt or does a MMIO access, we
> +     * require prior initialization in case of a virtual GICv3 or trigger
> +     * initialization when using a virtual GICv2.
> +     */
> +    for ( i = 0; i < nr_spis; i++ )
> +    {
> +        struct vgic_irq *irq = &dist->spis[i];
> +
> +        irq->intid = i + VGIC_NR_PRIVATE_IRQS;
> +        INIT_LIST_HEAD(&irq->ap_list);
> +        spin_lock_init(&irq->irq_lock);
> +        irq->vcpu = NULL;
> +        irq->target_vcpu = NULL;
> +        atomic_set(&irq->refcount, 0);
> +        if ( dist->version == GIC_V2 )
> +            irq->targets = 0;
> +        else
> +            irq->mpidr = 0;
> +    }
> +
> +    INIT_LIST_HEAD(&dist->lpi_list_head);
> +    spin_lock_init(&dist->lpi_list_lock);
> +
> +    if ( dist->version == GIC_V2 )
> +        ret = vgic_v2_map_resources(d);
> +    else
> +        ret = -ENXIO;
> +
> +    if ( ret )
> +        return ret;
> +
> +    /* allocated_irqs() is used by Xen to find available vIRQs */
> +    d->arch.vgic.allocated_irqs =
> +        xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
> +    if ( !d->arch.vgic.allocated_irqs )
> +        return -ENOMEM;
> +
> +    /* vIRQ0-15 (SGIs) are reserved */
> +    for ( i = 0; i < NR_GIC_SGI; i++ )
> +        set_bit(i, d->arch.vgic.allocated_irqs);
> +
> +    return 0;
> +}
> +
> +/**
> + * vcpu_vgic_init() - Register VCPU-specific KVM iodevs
> + * was: kvm_vgic_vcpu_init()
> + * Xen: adding vgic_vx_enable() call
> + * @vcpu: pointer to the VCPU being created and initialized
> + */
> +int vcpu_vgic_init(struct vcpu *vcpu)
> +{
> +    int ret = 0;
> +
> +    vgic_vcpu_early_init(vcpu);
> +
> +    if ( gic_hw_version() == GIC_V2 )
> +        vgic_v2_enable(vcpu);
> +    else
> +        ret = -ENXIO;
> +
> +    return ret;
> +}
> +
> +void domain_vgic_free(struct domain *d)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +        int i, ret;
> +
> +    for ( i = 0; i < dist->nr_spis; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(d, NULL, 32 + i);
> +
> +        if ( !irq->hw )
> +            continue;
> +
> +        ret = release_guest_irq(d, irq->hwintid);
> +        if ( ret )
> +            dprintk(XENLOG_G_WARNING,
> +                "d%u: Failed to release virq %u ret = %d\n",
> +                d->domain_id, 32 + i, ret);
> +    }
> +
> +    dist->ready = false;
> +    dist->initialized = false;
> +
> +    xfree(dist->spis);
> +    xfree(dist->allocated_irqs);
> +    dist->nr_spis = 0;
> +}
> +
> +int vcpu_vgic_free(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
> +
> +    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
> +
> +    return 0;
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 48/49] ARM: allocate two pages for struct vcpu

[Julien Grall]

Hi,

On 09/02/18 14:39, Andre Przywara wrote:
> At the moment we allocate exactly one page for struct vcpu on ARM, also
> have a check in place to prevent it growing beyond 4KB.
> As the struct includes the state of all 32 private (per-VCPU) interrupts,
> we are at 3840 bytes on arm64 at the moment already. Growing the per-IRQ
> VGIC structure even slightly makes the VCPU quickly exceed the 4K limit.
> The new VGIC will need more space per virtual IRQ. I spent a few hours
> trying to trim this down, but couldn't get it below 4KB, even with the
> nasty hacks piling up to save some bytes here and there.
> It turns out that beyond efficiency, maybe, there is no real technical
> reason this struct has to fit in one page, so lifting the limit to two
> pages seems like the most pragmatic solution.

I looked briefly:

sizeof(struct vcpu):
	arm32: 3809
	arm64: 5248

Clearly, bumping to 8K for 32-bit as well is a no go for me.

For arm64:

struct vgic_irq {
	spinlock_t                 irq_lock;     /*     0     8 */
	struct list_head           lpi_list;     /*     8    16 */
	struct list_head           ap_list;      /*    24    16 */
	struct vcpu *              vcpu;         /*    40     8 */
	struct vcpu *              target_vcpu;  /*    48     8 */
	u32                        intid;        /*    56     4 */
	_Bool                      line_level;   /*    60     1 */
	_Bool                      pending_latch;/*    61     1 */
	_Bool                      active;       /*    62     1 */
	_Bool                      enabled;      /*    63     1 */
	/* --- cacheline 1 boundary (64 bytes) --- */
	_Bool                      hw;           /*    64     1 */

	/* XXX 3 bytes hole, try to pack */

	atomic_t                   refcount;     /*    68     4 */
	u32                        hwintid;      /*    72     4 */
	union {
		u8                 targets;      /*           1 */
		u32                mpidr;        /*           4 */
	};                                       /*    76     4 */
	u8                         source;       /*    80     1 */
	u8                         priority;     /*    81     1 */

	/* XXX 2 bytes hole, try to pack */

	enum vgic_irq_config       config;       /*    84     4 */

	/* size: 88, cachelines: 2, members: 17 */
	/* sum members: 83, holes: 2, sum holes: 5 */
	/* last cacheline: 24 bytes */
};

   - There are 2 holes, with a total waste of 5 bytes per IRQ.
   - The bool fields could be turned into bool :1.
   - config is 4 bytes just for 2 values! Probably worth considered a 
different type (like a bool).

You multiple that saving per 32 and it will free a bit of space. Even if 
it is not going to reach the 4K, those small optimizing are not invasive.

Looking at the other structures, struct vgic_v{2,3}_cpu_if are 
duplicating struct gic_v{2,3}. The state is restored using the latter, 
but it looks like LRs will be saved in both structure... You probably 
want to look at streamlining that.

I might be convinced we need to bump the size of vCPU allocated, but for 
that I need to see that effort have been made to minimize the size of th 
new structures.

Cheers,

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>   xen/arch/arm/domain.c | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 87bd493924..4dd34393f1 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -502,10 +502,13 @@ void dump_pageframe_info(struct domain *d)
>   struct vcpu *alloc_vcpu_struct(void)
>   {
>       struct vcpu *v;
> -    BUILD_BUG_ON(sizeof(*v) > PAGE_SIZE);
> -    v = alloc_xenheap_pages(0, 0);
> -    if ( v != NULL )
> +
> +    BUILD_BUG_ON(sizeof(*v) > 2 * PAGE_SIZE);
> +    v = alloc_xenheap_pages(1, 0);
> +    if ( v != NULL ) {
>           clear_page(v);
> +        clear_page((void *)v + PAGE_SIZE);
> +    }
>       return v;
>   }
>   
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Julien Grall]

On 19/02/18 12:41, Andre Przywara wrote:
> Hi,

Hi,

> On 16/02/18 16:57, Julien Grall wrote:
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> +    spin_lock_irqsave(&desc->lock, flags);
>>> +    if ( enable )
>>> +    {
>>> +        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
>>> +                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);
>>
>> Indentation and I would prefer a helper to convert between the vgic
>> value and the IRQ_TYPE. This would make the code easier to read.
>>
>> Also, this code does not replicate correctly the current vGIC.
>> gic_set_irq_type is only allowed to be used when
>> irq_set_type_by_domain(d) returns true. If you consider this change
>> valid, then I would like to know why.
> 
> So what is/was the rationale for not allowing IRQ type changes for
> non-privileged guests? If you allow to pass through an hardware IRQ to a
> guest (which is the case this function handles), then I don't see why a
> guest would not be allowed to change the configuration? It seems rather
> odd, I guess it's up to the guest to know which type of IRQ this is?

If you can answer the question on top of irq_type_set_by_domain (i.e 
"See whether it is possible to let any domain configure the type) then 
we can remove it. We decided to only allow for the hardware domain 
because we trust it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 31/49] ARM: new VGIC: Add PENDING registers handlers

[Andre Przywara]

Hi,

On 16/02/18 17:16, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The pending register handlers are shared between the v2 and v3
>> emulation, so their implementation goes into vgic-mmio.c, to be easily
>> referenced from the v3 emulation as well later.
>> For level triggered interrupts the real line level is unaffected by
>> this write, so we keep this state separate and combine it with the
>> device's level to get the actual pending state.
>>
>> This is based on Linux commit 96b298000db4, written by Andre Przywara.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |  4 +--
>>   xen/arch/arm/vgic/vgic-mmio.c    | 62
>> ++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    | 11 +++++++
>>   3 files changed, 75 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index eca6840ff9..ceb86900a0 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -80,10 +80,10 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        vgic_mmio_read_pending, vgic_mmio_write_spending, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        vgic_mmio_read_pending, vgic_mmio_write_cpending, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index 3d9fa02a10..9a65e39d78 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -153,6 +153,68 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
>>       }
>>   }
>>   +unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
>> +                     paddr_t addr, unsigned int len)
> 
> Indentation.
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> 
> uint32_t
> 
>> +    u32 value = 0;
> 
> uint32_t
> 
>> +    int i;
> 
> unsigned int
> 
>> +
>> +    /* Loop over all IRQs affected by this read */
>> +    for ( i = 0; i < len * 8; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        if ( irq_is_pending(irq) )
>> +            value |= (1U << i);
> 
> Don't you need to propagate the value to the hardware for virtual
> interrupt mapped to physical interrupt?

Do you mean in the write functions below? (This is the read function, I
don't see how this would apply here.)

In case you meant the write_[cs]pending() functions:
I don't think this makes too much sense. Why would you want to trigger
an hardware IRQ? All you want it is to deliver it to the guest, which is
what those functions below do. So what do I miss here?

Cheers,
Andre.


>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return value;
>> +}
>> +
>> +void vgic_mmio_write_spending(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val)
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> 
> uint32_t
> 
>> +    int i;
> 
> unsigned int
> 
>> +    unsigned long flags;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +        irq->pending_latch = true;
> 
> Ditto.
> 
>> +
>> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>> +void vgic_mmio_write_cpending(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val)
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    int i;
>> +    unsigned long flags;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->pending_latch = false;
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 9f34bd1aec..209afbbb9a 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -148,6 +148,17 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
>>                    paddr_t addr, unsigned int len,
>>                    unsigned long val);
>>   +unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
>> +                     paddr_t addr, unsigned int len);
>> +
>> +void vgic_mmio_write_spending(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val);
>> +
>> +void vgic_mmio_write_cpending(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val);
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     /* Find the proper register handler entry given a certain address
>> offset */
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 31/49] ARM: new VGIC: Add PENDING registers handlers

[Julien Grall]

On 19/02/18 15:32, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 16/02/18 17:16, Julien Grall wrote:
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> +
>>> +    /* Loop over all IRQs affected by this read */
>>> +    for ( i = 0; i < len * 8; i++ )
>>> +    {
>>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>>> + i);
>>> +
>>> +        if ( irq_is_pending(irq) )
>>> +            value |= (1U << i);
>>
>> Don't you need to propagate the value to the hardware for virtual
>> interrupt mapped to physical interrupt?
> 
> Do you mean in the write functions below? (This is the read function, I
> don't see how this would apply here.)

Hmmm yes. Sorry I misplaced the comment.

> 
> In case you meant the write_[cs]pending() functions:
> I don't think this makes too much sense. Why would you want to trigger
> an hardware IRQ? All you want it is to deliver it to the guest, which is
> what those functions below do. So what do I miss here?

Imagine you clear the pending bit on an hardware mapped interrupt. If 
you never clear the active bit on the physical one, you will never 
receive that interrupt again.

For setting pending bit, I am not entirely sure. But it looks like KVM 
is doing it (see latest master). So I am wondering why Xen is diverging 
here.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 45/49] ARM: new VGIC: vgic-init: implement vgic_init

[Andre Przywara]

Hi,

On 19/02/18 13:21, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> This patch allocates and initializes the data structures used to model
>> the vgic distributor and virtual cpu interfaces. At that stage the
>> number of IRQs and number of virtual CPUs is frozen.
>>
>> This is based on Linux commit ad275b8bb1e6, written by Eric Auger.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-init.c | 197
>> ++++++++++++++++++++++++++++++++++++++++++
> 
> This file is exporting a lot of function, all the corresponding
> prototype should be declared within this patch as well.

But those are implementations of functions used by Xen code already, and
are all declared in xen/include/asm-arm/vgic.h.
Or did I miss any?

Cheers,
Andre.

> 
>>   1 file changed, 197 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-init.c
>> b/xen/arch/arm/vgic/vgic-init.c
>> index b5f1183a50..0cd2dfc600 100644
>> --- a/xen/arch/arm/vgic/vgic-init.c
>> +++ b/xen/arch/arm/vgic/vgic-init.c
>> @@ -1,5 +1,6 @@
>>   /*
>>    * Copyright (C) 2015, 2016 ARM Ltd.
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>>    *
>>    * This program is free software; you can redistribute it and/or modify
>>    * it under the terms of the GNU General Public License version 2 as
>> @@ -19,6 +20,77 @@
>>     #include "vgic.h"
>>   +/*
>> + * Initialization rules: there are multiple stages to the vgic
>> + * initialization, both for the distributor and the CPU interfaces. 
>> The basic
>> + * idea is that even though the VGIC is not functional or not
>> requested from
>> + * user space, the critical path of the run loop can still call VGIC
>> functions
>> + * that just won't do anything, without them having to check additional
>> + * initialization flags to ensure they don't look at uninitialized data
>> + * structures.
>> + *
>> + * Distributor:
>> + *
>> + * - vgic_early_init(): initialization of static data that doesn't
>> + *   depend on any sizing information or emulation type. No allocation
>> + *   is allowed there.
>> + *
>> + * - vgic_init(): allocation and initialization of the generic data
>> + *   structures that depend on sizing information (number of CPUs,
>> + *   number of interrupts). Also initializes the vcpu specific data
>> + *   structures. Can be executed lazily for GICv2.
>> + *
>> + * CPU Interface:
>> + *
>> + * - kvm_vgic_vcpu_early_init(): initialization of static data that
>> + *   doesn't depend on any sizing information or emulation type. No
>> + *   allocation is allowed there.
>> + */
>> +
>> +/**
>> + * vgic_vcpu_early_init() - Initialize static VGIC VCPU data structures
>> + * @vcpu: The VCPU whose VGIC data structures whould be initialized
>> + *
>> + * Only do initialization, but do not actually enable the VGIC CPU
>> interface
>> + * yet.
>> + */
>> +static void vgic_vcpu_early_init(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    int i;
>> +
>> +    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
>> +    spin_lock_init(&vgic_cpu->ap_list_lock);
>> +
>> +    /*
>> +     * Enable and configure all SGIs to be edge-triggered and
>> +     * configure all PPIs as level-triggered.
>> +     */
>> +    for ( i = 0; i < VGIC_NR_PRIVATE_IRQS; i++ )
>> +    {
>> +        struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
>> +
>> +        INIT_LIST_HEAD(&irq->ap_list);
>> +        spin_lock_init(&irq->irq_lock);
>> +        irq->intid = i;
>> +        irq->vcpu = NULL;
>> +        irq->target_vcpu = vcpu;
>> +        irq->targets = 1U << vcpu->vcpu_id;
>> +        atomic_set(&irq->refcount, 0);
>> +        if ( vgic_irq_is_sgi(i) )
>> +        {
>> +            /* SGIs */
>> +            irq->enabled = 1;
>> +            irq->config = VGIC_CONFIG_EDGE;
>> +        }
>> +        else
>> +        {
>> +            /* PPIs */
>> +            irq->config = VGIC_CONFIG_LEVEL;
>> +        }
>> +    }
>> +}
>> +
>>   /* CREATION */
>>     /**
>> @@ -52,6 +124,131 @@ int domain_vgic_register(struct domain *d, int
>> *mmio_count)
>>       return 0;
>>   }
>>   +/* INIT/DESTROY */
>> +
>> +/**
>> + * domain_vgic_init: initialize the dist data structures
>> + * @d: domain pointer
>> + * @nr_spis: number of SPIs
>> + */
>> +int domain_vgic_init(struct domain *d, unsigned int nr_spis)
>> +{
>> +    struct vgic_dist *dist = &d->arch.vgic;
>> +    int i, ret;
>> +
>> +    /* Limit the number of virtual SPIs supported to (1020 - 32) =
>> 988  */
>> +    if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
>> +        return -EINVAL;
>> +
>> +    dist->nr_spis = nr_spis;
>> +    dist->spis = xzalloc_array(struct vgic_irq, nr_spis);
>> +    if ( !dist->spis )
>> +        return  -ENOMEM;
>> +
>> +    /*
>> +     * In the following code we do not take the irq struct lock since
>> +     * no other action on irq structs can happen while the VGIC is
>> +     * not initialized yet:
>> +     * If someone wants to inject an interrupt or does a MMIO access, we
>> +     * require prior initialization in case of a virtual GICv3 or
>> trigger
>> +     * initialization when using a virtual GICv2.
>> +     */
>> +    for ( i = 0; i < nr_spis; i++ )
>> +    {
>> +        struct vgic_irq *irq = &dist->spis[i];
>> +
>> +        irq->intid = i + VGIC_NR_PRIVATE_IRQS;
>> +        INIT_LIST_HEAD(&irq->ap_list);
>> +        spin_lock_init(&irq->irq_lock);
>> +        irq->vcpu = NULL;
>> +        irq->target_vcpu = NULL;
>> +        atomic_set(&irq->refcount, 0);
>> +        if ( dist->version == GIC_V2 )
>> +            irq->targets = 0;
>> +        else
>> +            irq->mpidr = 0;
>> +    }
>> +
>> +    INIT_LIST_HEAD(&dist->lpi_list_head);
>> +    spin_lock_init(&dist->lpi_list_lock);
>> +
>> +    if ( dist->version == GIC_V2 )
>> +        ret = vgic_v2_map_resources(d);
>> +    else
>> +        ret = -ENXIO;
>> +
>> +    if ( ret )
>> +        return ret;
>> +
>> +    /* allocated_irqs() is used by Xen to find available vIRQs */
>> +    d->arch.vgic.allocated_irqs =
>> +        xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
>> +    if ( !d->arch.vgic.allocated_irqs )
>> +        return -ENOMEM;
>> +
>> +    /* vIRQ0-15 (SGIs) are reserved */
>> +    for ( i = 0; i < NR_GIC_SGI; i++ )
>> +        set_bit(i, d->arch.vgic.allocated_irqs);
>> +
>> +    return 0;
>> +}
>> +
>> +/**
>> + * vcpu_vgic_init() - Register VCPU-specific KVM iodevs
>> + * was: kvm_vgic_vcpu_init()
>> + * Xen: adding vgic_vx_enable() call
>> + * @vcpu: pointer to the VCPU being created and initialized
>> + */
>> +int vcpu_vgic_init(struct vcpu *vcpu)
>> +{
>> +    int ret = 0;
>> +
>> +    vgic_vcpu_early_init(vcpu);
>> +
>> +    if ( gic_hw_version() == GIC_V2 )
>> +        vgic_v2_enable(vcpu);
>> +    else
>> +        ret = -ENXIO;
>> +
>> +    return ret;
>> +}
>> +
>> +void domain_vgic_free(struct domain *d)
>> +{
>> +    struct vgic_dist *dist = &d->arch.vgic;
>> +        int i, ret;
>> +
>> +    for ( i = 0; i < dist->nr_spis; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(d, NULL, 32 + i);
>> +
>> +        if ( !irq->hw )
>> +            continue;
>> +
>> +        ret = release_guest_irq(d, irq->hwintid);
>> +        if ( ret )
>> +            dprintk(XENLOG_G_WARNING,
>> +                "d%u: Failed to release virq %u ret = %d\n",
>> +                d->domain_id, 32 + i, ret);
>> +    }
>> +
>> +    dist->ready = false;
>> +    dist->initialized = false;
>> +
>> +    xfree(dist->spis);
>> +    xfree(dist->allocated_irqs);
>> +    dist->nr_spis = 0;
>> +}
>> +
>> +int vcpu_vgic_free(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +
>> +    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
>> +
>> +    return 0;
>> +}
>> +
>>   /*
>>    * Local variables:
>>    * mode: C
>>
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 45/49] ARM: new VGIC: vgic-init: implement vgic_init

[Julien Grall]

On 19/02/18 15:53, Andre Przywara wrote:
> Hi,
> 
> On 19/02/18 13:21, Julien Grall wrote:
>> Hi,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> This patch allocates and initializes the data structures used to model
>>> the vgic distributor and virtual cpu interfaces. At that stage the
>>> number of IRQs and number of virtual CPUs is frozen.
>>>
>>> This is based on Linux commit ad275b8bb1e6, written by Eric Auger.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic-init.c | 197
>>> ++++++++++++++++++++++++++++++++++++++++++
>>
>> This file is exporting a lot of function, all the corresponding
>> prototype should be declared within this patch as well.
> 
> But those are implementations of functions used by Xen code already, and
> are all declared in xen/include/asm-arm/vgic.h.
> Or did I miss any?

Hmmm no. Can you mention it in the commit message please?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 35/49] ARM: new VGIC: Add TARGET registers handlers

[Andre Przywara]

Hi,

On 19/02/18 11:53, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The target register handlers are v2 emulation specific, so their
>> implementation lives entirely in vgic-mmio-v2.c.
>> We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
>> set in the target mask instead of making it possibly pending on
>> multiple VCPUs.
>>
>> This is based on Linux commit 2c234d6f1826, written by Andre Przywara.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c | 52
>> +++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 51 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index c0b88b347e..c59f2c1ba7 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -66,6 +66,56 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>>       }
>>   }
>>   +static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
>> +                       paddr_t addr, unsigned int len)
> 
> Indentation
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
> 
> uint32_t
> 
>> +    int i;
> 
> unsigned int
> 
>> +    u64 val = 0;
> 
> 
> Why 64-bit? IIRC, the target register is 32-bit.
> 
> 
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
>> +                            vcpu, intid + i);
> 
> Indenation.
> 
>> +
>> +        val |= (u64)irq->targets << (i * 8);
> 
> See above regarding u64.
> 
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return val;
>> +}
>> +
>> +static void vgic_mmio_write_target(struct vcpu *vcpu,
>> +                   paddr_t addr, unsigned int len,
>> +                   unsigned long val)
> 
> Indentation
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
> 
> uint32_t
> 
>> +    u8 cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);
> 
> uint32_t

I'd rather keep this at uint8_t, as this the natural data type for the
TARGET register.

> 
>> +    int i;
> 
> unsigned int
> 
>> +    unsigned long flags;
>> +
>> +    /* GICD_ITARGETSR[0-7] are read-only */
>> +    if ( intid < VGIC_NR_PRIVATE_IRQS )
>> +        return;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
>> +                            NULL, intid + i);
>> +        int target;
> 
> unsigned int
> 
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->targets = (val >> (i * 8)) & cpu_mask;
>> +        target = irq->targets ? (ffs(irq->targets) - 1) : 0;
> 
> Here you will route the IRQ to vCPU 0 if the mask is invalid. Is it
> intended? Do you have a pointer in the spec?

I think this was more of a KVM design decision to do so. Might be some
GICv2 speciality. I tossed this question over to the KVM guys.
For now I checked the code and AFAICS we handle target_vcpu being NULL
properly (in terms of not forwarding the interrupt). So I would try to
change this to properly use a NULL pointer here.

Fixed the rest.

Cheers,
Andre

>> +        irq->target_vcpu = vcpu->domain->vcpu[target];
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
>> @@ -95,7 +145,7 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
>>           8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>> +        vgic_mmio_read_target, vgic_mmio_write_target, NULL, NULL, 8,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
>>           vgic_mmio_read_config, vgic_mmio_write_config, NULL, NULL, 2,
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 37/49] ARM: new VGIC: Add SGIPENDR register handlers

[Andre Przywara]

Hi,

On 19/02/18 12:02, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> As this register is v2 specific, its implementation lives entirely
>> in vgic-mmio-v2.c.
>> This register allows setting the source mask of an IPI.
>>
>> This is based on Linux commit ed40213ef9b0, written by Andre Przywara.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c | 77
>> ++++++++++++++++++++++++++++++++++++++--
>>   1 file changed, 75 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index 3f67b4659a..c62307c3aa 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -161,6 +161,79 @@ static void vgic_mmio_write_target(struct vcpu
>> *vcpu,
>>       }
>>   }
>>   +static unsigned long vgic_mmio_read_sgipend(struct vcpu *vcpu,
>> +                        paddr_t addr, unsigned int len)
> 
> Indentation
> 
>> +{
>> +    u32 intid = addr & 0x0f;
> 
> uint32_t also why not using VGIC_ADDR_TO_INTID here?

Yeah, slight overkill, but probably better.
(Hopefully) fixed all the rest.

Cheers,
Andre.

> 
>> +    int i;
> 
> unsigned int.
> 
>> +    u64 val = 0;
> 
> Why 64-bit?
> 
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
>> +                            vcpu, intid + i);
> 
> Indentation.
> 
>> +
>> +        val |= (u64)irq->source << (i * 8);
> 
> See above for u64.
> 
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +    return val;
>> +}
>> +
>> +static void vgic_mmio_write_sgipendc(struct vcpu *vcpu,
>> +                     paddr_t addr, unsigned int len,
>> +                     unsigned long val)
> 
> Same remarks as above for this function.
> 
>> +{ > +    u32 intid = addr & 0x0f;> +    int i;
>> +    unsigned long flags;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
>> +                            vcpu, intid + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->source &= ~((val >> (i * 8)) & 0xff);
>> +        if ( !irq->source )
>> +            irq->pending_latch = false;
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>> +static void vgic_mmio_write_sgipends(struct vcpu *vcpu,
>> +                     paddr_t addr, unsigned int len,
>> +                     unsigned long val)
> 
> Ditto.
> 
>> +{
>> +    u32 intid = addr & 0x0f;
>> +    int i;
>> +    unsigned long flags;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain,
>> +                            vcpu, intid + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->source |= (val >> (i * 8)) & 0xff;
>> +
>> +        if ( irq->source )
>> +        {
>> +            irq->pending_latch = true;
>> +            vgic_queue_irq_unlock(vcpu->domain, irq, flags);
>> +        }
>> +        else
>> +        {
>> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        }
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
>> @@ -199,10 +272,10 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
>> +        vgic_mmio_read_sgipend, vgic_mmio_write_sgipendc, 16,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
>> +        vgic_mmio_read_sgipend, vgic_mmio_write_sgipends, 16,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>   };
>>  
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 33/49] ARM: new VGIC: Add PRIORITY registers handlers

[Andre Przywara]

Hi,

On 16/02/18 17:38, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The priority register handlers are shared between the v2 and v3
>> emulation,
>> so their implementation goes into vgic-mmio.c, to be easily referenced
>> from the v3 emulation as well later.
>> There is a corner case when we change the priority of a pending
>> interrupt which we don't handle at the moment.
>>
>> This is based on Linux commit dd238ec2b87b, written by Andre Przywara.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |  4 ++--
>>   xen/arch/arm/vgic/vgic-mmio.c    | 47
>> ++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
>>   xen/arch/arm/vgic/vgic.h         |  2 ++
>>   4 files changed, 58 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index eba24d9866..0574ff9b16 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -92,8 +92,8 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_active, vgic_mmio_write_cactive, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>> -        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>> +        vgic_mmio_read_priority, vgic_mmio_write_priority, NULL, NULL,
>> +        8, VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 8,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index ac3aa03fbc..14570d9d8e 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -309,6 +309,53 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>>       spin_unlock(&vcpu->domain->domain_lock);
>>   }
>>   +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
>> +                      paddr_t addr, unsigned int len)
> 
> Indentation.
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
> 
> uin32_t
> 
>> +    int i;
> 
> unsigned int
> 
>> +    u64 val = 0;
> 
> uint64_t
> 
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        val |= (u64)irq->priority << (i * 8);
> 
> uint64_t
> 
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return val;
>> +}
>> +
>> +/*
>> + * We currently don't handle changing the priority of an interrupt that
>> + * is already pending on a VCPU. If there is a need for this, we would
>> + * need to make this VCPU exit and re-evaluate the priorities,
>> potentially
>> + * leading to this interrupt getting presented now to the guest (if
>> it has
>> + * been masked by the priority mask before).
>> + */
>> +void vgic_mmio_write_priority(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val)
> 
> Indentation
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 8);
> 
> uint32_t
> 
>> +    int i;
> 
> unsigned int
> 
>> +    unsigned long flags;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
> 
> I believe this code will be follow the atomicity when 2 vCPUs (A and B)
> write to this register. The result should be either the value of write A
>  or write B. But not a mix of the both.

To somewhat summarise our IRL discussion:
Yes, technically I believe this is correct: We should first lock all
affected IRQs, then update the priorities, then unlock (in opposite
order). But: I don't think this is really critical: Two concurrent
writes to the same register are not reliable in the first place, every
sane driver (in the guest in our case) would always take a lock to
prevent this. So while technically the 32-bit MMIO write needs to be
atomic from an ARM ARM point of view, this does not extend to the actual
GIC effects, I believe.
And there are no bad effects to the HV from those being mixed up, so I
tend to ignore this for now.
I have a patch fixing this, but don't believe it's worth the effort and
has other recuperations like introducing a locking order on IRQs as well.

Fixed the rest.
Cheers,
Andre.

>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +        /* Narrow the priority range to what we actually support */
>> +        irq->priority = (val >> (i * 8)) & GENMASK(7, 8 -
>> VGIC_PRI_BITS);
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 39e854232e..30221096b9 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -170,6 +170,13 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>>                    paddr_t addr, unsigned int len,
>>                    unsigned long val);
>>   +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
>> +                      paddr_t addr, unsigned int len);
>> +
>> +void vgic_mmio_write_priority(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val);
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     /* Find the proper register handler entry given a certain address
>> offset */
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 82fe902e26..b104f8e964 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -20,6 +20,8 @@
>>   #define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
>>   #define IMPLEMENTER_ARM     0x43b
>>   +#define VGIC_PRI_BITS       5
>> +
>>   #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>>     static inline bool irq_is_pending(struct vgic_irq *irq)
>>
> 
> Cheers,
> 
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Andre Przywara]

Hi,

On 16/02/18 16:57, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> As the enable register handlers are shared between the v2 and v3
>> emulation, their implementation goes into vgic-mmio.c, to be easily
>> referenced from the v3 emulation as well later.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>>   xen/arch/arm/vgic/vgic-mmio.c    | 114
>> +++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
>>   3 files changed, 127 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index 0926b3243e..eca6840ff9 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -74,10 +74,10 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_rao, vgic_mmio_write_wi, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        vgic_mmio_read_enable, vgic_mmio_write_senable, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> +        vgic_mmio_read_enable, vgic_mmio_write_cenable, NULL, NULL, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, NULL, NULL, 1,
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index 59703a6909..3d9fa02a10 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -39,6 +39,120 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t
>> addr,
>>       /* Ignore */
>>   }
>>   +/*
>> + * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the
>> value
>> + * of the enabled bit, so there is only one function for both here.
>> + */
>> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
>> +                    paddr_t addr, unsigned int len)
> 
> Indentation.
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> 
> uint32_t here please.
> 
>> +    u32 value = 0;
> 
> Same here.
> 
>> +    int i;
>> +
>> +    /* Loop over all IRQs affected by this read */
>> +    for ( i = 0; i < len * 8; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        if ( irq->enabled )
>> +            value |= (1U << i);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return value;
>> +}
>> +
>> +static void vgic_handle_hardware_irq(irq_desc_t *desc, int irq_type,
> 
> Looking below irq_type should a enum vgic_irq_config and not an int.
> 
>> +                     bool enable)
> 
> Indentation.
> 
>> +{
>> +    unsigned long flags;
>> +
>> +//  irq_set_affinity(desc, cpumask_of(v_target->processor));
> 
> Why is that commented?
> 
>> +    spin_lock_irqsave(&desc->lock, flags);
>> +    if ( enable )
>> +    {
>> +        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
>> +                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);
> 
> Indentation and I would prefer a helper to convert between the vgic
> value and the IRQ_TYPE. This would make the code easier to read.
> 
> Also, this code does not replicate correctly the current vGIC.
> gic_set_irq_type is only allowed to be used when
> irq_set_type_by_domain(d) returns true. If you consider this change
> valid, then I would like to know why.
> 
>> +        desc->handler->enable(desc);
>> +    }
>> +    else
>> +        desc->handler->disable(desc);
>> +    spin_unlock_irqrestore(&desc->lock, flags);
>> +}
>> +
>> +void vgic_mmio_write_senable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val)
> 
> Indentation.
> 
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
> 
> uint32_t.
> 
>> +    irq_desc_t *desc;
>> +    int i;
>> +    unsigned long flags;
>> +    enum vgic_irq_config config;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq;
>> +
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +        irq->enabled = true;
>> +        if ( irq->hw )
>> +        {
>> +            /*
>> +             * The irq cannot be a PPI, we only support delivery
>> +             * of SPIs to guests.
>> +             */
>> +            ASSERT(irq->hwintid >= 32);
>> +
>> +            desc = irq_to_desc(irq->hwintid);
> 
> What is the rationale behind storing hwintid rather than the irq_desc
> directly?

Well, this is because KVM does it this way, for abstraction reasons,
mostly. Looking over the users I see that mostly we are indeed after the
struct irq_desc. But it would also increase struct vgic_irq by 4 bytes ;-)

I could try to make to make the change, but am not fully convinced.

What are your arguments for that change?

Cheers,
Andre.

>> +            config = irq->config;
>> +        }
>> +        else
>> +            desc = NULL;
>> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +
>> +        if ( desc )
>> +            vgic_handle_hardware_irq(desc, config, true);
> 
> This is slightly strange. You handle the hardware IRQ outside the
> virtual IRQ lock. It means that the hardware IRQ may end up enabled but
> the virtual IRQ disabled.
> 
>> +    }
>> +}
>> +
>> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val)
>> +{
>> +    u32 intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    int i;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq;
>> +        unsigned long flags;
>> +        irq_desc_t *desc;
>> +
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->enabled = false;
>> +
>> +        if ( irq->hw )
>> +            desc = irq_to_desc(irq->hwintid);
>> +        else
>> +            desc = NULL;
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +
>> +        if ( desc )
>> +            vgic_handle_hardware_irq(desc, 0, false);
> 
> Same remark here.
> 
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 10ac682296..9f34bd1aec 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -137,6 +137,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>>                           unsigned int len, unsigned long val);
>>   +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
>> +                    paddr_t addr, unsigned int len);
> 
> Indentation.
> 
>> +
>> +void vgic_mmio_write_senable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val);
> 
> Ditto.
> 
>> +
>> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val);
> 
> Ditto.
> 
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     /* Find the proper register handler entry given a certain address
>> offset */
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Andre Przywara]

Hi,

On 19/02/18 12:19, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
>> when a guest EOIs the virtual interrupt, it affects the state of that
>> corresponding interrupt on the hardware side at the same time.
>> Implement the interface that the Xen arch/core code expects to connect
>> the virtual and the physical world.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 63
>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>   1 file changed, 63 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index dc5e011fa3..8d5260a7db 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>>       }
>>   }
>>   +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu
>> *v,
>> +                                      unsigned int virq)
>> +{
>> +    struct irq_desc *desc = NULL;
>> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
>> +    unsigned long flags;
>> +
>> +    if ( !irq )
>> +        return NULL;
>> +
>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>> +    if ( irq->hw )
>> +        desc = irq_to_desc(irq->hwintid);
> 
> This is not going to work well for PPIs. We should consider to add at
> least an ASSERT(...) in the code to prevent bad use of it.

Yeah, done. But I wonder if we eventually should extend the
irq_to_desc() function to take the vCPU, since we will need it anyway
once we use hardware mapped timer IRQs (PPIs) in the future. But this
should not be in this series, I guess.

>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +    vgic_put_irq(d, irq);
>> +
>> +    return desc;
>> +}
>> +
>> +/*
>> + * was:
>> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq,
>> u32 phys_irq)
>> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int
>> virt_irq)
>> + */
>> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
>> +            unsigned int virt_irq, struct irq_desc *desc,
>> +            bool connect)
> 
> Indentation.
> 
>> +{
>> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
>> +    unsigned long flags;
>> +    int ret = 0;
>> +
>> +    if ( !irq )
>> +        return -EINVAL;
>> +
>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +    if ( connect )                      /* assign a mapped IRQ */
>> +    {
>> +        /* The VIRQ should not be already enabled by the guest */
>> +        if ( !irq->hw && !irq->enabled )
>> +        {
>> +            irq->hw = true;
>> +            irq->hwintid = desc->irq;
>> +        }
>> +        else
>> +        {
>> +            ret = -EBUSY;
>> +        }
> 
> I know that it should not matter for SPIs today. But aren't you meant to
> get a reference on that interrupt if you connect it?

No, the refcount feature is strictly for the pointer to the structure,
not for everything related to this virtual IRQ.
We store only the virtual IRQ number in the dev_id/info members, we will
get the struct vgic_irq pointer via the vIRQ number on do_IRQ().
Does that make sense?

>> +    }
>> +    else                                /* remove a mapped IRQ */
>> +    {
>> +        irq->hw = false;
>> +        irq->hwintid = 0;
> 
> Here you blindly remove the interrupt without been sure it was the
> correct physical one. We should have a check like in the current vGIC
> version.

Fixed.

Cheers,
Andre.

>> +    }
>> +
>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +    vgic_put_irq(d, irq);
>> +
>> +    return ret;
>> +}
>> +
>>   /*
>>    * Local variables:
>>    * mode: C
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Julien Grall]

On 23/02/18 18:02, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 19/02/18 12:19, Julien Grall wrote:
>> Hi,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
>>> when a guest EOIs the virtual interrupt, it affects the state of that
>>> corresponding interrupt on the hardware side at the same time.
>>> Implement the interface that the Xen arch/core code expects to connect
>>> the virtual and the physical world.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic.c | 63
>>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>>    1 file changed, 63 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> index dc5e011fa3..8d5260a7db 100644
>>> --- a/xen/arch/arm/vgic/vgic.c
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>>>        }
>>>    }
>>>    +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu
>>> *v,
>>> +                                      unsigned int virq)
>>> +{
>>> +    struct irq_desc *desc = NULL;
>>> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
>>> +    unsigned long flags;
>>> +
>>> +    if ( !irq )
>>> +        return NULL;
>>> +
>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>> +    if ( irq->hw )
>>> +        desc = irq_to_desc(irq->hwintid);
>>
>> This is not going to work well for PPIs. We should consider to add at
>> least an ASSERT(...) in the code to prevent bad use of it.
> 
> Yeah, done. But I wonder if we eventually should extend the
> irq_to_desc() function to take the vCPU, since we will need it anyway
> once we use hardware mapped timer IRQs (PPIs) in the future. But this
> should not be in this series, I guess.

irq_to_desc only deal with hardware interrupt, so you mean pCPU instead 
of vCPU?

> 
>>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>>> +
>>> +    vgic_put_irq(d, irq);
>>> +
>>> +    return desc;
>>> +}
>>> +
>>> +/*
>>> + * was:
>>> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq,
>>> u32 phys_irq)
>>> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int
>>> virt_irq)
>>> + */
>>> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
>>> +            unsigned int virt_irq, struct irq_desc *desc,
>>> +            bool connect)
>>
>> Indentation.
>>
>>> +{
>>> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
>>> +    unsigned long flags;
>>> +    int ret = 0;
>>> +
>>> +    if ( !irq )
>>> +        return -EINVAL;
>>> +
>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>> +
>>> +    if ( connect )                      /* assign a mapped IRQ */
>>> +    {
>>> +        /* The VIRQ should not be already enabled by the guest */
>>> +        if ( !irq->hw && !irq->enabled )
>>> +        {
>>> +            irq->hw = true;
>>> +            irq->hwintid = desc->irq;
>>> +        }
>>> +        else
>>> +        {
>>> +            ret = -EBUSY;
>>> +        }
>>
>> I know that it should not matter for SPIs today. But aren't you meant to
>> get a reference on that interrupt if you connect it?
> 
> No, the refcount feature is strictly for the pointer to the structure,
> not for everything related to this virtual IRQ.
> We store only the virtual IRQ number in the dev_id/info members, we will
> get the struct vgic_irq pointer via the vIRQ number on do_IRQ().
> Does that make sense?

But technically you "allocate" the virtual SPI at that time, right? So 
this would mean you need to get a reference, otherwise it might disappear.

So I am not entirely sure why the reference is not necessary here.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Julien Grall]

Hi Andre,

On 23/02/18 15:18, Andre Przywara wrote:
>>> +    irq_desc_t *desc;
>>> +    int i;
>>> +    unsigned long flags;
>>> +    enum vgic_irq_config config;
>>> +
>>> +    for_each_set_bit( i, &val, len * 8 )
>>> +    {
>>> +        struct vgic_irq *irq;
>>> +
>>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
>>> +
>>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>>> +        irq->enabled = true;
>>> +        if ( irq->hw )
>>> +        {
>>> +            /*
>>> +             * The irq cannot be a PPI, we only support delivery
>>> +             * of SPIs to guests.
>>> +             */
>>> +            ASSERT(irq->hwintid >= 32);
>>> +
>>> +            desc = irq_to_desc(irq->hwintid);
>>
>> What is the rationale behind storing hwintid rather than the irq_desc
>> directly?
> 
> Well, this is because KVM does it this way, for abstraction reasons,
> mostly. Looking over the users I see that mostly we are indeed after the
> struct irq_desc. But it would also increase struct vgic_irq by 4 bytes ;-)
> 
> I could try to make to make the change, but am not fully convinced.
> 
> What are your arguments for that change?

To be honest, I don't have much arguments :). My main concern is using 
irq_to_desc wrongly when we will add support for routing PPI to domains.
This would useful to support the virtual timer without the hack we 
currently have.

In the PPI context, irq_to_desc would always return the PPI irq_desc of 
the current CPU. I am not entirely if this will always be ok for us. But 
I might be over cautious :).

So I guess, we can keep it like that for now.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Hi,

On 13/02/18 14:31, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Processing maintenance interrupts and accessing the list registers
>> are dependent on the host's GIC version.
>> Introduce vgic-v2.c to contain GICv2 specific functions.
>> Implement the GICv2 specific code for syncing the emulation state
>> into the VGIC registers.
>> This also adds the hook to let Xen setup the host GIC addresses.
>>
>> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-v2.c | 261
>> ++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.c    |  20 ++++
>>   xen/arch/arm/vgic/vgic.h    |   8 ++
>>   3 files changed, 289 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>>
>> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
>> new file mode 100644
>> index 0000000000..10fc467ffa
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic-v2.c
>> @@ -0,0 +1,261 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/arm_vgic.h>
>> +#include <asm/bug.h>
>> +#include <asm/io.h>
>> +#include <xen/sched.h>
>> +#include <xen/sizes.h>
>> +
>> +#include "vgic.h"
>> +
>> +#define GICH_ELRSR0                     0x30
>> +#define GICH_ELRSR1                     0x34
>> +#define GICH_LR0                        0x100
>> +
>> +#define GICH_LR_VIRTUALID               (0x3ff << 0)
>> +#define GICH_LR_PHYSID_CPUID_SHIFT      (10)
>> +#define GICH_LR_PHYSID_CPUID            (0x3ff <<
>> GICH_LR_PHYSID_CPUID_SHIFT)
>> +#define GICH_LR_PRIORITY_SHIFT          23
>> +#define GICH_LR_STATE                   (3 << 28)
>> +#define GICH_LR_PENDING_BIT             (1 << 28)
>> +#define GICH_LR_ACTIVE_BIT              (1 << 29)
>> +#define GICH_LR_EOI                     (1 << 19)
>> +#define GICH_LR_HW                      (1 << 31)
> 
> Can we define them in either in gic.h or a new header gic-v2.h?

Yes, but they clash with some ill-named GICv3 LR bits. So expect another
patch which renames GICH_LR_STATE_SHIFT to ICH_LR_STATE_SHIFT. Which is
the actual spec name for that system register in GICv3, there is no
GICH_LR_ with the GICv3 bit positions.


>> +
>> +static struct {
>> +    bool enabled;
>> +    paddr_t dbase;          /* Distributor interface address */
>> +    paddr_t cbase;          /* CPU interface address & size */
>> +    paddr_t csize;
>> +    paddr_t vbase;          /* Virtual CPU interface address */
>> +    void __iomem *hbase;        /* Hypervisor control interface */
>> +
>> +    /* Offset to add to get an 8kB contiguous region if GIC is
>> aliased */
>> +    uint32_t aliased_offset;
>> +} gic_v2_hw_data;
>> +
>> +void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
>> +              paddr_t vbase, void __iomem *hbase,
>> +              uint32_t aliased_offset)
>> +{
>> +    gic_v2_hw_data.enabled = true;
>> +    gic_v2_hw_data.dbase = dbase;
>> +    gic_v2_hw_data.cbase = cbase;
>> +    gic_v2_hw_data.csize = csize;
>> +    gic_v2_hw_data.vbase = vbase;
>> +    gic_v2_hw_data.hbase = hbase;
>> +    gic_v2_hw_data.aliased_offset = aliased_offset;
>> +}
>> +
>> +void vgic_v2_set_underflow(struct vcpu *vcpu)
>> +{
>> +    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
>> +}
>> +
>> +/*
>> + * transfer the content of the LRs back into the corresponding ap_list:
>> + * - active bit is transferred as is
>> + * - pending bit is
>> + *   - transferred as is in case of edge sensitive IRQs
>> + *   - set to the line-level (resample time) for level sensitive IRQs
>> + */
>> +void vgic_v2_fold_lr_state(struct vcpu *vcpu)
> 
> I am wondering how much we could share this code with
> vgic_v3_fold_lr_state.

I think we discussed this and dismissed the idea:
- The actual LR encoding is much different between GICv3 and GICv2, up
to the point where we have some fields in one which are not in the
other. That really clutters the code.
- Originally this function was much shorter and didn't have that many
special cases. So the code duplication was really minimal.

I see your point, but don't really want to go there now for two reasons:
- It is probably nasty to implement, since we always have to check which
GIC we are running on when masking the LR value.
- It would deviate further from the KVM implementation, in a core
function. For any bugs introduced we are on our own here.

I will try to bring this up with the KVM people, to see whether it's
worth to revisit this decision. There is indeed quite some code
duplication these days.
But this may come as an optimization later.

>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_v2_cpu_if *cpuif = &vgic_cpu->vgic_v2;
>> +    int lr;
> 
> unsigned please.
> 
>> +    unsigned long flags;
>> +
>> +    cpuif->vgic_hcr &= ~GICH_HCR_UIE;
>> +
>> +    for ( lr = 0; lr < vgic_cpu->used_lrs; lr++ )
>> +    {
>> +        u32 val = cpuif->vgic_lr[lr];
>> +        u32 intid = val & GICH_LR_VIRTUALID;
>> +        struct vgic_irq *irq;
>> +
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        /* Always preserve the active bit */
>> +        irq->active = !!(val & GICH_LR_ACTIVE_BIT);
>> +
>> +        /* Edge is the only case where we preserve the pending bit */
>> +        if ( irq->config == VGIC_CONFIG_EDGE && (val &
>> GICH_LR_PENDING_BIT) )
>> +        {
>> +            irq->pending_latch = true;
>> +
>> +            if ( vgic_irq_is_sgi(intid) )
>> +            {
>> +                u32 cpuid = val & GICH_LR_PHYSID_CPUID;
>> +
>> +                cpuid >>= GICH_LR_PHYSID_CPUID_SHIFT;
>> +                irq->source |= (1 << cpuid);
>> +            }
>> +        }
>> +
> 
> May I ask to keep the big comments from KVM around? It looks quite
> useful to have it.

Indeed I somehow lost that. Added now.

>> +        if ( irq->hw && irq->config == VGIC_CONFIG_LEVEL &&
> 
> You probably want to have the helper vgic_irq_is_mapped_level(...) as in
> KVM.

Yes.

Cheers,
Andre

>> +            (val & GICH_LR_PENDING_BIT) )
>> +        {
>> +            irq->line_level = gic_read_pending_state(irq->hwintid);
>> +
>> +            if ( !irq->line_level )
>> +                            gic_set_active_state(irq->hwintid, true);
>> +        }
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    vgic_cpu->used_lrs = 0;
>> +}
>> +
>> +/*
>> + * Populates the particular LR with the state of a given IRQ:
>> + * - for an edge sensitive IRQ the pending state is cleared in struct
>> vgic_irq
>> + * - for a level sensitive IRQ the pending state value is unchanged;
>> + *   it is dictated directly by the input level
>> + *
>> + * If @irq describes an SGI with multiple sources, we choose the
>> + * lowest-numbered source VCPU and clear that bit in the source bitmap.
>> + *
>> + * The irq_lock must be held by the caller.
>> + */
>> +void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int
>> lr)
> 
> I am wondering how much we could share this code with vgic_v3_populate_lr.
> 
>> +{
>> +    u32 val = irq->intid;
>> +
>> +    if ( irq_is_pending(irq) )
>> +    {
>> +        val |= GICH_LR_PENDING_BIT;
>> +
>> +        if ( irq->config == VGIC_CONFIG_EDGE )
>> +            irq->pending_latch = false;
>> +
>> +        if ( vgic_irq_is_sgi(irq->intid) )
>> +        {
>> +            u32 src = ffs(irq->source);
>> +
>> +            BUG_ON(!src);
>> +            val |= (src - 1) << GICH_LR_PHYSID_CPUID_SHIFT;
>> +            irq->source &= ~(1 << (src - 1));
>> +            if ( irq->source )
>> +                irq->pending_latch = true;
>> +        }
>> +    }
>> +
>> +    if ( irq->active )
>> +        val |= GICH_LR_ACTIVE_BIT;
>> +
>> +    if ( irq->hw )
>> +    {
>> +        val |= GICH_LR_HW;
>> +        val |= irq->hwintid << GICH_LR_PHYSID_CPUID_SHIFT;
>> +        /*
>> +         * Never set pending+active on a HW interrupt, as the
>> +         * pending state is kept at the physical distributor
>> +         * level.
>> +         */
>> +        if ( irq->active && irq_is_pending(irq) )
>> +            val &= ~GICH_LR_PENDING_BIT;
>> +    }
>> +    else
>> +    {
>> +        if ( irq->config == VGIC_CONFIG_LEVEL )
>> +            val |= GICH_LR_EOI;
>> +    }
>> +
>> +    /*
>> +     * Level-triggered mapped IRQs are special because we only observe
>> +     * rising edges as input to the VGIC.  We therefore lower the line
>> +     * level here, so that we can take new virtual IRQs.  See
>> +     * vgic_v2_fold_lr_state for more info.
>> +     */
>> +    if ( irq->hw && irq->config == VGIC_CONFIG_LEVEL &&
> 
> Same remark for the helper.
> 
>> +        (val & GICH_LR_PENDING_BIT) )
>> +        irq->line_level = false;
>> +
>> +    /* The GICv2 LR only holds five bits of priority. */
>> +    val |= (irq->priority >> 3) << GICH_LR_PRIORITY_SHIFT;
>> +
>> +    vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = val;
>> +}
>> +
>> +void vgic_v2_clear_lr(struct vcpu *vcpu, int lr)
>> +{
>> +    vcpu->arch.vgic_cpu.vgic_v2.vgic_lr[lr] = 0;
>> +}
>> +
>> +static void save_lrs(struct vcpu *vcpu, void __iomem *base)
>> +{
>> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>> +    u64 elrsr;
>> +    int i;
>> +
>> +    elrsr = readl_relaxed(base + GICH_ELRSR0);
>> +    if ( unlikely(used_lrs > 32) )
>> +        elrsr |= ((u64)readl_relaxed(base + GICH_ELRSR1)) << 32;
>> +
>> +    for ( i = 0; i < used_lrs; i++ )
>> +    {
>> +        if ( elrsr & (1UL << i) )
>> +            cpu_if->vgic_lr[i] &= ~GICH_LR_STATE;
>> +        else
>> +            cpu_if->vgic_lr[i] = readl_relaxed(base + GICH_LR0 + (i *
>> 4));
>> +
>> +        writel_relaxed(0, base + GICH_LR0 + (i * 4));
>> +    }
>> +}
>> +
>> +void vgic_v2_save_state(struct vcpu *vcpu)
>> +{
>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>> +
>> +    if ( used_lrs )
>> +    {
>> +        save_lrs(vcpu, gic_v2_hw_data.hbase);
>> +        writel_relaxed(0, gic_v2_hw_data.hbase + GICH_HCR);
>> +    }
>> +}
> 
> I am not entirely convinced that have a separate function to save the
> LRs is necessary. This could be done in fold_lr_state().

>> +
>> +void vgic_v2_restore_state(struct vcpu *vcpu)
>> +{
>> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>> +    int i;
>> +
>> +    if ( used_lrs )
>> +    {
>> +        writel_relaxed(cpu_if->vgic_hcr,
>> +                       gic_v2_hw_data.hbase + GICH_HCR);
>> +        for ( i = 0; i < used_lrs; i++ )
>> +            writel_relaxed(cpu_if->vgic_lr[i],
>> +                           gic_v2_hw_data.hbase + GICH_LR0 + (i * 4));
>> +    }
> 
> Same here but with populate_lr_state(). This would make the code easier
> to follow and also avoid a lot ifery in the vgic.c code.
> 
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index a1f77130d4..f4f2a04a60 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -488,6 +488,7 @@ retry:
>>     static inline void vgic_fold_lr_state(struct vcpu *vcpu)
>>   {
>> +    vgic_v2_fold_lr_state(vcpu);
>>   }
>>     /* Requires the irq_lock to be held. */
>> @@ -495,14 +496,18 @@ static inline void vgic_populate_lr(struct vcpu
>> *vcpu,
>>                                       struct vgic_irq *irq, int lr)
>>   {
>>       ASSERT(spin_is_locked(&irq->irq_lock));
>> +
>> +    vgic_v2_populate_lr(vcpu, irq, lr);
>>   }
>>     static inline void vgic_clear_lr(struct vcpu *vcpu, int lr)
>>   {
>> +    vgic_v2_clear_lr(vcpu, lr);
>>   }
>>     static inline void vgic_set_underflow(struct vcpu *vcpu)
>>   {
>> +    vgic_v2_set_underflow(vcpu);
>>   }
>>     /* Requires the ap_list_lock to be held. */
>> @@ -573,6 +578,11 @@ next:
>>           vgic_clear_lr(vcpu, count);
>>   }
>>   +static inline void vgic_save_state(struct vcpu *vcpu)
>> +{
>> +    vgic_v2_save_state(vcpu);
>> +}
>> +
>>   /*
>>    * gic_clear_lrs() - Update the VGIC state from hardware after a
>> guest's run.
>>    * @vcpu: the VCPU.
>> @@ -592,11 +602,18 @@ void gic_clear_lrs(struct vcpu *vcpu)
>>       if ( list_empty(&vcpu->arch.vgic_cpu.ap_list_head) )
>>           return;
>>   +    vgic_save_state(vcpu);
>> +
>>       if ( vgic_cpu->used_lrs )
>>           vgic_fold_lr_state(vcpu);
>>       vgic_prune_ap_list(vcpu);
>>   }
>>   +static inline void vgic_restore_state(struct vcpu *vcpu)
>> +{
>> +    vgic_v2_restore_state(vcpu);
>> +}
>> +
>>   /*
>>    * gic_inject() - flush the emulation state into the hardware on
>> guest entry
>>    *
>> @@ -625,7 +642,10 @@ void gic_inject(void)
>>       spin_lock(&current->arch.vgic_cpu.ap_list_lock);
>>       vgic_flush_lr_state(current);
>>       spin_unlock(&current->arch.vgic_cpu.ap_list_lock);
>> +
>> +    vgic_restore_state(current);
>>   }
>> +
>>   /*
>>    * Local variables:
>>    * mode: C
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 47fc58b81e..771ca6f046 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -41,4 +41,12 @@ static inline void vgic_get_irq_kref(struct
>> vgic_irq *irq)
>>       atomic_inc(&irq->refcount);
>>   }
>>   +void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>> +void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int
>> lr);
>> +void vgic_v2_clear_lr(struct vcpu *vcpu, int lr);
>> +void vgic_v2_set_underflow(struct vcpu *vcpu);
>> +
>> +void vgic_v2_save_state(struct vcpu *vcpu);
>> +void vgic_v2_restore_state(struct vcpu *vcpu);
>> +
>>   #endif
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Hi,

forgot to mention:

On 13/02/18 14:31, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Processing maintenance interrupts and accessing the list registers
>> are dependent on the host's GIC version.
>> Introduce vgic-v2.c to contain GICv2 specific functions.
>> Implement the GICv2 specific code for syncing the emulation state
>> into the VGIC registers.
>> This also adds the hook to let Xen setup the host GIC addresses.
>>
>> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-v2.c | 261
>> ++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.c    |  20 ++++
>>   xen/arch/arm/vgic/vgic.h    |   8 ++
>>   3 files changed, 289 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>>
>> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
>> new file mode 100644
>> index 0000000000..10fc467ffa
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic-v2.c

....

>> +void vgic_v2_save_state(struct vcpu *vcpu)
>> +{
>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>> +
>> +    if ( used_lrs )
>> +    {
>> +        save_lrs(vcpu, gic_v2_hw_data.hbase);
>> +        writel_relaxed(0, gic_v2_hw_data.hbase + GICH_HCR);
>> +    }
>> +}
> 
> I am not entirely convinced that have a separate function to save the
> LRs is necessary. This could be done in fold_lr_state().
> 
>> +
>> +void vgic_v2_restore_state(struct vcpu *vcpu)
>> +{
>> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>> +    int i;
>> +
>> +    if ( used_lrs )
>> +    {
>> +        writel_relaxed(cpu_if->vgic_hcr,
>> +                       gic_v2_hw_data.hbase + GICH_HCR);
>> +        for ( i = 0; i < used_lrs; i++ )
>> +            writel_relaxed(cpu_if->vgic_lr[i],
>> +                           gic_v2_hw_data.hbase + GICH_LR0 + (i * 4));
>> +    }
> 
> Same here but with populate_lr_state(). This would make the code easier
> to follow and also avoid a lot ifery in the vgic.c code.

This is mostly due to KVM's inability to directly access the GICv3 LRs
when running in EL1. I will take a look whether what it would take to
merge this. Sounds tempting, but there might be side effects.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Andre Przywara]

Hi,

On 13/02/18 16:35, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Tell Xen whether a particular VCPU has an IRQ that needs handling
>> in the guest. This is used to decide whether a VCPU is runnable.
>>
>> This is based on Linux commit 90eee56c5f90, written by Eric Auger.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 32 ++++++++++++++++++++++++++++++++
>>   1 file changed, 32 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index f4f2a04a60..9e7fb1edcb 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>       vgic_restore_state(current);
>>   }
>>   +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>> +    struct vgic_irq *irq;
>> +    bool pending = false;
>> +    unsigned long flags;
>> +
>> +    if ( !vcpu->domain->arch.vgic.enabled )
>> +        return false;
>> +
>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>> +
>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>> +    {
>> +        spin_lock(&irq->irq_lock);
>> +        pending = irq_is_pending(irq) && irq->enabled;
>> +        spin_unlock(&irq->irq_lock);
>> +
>> +        if ( pending )
>> +            break;
>> +    }
>> +
>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>> +
>> +    return pending;
>> +}
>> +
>> +int gic_events_need_delivery(void)
> 
> You probably want to rename that function or just expose
> vgic_vcpu_pending_irq().

Rename to what? I need both functions: vgic_vcpu_pending_irq() is also
called by vgic_kick_vcpus() (later in the series).
And gic_events_need_delivery(void) is the interface that the arch code
expects. Shall I rename this there? To what?

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Julien Grall]

Hi,

On 02/26/2018 03:29 PM, Andre Przywara wrote:
> On 13/02/18 16:35, Julien Grall wrote:
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> index f4f2a04a60..9e7fb1edcb 100644
>>> --- a/xen/arch/arm/vgic/vgic.c
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>>        vgic_restore_state(current);
>>>    }
>>>    +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>>> +{
>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>> +    struct vgic_irq *irq;
>>> +    bool pending = false;
>>> +    unsigned long flags;
>>> +
>>> +    if ( !vcpu->domain->arch.vgic.enabled )
>>> +        return false;
>>> +
>>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>>> +
>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>> +    {
>>> +        spin_lock(&irq->irq_lock);
>>> +        pending = irq_is_pending(irq) && irq->enabled;
>>> +        spin_unlock(&irq->irq_lock);
>>> +
>>> +        if ( pending )
>>> +            break;
>>> +    }
>>> +
>>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>>> +
>>> +    return pending;
>>> +}
>>> +
>>> +int gic_events_need_delivery(void)
>>
>> You probably want to rename that function or just expose
>> vgic_vcpu_pending_irq().
> 
> Rename to what? I need both functions: vgic_vcpu_pending_irq() is also
> called by vgic_kick_vcpus() (later in the series).
> And gic_events_need_delivery(void) is the interface that the arch code
> expects. Shall I rename this there? To what?

Let me start with it is a bit odd to have a function name 'gic_*' in the 
virtual GIC code. So at least renaming to vgic_events_need_delivery 
would be an improvement.

Regarding the interface itself, it is ARM specific and not set in stone. 
It would not be too bad to use vgic_vcpu_pending_irq(current). Is there 
any reason for not doing that?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Julien Grall]

On 02/26/2018 03:16 PM, Andre Przywara wrote:
> Hi,

Hi,

> forgot to mention:
> 
> On 13/02/18 14:31, Julien Grall wrote:
>> Hi,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> Processing maintenance interrupts and accessing the list registers
>>> are dependent on the host's GIC version.
>>> Introduce vgic-v2.c to contain GICv2 specific functions.
>>> Implement the GICv2 specific code for syncing the emulation state
>>> into the VGIC registers.
>>> This also adds the hook to let Xen setup the host GIC addresses.
>>>
>>> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic-v2.c | 261
>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/vgic/vgic.c    |  20 ++++
>>>    xen/arch/arm/vgic/vgic.h    |   8 ++
>>>    3 files changed, 289 insertions(+)
>>>    create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
>>> new file mode 100644
>>> index 0000000000..10fc467ffa
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vgic/vgic-v2.c
> 
> ....
> 
>>> +void vgic_v2_save_state(struct vcpu *vcpu)
>>> +{
>>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>>> +
>>> +    if ( used_lrs )
>>> +    {
>>> +        save_lrs(vcpu, gic_v2_hw_data.hbase);
>>> +        writel_relaxed(0, gic_v2_hw_data.hbase + GICH_HCR);
>>> +    }
>>> +}
>>
>> I am not entirely convinced that have a separate function to save the
>> LRs is necessary. This could be done in fold_lr_state().
>>
>>> +
>>> +void vgic_v2_restore_state(struct vcpu *vcpu)
>>> +{
>>> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;,
>>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>>> +    int i;
>>> +
>>> +    if ( used_lrs )
>>> +    {
>>> +        writel_relaxed(cpu_if->vgic_hcr,
>>> +                       gic_v2_hw_data.hbase + GICH_HCR);
>>> +        for ( i = 0; i < used_lrs; i++ )
>>> +            writel_relaxed(cpu_if->vgic_lr[i],
>>> +                           gic_v2_hw_data.hbase + GICH_LR0 + (i * 4));
>>> +    }
>>
>> Same here but with populate_lr_state(). This would make the code easier
>> to follow and also avoid a lot ifery in the vgic.c code.
> 
> This is mostly due to KVM's inability to directly access the GICv3 LRs
> when running in EL1. I will take a look whether what it would take to
> merge this. Sounds tempting, but there might be side effects.

I am not sure what would be the side effects. You basically
call save_state and right after fold_lr_state. This would streamline a 
bit more the code.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Julien Grall]

Hi Andre,

On 02/26/2018 03:13 PM, Andre Przywara wrote:
> Hi,
> 
> On 13/02/18 14:31, Julien Grall wrote:
>> Hi,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> Processing maintenance interrupts and accessing the list registers
>>> are dependent on the host's GIC version.
>>> Introduce vgic-v2.c to contain GICv2 specific functions.
>>> Implement the GICv2 specific code for syncing the emulation state
>>> into the VGIC registers.
>>> This also adds the hook to let Xen setup the host GIC addresses.
>>>
>>> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic-v2.c | 261
>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/vgic/vgic.c    |  20 ++++
>>>    xen/arch/arm/vgic/vgic.h    |   8 ++
>>>    3 files changed, 289 insertions(+)
>>>    create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
>>> new file mode 100644
>>> index 0000000000..10fc467ffa
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vgic/vgic-v2.c
>>> @@ -0,0 +1,261 @@
>>> +/*
>>> + * Copyright (C) 2015, 2016 ARM Ltd.
>>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <asm/arm_vgic.h>
>>> +#include <asm/bug.h>
>>> +#include <asm/io.h>
>>> +#include <xen/sched.h>
>>> +#include <xen/sizes.h>
>>> +
>>> +#include "vgic.h"
>>> +
>>> +#define GICH_ELRSR0                     0x30
>>> +#define GICH_ELRSR1                     0x34
>>> +#define GICH_LR0                        0x100
>>> +
>>> +#define GICH_LR_VIRTUALID               (0x3ff << 0)
>>> +#define GICH_LR_PHYSID_CPUID_SHIFT      (10)
>>> +#define GICH_LR_PHYSID_CPUID            (0x3ff <<
>>> GICH_LR_PHYSID_CPUID_SHIFT)
>>> +#define GICH_LR_PRIORITY_SHIFT          23
>>> +#define GICH_LR_STATE                   (3 << 28)
>>> +#define GICH_LR_PENDING_BIT             (1 << 28)
>>> +#define GICH_LR_ACTIVE_BIT              (1 << 29)
>>> +#define GICH_LR_EOI                     (1 << 19)
>>> +#define GICH_LR_HW                      (1 << 31)
>>
>> Can we define them in either in gic.h or a new header gic-v2.h?
> 
> Yes, but they clash with some ill-named GICv3 LR bits. So expect another
> patch which renames GICH_LR_STATE_SHIFT to ICH_LR_STATE_SHIFT. Which is
> the actual spec name for that system register in GICv3, there is no
> GICH_LR_ with the GICv3 bit positions.

While this would be a nice clean-up. Wouldn't create a new gic-v2.h 
sufficient?

> 
> 
>>> +
>>> +static struct {
>>> +    bool enabled;
>>> +    paddr_t dbase;          /* Distributor interface address */
>>> +    paddr_t cbase;          /* CPU interface address & size */
>>> +    paddr_t csize;
>>> +    paddr_t vbase;          /* Virtual CPU interface address */
>>> +    void __iomem *hbase;        /* Hypervisor control interface */
>>> +
>>> +    /* Offset to add to get an 8kB contiguous region if GIC is
>>> aliased */
>>> +    uint32_t aliased_offset;
>>> +} gic_v2_hw_data;
>>> +
>>> +void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
>>> +              paddr_t vbase, void __iomem *hbase,
>>> +              uint32_t aliased_offset)
>>> +{
>>> +    gic_v2_hw_data.enabled = true;
>>> +    gic_v2_hw_data.dbase = dbase;
>>> +    gic_v2_hw_data.cbase = cbase;
>>> +    gic_v2_hw_data.csize = csize;
>>> +    gic_v2_hw_data.vbase = vbase;
>>> +    gic_v2_hw_data.hbase = hbase;
>>> +    gic_v2_hw_data.aliased_offset = aliased_offset;
>>> +}
>>> +
>>> +void vgic_v2_set_underflow(struct vcpu *vcpu)
>>> +{
>>> +    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
>>> +}
>>> +
>>> +/*
>>> + * transfer the content of the LRs back into the corresponding ap_list:
>>> + * - active bit is transferred as is
>>> + * - pending bit is
>>> + *   - transferred as is in case of edge sensitive IRQs
>>> + *   - set to the line-level (resample time) for level sensitive IRQs
>>> + */
>>> +void vgic_v2_fold_lr_state(struct vcpu *vcpu)
>>
>> I am wondering how much we could share this code with
>> vgic_v3_fold_lr_state.
> 
> I think we discussed this and dismissed the idea:
> - The actual LR encoding is much different between GICv3 and GICv2, up
> to the point where we have some fields in one which are not in the
> other. That really clutters the code.
> - Originally this function was much shorter and didn't have that many
> special cases. So the code duplication was really minimal.
> 
> I see your point, but don't really want to go there now for two reasons:
> - It is probably nasty to implement, since we always have to check which
> GIC we are running on when masking the LR value.
> - It would deviate further from the KVM implementation, in a core
> function. For any bugs introduced we are on our own here.
> 
> I will try to bring this up with the KVM people, to see whether it's
> worth to revisit this decision. There is indeed quite some code
> duplication these days.
> But this may come as an optimization later.

Fine with me. It was mostly to avoid having to review twice the same 
hairy code.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Hi,

On 26/02/18 16:02, Julien Grall wrote:
> Hi Andre,
> 
> On 02/26/2018 03:13 PM, Andre Przywara wrote:
>> Hi,
>>
>> On 13/02/18 14:31, Julien Grall wrote:
>>> Hi,
>>>
>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>> Processing maintenance interrupts and accessing the list registers
>>>> are dependent on the host's GIC version.
>>>> Introduce vgic-v2.c to contain GICv2 specific functions.
>>>> Implement the GICv2 specific code for syncing the emulation state
>>>> into the VGIC registers.
>>>> This also adds the hook to let Xen setup the host GIC addresses.
>>>>
>>>> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>> ---
>>>>    xen/arch/arm/vgic/vgic-v2.c | 261
>>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>>    xen/arch/arm/vgic/vgic.c    |  20 ++++
>>>>    xen/arch/arm/vgic/vgic.h    |   8 ++
>>>>    3 files changed, 289 insertions(+)
>>>>    create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>>>>
>>>> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
>>>> new file mode 100644
>>>> index 0000000000..10fc467ffa
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vgic/vgic-v2.c
>>>> @@ -0,0 +1,261 @@
>>>> +/*
>>>> + * Copyright (C) 2015, 2016 ARM Ltd.
>>>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or
>>>> modify
>>>> + * it under the terms of the GNU General Public License version 2 as
>>>> + * published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>> + * GNU General Public License for more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License
>>>> + * along with this program.  If not, see
>>>> <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <asm/arm_vgic.h>
>>>> +#include <asm/bug.h>
>>>> +#include <asm/io.h>
>>>> +#include <xen/sched.h>
>>>> +#include <xen/sizes.h>
>>>> +
>>>> +#include "vgic.h"
>>>> +
>>>> +#define GICH_ELRSR0                     0x30
>>>> +#define GICH_ELRSR1                     0x34
>>>> +#define GICH_LR0                        0x100
>>>> +
>>>> +#define GICH_LR_VIRTUALID               (0x3ff << 0)
>>>> +#define GICH_LR_PHYSID_CPUID_SHIFT      (10)
>>>> +#define GICH_LR_PHYSID_CPUID            (0x3ff <<
>>>> GICH_LR_PHYSID_CPUID_SHIFT)
>>>> +#define GICH_LR_PRIORITY_SHIFT          23
>>>> +#define GICH_LR_STATE                   (3 << 28)
>>>> +#define GICH_LR_PENDING_BIT             (1 << 28)
>>>> +#define GICH_LR_ACTIVE_BIT              (1 << 29)
>>>> +#define GICH_LR_EOI                     (1 << 19)
>>>> +#define GICH_LR_HW                      (1 << 31)
>>>
>>> Can we define them in either in gic.h or a new header gic-v2.h?
>>
>> Yes, but they clash with some ill-named GICv3 LR bits. So expect another
>> patch which renames GICH_LR_STATE_SHIFT to ICH_LR_STATE_SHIFT. Which is
>> the actual spec name for that system register in GICv3, there is no
>> GICH_LR_ with the GICv3 bit positions.
> 
> While this would be a nice clean-up. Wouldn't create a new gic-v2.h
> sufficient?

I don't think that would be right. We actually already have some GICH_
definitions in xen/include/asm-arm/gic.h, so just adding the missing
ones there sounds natural. I now remember that I just didn't do this
initially because of the clash and and at this time I just wanted to
make it compile ;-)

And since assigning GICH_ names to GICv3 ICH_ register bits sounds wrong
in the first place, I consider this a good opportunity to fix this.

Cheers,
Andre.

> 
>>
>>
>>>> +
>>>> +static struct {
>>>> +    bool enabled;
>>>> +    paddr_t dbase;          /* Distributor interface address */
>>>> +    paddr_t cbase;          /* CPU interface address & size */
>>>> +    paddr_t csize;
>>>> +    paddr_t vbase;          /* Virtual CPU interface address */
>>>> +    void __iomem *hbase;        /* Hypervisor control interface */
>>>> +
>>>> +    /* Offset to add to get an 8kB contiguous region if GIC is
>>>> aliased */
>>>> +    uint32_t aliased_offset;
>>>> +} gic_v2_hw_data;
>>>> +
>>>> +void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
>>>> +              paddr_t vbase, void __iomem *hbase,
>>>> +              uint32_t aliased_offset)
>>>> +{
>>>> +    gic_v2_hw_data.enabled = true;
>>>> +    gic_v2_hw_data.dbase = dbase;
>>>> +    gic_v2_hw_data.cbase = cbase;
>>>> +    gic_v2_hw_data.csize = csize;
>>>> +    gic_v2_hw_data.vbase = vbase;
>>>> +    gic_v2_hw_data.hbase = hbase;
>>>> +    gic_v2_hw_data.aliased_offset = aliased_offset;
>>>> +}
>>>> +
>>>> +void vgic_v2_set_underflow(struct vcpu *vcpu)
>>>> +{
>>>> +    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
>>>> +}
>>>> +
>>>> +/*
>>>> + * transfer the content of the LRs back into the corresponding
>>>> ap_list:
>>>> + * - active bit is transferred as is
>>>> + * - pending bit is
>>>> + *   - transferred as is in case of edge sensitive IRQs
>>>> + *   - set to the line-level (resample time) for level sensitive IRQs
>>>> + */
>>>> +void vgic_v2_fold_lr_state(struct vcpu *vcpu)
>>>
>>> I am wondering how much we could share this code with
>>> vgic_v3_fold_lr_state.
>>
>> I think we discussed this and dismissed the idea:
>> - The actual LR encoding is much different between GICv3 and GICv2, up
>> to the point where we have some fields in one which are not in the
>> other. That really clutters the code.
>> - Originally this function was much shorter and didn't have that many
>> special cases. So the code duplication was really minimal.
>>
>> I see your point, but don't really want to go there now for two reasons:
>> - It is probably nasty to implement, since we always have to check which
>> GIC we are running on when masking the LR value.
>> - It would deviate further from the KVM implementation, in a core
>> function. For any bugs introduced we are on our own here.
>>
>> I will try to bring this up with the KVM people, to see whether it's
>> worth to revisit this decision. There is indeed quite some code
>> duplication these days.
>> But this may come as an optimization later.
> 
> Fine with me. It was mostly to avoid having to review twice the same
> hairy code.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 25/49] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Hi,

On 26/02/18 15:59, Julien Grall wrote:
> 
> 
> On 02/26/2018 03:16 PM, Andre Przywara wrote:
>> Hi,
> 
> Hi,
> 
>> forgot to mention:
>>
>> On 13/02/18 14:31, Julien Grall wrote:
>>> Hi,
>>>
>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>> Processing maintenance interrupts and accessing the list registers
>>>> are dependent on the host's GIC version.
>>>> Introduce vgic-v2.c to contain GICv2 specific functions.
>>>> Implement the GICv2 specific code for syncing the emulation state
>>>> into the VGIC registers.
>>>> This also adds the hook to let Xen setup the host GIC addresses.
>>>>
>>>> This is based on Linux commit 140b086dd197, written by Marc Zyngier.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>> ---
>>>>    xen/arch/arm/vgic/vgic-v2.c | 261
>>>> ++++++++++++++++++++++++++++++++++++++++++++
>>>>    xen/arch/arm/vgic/vgic.c    |  20 ++++
>>>>    xen/arch/arm/vgic/vgic.h    |   8 ++
>>>>    3 files changed, 289 insertions(+)
>>>>    create mode 100644 xen/arch/arm/vgic/vgic-v2.c
>>>>
>>>> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
>>>> new file mode 100644
>>>> index 0000000000..10fc467ffa
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vgic/vgic-v2.c
>>
>> ....
>>
>>>> +void vgic_v2_save_state(struct vcpu *vcpu)
>>>> +{
>>>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>>>> +
>>>> +    if ( used_lrs )
>>>> +    {
>>>> +        save_lrs(vcpu, gic_v2_hw_data.hbase);
>>>> +        writel_relaxed(0, gic_v2_hw_data.hbase + GICH_HCR);
>>>> +    }
>>>> +}
>>>
>>> I am not entirely convinced that have a separate function to save the
>>> LRs is necessary. This could be done in fold_lr_state().
>>>
>>>> +
>>>> +void vgic_v2_restore_state(struct vcpu *vcpu)
>>>> +{
>>>> +    struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;,
>>>> +    u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
>>>> +    int i;
>>>> +
>>>> +    if ( used_lrs )
>>>> +    {
>>>> +        writel_relaxed(cpu_if->vgic_hcr,
>>>> +                       gic_v2_hw_data.hbase + GICH_HCR);
>>>> +        for ( i = 0; i < used_lrs; i++ )
>>>> +            writel_relaxed(cpu_if->vgic_lr[i],
>>>> +                           gic_v2_hw_data.hbase + GICH_LR0 + (i * 4));
>>>> +    }
>>>
>>> Same here but with populate_lr_state(). This would make the code easier
>>> to follow and also avoid a lot ifery in the vgic.c code.
>>
>> This is mostly due to KVM's inability to directly access the GICv3 LRs
>> when running in EL1. I will take a look whether what it would take to
>> merge this. Sounds tempting, but there might be side effects.
> 
> I am not sure what would be the side effects. You basically
> call save_state and right after fold_lr_state. This would streamline a
> bit more the code.

The possible side effects are that we actually now have a shadow copy of
the LRs in our data structures. I have the gut feeling we don't need
this in Xen, but need to check more thoroughly.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Andre Przywara]

Hi,

On 26/02/18 15:55, Julien Grall wrote:
> Hi,
> 
> On 02/26/2018 03:29 PM, Andre Przywara wrote:
>> On 13/02/18 16:35, Julien Grall wrote:
>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>> index f4f2a04a60..9e7fb1edcb 100644
>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>>>        vgic_restore_state(current);
>>>>    }
>>>>    +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>>>> +{
>>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>>> +    struct vgic_irq *irq;
>>>> +    bool pending = false;
>>>> +    unsigned long flags;
>>>> +
>>>> +    if ( !vcpu->domain->arch.vgic.enabled )
>>>> +        return false;
>>>> +
>>>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>>>> +
>>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>>> +    {
>>>> +        spin_lock(&irq->irq_lock);
>>>> +        pending = irq_is_pending(irq) && irq->enabled;
>>>> +        spin_unlock(&irq->irq_lock);
>>>> +
>>>> +        if ( pending )
>>>> +            break;
>>>> +    }
>>>> +
>>>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>>>> +
>>>> +    return pending;
>>>> +}
>>>> +
>>>> +int gic_events_need_delivery(void)
>>>
>>> You probably want to rename that function or just expose
>>> vgic_vcpu_pending_irq().
>>
>> Rename to what? I need both functions: vgic_vcpu_pending_irq() is also
>> called by vgic_kick_vcpus() (later in the series).
>> And gic_events_need_delivery(void) is the interface that the arch code
>> expects. Shall I rename this there? To what?
> 
> Let me start with it is a bit odd to have a function name 'gic_*' in the
> virtual GIC code. So at least renaming to vgic_events_need_delivery
> would be an improvement.
> 
> Regarding the interface itself, it is ARM specific and not set in stone.
> It would not be too bad to use vgic_vcpu_pending_irq(current). Is there
> any reason for not doing that?

Not really, but I am a bit reluctant to change too much original Xen
code, don't want to step on anyone's toes ;-)

But if that's fine with you, I am OK with the renaming - though it adds
yet another patch ;-)

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Julien Grall]

On 02/26/2018 04:25 PM, Andre Przywara wrote:
> Hi,
> 
> On 26/02/18 15:55, Julien Grall wrote:
>> Hi,
>>
>> On 02/26/2018 03:29 PM, Andre Przywara wrote:
>>> On 13/02/18 16:35, Julien Grall wrote:
>>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>>> index f4f2a04a60..9e7fb1edcb 100644
>>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>>>>         vgic_restore_state(current);
>>>>>     }
>>>>>     +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>>>>> +{
>>>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>>>> +    struct vgic_irq *irq;
>>>>> +    bool pending = false;
>>>>> +    unsigned long flags;
>>>>> +
>>>>> +    if ( !vcpu->domain->arch.vgic.enabled )
>>>>> +        return false;
>>>>> +
>>>>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>>>>> +
>>>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>>>> +    {
>>>>> +        spin_lock(&irq->irq_lock);
>>>>> +        pending = irq_is_pending(irq) && irq->enabled;
>>>>> +        spin_unlock(&irq->irq_lock);
>>>>> +
>>>>> +        if ( pending )
>>>>> +            break;
>>>>> +    }
>>>>> +
>>>>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>>>>> +
>>>>> +    return pending;
>>>>> +}
>>>>> +
>>>>> +int gic_events_need_delivery(void)
>>>>
>>>> You probably want to rename that function or just expose
>>>> vgic_vcpu_pending_irq().
>>>
>>> Rename to what? I need both functions: vgic_vcpu_pending_irq() is also
>>> called by vgic_kick_vcpus() (later in the series).
>>> And gic_events_need_delivery(void) is the interface that the arch code
>>> expects. Shall I rename this there? To what?
>>
>> Let me start with it is a bit odd to have a function name 'gic_*' in the
>> virtual GIC code. So at least renaming to vgic_events_need_delivery
>> would be an improvement.
>>
>> Regarding the interface itself, it is ARM specific and not set in stone.
>> It would not be too bad to use vgic_vcpu_pending_irq(current). Is there
>> any reason for not doing that?
> 
> Not really, but I am a bit reluctant to change too much original Xen
> code, don't want to step on anyone's toes ;-)

The original code is going to get kill at some point. So better use name 
that makes sense in the new context. It is quite similar to the 
gic_inject change.

> 
> But if that's fine with you, I am OK with the renaming - though it adds
> yet another patch ;-)

The end goal is a better world, so the number of patches does not matter 
here :).

If you put them at the beginning, we can merge them right away.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Andre Przywara]

Hi,

On 23/02/18 18:14, Julien Grall wrote:
> 
> 
> On 23/02/18 18:02, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 19/02/18 12:19, Julien Grall wrote:
>>> Hi,
>>>
>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
>>>> when a guest EOIs the virtual interrupt, it affects the state of that
>>>> corresponding interrupt on the hardware side at the same time.
>>>> Implement the interface that the Xen arch/core code expects to connect
>>>> the virtual and the physical world.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>> ---
>>>>    xen/arch/arm/vgic/vgic.c | 63
>>>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>    1 file changed, 63 insertions(+)
>>>>
>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>> index dc5e011fa3..8d5260a7db 100644
>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>>>>        }
>>>>    }
>>>>    +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu
>>>> *v,
>>>> +                                      unsigned int virq)
>>>> +{
>>>> +    struct irq_desc *desc = NULL;
>>>> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
>>>> +    unsigned long flags;
>>>> +
>>>> +    if ( !irq )
>>>> +        return NULL;
>>>> +
>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>> +    if ( irq->hw )
>>>> +        desc = irq_to_desc(irq->hwintid);
>>>
>>> This is not going to work well for PPIs. We should consider to add at
>>> least an ASSERT(...) in the code to prevent bad use of it.
>>
>> Yeah, done. But I wonder if we eventually should extend the
>> irq_to_desc() function to take the vCPU, since we will need it anyway
>> once we use hardware mapped timer IRQs (PPIs) in the future. But this
>> should not be in this series, I guess.
> 
> irq_to_desc only deal with hardware interrupt, so you mean pCPU instead
> of vCPU?

Yes, indeed. But I think this points to the problem of this approach:
the virtual IRQ is tied to a VCPU, and we have to make sure that not
only the affinity is updated on a CPU migration (as we do for SPIs), but
actually the interrupt itself is changed: since CPU0/PPI9 has a
different irq_desc* from, say, CPU1/PPI9.
So there is more than just adding a parameter to irq_to_desc().

>>>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>>>> +
>>>> +    vgic_put_irq(d, irq);
>>>> +
>>>> +    return desc;
>>>> +}
>>>> +
>>>> +/*
>>>> + * was:
>>>> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq,
>>>> u32 phys_irq)
>>>> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int
>>>> virt_irq)
>>>> + */
>>>> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
>>>> +            unsigned int virt_irq, struct irq_desc *desc,
>>>> +            bool connect)
>>>
>>> Indentation.
>>>
>>>> +{
>>>> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
>>>> +    unsigned long flags;
>>>> +    int ret = 0;
>>>> +
>>>> +    if ( !irq )
>>>> +        return -EINVAL;
>>>> +
>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>> +
>>>> +    if ( connect )                      /* assign a mapped IRQ */
>>>> +    {
>>>> +        /* The VIRQ should not be already enabled by the guest */
>>>> +        if ( !irq->hw && !irq->enabled )
>>>> +        {
>>>> +            irq->hw = true;
>>>> +            irq->hwintid = desc->irq;
>>>> +        }
>>>> +        else
>>>> +        {
>>>> +            ret = -EBUSY;
>>>> +        }
>>>
>>> I know that it should not matter for SPIs today. But aren't you meant to
>>> get a reference on that interrupt if you connect it?
>>
>> No, the refcount feature is strictly for the pointer to the structure,
>> not for everything related to this virtual IRQ.
>> We store only the virtual IRQ number in the dev_id/info members, we will
>> get the struct vgic_irq pointer via the vIRQ number on do_IRQ().
>> Does that make sense?
> 
> But technically you "allocate" the virtual SPI at that time, right? So
> this would mean you need to get a reference, otherwise it might disappear.

We will realise that is has disappeared when vgic_get_irq() called with
that virtual number returns NULL. The refcount is really just to know
when you can free dynamically allocated struct vgic_irqs, so it's
strictly about the *pointer* to the *memory*, not about the logical
entity of that particular virtual IRQ.
Actually it should not really happen that you end up with a hardware IRQ
still assigned to an abandoned virtual IRQ, as you would expect to free
that connection *before* disbanding the virtual IRQ.

> So I am not entirely sure why the reference is not necessary here.

Typically to remove a virtual IRQ, you arrange for vgic_get_irq() to
return NULL on that number. Then you "wait" for the refcount to drop to
zero, at which point it's safe to free the memory allocated for that
vgic_irq. As mentioned, only really useful for LPIs, but it's a central
property of the new VGIC architecture, because we need to have those
gets/puts in virtually every function.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Julien Grall]

On 02/26/2018 04:48 PM, Andre Przywara wrote:
> Hi,
> 
> On 23/02/18 18:14, Julien Grall wrote:
>>
>>
>> On 23/02/18 18:02, Andre Przywara wrote:
>>> Hi,
>>
>> Hi Andre,
>>
>>> On 19/02/18 12:19, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>>> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
>>>>> when a guest EOIs the virtual interrupt, it affects the state of that
>>>>> corresponding interrupt on the hardware side at the same time.
>>>>> Implement the interface that the Xen arch/core code expects to connect
>>>>> the virtual and the physical world.
>>>>>
>>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>>> ---
>>>>>     xen/arch/arm/vgic/vgic.c | 63
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>     1 file changed, 63 insertions(+)
>>>>>
>>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>>> index dc5e011fa3..8d5260a7db 100644
>>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>>> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>>>>>         }
>>>>>     }
>>>>>     +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu
>>>>> *v,
>>>>> +                                      unsigned int virq)
>>>>> +{
>>>>> +    struct irq_desc *desc = NULL;
>>>>> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
>>>>> +    unsigned long flags;
>>>>> +
>>>>> +    if ( !irq )
>>>>> +        return NULL;
>>>>> +
>>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>>> +    if ( irq->hw )
>>>>> +        desc = irq_to_desc(irq->hwintid);
>>>>
>>>> This is not going to work well for PPIs. We should consider to add at
>>>> least an ASSERT(...) in the code to prevent bad use of it.
>>>
>>> Yeah, done. But I wonder if we eventually should extend the
>>> irq_to_desc() function to take the vCPU, since we will need it anyway
>>> once we use hardware mapped timer IRQs (PPIs) in the future. But this
>>> should not be in this series, I guess.
>>
>> irq_to_desc only deal with hardware interrupt, so you mean pCPU instead
>> of vCPU?
> 
> Yes, indeed. But I think this points to the problem of this approach:
> the virtual IRQ is tied to a VCPU, and we have to make sure that not
> only the affinity is updated on a CPU migration (as we do for SPIs), but
> actually the interrupt itself is changed: since CPU0/PPI9 has a
> different irq_desc* from, say, CPU1/PPI9.
> So there is more than just adding a parameter to irq_to_desc().

Change in the irq_to_desc() interface needs to be justify. The use case 
I have in mind for PPI is the virtual timer. In that case, you will 
always receive the PPI on the right pCPU.

Do you really see a use case where a vCPU is running on pCPU A but the 
PPI is routed to pCPU B?


> 
>>>>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>>>>> +
>>>>> +    vgic_put_irq(d, irq);
>>>>> +
>>>>> +    return desc;
>>>>> +}
>>>>> +
>>>>> +/*
>>>>> + * was:
>>>>> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq,
>>>>> u32 phys_irq)
>>>>> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int
>>>>> virt_irq)
>>>>> + */
>>>>> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
>>>>> +            unsigned int virt_irq, struct irq_desc *desc,
>>>>> +            bool connect)
>>>>
>>>> Indentation.
>>>>
>>>>> +{
>>>>> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
>>>>> +    unsigned long flags;
>>>>> +    int ret = 0;
>>>>> +
>>>>> +    if ( !irq )
>>>>> +        return -EINVAL;
>>>>> +
>>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>>> +
>>>>> +    if ( connect )                      /* assign a mapped IRQ */
>>>>> +    {
>>>>> +        /* The VIRQ should not be already enabled by the guest */
>>>>> +        if ( !irq->hw && !irq->enabled )
>>>>> +        {
>>>>> +            irq->hw = true;
>>>>> +            irq->hwintid = desc->irq;
>>>>> +        }
>>>>> +        else
>>>>> +        {
>>>>> +            ret = -EBUSY;
>>>>> +        }
>>>>
>>>> I know that it should not matter for SPIs today. But aren't you meant to
>>>> get a reference on that interrupt if you connect it?
>>>
>>> No, the refcount feature is strictly for the pointer to the structure,
>>> not for everything related to this virtual IRQ.
>>> We store only the virtual IRQ number in the dev_id/info members, we will
>>> get the struct vgic_irq pointer via the vIRQ number on do_IRQ().
>>> Does that make sense?
>>
>> But technically you "allocate" the virtual SPI at that time, right? So
>> this would mean you need to get a reference, otherwise it might disappear.
> 
> We will realise that is has disappeared when vgic_get_irq() called with
> that virtual number returns NULL. The refcount is really just to know
> when you can free dynamically allocated struct vgic_irqs, so it's
> strictly about the *pointer* to the *memory*, not about the logical
> entity of that particular virtual IRQ.
> Actually it should not really happen that you end up with a hardware IRQ
> still assigned to an abandoned virtual IRQ, as you would expect to free
> that connection *before* disbanding the virtual IRQ.
> 
>> So I am not entirely sure why the reference is not necessary here.
> 
> Typically to remove a virtual IRQ, you arrange for vgic_get_irq() to
> return NULL on that number. Then you "wait" for the refcount to drop to
> zero, at which point it's safe to free the memory allocated for that
> vgic_irq. As mentioned, only really useful for LPIs, but it's a central
> property of the new VGIC architecture, because we need to have those
> gets/puts in virtually every function.

Thank you for the explanation.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 41/49] ARM: new VGIC: dump virtual IRQ info

[Andre Przywara]

Hi,

On 19/02/18 12:26, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> When we dump guest state on the Xen console, we also print the state of
>> IRQs that are on a VCPU.
>> Add the code to dump the state of an IRQ handled by the new VGIC.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index 3b475ed1a4..97ffdba5ad 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -757,6 +757,19 @@ void vgic_free_virq(struct domain *d, unsigned
>> int virq)
>>       clear_bit(virq, d->arch.vgic.allocated_irqs);
>>   }
>>   +void gic_dump_vgic_info(struct vcpu *v)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &v->arch.vgic_cpu;
>> +    struct vgic_irq *irq;
>> +
>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
> 
> I don't think you can assume that the vCPU is not running somewhere
> else. So likely you want to take the lock while dumping the info.

Oh, good point. Totally forgot the locking here :-(
Same for the IRQs within.
Thanks for pointing this out.

> 
>> +        printk("   on CPU: %s %s irq %u: %spending, %sactive,
>> %senabled\n",
> 
> I am not sure the value of "on CPU".

That is meant to be a short phrase for "being on the ap_list", which is
an implementation specific term. "Active" or "pending" alone are
confusing or misleading. If you have a better term (not too long!), I am
happy to take that.

Cheers,
Andre.

> 
>> +               irq->hw ? "hardware" : "virtual",
>> +               irq->config == VGIC_CONFIG_LEVEL ? "level" : "edge",
>> +               irq->intid, irq_is_pending(irq) ? "" : "not ",
>> +               irq->active ? "" : "not ", irq->enabled ? "" : "not ");
>> +}
>> +
>>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>>                                         unsigned int virq)
>>   {
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 41/49] ARM: new VGIC: dump virtual IRQ info

[Julien Grall]

On 02/26/2018 04:58 PM, Andre Przywara wrote:
> Hi,
> 
> On 19/02/18 12:26, Julien Grall wrote:
>> Hi Andre,
>>
>> On 09/02/18 14:39, Andre Przywara wrote:
>>> When we dump guest state on the Xen console, we also print the state of
>>> IRQs that are on a VCPU.
>>> Add the code to dump the state of an IRQ handled by the new VGIC.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
>>>    1 file changed, 13 insertions(+)
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> index 3b475ed1a4..97ffdba5ad 100644
>>> --- a/xen/arch/arm/vgic/vgic.c
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -757,6 +757,19 @@ void vgic_free_virq(struct domain *d, unsigned
>>> int virq)
>>>        clear_bit(virq, d->arch.vgic.allocated_irqs);
>>>    }
>>>    +void gic_dump_vgic_info(struct vcpu *v)
>>> +{
>>> +    struct vgic_cpu *vgic_cpu = &v->arch.vgic_cpu;
>>> +    struct vgic_irq *irq;
>>> +
>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>
>> I don't think you can assume that the vCPU is not running somewhere
>> else. So likely you want to take the lock while dumping the info.
> 
> Oh, good point. Totally forgot the locking here :-(
> Same for the IRQs within.
> Thanks for pointing this out.
> 
>>
>>> +        printk("   on CPU: %s %s irq %u: %spending, %sactive,
>>> %senabled\n",
>>
>> I am not sure the value of "on CPU".
> 
> That is meant to be a short phrase for "being on the ap_list", which is
> an implementation specific term. "Active" or "pending" alone are
> confusing or misleading. If you have a better term (not too long!), I am
> happy to take that.

How about a print before dumping the list? This would avoid the on CPU 
on each line and it can be longer :).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Andre Przywara]

Hi,

On 26/02/18 16:57, Julien Grall wrote:
> 
> 
> On 02/26/2018 04:48 PM, Andre Przywara wrote:
>> Hi,
>>
>> On 23/02/18 18:14, Julien Grall wrote:
>>>
>>>
>>> On 23/02/18 18:02, Andre Przywara wrote:
>>>> Hi,
>>>
>>> Hi Andre,
>>>
>>>> On 19/02/18 12:19, Julien Grall wrote:
>>>>> Hi,
>>>>>
>>>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>>>> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
>>>>>> when a guest EOIs the virtual interrupt, it affects the state of that
>>>>>> corresponding interrupt on the hardware side at the same time.
>>>>>> Implement the interface that the Xen arch/core code expects to
>>>>>> connect
>>>>>> the virtual and the physical world.
>>>>>>
>>>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>>>> ---
>>>>>>     xen/arch/arm/vgic/vgic.c | 63
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>     1 file changed, 63 insertions(+)
>>>>>>
>>>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>>>> index dc5e011fa3..8d5260a7db 100644
>>>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>>>> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>>>>>>         }
>>>>>>     }
>>>>>>     +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d,
>>>>>> struct vcpu
>>>>>> *v,
>>>>>> +                                      unsigned int virq)
>>>>>> +{
>>>>>> +    struct irq_desc *desc = NULL;
>>>>>> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
>>>>>> +    unsigned long flags;
>>>>>> +
>>>>>> +    if ( !irq )
>>>>>> +        return NULL;
>>>>>> +
>>>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>>>> +    if ( irq->hw )
>>>>>> +        desc = irq_to_desc(irq->hwintid);
>>>>>
>>>>> This is not going to work well for PPIs. We should consider to add at
>>>>> least an ASSERT(...) in the code to prevent bad use of it.
>>>>
>>>> Yeah, done. But I wonder if we eventually should extend the
>>>> irq_to_desc() function to take the vCPU, since we will need it anyway
>>>> once we use hardware mapped timer IRQs (PPIs) in the future. But this
>>>> should not be in this series, I guess.
>>>
>>> irq_to_desc only deal with hardware interrupt, so you mean pCPU instead
>>> of vCPU?
>>
>> Yes, indeed. But I think this points to the problem of this approach:
>> the virtual IRQ is tied to a VCPU, and we have to make sure that not
>> only the affinity is updated on a CPU migration (as we do for SPIs), but
>> actually the interrupt itself is changed: since CPU0/PPI9 has a
>> different irq_desc* from, say, CPU1/PPI9.
>> So there is more than just adding a parameter to irq_to_desc().
> 
> Change in the irq_to_desc() interface needs to be justify. The use case
> I have in mind for PPI is the virtual timer. In that case, you will
> always receive the PPI on the right pCPU.

Yes, but the connection between virtual and physical IRQ is realised as
a connection between struct pending_irq/vgic_irq and struct irq_desc.
For an SPI this is always the same irq_desc, regardless of the affinity
or running CPU. But for PPIs you would need to change the actual
irq_desc pointer when changing the affinity. Not really rocket science
(though it may become nasty with the locking), but needs to be implemented.

> Do you really see a use case where a vCPU is running on pCPU A but the
> PPI is routed to pCPU B?

Not at the moment, I guess the very nature of PPIs would avoid this. The
other PPIs I know about are PMUs and the VGIC. The latter is not of a
concern for us yet (fortunately). PMUs should have the same local
property as the arch timer.

Cheers,
Andre.

>>>>>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>>>>>> +
>>>>>> +    vgic_put_irq(d, irq);
>>>>>> +
>>>>>> +    return desc;
>>>>>> +}
>>>>>> +
>>>>>> +/*
>>>>>> + * was:
>>>>>> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq,
>>>>>> u32 phys_irq)
>>>>>> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int
>>>>>> virt_irq)
>>>>>> + */
>>>>>> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
>>>>>> +            unsigned int virt_irq, struct irq_desc *desc,
>>>>>> +            bool connect)
>>>>>
>>>>> Indentation.
>>>>>
>>>>>> +{
>>>>>> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
>>>>>> +    unsigned long flags;
>>>>>> +    int ret = 0;
>>>>>> +
>>>>>> +    if ( !irq )
>>>>>> +        return -EINVAL;
>>>>>> +
>>>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>>>> +
>>>>>> +    if ( connect )                      /* assign a mapped IRQ */
>>>>>> +    {
>>>>>> +        /* The VIRQ should not be already enabled by the guest */
>>>>>> +        if ( !irq->hw && !irq->enabled )
>>>>>> +        {
>>>>>> +            irq->hw = true;
>>>>>> +            irq->hwintid = desc->irq;
>>>>>> +        }
>>>>>> +        else
>>>>>> +        {
>>>>>> +            ret = -EBUSY;
>>>>>> +        }
>>>>>
>>>>> I know that it should not matter for SPIs today. But aren't you
>>>>> meant to
>>>>> get a reference on that interrupt if you connect it?
>>>>
>>>> No, the refcount feature is strictly for the pointer to the structure,
>>>> not for everything related to this virtual IRQ.
>>>> We store only the virtual IRQ number in the dev_id/info members, we
>>>> will
>>>> get the struct vgic_irq pointer via the vIRQ number on do_IRQ().
>>>> Does that make sense?
>>>
>>> But technically you "allocate" the virtual SPI at that time, right? So
>>> this would mean you need to get a reference, otherwise it might
>>> disappear.
>>
>> We will realise that is has disappeared when vgic_get_irq() called with
>> that virtual number returns NULL. The refcount is really just to know
>> when you can free dynamically allocated struct vgic_irqs, so it's
>> strictly about the *pointer* to the *memory*, not about the logical
>> entity of that particular virtual IRQ.
>> Actually it should not really happen that you end up with a hardware IRQ
>> still assigned to an abandoned virtual IRQ, as you would expect to free
>> that connection *before* disbanding the virtual IRQ.
>>
>>> So I am not entirely sure why the reference is not necessary here.
>>
>> Typically to remove a virtual IRQ, you arrange for vgic_get_irq() to
>> return NULL on that number. Then you "wait" for the refcount to drop to
>> zero, at which point it's safe to free the memory allocated for that
>> vgic_irq. As mentioned, only really useful for LPIs, but it's a central
>> property of the new VGIC architecture, because we need to have those
>> gets/puts in virtually every function.
> 
> Thank you for the explanation.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 38/49] ARM: new VGIC: handle hardware mapped IRQs

[Julien Grall]

Hi,

On 02/26/2018 05:19 PM, Andre Przywara wrote:
> Hi,
> 
> On 26/02/18 16:57, Julien Grall wrote:
>>
>>
>> On 02/26/2018 04:48 PM, Andre Przywara wrote:
>>> Hi,
>>>
>>> On 23/02/18 18:14, Julien Grall wrote:
>>>>
>>>>
>>>> On 23/02/18 18:02, Andre Przywara wrote:
>>>>> Hi,
>>>>
>>>> Hi Andre,
>>>>
>>>>> On 19/02/18 12:19, Julien Grall wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>>>>> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
>>>>>>> when a guest EOIs the virtual interrupt, it affects the state of that
>>>>>>> corresponding interrupt on the hardware side at the same time.
>>>>>>> Implement the interface that the Xen arch/core code expects to
>>>>>>> connect
>>>>>>> the virtual and the physical world.
>>>>>>>
>>>>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>>>>> ---
>>>>>>>      xen/arch/arm/vgic/vgic.c | 63
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>>      1 file changed, 63 insertions(+)
>>>>>>>
>>>>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>>>>> index dc5e011fa3..8d5260a7db 100644
>>>>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>>>>> @@ -693,6 +693,69 @@ void vgic_kick_vcpus(struct domain *d)
>>>>>>>          }
>>>>>>>      }
>>>>>>>      +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d,
>>>>>>> struct vcpu
>>>>>>> *v,
>>>>>>> +                                      unsigned int virq)
>>>>>>> +{
>>>>>>> +    struct irq_desc *desc = NULL;
>>>>>>> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
>>>>>>> +    unsigned long flags;
>>>>>>> +
>>>>>>> +    if ( !irq )
>>>>>>> +        return NULL;
>>>>>>> +
>>>>>>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>>>>>>> +    if ( irq->hw )
>>>>>>> +        desc = irq_to_desc(irq->hwintid);
>>>>>>
>>>>>> This is not going to work well for PPIs. We should consider to add at
>>>>>> least an ASSERT(...) in the code to prevent bad use of it.
>>>>>
>>>>> Yeah, done. But I wonder if we eventually should extend the
>>>>> irq_to_desc() function to take the vCPU, since we will need it anyway
>>>>> once we use hardware mapped timer IRQs (PPIs) in the future. But this
>>>>> should not be in this series, I guess.
>>>>
>>>> irq_to_desc only deal with hardware interrupt, so you mean pCPU instead
>>>> of vCPU?
>>>
>>> Yes, indeed. But I think this points to the problem of this approach:
>>> the virtual IRQ is tied to a VCPU, and we have to make sure that not
>>> only the affinity is updated on a CPU migration (as we do for SPIs), but
>>> actually the interrupt itself is changed: since CPU0/PPI9 has a
>>> different irq_desc* from, say, CPU1/PPI9.
>>> So there is more than just adding a parameter to irq_to_desc().
>>
>> Change in the irq_to_desc() interface needs to be justify. The use case
>> I have in mind for PPI is the virtual timer. In that case, you will
>> always receive the PPI on the right pCPU.
> 
> Yes, but the connection between virtual and physical IRQ is realised as
> a connection between struct pending_irq/vgic_irq and struct irq_desc.
> For an SPI this is always the same irq_desc, regardless of the affinity
> or running CPU. But for PPIs you would need to change the actual
> irq_desc pointer when changing the affinity. Not really rocket science
> (though it may become nasty with the locking), but needs to be implemented.

I don't think it is a big deal. You would remove the "link" when saving 
the vCPU state and add the "link" when restoring. In both case, you 
would be on the right pCPU. Have a look at:

https://lists.xenproject.org/archives/html/xen-devel/2015-11/msg00925.html

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 44/49] ARM: new VGIC: vgic-init: register VGIC

[Andre Przywara]

Hi,

On 19/02/18 12:39, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> This patch implements the function which is called by Xen when it wants
>> to register the virtual GIC.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic-init.c | 62
>> +++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.h      |  3 +++
>>   2 files changed, 65 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic-init.c
>>
>> diff --git a/xen/arch/arm/vgic/vgic-init.c
>> b/xen/arch/arm/vgic/vgic-init.c
>> new file mode 100644
>> index 0000000000..b5f1183a50
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic-init.c
>> @@ -0,0 +1,62 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/sched.h>
>> +#include <asm/arm_vgic.h>
>> +
>> +#include "vgic.h"
>> +
>> +/* CREATION */
>> +
>> +/**
>> + * domain_vgic_register: create a virtual GIC
>> + * @d: domain pointer
>> + * @mmio_count: pointer to add number of required MMIO regions
>> + *
>> + * was: kvm_vgic_create
>> + */
>> +int domain_vgic_register(struct domain *d, int *mmio_count)
> 
> mmio_count should be set to the number of I/O region you will register.
> 
>> +{
>> +    switch ( d->arch.vgic.version )
>> +    {
>> +#ifdef CONFIG_HAS_GICV3
>> +    case GIC_V3:
>> +        d->arch.max_vcpus = VGIC_V3_MAX_CPUS;
>> +        break;
>> +#endif
>> +    case GIC_V2:
>> +        d->arch.max_vcpus = VGIC_V2_MAX_CPUS;
>> +        break;
>> +    }
>> +
>> +    if ( d->max_vcpus > d->arch.max_vcpus )
>> +        return -E2BIG;
>> +
>> +    d->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
>> +    d->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
>> +    d->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
> 
> Is there any reason to store an address rather than a frame? The latter
> would add a be more safety.

Possibly, but the existing VGIC doesn't do it as well, which means that
at the moment we deal with addresses (and not frame numbers) everywhere
(for instance GUEST_GICD_BASE).
Changing this for the distributor internally already (as requested for
some other patch) caused more changes than I liked, and at the moment I
am trying to make the TODO list smaller, not bigger ;-)

Cheers,
Andre.

>> +
>> +    return 0;
>> +}
>> +
>> +/*
>> + * Local variables:
>> + * mode: C
>> + * c-file-style: "BSD"
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index b104f8e964..205ce10ffa 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -20,6 +20,9 @@
>>   #define PRODUCT_ID_KVM      0x4b    /* ASCII code K */
>>   #define IMPLEMENTER_ARM     0x43b
>>   +#define VGIC_ADDR_UNDEF     (-1)
> 
> Please use INVALID_PADDR here.
> 
>> +#define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
>> +
>>   #define VGIC_PRI_BITS       5
>>     #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 22/49] ARM: new VGIC: Implement virtual IRQ injection

[Andre Przywara]

Hi,

On 12/02/18 18:59, Julien Grall wrote:
> Hi Andre,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> Provide a vgic_queue_irq_unlock() function which decides whether a
>> given IRQ needs to be queued to a VCPU's ap_list.
>> This should be called whenever an IRQ becomes pending or enabled,
>> either as a result of a hardware IRQ injection, from devices emulated by
>> Xen (like the architected timer) or from MMIO accesses to the distributor
>> emulation.
>> Also provides the necessary functions to allow to inject an IRQ to a
>> guest.
>> Since this is the first code that starts using our locking mechanism,
>> we add some (hopefully) clear documentation of our locking strategy and
>> requirements along with this patch.
>>
>> This is based on Linux commit 81eeb95ddbab, written by Christoffer Dall.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 224
>> +++++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.h |  10 +++
>>   2 files changed, 234 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index 3075091caa..f517df6d00 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -21,6 +21,32 @@
>>   #include <asm/arm_vgic.h>
>>   #include "vgic.h"
>>   +/*
>> + * Locking order is always:
>> + * kvm->lock (mutex)
> 
> You probably want to update the locking order to match Xen one. In that
> case, I am not sure if we need to take the domain lock in the code?
> 
> 
>> + *   its->cmd_lock (mutex)
>> + *     its->its_lock (mutex)
> 
>> + *       vgic_cpu->ap_list_lock
>> + *         kvm->lpi_list_lock
>> + *           vgic_irq->irq_lock
>> + *
>> + * If you need to take multiple locks, always take the upper lock first,
>> + * then the lower ones, e.g. first take the its_lock, then the irq_lock.
>> + * If you are already holding a lock and need to take a higher one, you
>> + * have to drop the lower ranking lock first and re-aquire it after
>> having
> 
> s/re-aquite/acquire/
> 
>> + * taken the upper one.
>> + *
>> + * When taking more than one ap_list_lock at the same time, always
>> take the
>> + * lowest numbered VCPU's ap_list_lock first, so:
>> + *   vcpuX->vcpu_id < vcpuY->vcpu_id:
>> + *     spin_lock(vcpuX->arch.vgic_cpu.ap_list_lock);
>> + *     spin_lock(vcpuY->arch.vgic_cpu.ap_list_lock);
>> + *
>> + * Since the VGIC must support injecting virtual interrupts from
>> ISRs, we have
>> + * to use the spin_lock_irqsave/spin_unlock_irqrestore versions of outer
>> + * spinlocks for any lock that may be taken while injecting an
>> interrupt.
> 
> It is quite nice to see the locking explained in the file and in general
> a lot of explanation within the code :).
> 
>> + */
>> +
>>   /*
>>    * Iterate over the VM's list of mapped LPIs to find the one with a
>>    * matching interrupt ID and return a reference to the IRQ structure.
>> @@ -97,6 +123,204 @@ void vgic_put_irq(struct domain *d, struct
>> vgic_irq *irq)
>>       xfree(irq);
>>   }
>>   +/**
>> + * vgic_target_oracle - compute the target vcpu for an irq
>> + *
>> + * @irq:    The irq to route. Must be already locked.
>> + *
>> + * Based on the current state of the interrupt (enabled, pending,
>> + * active, vcpu and target_vcpu), compute the next vcpu this should be
>> + * given to. Return NULL if this shouldn't be injected at all.
>> + *
>> + * Requires the IRQ lock to be held.
>> + */
>> +static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
>> +{
>> +    ASSERT(spin_is_locked(&irq->irq_lock));
>> +
>> +    /* If the interrupt is active, it must stay on the current vcpu */
>> +    if ( irq->active )
>> +        return irq->vcpu ? : irq->target_vcpu;
> I am not sure to understand why you check whether irq->vcpu is NULL. If
> the interrupt is active, then irq->vcpu should be NULL. Did I miss
                                                ^
                                               not           you mean?
> anything?

Not if it has been explicitly activated via ISACTIVER. This is not
implemented in Xen at the moment, but would be in the future. So I like
to keep this in.

>> +
>> +    /*
>> +     * If the IRQ is not active but enabled and pending, we should
>> direct
>> +     * it to its configured target VCPU.
>> +     * If the distributor is disabled, pending interrupts shouldn't be
>> +     * forwarded.
>> +     */
>> +    if ( irq->enabled && irq_is_pending(irq) )
>> +    {
>> +        if ( unlikely(irq->target_vcpu &&
>> +                 !irq->target_vcpu->domain->arch.vgic.enabled) )
> 
> The indentation looks wrong here.
> 
>> +            return NULL;
>> +
>> +        return irq->target_vcpu;
>> +    }
>> +
>> +    /* If neither active nor pending and enabled, then this IRQ
>> should not
> 
> Comment style:
> 
> /*
>  * ...
> 
>> +     * be queued to any VCPU.
>> +     */
>> +    return NULL;
>> +}
>> +
>> +/*
>> + * Only valid injection if changing level for level-triggered IRQs or
>> for a
>> + * rising edge.
>> + */
>> +static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
>> +{
>> +    switch (irq->config)
> 
> switch ( ... )
> 
>> +    {
>> +    case VGIC_CONFIG_LEVEL:
>> +        return irq->line_level != level;
>> +    case VGIC_CONFIG_EDGE:
>> +        return level;
>> +    }
>> +
> 
> I would add an ASSERT_UNREACHABLE().
> 
>> +    return false;
>> +}
>> +
>> +/*
>> + * Check whether an IRQ needs to (and can) be queued to a VCPU's ap
>> list.
>> + * Do the queuing if necessary, taking the right locks in the right
>> order.
>> + * Returns true when the IRQ was queued, false otherwise.
>> + *
>> + * Needs to be entered with the IRQ lock already held, but will return
>> + * with all locks dropped.
>> + */
>> +bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>> +               unsigned long flags)
> 
> Indentation. Also same remark as from vgic_inject_irq. No-one seems to
> care about the return (even in KVM :)).
> 
>> +{
>> +    struct vcpu *vcpu;
>> +    bool running;
>> +
>> +    ASSERT(spin_is_locked(&irq->irq_lock));
>> +
>> +retry:
>> +    vcpu = vgic_target_oracle(irq);
>> +    if ( irq->vcpu || !vcpu )
>> +    {
>> +        /*
>> +         * If this IRQ is already on a VCPU's ap_list, then it
>> +         * cannot be moved or modified and there is no more work for
>> +         * us to do.
>> +         *
>> +         * Otherwise, if the irq is not pending and enabled, it does
>> +         * not need to be inserted into an ap_list and there is also
>> +         * no more work for us to do.
>> +         */
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +        /*
>> +         * We have to kick the VCPU here, because we could be
>> +         * queueing an edge-triggered interrupt for which we
>> +         * get no EOI maintenance interrupt. In that case,
>> +         * while the IRQ is already on the VCPU's AP list, the
>> +         * VCPU could have EOI'ed the original interrupt and
>> +         * won't see this one until it exits for some other
>> +         * reason.
>> +         */
>> +        if ( vcpu )
>> +            vcpu_unblock(vcpu);
> 
> vcpu_unblock will only "unblock" a vCPU that is blocked. It won't notify
> a running vCPU. So you want to have something similar to:
> 
> vcpu_unblock(vcpu);
> if ( running && vcpu != current )
>   smp_send_event_check_mask(...);
> 
> It is probably worth to introduce an helper for that.
> 
>> +        return false;
>> +    }
>> +
>> +    /*
>> +     * We must unlock the irq lock to take the ap_list_lock where
>> +     * we are going to insert this new pending interrupt.
>> +     */
>> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +    /* someone can do stuff here, which we re-check below */
>> +
>> +    spin_lock_irqsave(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
>> +    spin_lock(&irq->irq_lock);
>> +
>> +    /*
>> +     * Did something change behind our backs?
>> +     *
>> +     * There are two cases:
>> +     * 1) The irq lost its pending state or was disabled behind our
>> +     *    backs and/or it was queued to another VCPU's ap_list.
>> +     * 2) Someone changed the affinity on this irq behind our
>> +     *    backs and we are now holding the wrong ap_list_lock.
>> +     *
>> +     * In both cases, drop the locks and retry.
>> +     */
>> +
>> +    if ( unlikely(irq->vcpu || vcpu != vgic_target_oracle(irq)) )
>> +    {
>> +        spin_unlock(&irq->irq_lock);
>> +        spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock,
>> flags);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +        goto retry;
>> +    }
>> +
>> +    /*
>> +     * Grab a reference to the irq to reflect the fact that it is
>> +     * now in the ap_list.
>> +     */
>> +    vgic_get_irq_kref(irq);
>> +    list_add_tail(&irq->ap_list, &vcpu->arch.vgic_cpu.ap_list_head);
> 
> I was expecting the list to be sorted here. But you seem to do it only
> in vgic_flush_lr_state() which is quite interesting.

The list is not sorted on insertion because this is not necessary most
of the time. In fact the hardware VGIC will do the sorting (kind of)
within the LRs. So as long as we don't have more than <number of LRS>
IRQs in the list, sorting is a waste of time. Experiments in the past
showed that the number of used LRs is less than 4 almost every time. And
since 4 is the mostly used number of implemented LRs, we virtually never
need the sorting. So we avoid doing that on every insertion, instead
doing that only if it's necessary.

> I can foresee quite a few issues with this choice on Xen:
>     1) You compute the size of ap list in vgic_flush_lr_state() and take
> lock on every IRQ one by one. A guest could be nasty and make that list
> quite big by make IRQs pending but never "active" them (i.e read IAR).

Yeah, we could try to shortcut a bit here.

>     2) This might be an issue while checking whether you need to deliver
> an interrupt (vgic_vcpu_pending_irq) because the list is not sorted.

Most of the time the list is very short, storing one, two or actually no
IRQs. In the function where we check for pending IRQs we bail out as
soon as we found the first eligible interrupt. So sorting does not help
in the majority of cases.

If you are really concerned about that list growing too long, we could
think about mitigations:
1) Try to avoid iterating the whole list while checking whether it needs
to be sorted.
2) Store a flag that notes if the list has already been sorted. As long
as we don't change anything, we don't need to sort again. Would be good
to test whether this is actually helpful. But we would need to keep this
flag up-to-date, which sounds a bit fragile to get right.
3) Switch to sort-on-insertion once we reached a certain number of IRQs
on the list, to mitigate DOS attacks from the guest. This should avoid
list iterations in hot paths, with IRQs disabled.

But all of these sound a bit hackish to me and just would spoil the very
clean and robust code we have today. Also I am not sure we can avoid
list iterations every time (for instance in prune_ap_list()). There is
an upper limit today (number of SPIs), so we might be happy with that
for now.
To be honest I would very much dislike changing the code at this point.
I believe a patch series afterwards would be better, also to actually
have some numbers on the impact of this.

Cheers,
Andre.

>> +    irq->vcpu = vcpu;
>> +
>> +    spin_unlock(&irq->irq_lock);
>> +    spin_unlock_irqrestore(&vcpu->arch.vgic_cpu.ap_list_lock, flags);
>> +
>> +    running = vcpu->is_running;
>> +    vcpu_unblock(vcpu);
>> +    if ( running && vcpu != current )
>> +        smp_send_event_check_mask(cpumask_of(vcpu->processor));
>> +
>> +    return true;
>> +}
>> +
>> +/**
>> + * vgic_inject_irq - Inject an IRQ from a device to the vgic
>> + * @d:       The domain pointer
>> + * @vcpu:    The vCPU for PPIs
>> + * @intid:   The INTID to inject a new state to.
>> + * @level:   Edge-triggered:  true:  to trigger the interrupt
>> + *                false: to ignore the call
>> + *       Level-sensitive  true:  raise the input signal
>> + *                false: lower the input signal
>> + *
>> + * The VGIC is not concerned with devices being active-LOW or
>> active-HIGH for
>> + * level-sensitive interrupts.  You can think of the level parameter
>> as 1
>> + * being HIGH and 0 being LOW and all devices being active-HIGH.
>> + */
>> +int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int
>> intid,
>> +            bool level)
> 
> Indentation.
> 
>> +{
>> +    struct vgic_irq *irq;
>> +    unsigned long flags;
>> +
>> +    irq = vgic_get_irq(d, vcpu, intid);
>> +    if ( !irq )
>> +        return -EINVAL;
>> +
>> +    spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +    if ( !vgic_validate_injection(irq, level) )
>> +    {
>> +        /* Nothing to see here, move along... */
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(d, irq);
>> +        return 0;
>> +    }
>> +
>> +    if ( irq->config == VGIC_CONFIG_LEVEL )
>> +        irq->line_level = level;
>> +    else
>> +        irq->pending_latch = true;
>> +
>> +    vgic_queue_irq_unlock(d, irq, flags);
>> +    vgic_put_irq(d, irq);
>> +
>> +    return 0;
>> +}
>> +
>>   /*
>>    * Local variables:
>>    * mode: C
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 7a15cfdd79..5127739f0f 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -17,9 +17,19 @@
>>   #ifndef __XEN_ARM_VGIC_NEW_H__
>>   #define __XEN_ARM_VGIC_NEW_H__
>>   +static inline bool irq_is_pending(struct vgic_irq *irq)
>> +{
>> +    if ( irq->config == VGIC_CONFIG_EDGE )
>> +        return irq->pending_latch;
>> +    else
>> +        return irq->pending_latch || irq->line_level;
>> +}
>> +
>>   struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>>                                 u32 intid);
>>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
>> +bool vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>> +               unsigned long flags);
>>     static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>>   {
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 22/49] ARM: new VGIC: Implement virtual IRQ injection

[Julien Grall]

On 27/02/18 10:17, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 12/02/18 18:59, Julien Grall wrote:
>> On 09/02/18 14:39, Andre Przywara wrote:
>>>    /*
>>>     * Iterate over the VM's list of mapped LPIs to find the one with a
>>>     * matching interrupt ID and return a reference to the IRQ structure.
>>> @@ -97,6 +123,204 @@ void vgic_put_irq(struct domain *d, struct
>>> vgic_irq *irq)
>>>        xfree(irq);
>>>    }
>>>    +/**
>>> + * vgic_target_oracle - compute the target vcpu for an irq
>>> + *
>>> + * @irq:    The irq to route. Must be already locked.
>>> + *
>>> + * Based on the current state of the interrupt (enabled, pending,
>>> + * active, vcpu and target_vcpu), compute the next vcpu this should be
>>> + * given to. Return NULL if this shouldn't be injected at all.
>>> + *
>>> + * Requires the IRQ lock to be held.
>>> + */
>>> +static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
>>> +{
>>> +    ASSERT(spin_is_locked(&irq->irq_lock));
>>> +
>>> +    /* If the interrupt is active, it must stay on the current vcpu */
>>> +    if ( irq->active )
>>> +        return irq->vcpu ? : irq->target_vcpu;
>> I am not sure to understand why you check whether irq->vcpu is NULL. If
>> the interrupt is active, then irq->vcpu should be NULL. Did I miss
>                                                  ^
>                                                 not           you mean?

Yes not NULL.

>> anything?
> 
> Not if it has been explicitly activated via ISACTIVER. This is not
> implemented in Xen at the moment, but would be in the future. So I like
> to keep this in.

Oh, I missed that case. Thank you for the explanation :).

[...]

> 
> The list is not sorted on insertion because this is not necessary most
> of the time. In fact the hardware VGIC will do the sorting (kind of)
> within the LRs. So as long as we don't have more than <number of LRS>
> IRQs in the list, sorting is a waste of time. Experiments in the past
> showed that the number of used LRs is less than 4 almost every time. And
> since 4 is the mostly used number of implemented LRs, we virtually never
> need the sorting. So we avoid doing that on every insertion, instead
> doing that only if it's necessary.
> 
>> I can foresee quite a few issues with this choice on Xen:
>>      1) You compute the size of ap list in vgic_flush_lr_state() and take
>> lock on every IRQ one by one. A guest could be nasty and make that list
>> quite big by make IRQs pending but never "active" them (i.e read IAR).
> 
> Yeah, we could try to shortcut a bit here.
> 
>>      2) This might be an issue while checking whether you need to deliver
>> an interrupt (vgic_vcpu_pending_irq) because the list is not sorted.
> 
> Most of the time the list is very short, storing one, two or actually no
> IRQs. In the function where we check for pending IRQs we bail out as
> soon as we found the first eligible interrupt. So sorting does not help
> in the majority of cases.

As you say "most of the time". Malicious guest are unusual but just 
enough to keep busy both the hypervisor and the security team.

> 
> If you are really concerned about that list growing too long, we could
> think about mitigations:
> 1) Try to avoid iterating the whole list while checking whether it needs
> to be sorted.
> 2) Store a flag that notes if the list has already been sorted. As long
> as we don't change anything, we don't need to sort again. Would be good
> to test whether this is actually helpful. But we would need to keep this
> flag up-to-date, which sounds a bit fragile to get right.
> 3) Switch to sort-on-insertion once we reached a certain number of IRQs
> on the list, to mitigate DOS attacks from the guest. This should avoid
> list iterations in hot paths, with IRQs disabled.
> 
> But all of these sound a bit hackish to me and just would spoil the very
> clean and robust code we have today. Also I am not sure we can avoid
> list iterations every time (for instance in prune_ap_list()). There is
> an upper limit today (number of SPIs), so we might be happy with that
> for now.
> To be honest I would very much dislike changing the code at this point.
> I believe a patch series afterwards would be better, also to actually
> have some numbers on the impact of this.

More robust than the current vGIC yes. However, clean code is not an 
excuse to dismiss valid (but unusual) use case. So even if I quite like 
the new vGIC, I really don't want to deal with yet another security.

Thankfully passthrough case is not currently security supported (see 
SUPPORT.md). So we probably can defer, although I would like to keep 
track of known pitfalls of the new vGIC.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Andre Przywara]

Hi,

On 19/02/18 14:13, Julien Grall wrote:
> 
> 
> On 19/02/18 12:41, Andre Przywara wrote:
>> Hi,
> 
> Hi,
> 
>> On 16/02/18 16:57, Julien Grall wrote:
>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>> +    spin_lock_irqsave(&desc->lock, flags);
>>>> +    if ( enable )
>>>> +    {
>>>> +        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
>>>> +                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);
>>>
>>> Indentation and I would prefer a helper to convert between the vgic
>>> value and the IRQ_TYPE. This would make the code easier to read.
>>>
>>> Also, this code does not replicate correctly the current vGIC.
>>> gic_set_irq_type is only allowed to be used when
>>> irq_set_type_by_domain(d) returns true. If you consider this change
>>> valid, then I would like to know why.
>>
>> So what is/was the rationale for not allowing IRQ type changes for
>> non-privileged guests? If you allow to pass through an hardware IRQ to a
>> guest (which is the case this function handles), then I don't see why a
>> guest would not be allowed to change the configuration? It seems rather
>> odd, I guess it's up to the guest to know which type of IRQ this is?
> 
> If you can answer the question on top of irq_type_set_by_domain (i.e
> "See whether it is possible to let any domain configure the type) then
> we can remove it. We decided to only allow for the hardware domain
> because we trust it.

But why would you mistrust a DomU in this respect?
The only point I see is that a guest has *some* influence on a hardware
access, but I fail to see how a single MMIO read-modify-write sequence
would actually impact the host. Especially since we do it only on
enabling an IRQ.
Looking more closely at the existing VGIC code we might want to check if
the hardware IRQ was already enabled before entering the
"if ( p->desc != NULL )" branch, btw.

So is this the concern? Commit b0003bdd690 wasn't really enlightening in
this respect.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 30/49] ARM: new VGIC: Add ENABLE registers handlers

[Julien Grall]

On 27/02/18 13:54, Andre Przywara wrote:
> Hi,
> 
> On 19/02/18 14:13, Julien Grall wrote:
>>
>>
>> On 19/02/18 12:41, Andre Przywara wrote:
>>> Hi,
>>
>> Hi,
>>
>>> On 16/02/18 16:57, Julien Grall wrote:
>>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>>> +    spin_lock_irqsave(&desc->lock, flags);
>>>>> +    if ( enable )
>>>>> +    {
>>>>> +        gic_set_irq_type(desc, irq_type == VGIC_CONFIG_LEVEL ?
>>>>> +                 IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING);
>>>>
>>>> Indentation and I would prefer a helper to convert between the vgic
>>>> value and the IRQ_TYPE. This would make the code easier to read.
>>>>
>>>> Also, this code does not replicate correctly the current vGIC.
>>>> gic_set_irq_type is only allowed to be used when
>>>> irq_set_type_by_domain(d) returns true. If you consider this change
>>>> valid, then I would like to know why.
>>>
>>> So what is/was the rationale for not allowing IRQ type changes for
>>> non-privileged guests? If you allow to pass through an hardware IRQ to a
>>> guest (which is the case this function handles), then I don't see why a
>>> guest would not be allowed to change the configuration? It seems rather
>>> odd, I guess it's up to the guest to know which type of IRQ this is?
>>
>> If you can answer the question on top of irq_type_set_by_domain (i.e
>> "See whether it is possible to let any domain configure the type) then
>> we can remove it. We decided to only allow for the hardware domain
>> because we trust it.
> 
> But why would you mistrust a DomU in this respect?
> The only point I see is that a guest has *some* influence on a hardware
> access, but I fail to see how a single MMIO read-modify-write sequence
> would actually impact the host. Especially since we do it only on
> enabling an IRQ.
> Looking more closely at the existing VGIC code we might want to check if
> the hardware IRQ was already enabled before entering the
> "if ( p->desc != NULL )" branch, btw.

That's not an issue here. You can only enter in vgic_enable_irqs if the 
virtual interrupt was previously disabled. Because the physical 
interrupt is routed to the guest, it will also be disabled at that time.

> 
> So is this the concern? Commit b0003bdd690 wasn't really enlightening in
> this respect.

It was not really clear if it would be an issue when I wrote the patch. 
We trust the hardware domain so it is fine to let him configure the 
interrupt. For the guests, this will be taken from the DT (see 
gic_route_irq_to_guest). So this is likely to get configured correctly 
for the guest.

What I was worry about is whether we need to sanitize the ICFGR when the 
interrupt is routed to another domain. But if you can clear that, then I 
guess it should be ok.

However, I would prefer to do this in a separate patch and keep 
irq_type_set_by_domain around. This is to match the current vGIC and not 
changing too much Xen behavior.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 43/49] ARM: new VGIC: Add preliminary stub implementations

[Andre Przywara]

Hi,

On 19/02/18 12:34, Julien Grall wrote:
> Hi,
> 
> On 09/02/18 14:39, Andre Przywara wrote:
>> The Xen core code requires an interrupt controller emulation to implement
>> arch_move_irqs(), to move the affinity of an hardware mapped virtual IRQ
>> to another core. In the moment we don't implement this
>> physical-follow-virtual regime in our new VGIC, so just provide an empty
>> stub implementation to make the linker happy.
> 
> physical-follow-virtual is a must feature for the new vGIC. This has
> shown better interrupt latency.
> 
>> Similarily vgic_clear_pending_irqs() is required by the ARM code,
>> although it is suspected that it is actually not necessary. Go with a
>> stub for now.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/vgic/vgic.c | 13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index d91028bd43..77fa756329 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -770,6 +770,19 @@ void gic_dump_vgic_info(struct vcpu *v)
>>                  irq->active ? "" : "not ", irq->enabled ? "" : "not ");
>>   }
>>   +void vgic_clear_pending_irqs(struct vcpu *v)
>> +{
>> +    /*
>> +     * TODO: It is unclear whether we really need this, so we might
>> instead
>> +     * remove it on the caller site.
>> +     */
> 
> I remember some issue with the current vGIC when not removing pending
> interrupts on PSCI CPU ON. But that was a while ago. I will have another
> try and see if we can drop it.

So that's what I came up with:
CPU_ON can be called from two states:
- The initial state of a core before it's being brought up for the first
time.
- After the OS has called CPU_OFF.
 The PSCI spec says that before calling CPU_OFF an OS all IRQs must have
been migrated away from that core. I take this as no IRQs are allowed,
hence we don't have to clear anything on CPU_ON.

In both cases the CPU is expected to enter in a defined state, which
includes all interrupts masked on the CPU level (SPSR.ELx.[DAIF] = 1).
The GIC defaults to ISPENDR being 0.

So I take we should not be held responsible for clearing the pending
state upon CPU_ON.

What is your opinion?

>> +}
>> +
>> +void arch_move_irqs(struct vcpu *v)
>> +{
>> +    /* TODO: implement this (?) */
> 
> Here you would need to go through the interrupt and modify the affinity
> of the physical IRQs routed to that vCPU.

Since this is apparently the next best thing after sliced bread, I
implemented this now. Coming to a theatre near you any time soon.

> 
>> +}
>> +
>>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>>                                         unsigned int virq)
>>   {
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 08/49] ARM: VGIC: move max_vcpus VGIC limit to struct arch_domain

[Andre Przywara]

Hi,

On 09/02/18 19:27, Julien Grall wrote:
> Hi,
> 
> On 02/09/2018 02:38 PM, Andre Przywara wrote:
>> The VGIC model used for a domain (GICv2 or GICv3) determines the maximum
>> number of VCPUs for that guest, as GICv2 can only handle 8 processors.
>> In the moment we carry this per-VGIC-model limit in the vgic_ops,
>> alongside the model specific functions. That makes some sense, but
>> exposes some current VGIC implementation details to generic Xen code.
>> Add a new arch specific field in our domain structure to hold this
>> vcpu limit,
>> and initialize it when we set the ops. This allows us to plug in the new
>> VGIC later without also needing to carry some ops structure.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>>   xen/arch/arm/domain.c        | 5 ++---
>>   xen/arch/arm/vgic.c          | 3 ++-
>>   xen/include/asm-arm/domain.h | 1 +
>>   3 files changed, 5 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>> index a010443bfd..9ad4cd0a6e 100644
>> --- a/xen/arch/arm/domain.c
>> +++ b/xen/arch/arm/domain.c
>> @@ -975,11 +975,10 @@ unsigned int domain_max_vcpus(const struct
>> domain *d)
>>        * allocation when the vgic_ops haven't been initialised yet,
>>        * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
>>        */
>> -    if ( !d->arch.vgic.handler )
>> +    if ( !d->arch.max_vcpus )
>>           return MAX_VIRT_CPUS;
>>       else
>> -        return min_t(unsigned int, MAX_VIRT_CPUS,
>> -                     d->arch.vgic.handler->max_vcpus);
>> +        return d->arch.max_vcpus;
>>   }
>>     /*
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 9921769b15..5f47aa84a9 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -166,7 +166,8 @@ int domain_vgic_init(struct domain *d, unsigned
>> int nr_spis)
>>     void register_vgic_ops(struct domain *d, const struct vgic_ops *ops)
>>   {
>> -   d->arch.vgic.handler = ops;
>> +    d->arch.vgic.handler = ops;
>> +    d->arch.max_vcpus = ops->max_vcpus;
>>   }
>>     void domain_vgic_free(struct domain *d)
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 1dd9683d25..2fef32eaee 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -149,6 +149,7 @@ struct arch_domain
>>   #ifdef CONFIG_SBSA_VUART_CONSOLE
>>       struct vpl011 vpl011;
>>   #endif
>> +    unsigned int max_vcpus;
> 
> Now, you have max_vcpus defined in both arch_domain and domain. Which
> makes the code very confusing to read. This is becoming apparent in the
> check if (d->arch.max_vcpus > d->max_vcpus).

True, though I was just copying the name from the vgic_ops ;-)
I could suggest arch.vgic_vcpu_limit instead, but...

> If you plan to ditch the ops, then I would prefer a check on the vGIC
> version. Even if it means carrying vGIC specific implementation in
> generic Xen code.

So what about just providing per-VGIC implementations of
domain_max_vcpus()? That seems to be the only user of this information,
AFAICS? So I move this from xen/arch/arm/domain.c to
xen/arch/arm{/vgic,}/vgic.c, where we can do all kind of internal VGIC
tricks to learn this bit of information.

> This would also avoid to grow the struct domain just for a "constant
> field" used mostly at guest creation.

"Here’s a nickel, kid. Get yourself some memory." ;-)

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 08/49] ARM: VGIC: move max_vcpus VGIC limit to struct arch_domain

[Julien Grall]

Hi,

On 02/28/2018 12:32 PM, Andre Przywara wrote:
> Hi,
> 
> On 09/02/18 19:27, Julien Grall wrote:
>> Hi,
>>
>> On 02/09/2018 02:38 PM, Andre Przywara wrote:
>>> The VGIC model used for a domain (GICv2 or GICv3) determines the maximum
>>> number of VCPUs for that guest, as GICv2 can only handle 8 processors.
>>> In the moment we carry this per-VGIC-model limit in the vgic_ops,
>>> alongside the model specific functions. That makes some sense, but
>>> exposes some current VGIC implementation details to generic Xen code.
>>> Add a new arch specific field in our domain structure to hold this
>>> vcpu limit,
>>> and initialize it when we set the ops. This allows us to plug in the new
>>> VGIC later without also needing to carry some ops structure.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>>    xen/arch/arm/domain.c        | 5 ++---
>>>    xen/arch/arm/vgic.c          | 3 ++-
>>>    xen/include/asm-arm/domain.h | 1 +
>>>    3 files changed, 5 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
>>> index a010443bfd..9ad4cd0a6e 100644
>>> --- a/xen/arch/arm/domain.c
>>> +++ b/xen/arch/arm/domain.c
>>> @@ -975,11 +975,10 @@ unsigned int domain_max_vcpus(const struct
>>> domain *d)
>>>         * allocation when the vgic_ops haven't been initialised yet,
>>>         * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
>>>         */
>>> -    if ( !d->arch.vgic.handler )
>>> +    if ( !d->arch.max_vcpus )
>>>            return MAX_VIRT_CPUS;
>>>        else
>>> -        return min_t(unsigned int, MAX_VIRT_CPUS,
>>> -                     d->arch.vgic.handler->max_vcpus);
>>> +        return d->arch.max_vcpus;
>>>    }
>>>      /*
>>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>>> index 9921769b15..5f47aa84a9 100644
>>> --- a/xen/arch/arm/vgic.c
>>> +++ b/xen/arch/arm/vgic.c
>>> @@ -166,7 +166,8 @@ int domain_vgic_init(struct domain *d, unsigned
>>> int nr_spis)
>>>      void register_vgic_ops(struct domain *d, const struct vgic_ops *ops)
>>>    {
>>> -   d->arch.vgic.handler = ops;
>>> +    d->arch.vgic.handler = ops;
>>> +    d->arch.max_vcpus = ops->max_vcpus;
>>>    }
>>>      void domain_vgic_free(struct domain *d)
>>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>>> index 1dd9683d25..2fef32eaee 100644
>>> --- a/xen/include/asm-arm/domain.h
>>> +++ b/xen/include/asm-arm/domain.h
>>> @@ -149,6 +149,7 @@ struct arch_domain
>>>    #ifdef CONFIG_SBSA_VUART_CONSOLE
>>>        struct vpl011 vpl011;
>>>    #endif
>>> +    unsigned int max_vcpus;
>>
>> Now, you have max_vcpus defined in both arch_domain and domain. Which
>> makes the code very confusing to read. This is becoming apparent in the
>> check if (d->arch.max_vcpus > d->max_vcpus).
> 
> True, though I was just copying the name from the vgic_ops ;-)
> I could suggest arch.vgic_vcpu_limit instead, but...
> 
>> If you plan to ditch the ops, then I would prefer a check on the vGIC
>> version. Even if it means carrying vGIC specific implementation in
>> generic Xen code.
> 
> So what about just providing per-VGIC implementations of
> domain_max_vcpus()? That seems to be the only user of this information,
> AFAICS? So I move this from xen/arch/arm/domain.c to
> xen/arch/arm{/vgic,}/vgic.c, where we can do all kind of internal VGIC
> tricks to learn this bit of information.

Sounds good to me.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Andre Przywara]

Hi,

On 26/02/18 16:30, Julien Grall wrote:
> 
> 
> On 02/26/2018 04:25 PM, Andre Przywara wrote:
>> Hi,
>>
>> On 26/02/18 15:55, Julien Grall wrote:
>>> Hi,
>>>
>>> On 02/26/2018 03:29 PM, Andre Przywara wrote:
>>>> On 13/02/18 16:35, Julien Grall wrote:
>>>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>>>> index f4f2a04a60..9e7fb1edcb 100644
>>>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>>>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>>>>>         vgic_restore_state(current);
>>>>>>     }
>>>>>>     +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>>>>>> +{
>>>>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>>>>> +    struct vgic_irq *irq;
>>>>>> +    bool pending = false;
>>>>>> +    unsigned long flags;
>>>>>> +
>>>>>> +    if ( !vcpu->domain->arch.vgic.enabled )
>>>>>> +        return false;
>>>>>> +
>>>>>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>>>>>> +
>>>>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>>>>> +    {
>>>>>> +        spin_lock(&irq->irq_lock);
>>>>>> +        pending = irq_is_pending(irq) && irq->enabled;
>>>>>> +        spin_unlock(&irq->irq_lock);
>>>>>> +
>>>>>> +        if ( pending )
>>>>>> +            break;
>>>>>> +    }
>>>>>> +
>>>>>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>>>>>> +
>>>>>> +    return pending;
>>>>>> +}
>>>>>> +
>>>>>> +int gic_events_need_delivery(void)
>>>>>
>>>>> You probably want to rename that function or just expose
>>>>> vgic_vcpu_pending_irq().
>>>>
>>>> Rename to what? I need both functions: vgic_vcpu_pending_irq() is also
>>>> called by vgic_kick_vcpus() (later in the series).
>>>> And gic_events_need_delivery(void) is the interface that the arch code
>>>> expects. Shall I rename this there? To what?
>>>
>>> Let me start with it is a bit odd to have a function name 'gic_*' in the
>>> virtual GIC code. So at least renaming to vgic_events_need_delivery
>>> would be an improvement.
>>>
>>> Regarding the interface itself, it is ARM specific and not set in stone.
>>> It would not be too bad to use vgic_vcpu_pending_irq(current). Is there
>>> any reason for not doing that?

The two interfaces used for that purpose are different in the two VGICs:
- The old VGIC only works on the current VCPU, since it peeks into the
GICH_ register to learn the priority (regardless of whether this is
really needed or useful).
- The new VGIC can use this function for any VCPU, and we need this
functionality later on (when we iterate over all VCPUs).
So we can't use a function hardwiring "current", that would break
vgic_kick_vcpus() in the new VGIC. And we can't pass a VCPU parameter,
that would not work for the old VGIC.
So I believe having this small wrapper here is the easiest solution.
I will add a patch to rename this function to vgic_pending_irq(),
though, so this one here looks like:
int vgic_pending_irq(void)
{
    return vgic_vcpu_pending_irq(current);
}

We can clean this up when the old VGIC gets removed.

Cheers,
Andre.

>> Not really, but I am a bit reluctant to change too much original Xen
>> code, don't want to step on anyone's toes ;-)
> 
> The original code is going to get kill at some point. So better use name
> that makes sense in the new context. It is quite similar to the
> gic_inject change.
> 
>>
>> But if that's fine with you, I am OK with the renaming - though it adds
>> yet another patch ;-)
> 
> The end goal is a better world, so the number of patches does not matter
> here :).
> 
> If you put them at the beginning, we can merge them right away.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 26/49] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Julien Grall]

On 02/03/18 13:53, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 26/02/18 16:30, Julien Grall wrote:
>>
>>
>> On 02/26/2018 04:25 PM, Andre Przywara wrote:
>>> Hi,
>>>
>>> On 26/02/18 15:55, Julien Grall wrote:
>>>> Hi,
>>>>
>>>> On 02/26/2018 03:29 PM, Andre Przywara wrote:
>>>>> On 13/02/18 16:35, Julien Grall wrote:
>>>>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>>>>> index f4f2a04a60..9e7fb1edcb 100644
>>>>>>> --- a/xen/arch/arm/vgic/vgic.c
>>>>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>>>>> @@ -646,6 +646,38 @@ void gic_inject(void)
>>>>>>>          vgic_restore_state(current);
>>>>>>>      }
>>>>>>>      +static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
>>>>>>> +{
>>>>>>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic_cpu;
>>>>>>> +    struct vgic_irq *irq;
>>>>>>> +    bool pending = false;
>>>>>>> +    unsigned long flags;
>>>>>>> +
>>>>>>> +    if ( !vcpu->domain->arch.vgic.enabled )
>>>>>>> +        return false;
>>>>>>> +
>>>>>>> +    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
>>>>>>> +
>>>>>>> +    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
>>>>>>> +    {
>>>>>>> +        spin_lock(&irq->irq_lock);
>>>>>>> +        pending = irq_is_pending(irq) && irq->enabled;
>>>>>>> +        spin_unlock(&irq->irq_lock);
>>>>>>> +
>>>>>>> +        if ( pending )
>>>>>>> +            break;
>>>>>>> +    }
>>>>>>> +
>>>>>>> +    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
>>>>>>> +
>>>>>>> +    return pending;
>>>>>>> +}
>>>>>>> +
>>>>>>> +int gic_events_need_delivery(void)
>>>>>>
>>>>>> You probably want to rename that function or just expose
>>>>>> vgic_vcpu_pending_irq().
>>>>>
>>>>> Rename to what? I need both functions: vgic_vcpu_pending_irq() is also
>>>>> called by vgic_kick_vcpus() (later in the series).
>>>>> And gic_events_need_delivery(void) is the interface that the arch code
>>>>> expects. Shall I rename this there? To what?
>>>>
>>>> Let me start with it is a bit odd to have a function name 'gic_*' in the
>>>> virtual GIC code. So at least renaming to vgic_events_need_delivery
>>>> would be an improvement.
>>>>
>>>> Regarding the interface itself, it is ARM specific and not set in stone.
>>>> It would not be too bad to use vgic_vcpu_pending_irq(current). Is there
>>>> any reason for not doing that?
> 
> The two interfaces used for that purpose are different in the two VGICs:
> - The old VGIC only works on the current VCPU, since it peeks into the
> GICH_ register to learn the priority (regardless of whether this is
> really needed or useful).
> - The new VGIC can use this function for any VCPU, and we need this
> functionality later on (when we iterate over all VCPUs).
> So we can't use a function hardwiring "current", that would break
> vgic_kick_vcpus() in the new VGIC. And we can't pass a VCPU parameter,
> that would not work for the old VGIC.
> So I believe having this small wrapper here is the easiest solution.
> I will add a patch to rename this function to vgic_pending_irq(),
> though, so this one here looks like:
> int vgic_pending_irq(void)
> {
>      return vgic_vcpu_pending_irq(current);
> }
> 
> We can clean this up when the old VGIC gets removed.

Likely no-one in the old vGIC are going to call that function with v != 
current. This would not be the only place in Xen where a vCPU is taken 
in parameter but effectively v can only be current. That's where 
ASSERT(v == current) comes into place.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [RFC PATCH 31/49] ARM: new VGIC: Add PENDING registers handlers

[Andre Przywara]

Hi,

On 19/02/18 15:43, Julien Grall wrote:
> 
> 
> On 19/02/18 15:32, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 16/02/18 17:16, Julien Grall wrote:
>>> On 09/02/18 14:39, Andre Przywara wrote:
>>>> +
>>>> +    /* Loop over all IRQs affected by this read */
>>>> +    for ( i = 0; i < len * 8; i++ )
>>>> +    {
>>>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>>>> + i);
>>>> +
>>>> +        if ( irq_is_pending(irq) )
>>>> +            value |= (1U << i);
>>>
>>> Don't you need to propagate the value to the hardware for virtual
>>> interrupt mapped to physical interrupt?
>>
>> Do you mean in the write functions below? (This is the read function, I
>> don't see how this would apply here.)
> 
> Hmmm yes. Sorry I misplaced the comment.
> 
>>
>> In case you meant the write_[cs]pending() functions:
>> I don't think this makes too much sense. Why would you want to trigger
>> an hardware IRQ? All you want it is to deliver it to the guest, which is
>> what those functions below do. So what do I miss here?
> 
> Imagine you clear the pending bit on an hardware mapped interrupt. If
> you never clear the active bit on the physical one, you will never
> receive that interrupt again.
> 
> For setting pending bit, I am not entirely sure. But it looks like KVM
> is doing it (see latest master). So I am wondering why Xen is diverging
> here.

The simple reason is that "latest master" was something different when I
imported the VGIC, obviously. So this was a later addition. I now ported
this new patch over, though due to the locking order of the desc lock
this isn't so pretty (but still not too bad). But actually this function
should be extremely rare, up to the point that I currently cannot test
this easily.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel