[PATCH 00/57] New VGIC(-v2) implementation

[PATCH 00/57] New VGIC(-v2) implementation

[Andre Przywara]

tl;dr: Coarse changelog below, individual patches have changelogs as
well.

This is an updated version of the new VGIC-v2 implementation.
Compared to the RFC posted a month ago, many things have been changed to
address the review comments. The most important things are:
- The GICv3 redistributor cleanup patches at the beginning of this series
have been fixed to address the previous review comments.
- The "physical-follows-virtual" IRQ affinity functionality is now implemented.
- Trying to shrink data structures, namely struct vgic_irq.
- Removing not needed data structures and code stubs for parts dealing
with ITS or the CPU interface emulation.
- Renaming of some existing Xen function names to be more readable.
- Use existing LR accessor functions when updating/sycing LR content.
- Dumping extra save/restore_state functions when syncing to/from LRs.
- Fixes and adjustments to locking scheme.
- Improving ACTIVE MMIO handling, documenting limitations.
- Many minor changes to address whitespace issues, data types (uint32_t
vs. u32, unsigned vs. signed), extended comments and commit messages.

An summarising changelog can be found below, each individual patch has
its own changelog as well.

There are some things that have (still) not been covered yet:
- struct VCPU still allocates two pages now. We can either limit this to
ARM64 && the new VGIC, or try to look if we can allocate some parts of
struct vcpu instead of embedding sub-structures into it.
- vGICv3 support is not implemented, but should be fairly straight-forward to
add, as the design incorporated this already. Will look at this next.
- There is a possible DOS vector on the VCPU ap_list, which holds pending
vIRQs. A guest can make this list rather long, which forces the hypervisor
to hold the list lock when iterating the list. This should be bounded by
the number of emulated vIRQs though, and there are ideas how to mitigate
this issue. Those fixes would be posted on top as fixes later.
- There is no ITS support, though the VGIC code itself is more ready for that
than the old VGIC ever was. However due to differences between the Xen
and KVM architecture the ITS bits are not easy to port over to Xen.

Cheers,
Andre

=====================
During development of the Dom0 ITS MSI support last year we realised
that the existing GIC interrupt controller emulation has some shortcomings.
After some tries to fix those in the existing code, it was agreed upon
that the problems are fundamental and a new implementation based on the
"new VGIC" in KVM is the best choice.
This is the first drop of this new VGIC implementation. It lives in the
xen/arch/arm/vgic/ directory and is written to be a compile time option,
so people can choose whether to use the new VGIC or the existing
implementation. This is just for a transitional period, the old VGIC is
expected to be removed after confidence in the new implementation has grown.

This series starts with some GICv3 redistributor cleanup, which I posted
before. I need to incorporate the comments from the list, but for now I left
those patches as it from the previous post.

Starting with patch 07 there are some more cleanups and preparations for
the existing VGIC/GIC code. A big part of those patches are preparations to
properly support level triggered interrupts. This is one of the biggest
problems in the existing VGIC, which only correctly emulates edge triggered
IRQs. This affects both arch code and some users like the timer and the
event channel.

Starting with patch 27 we plumb in the new VGIC then. This is done in a
new directory, with all the files actually not wired into the build system
until the very last patch. The idea is to split the series into reviewable
chunks without resorting to nasty hacks to keep bisectability.
The code was forked from Linux' virt/kvm/arm/vgic/, as of 4.14-rc7, plus
some recent changes to improve support for level triggered and hardware
mapped interrupts, which is what we use heavily in Dom0. The code was
heavily adapted to fit into Xen, starting with using the Xen coding style
and using Xen structure and variable names (struct domain instead of
struct kvm, for instance). Where interfacing functions were similar enough,
they were changed over to the existing Xen name and prototypes (for instance
kvm_vgic_create() was renamed to domain_vgic_register()). As far as possible
the code layout and split was re-used from KVM, so patches in Linux should
be relatively easy to port into Xen. Due to the mentioned changes this can
not be done easily in an automatic way, but it should be not too complicated
to extract the gist of the patch and re-apply this to our code base.

The actual VGIC code splits into several parts:
- The core is the struct vgic_irq, which holds every information about a
virtual IRQ, including a per-IRQ lock. Also there is on (ordered) per-VCPU
list (ap_list), which links the interrupts to be considered by a VCPU.
There are functions to deal with queuing and removing IRQs from those lists
safely, obeying the locking order. (patches 27-30)
- There are functions to push vIRQs on a VCPU list to the list registers,
and handle their state changes. (patches 31-33)
- The distributor MMIO emulation is using separate functions per register,
also having read and write split. (patches 34-44)
- There are functions to deal with Xen specialities. (patches 45-51)
- The data structures and the wiring of the emulation into the hypervisor
  and the guests are done in vgic-init.c. (patches 52-55)
- Finally patch 57 enables the build of the new VGIC. This requires to
  increase the size limit for struct vcpu in patch 56.

Andre

Changelog RFC ... v1:
- observe review comments on GICv3 redistributor patches
- implement physical-follows-virtual IRQ affinity
- actually implement arch_move_irq()
- move max_domain_vcpus() into vgic.c, to make it VGIC specific
- improved many commit messages
- add ACKs so far
- added and extended many comments
- use C99 data types (uint32_t)
- use unsigned data types
- use symbolic names for constants
- white space fixes (indentation mostly)
- adapt later patches to changes earlier in the series (renames etc.)
- use 32 bit data types where sufficient
- add helper functions as requested (for instance gicv2/3_peek/poke_irq)
- use struct irq_desc * in interface of hardware facing functions
- rename some existing Xen function names to be more readable
- rename new header file from arm_vgic.h to new_vgic.h
- drop code or variables dealing with unimplemented features (ITS, CPU i/f)
- reorder struct vgic_irq and use bitfield to shrink data structure size
- remove not needed functions (gic_clear_lrs(), save/restore_state())
- add ASSERTS as requested
- add locking where missing (dump_vgic_info, read pending state, enabling GIC)
- keep Linux coding style for list_sort.c
- add set_pending_state() GIC abstraction function
- factor out and use kick_vcpu()
- use frame number instead of physical address
- use existing LR accessor functions, drop GICH_ accesses from vgic-v2.c
- skip already disabled/enabled IRQs and setting enabled state
- use PRODUCT_ID_XEN
- simplify and clarify on ACTIVE bit MMIO accesses
- use interface for HCR bit changes
- iterate over set CPU bits in SGI injection handler

Andre Przywara (57):
  tools: ARM: vGICv3: Avoid inserting optional DT properties
  ARM: vGICv3: clarify on GUEST_GICV3_RDIST_REGIONS symbol
  ARM: GICv3: use hardware GICv3 redistributor values for Dom0
  ARM: GICv3: simplify GICv3 redistributor stride handling
  ARM: vGICv3: always use architected redist stride
  ARM: vGICv3: remove rdist_stride from VGIC structure
  ARM: VGIC: rename gic_inject() and gic_clear_lrs()
  ARM: VGIC: Move gic_remove_from_lr_pending() prototype
  ARM: VGIC: Move domain_max_vcpus() to be VGIC specific
  ARM: VGIC: rename gic_event_needs_delivery()
  ARM: VGIC: change to level-IRQ compatible IRQ injection interface
  ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist
  ARM: VGIC: reorder prototypes in vgic.h
  ARM: VGIC: Introduce gic_get_nr_lrs()
  ARM: GICv2: Extend and adjust register definitions
  ARM: GICv3: rename HYP interface definitions to use ICH_ prefix
  ARM: Introduce kick_vcpu()
  ARM: GICv2: introduce gicv2_poke_irq()
  ARM: GICv3: poke_irq: make RWP optional
  ARM: GICv2: fix GICH_V2_LR definitions
  ARM: GICv2: extend LR read/write functions to cover EOI and source
  ARM: GIC: Allow tweaking the active and pending state of an IRQ
  ARM: GIC: allow reading pending state of a hardware IRQ
  ARM: timer: Handle level triggered IRQs correctly
  ARM: evtchn: Handle level triggered IRQs correctly
  ARM: vPL011: Use the VGIC's level triggered IRQs handling if available
  ARM: new VGIC: Add data structure definitions
  ARM: new VGIC: Add acccessor to new struct vgic_irq instance
  ARM: new VGIC: Implement virtual IRQ injection
  ARM: new VGIC: Add IRQ sorting
  ARM: new VGIC: Add IRQ sync/flush framework
  ARM: new VGIC: Add GICv2 world switch backend
  ARM: new VGIC: Implement vgic_vcpu_pending_irq
  ARM: new VGIC: Add MMIO handling framework
  ARM: new VGIC: Add GICv2 MMIO handling framework
  ARM: new VGIC: Add CTLR, TYPER and IIDR handlers
  ARM: new VGIC: Add ENABLE registers handlers
  ARM: new VGIC: Add PENDING registers handlers
  ARM: new VGIC: Add ACTIVE registers handlers
  ARM: new VGIC: Add PRIORITY registers handlers
  ARM: new VGIC: Add CONFIG registers handlers
  ARM: new VGIC: Add TARGET registers handlers
  ARM: new VGIC: Add SGIR register handler
  ARM: new VGIC: Add SGIPENDR register handlers
  ARM: new VGIC: Handle hardware mapped IRQs
  ARM: new VGIC: Add event channel IRQ handling
  ARM: new VGIC: Handle virtual IRQ allocation/reservation
  ARM: new VGIC: Dump virtual IRQ info
  ARM: new VGIC: provide system register emulation stub
  ARM: new VGIC: Implement arch_move_irqs()
  ARM: new VGIC: Add preliminary stub implementation
  ARM: new VGIC: vgic-init: register VGIC
  ARM: new VGIC: vgic-init: implement vgic_init
  ARM: new VGIC: vgic-init: implement map_resources
  ARM: new VGIC: Add vgic_v2_enable
  ARM: allocate two pages for struct vcpu
  ARM: VGIC: wire new VGIC(-v2) files into Xen build system

 tools/libxl/libxl_arm.c           |   8 -
 xen/arch/arm/Kconfig              |   6 +-
 xen/arch/arm/Makefile             |  10 +-
 xen/arch/arm/domain.c             |  34 +-
 xen/arch/arm/gic-v2.c             |  76 ++-
 xen/arch/arm/gic-v3-lpi.c         |   2 +-
 xen/arch/arm/gic-v3.c             | 147 ++++--
 xen/arch/arm/gic-vgic.c           |  16 +-
 xen/arch/arm/gic.c                |  15 +
 xen/arch/arm/irq.c                |   2 +-
 xen/arch/arm/smp.c                |  14 +
 xen/arch/arm/time.c               |  38 +-
 xen/arch/arm/traps.c              |  11 +-
 xen/arch/arm/vgic-v3.c            |  40 +-
 xen/arch/arm/vgic.c               |  67 ++-
 xen/arch/arm/vgic/vgic-init.c     | 256 ++++++++++
 xen/arch/arm/vgic/vgic-mmio-v2.c  | 317 ++++++++++++
 xen/arch/arm/vgic/vgic-mmio.c     | 651 +++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h     | 148 ++++++
 xen/arch/arm/vgic/vgic-v2.c       | 303 ++++++++++++
 xen/arch/arm/vgic/vgic.c          | 996 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h          |  78 +++
 xen/arch/arm/vpl011.c             |   6 +-
 xen/arch/arm/vtimer.c             |   4 +-
 xen/common/Makefile               |   1 +
 xen/common/list_sort.c            | 157 ++++++
 xen/include/asm-arm/domain.h      |  87 +---
 xen/include/asm-arm/event.h       |   3 +-
 xen/include/asm-arm/gic.h         |  55 ++-
 xen/include/asm-arm/gic_v3_defs.h |  54 ++-
 xen/include/asm-arm/new_vgic.h    | 198 ++++++++
 xen/include/asm-arm/smp.h         |   3 +
 xen/include/asm-arm/vgic.h        | 150 +++++-
 xen/include/public/arch-arm.h     |   1 -
 xen/include/xen/list_sort.h       |  11 +
 xen/include/xen/timer.h           |   2 +
 36 files changed, 3666 insertions(+), 301 deletions(-)
 create mode 100644 xen/arch/arm/vgic/vgic-init.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.h
 create mode 100644 xen/arch/arm/vgic/vgic-v2.c
 create mode 100644 xen/arch/arm/vgic/vgic.c
 create mode 100644 xen/arch/arm/vgic/vgic.h
 create mode 100644 xen/common/list_sort.c
 create mode 100644 xen/include/asm-arm/new_vgic.h
 create mode 100644 xen/include/xen/list_sort.h

-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 01/57] tools: ARM: vGICv3: Avoid inserting optional DT properties

[Andre Przywara]

When creating a GICv3 devicetree node, we currently insert the
redistributor-stride and #redistributor-regions properties, with fixed
values which are actually the architected ones. Since those properties are
optional, and in the case of the stride only needed to cover for broken
platforms, we don't need to describe them if they don't differ from the
default values. This will always be the case for our constructed
DomU memory map.
So we drop those properties altogether and provide a clean and architected
GICv3 DT node for DomUs.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- improve commit message

 tools/libxl/libxl_arm.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
index 86f59c0d80..906fd0dcdf 100644
--- a/tools/libxl/libxl_arm.c
+++ b/tools/libxl/libxl_arm.c
@@ -525,14 +525,6 @@ static int make_gicv3_node(libxl__gc *gc, void *fdt)
     res = fdt_property(fdt, "interrupt-controller", NULL, 0);
     if (res) return res;
 
-    res = fdt_property_cell(fdt, "redistributor-stride",
-                            GUEST_GICV3_RDIST_STRIDE);
-    if (res) return res;
-
-    res = fdt_property_cell(fdt, "#redistributor-regions",
-                            GUEST_GICV3_RDIST_REGIONS);
-    if (res) return res;
-
     res = fdt_property_regs(gc, fdt, ROOT_ADDRESS_CELLS, ROOT_SIZE_CELLS,
                             2,
                             gicd_base, gicd_size,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 02/57] ARM: vGICv3: clarify on GUEST_GICV3_RDIST_REGIONS symbol

[Andre Przywara]

Normally there is only one GICv3 redistributor region, and we use
that for DomU guests using a GICv3.
Explain the background in a comment and why we need to keep the number
of hardware regions for Dom0.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- Keep GUEST_GICV3_RDIST_REGIONS symbol around, just extend comments

 xen/arch/arm/vgic-v3.c | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 2ad8a6be62..d5b34a7d0f 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1632,6 +1632,16 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
 
 static inline unsigned int vgic_v3_rdist_count(struct domain *d)
 {
+    /*
+     * Normally there is only one GICv3 redistributor region.
+     * The GICv3 DT binding provisions for multiple regions, since there are
+     * platforms out there which need those (multi-socket systems).
+     * For Dom0 we have to live with the MMIO layout the hardware provides,
+     * so we have to copy the multiple regions - as the first region may not
+     * provide enough space to hold all redistributors we need.
+     * However DomU get a constructed memory map, so we can go with
+     * the architected single redistributor region.
+     */
     return is_hardware_domain(d) ? vgic_v3_hw.nr_rdist_regions :
                GUEST_GICV3_RDIST_REGIONS;
 }
@@ -1692,7 +1702,7 @@ static int vgic_v3_domain_init(struct domain *d)
     {
         d->arch.vgic.dbase = GUEST_GICV3_GICD_BASE;
 
-        /* XXX: Only one Re-distributor region mapped for the guest */
+        /* A single Re-distributor region is mapped for the guest. */
         BUILD_BUG_ON(GUEST_GICV3_RDIST_REGIONS != 1);
 
         d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 03/57] ARM: GICv3: use hardware GICv3 redistributor values for Dom0

[Andre Przywara]

The code to generate the DT node or MADT table for Dom0 reaches into the
domain's VGIC structure to learn the number of redistributor regions and
their base addresses.
Since those values are copied from the hardware, we can as well use
those hardware values directly when setting up the hardware domain.

This avoids the hardware GIC code to reference vGIC data structures.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- Use GIC hardware values consistently in this function.

 xen/arch/arm/gic-v3.c | 17 +++++++----------
 1 file changed, 7 insertions(+), 10 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 25c30bb9ea..b1f8a86409 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1162,13 +1162,11 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
     if ( res )
         return res;
 
-    res = fdt_property_cell(fdt, "redistributor-stride",
-                            d->arch.vgic.rdist_stride);
+    res = fdt_property_cell(fdt, "redistributor-stride", gicv3.rdist_stride);
     if ( res )
         return res;
 
-    res = fdt_property_cell(fdt, "#redistributor-regions",
-                            d->arch.vgic.nr_regions);
+    res = fdt_property_cell(fdt, "#redistributor-regions", gicv3.rdist_count);
     if ( res )
         return res;
 
@@ -1178,7 +1176,7 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
      * CPU interface and virtual cpu interfaces accessesed as System registers
      * So cells are created only for Distributor and rdist regions
      */
-    new_len = new_len * (d->arch.vgic.nr_regions + 1);
+    new_len = new_len * (gicv3.rdist_count + 1);
 
     hw_reg = dt_get_property(gic, "reg", &len);
     if ( !hw_reg )
@@ -1406,13 +1404,13 @@ static int gicv3_make_hwdom_madt(const struct domain *d, u32 offset)
 
     /* Add Generic Redistributor */
     size = sizeof(struct acpi_madt_generic_redistributor);
-    for ( i = 0; i < d->arch.vgic.nr_regions; i++ )
+    for ( i = 0; i < gicv3.rdist_count; i++ )
     {
         gicr = (struct acpi_madt_generic_redistributor *)(base_ptr + table_len);
         gicr->header.type = ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR;
         gicr->header.length = size;
-        gicr->base_address = d->arch.vgic.rdist_regions[i].base;
-        gicr->length = d->arch.vgic.rdist_regions[i].size;
+        gicr->base_address = gicv3.rdist_regions[i].base;
+        gicr->length = gicv3.rdist_regions[i].size;
         table_len += size;
     }
 
@@ -1425,8 +1423,7 @@ static unsigned long gicv3_get_hwdom_extra_madt_size(const struct domain *d)
 {
     unsigned long size;
 
-    size = sizeof(struct acpi_madt_generic_redistributor)
-           * d->arch.vgic.nr_regions;
+    size = sizeof(struct acpi_madt_generic_redistributor) * gicv3.rdist_count;
 
     size += sizeof(struct acpi_madt_generic_translator)
             * vgic_v3_its_count(d);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 04/57] ARM: GICv3: simplify GICv3 redistributor stride handling

[Andre Przywara]

Instead of hard coding the architected redistributor stride into the
code, lets use a clear #define to the two values for GICv3 and GICv4 and
clarify the algorithm to determine the needed stride value.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/gic-v3.c             | 18 ++++++++++--------
 xen/include/asm-arm/gic_v3_defs.h |  5 +++++
 2 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index b1f8a86409..be1787b39a 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -690,6 +690,15 @@ static int __init gicv3_populate_rdist(void)
         do {
             typer = readq_relaxed(ptr + GICR_TYPER);
 
+            /* Set the architectural redist size if not overridden by DT. */
+            if ( !gicv3.rdist_stride )
+            {
+                if ( typer & GICR_TYPER_VLPIS )
+                    gicv3.rdist_stride = GICV4_GICR_SIZE;
+                else
+                    gicv3.rdist_stride = GICV3_GICR_SIZE;
+            }
+
             if ( (typer >> 32) == aff )
             {
                 this_cpu(rbase) = ptr;
@@ -732,14 +741,7 @@ static int __init gicv3_populate_rdist(void)
             if ( gicv3.rdist_regions[i].single_rdist )
                 break;
 
-            if ( gicv3.rdist_stride )
-                ptr += gicv3.rdist_stride;
-            else
-            {
-                ptr += SZ_64K * 2; /* Skip RD_base + SGI_base */
-                if ( typer & GICR_TYPER_VLPIS )
-                    ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */
-            }
+            ptr += gicv3.rdist_stride;
 
         } while ( !(typer & GICR_TYPER_LAST) );
     }
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
index 65c9dc47cf..412e41afed 100644
--- a/xen/include/asm-arm/gic_v3_defs.h
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -18,6 +18,8 @@
 #ifndef __ASM_ARM_GIC_V3_DEFS_H__
 #define __ASM_ARM_GIC_V3_DEFS_H__
 
+#include <xen/sizes.h>
+
 /*
  * Additional registers defined in GIC v3.
  * Common GICD registers are defined in gic.h
@@ -68,6 +70,9 @@
 #define GICV3_GICD_IIDR_VAL          0x34c
 #define GICV3_GICR_IIDR_VAL          GICV3_GICD_IIDR_VAL
 
+#define GICV3_GICR_SIZE              (2 * SZ_64K)
+#define GICV4_GICR_SIZE              (4 * SZ_64K)
+
 #define GICR_CTLR                    (0x0000)
 #define GICR_IIDR                    (0x0004)
 #define GICR_TYPER                   (0x0008)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 05/57] ARM: vGICv3: always use architected redist stride

[Andre Przywara]

The redistributor-stride property in a GICv3 DT node is only there to
cover broken platforms where this value deviates from the architected one.
Since we emulate the GICv3 distributor even for Dom0, we don't need to
copy the broken behaviour. All the special handling for Dom0s using
GICv3 is just for using the hardware's memory map, which is unaffected
by the redistributor stride - it can never be smaller than the
architected two pages.
Remove the redistributor-stride property from Dom0's DT node and also
remove the code that tried to reuse the hardware value for Dom0's GICv3
emulation.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
Acked-by: Julien Grall <julien.grall@linaro.org>
---
Changelog RFC ... v1:
- Add Julien's ACK

 xen/arch/arm/gic-v3.c  |  4 ----
 xen/arch/arm/vgic-v3.c | 14 ++++++--------
 2 files changed, 6 insertions(+), 12 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index be1787b39a..02c85e4c0c 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1164,10 +1164,6 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
     if ( res )
         return res;
 
-    res = fdt_property_cell(fdt, "redistributor-stride", gicv3.rdist_stride);
-    if ( res )
-        return res;
-
     res = fdt_property_cell(fdt, "#redistributor-regions", gicv3.rdist_count);
     if ( res )
         return res;
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index d5b34a7d0f..56cc38ffcc 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1024,10 +1024,9 @@ static struct vcpu *get_vcpu_from_rdist(struct domain *d,
     paddr_t gpa, uint32_t *offset)
 {
     struct vcpu *v;
-    uint32_t stride = d->arch.vgic.rdist_stride;
     unsigned int vcpu_id;
 
-    vcpu_id = region->first_cpu + ((gpa - region->base) / stride);
+    vcpu_id = region->first_cpu + ((gpa - region->base) / GICV3_GICR_SIZE);
     if ( unlikely(vcpu_id >= d->max_vcpus) )
         return NULL;
 
@@ -1586,7 +1585,6 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
 
     /* Convenient alias */
     struct domain *d = v->domain;
-    uint32_t rdist_stride = d->arch.vgic.rdist_stride;
 
     /*
      * Find the region where the re-distributor lives. For this purpose,
@@ -1602,11 +1600,11 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
 
     /* Get the base address of the redistributor */
     rdist_base = region->base;
-    rdist_base += (v->vcpu_id - region->first_cpu) * rdist_stride;
+    rdist_base += (v->vcpu_id - region->first_cpu) * GICV3_GICR_SIZE;
 
     /* Check if a valid region was found for the re-distributor */
     if ( (rdist_base < region->base) ||
-         ((rdist_base + rdist_stride) > (region->base + region->size)) )
+         ((rdist_base + GICV3_GICR_SIZE) > (region->base + region->size)) )
     {
         dprintk(XENLOG_ERR,
                 "d%u: Unable to find a re-distributor for VCPU %u\n",
@@ -1622,7 +1620,7 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
      * VGIC_V3_RDIST_LAST flags.
      * Note that we are assuming max_vcpus will never change.
      */
-    last_cpu = (region->size / rdist_stride) + region->first_cpu - 1;
+    last_cpu = (region->size / GICV3_GICR_SIZE) + region->first_cpu - 1;
 
     if ( v->vcpu_id == last_cpu || (v->vcpu_id == (d->max_vcpus - 1)) )
         v->arch.vgic.flags |= VGIC_V3_RDIST_LAST;
@@ -1693,7 +1691,7 @@ static int vgic_v3_domain_init(struct domain *d)
             /* Set the first CPU handled by this region */
             d->arch.vgic.rdist_regions[i].first_cpu = first_cpu;
 
-            first_cpu += size / d->arch.vgic.rdist_stride;
+            first_cpu += size / GICV3_GICR_SIZE;
         }
 
         d->arch.vgic.intid_bits = vgic_v3_hw.intid_bits;
@@ -1708,7 +1706,7 @@ static int vgic_v3_domain_init(struct domain *d)
         d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
 
         /* The first redistributor should contain enough space for all CPUs */
-        BUILD_BUG_ON((GUEST_GICV3_GICR0_SIZE / GUEST_GICV3_RDIST_STRIDE) < MAX_VIRT_CPUS);
+        BUILD_BUG_ON((GUEST_GICV3_GICR0_SIZE / GICV3_GICR_SIZE) < MAX_VIRT_CPUS);
         d->arch.vgic.rdist_regions[0].base = GUEST_GICV3_GICR0_BASE;
         d->arch.vgic.rdist_regions[0].size = GUEST_GICV3_GICR0_SIZE;
         d->arch.vgic.rdist_regions[0].first_cpu = 0;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 06/57] ARM: vGICv3: remove rdist_stride from VGIC structure

[Andre Przywara]

The last patch removed the usage of the hardware's redistributor-stride
value from our (Dom0) GICv3 emulation. This means we no longer need to
store this value in the VGIC data structure.
Remove that variable and every code snippet that handled that, instead
simply always use the architected value.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
Acked-by: Julien Grall <julien.grall@linaro.org>
---
Changelog RFC ... v1:
- Add Julien's ACK

 xen/arch/arm/gic-v3.c         |  3 +--
 xen/arch/arm/vgic-v3.c        | 14 --------------
 xen/include/asm-arm/domain.h  |  1 -
 xen/include/asm-arm/vgic.h    |  1 -
 xen/include/public/arch-arm.h |  1 -
 5 files changed, 1 insertion(+), 19 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 02c85e4c0c..ea14ab4028 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1682,8 +1682,7 @@ static int __init gicv3_init(void)
     reg = readl_relaxed(GICD + GICD_TYPER);
     intid_bits = GICD_TYPE_ID_BITS(reg);
 
-    vgic_v3_setup_hw(dbase, gicv3.rdist_count, gicv3.rdist_regions,
-                     gicv3.rdist_stride, intid_bits);
+    vgic_v3_setup_hw(dbase, gicv3.rdist_count, gicv3.rdist_regions, intid_bits);
     gicv3_init_v2();
 
     spin_lock_init(&gicv3.lock);
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 56cc38ffcc..4b42739a52 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -58,21 +58,18 @@ static struct {
     /* Re-distributor regions */
     unsigned int nr_rdist_regions;
     const struct rdist_region *regions;
-    uint32_t rdist_stride; /* Re-distributor stride */
     unsigned int intid_bits;  /* Number of interrupt ID bits */
 } vgic_v3_hw;
 
 void vgic_v3_setup_hw(paddr_t dbase,
                       unsigned int nr_rdist_regions,
                       const struct rdist_region *regions,
-                      uint32_t rdist_stride,
                       unsigned int intid_bits)
 {
     vgic_v3_hw.enabled = true;
     vgic_v3_hw.dbase = dbase;
     vgic_v3_hw.nr_rdist_regions = nr_rdist_regions;
     vgic_v3_hw.regions = regions;
-    vgic_v3_hw.rdist_stride = rdist_stride;
     vgic_v3_hw.intid_bits = intid_bits;
 }
 
@@ -1672,15 +1669,6 @@ static int vgic_v3_domain_init(struct domain *d)
 
         d->arch.vgic.dbase = vgic_v3_hw.dbase;
 
-        d->arch.vgic.rdist_stride = vgic_v3_hw.rdist_stride;
-        /*
-         * If the stride is not set, the default stride for GICv3 is 2 * 64K:
-         *     - first 64k page for Control and Physical LPIs
-         *     - second 64k page for Control and Generation of SGIs
-         */
-        if ( !d->arch.vgic.rdist_stride )
-            d->arch.vgic.rdist_stride = 2 * SZ_64K;
-
         for ( i = 0; i < vgic_v3_hw.nr_rdist_regions; i++ )
         {
             paddr_t size = vgic_v3_hw.regions[i].size;
@@ -1703,8 +1691,6 @@ static int vgic_v3_domain_init(struct domain *d)
         /* A single Re-distributor region is mapped for the guest. */
         BUILD_BUG_ON(GUEST_GICV3_RDIST_REGIONS != 1);
 
-        d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
-
         /* The first redistributor should contain enough space for all CPUs */
         BUILD_BUG_ON((GUEST_GICV3_GICR0_SIZE / GICV3_GICR_SIZE) < MAX_VIRT_CPUS);
         d->arch.vgic.rdist_regions[0].base = GUEST_GICV3_GICR0_BASE;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 4fe189b1c3..3eda7196ff 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -108,7 +108,6 @@ struct arch_domain
             unsigned int first_cpu;         /* First CPU handled */
         } *rdist_regions;
         int nr_regions;                     /* Number of rdist regions */
-        uint32_t rdist_stride;              /* Re-Distributor stride */
         unsigned long int nr_lpis;
         uint64_t rdist_propbase;
         struct rb_root its_devices;         /* Devices mapped to an ITS */
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 6ea9f140a7..d61b54867b 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -261,7 +261,6 @@ struct rdist_region;
 void vgic_v3_setup_hw(paddr_t dbase,
                       unsigned int nr_rdist_regions,
                       const struct rdist_region *regions,
-                      uint32_t rdist_stride,
                       unsigned int intid_bits);
 #endif
 
diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h
index 05fd11ca38..eb424e8286 100644
--- a/xen/include/public/arch-arm.h
+++ b/xen/include/public/arch-arm.h
@@ -401,7 +401,6 @@ typedef uint64_t xen_callback_t;
 #define GUEST_GICV3_GICD_BASE      xen_mk_ullong(0x03001000)
 #define GUEST_GICV3_GICD_SIZE      xen_mk_ullong(0x00010000)
 
-#define GUEST_GICV3_RDIST_STRIDE   xen_mk_ullong(0x00020000)
 #define GUEST_GICV3_RDIST_REGIONS  1
 
 #define GUEST_GICV3_GICR0_BASE     xen_mk_ullong(0x03020000) /* vCPU0..127 */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 07/57] ARM: VGIC: rename gic_inject() and gic_clear_lrs()

[Andre Przywara]

The two central functions to synchronise our emulated VGIC state with
the GIC hardware (the LRs, really), are named somewhat confusingly.
Rename them from gic_inject() to vgic_sync_to_lrs() and from
gic_clear_lrs() to vgic_sync_from_lrs(), to make the code more readable.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/gic-vgic.c   | 4 ++--
 xen/arch/arm/traps.c      | 4 ++--
 xen/include/asm-arm/gic.h | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index d273863556..c0fe38fd37 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -247,7 +247,7 @@ static void gic_update_one_lr(struct vcpu *v, int i)
     }
 }
 
-void gic_clear_lrs(struct vcpu *v)
+void vgic_sync_from_lrs(struct vcpu *v)
 {
     int i = 0;
     unsigned long flags;
@@ -377,7 +377,7 @@ out:
     return rc;
 }
 
-void gic_inject(void)
+void vgic_sync_to_lrs(void)
 {
     ASSERT(!local_irq_is_enabled());
 
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 1cba7e584d..7411bff7a7 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2024,7 +2024,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
         if ( current->arch.hcr_el2 & HCR_VA )
             current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
 
-        gic_clear_lrs(current);
+        vgic_sync_from_lrs(current);
     }
 }
 
@@ -2234,7 +2234,7 @@ void leave_hypervisor_tail(void)
     {
         local_irq_disable();
         if (!softirq_pending(smp_processor_id())) {
-            gic_inject();
+            vgic_sync_to_lrs();
 
             /*
              * If the SErrors handle option is "DIVERSE", we have to prevent
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 497f195bc1..e2ae4254ed 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -237,7 +237,7 @@ extern int gic_route_irq_to_guest(struct domain *, unsigned int virq,
 int gic_remove_irq_from_guest(struct domain *d, unsigned int virq,
                               struct irq_desc *desc);
 
-extern void gic_inject(void);
+extern void vgic_sync_to_lrs(void);
 extern void gic_clear_pending_irqs(struct vcpu *v);
 extern int gic_events_need_delivery(void);
 
@@ -295,7 +295,7 @@ extern unsigned int gic_number_lines(void);
 /* IRQ translation function for the device tree */
 int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
                   unsigned int *out_hwirq, unsigned int *out_type);
-void gic_clear_lrs(struct vcpu *v);
+void vgic_sync_from_lrs(struct vcpu *v);
 
 struct gic_info {
     /* GIC version */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 08/57] ARM: VGIC: Move gic_remove_from_lr_pending() prototype

[Andre Przywara]

The prototype for gic_remove_from_lr_pending() is the last function in
gic.h which references a VGIC data structure.
Move it over to vgic.h, so that we can remove the inclusion of vgic.h
from gic.h. We add it to asm/domain.h instead, where it is actually
needed.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
Reviewed-by: Julien Grall <julien.grall@arm.com>
---
Changelog RFC ... v1:
- Add Julien's Reviewed-by:

 xen/include/asm-arm/domain.h | 1 +
 xen/include/asm-arm/gic.h    | 2 --
 xen/include/asm-arm/vgic.h   | 1 +
 3 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 3eda7196ff..1dd9683d25 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -8,6 +8,7 @@
 #include <asm/vfp.h>
 #include <asm/mmio.h>
 #include <asm/gic.h>
+#include <asm/vgic.h>
 #include <public/hvm/params.h>
 #include <xen/serial.h>
 #include <xen/rbtree.h>
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index e2ae4254ed..3b2d0217a6 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -156,7 +156,6 @@
 #ifndef __ASSEMBLY__
 #include <xen/device_tree.h>
 #include <xen/irq.h>
-#include <asm-arm/vgic.h>
 
 #define DT_COMPAT_GIC_CORTEX_A15 "arm,cortex-a15-gic"
 
@@ -245,7 +244,6 @@ extern void init_maintenance_interrupt(void);
 extern void gic_raise_guest_irq(struct vcpu *v, unsigned int irq,
         unsigned int priority);
 extern void gic_raise_inflight_irq(struct vcpu *v, unsigned int virtual_irq);
-extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 
 /* Accept an interrupt from the GIC and dispatch its handler */
 extern void gic_interrupt(struct cpu_user_regs *regs, int is_fiq);
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index d61b54867b..d03298e12c 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -205,6 +205,7 @@ extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
 extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
 extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
 extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
+extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
 extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
 extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 09/57] ARM: VGIC: Move domain_max_vcpus() to be VGIC specific

[Andre Przywara]

domain_max_vcpus(), which is used by generic Xen code, returns the
maximum number of VCPUs for a domain, which on ARM is mostly limited by
the VGIC model emulated (a (v)GICv2 can only handle 8 CPUs).
Our current implementation lives in arch/arm/domain.c, but reaches into
VGIC internal data structures.
Move this function into vgic.c, to keep this VGIC internal.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- dump previous approach, move function to VGIC specific file instead

 xen/arch/arm/domain.c | 14 --------------
 xen/arch/arm/vgic.c   | 14 ++++++++++++++
 2 files changed, 14 insertions(+), 14 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index a010443bfd..8546443bad 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -968,20 +968,6 @@ void vcpu_block_unless_event_pending(struct vcpu *v)
         vcpu_unblock(current);
 }
 
-unsigned int domain_max_vcpus(const struct domain *d)
-{
-    /*
-     * Since evtchn_init would call domain_max_vcpus for poll_mask
-     * allocation when the vgic_ops haven't been initialised yet,
-     * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
-     */
-    if ( !d->arch.vgic.handler )
-        return MAX_VIRT_CPUS;
-    else
-        return min_t(unsigned int, MAX_VIRT_CPUS,
-                     d->arch.vgic.handler->max_vcpus);
-}
-
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 34269bcf27..c3fdcebbde 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -665,6 +665,20 @@ void vgic_free_virq(struct domain *d, unsigned int virq)
     clear_bit(virq, d->arch.vgic.allocated_irqs);
 }
 
+unsigned int domain_max_vcpus(const struct domain *d)
+{
+    /*
+     * Since evtchn_init would call domain_max_vcpus for poll_mask
+     * allocation when the vgic_ops haven't been initialised yet,
+     * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
+     */
+    if ( !d->arch.vgic.handler )
+        return MAX_VIRT_CPUS;
+    else
+        return min_t(unsigned int, MAX_VIRT_CPUS,
+                     d->arch.vgic.handler->max_vcpus);
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 10/57] ARM: VGIC: rename gic_event_needs_delivery()

[Andre Przywara]

gic_event_needs_delivery() is not named very intuitively, especially
the gic_ prefix is somewhat misleading.
Rename it to vgic_pending_irq(), which makes it clear that this relates
to the virtual GIC and is about interrupts.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/gic-vgic.c     | 2 +-
 xen/include/asm-arm/event.h | 2 +-
 xen/include/asm-arm/gic.h   | 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index c0fe38fd37..60c6c463e9 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -339,7 +339,7 @@ void gic_clear_pending_irqs(struct vcpu *v)
         gic_remove_from_lr_pending(v, p);
 }
 
-int gic_events_need_delivery(void)
+int vgic_pending_irq(void)
 {
     struct vcpu *v = current;
     struct pending_irq *p;
diff --git a/xen/include/asm-arm/event.h b/xen/include/asm-arm/event.h
index e8c2a6cb44..c4c79fa87d 100644
--- a/xen/include/asm-arm/event.h
+++ b/xen/include/asm-arm/event.h
@@ -24,7 +24,7 @@ static inline int local_events_need_delivery_nomask(void)
      * interrupts disabled so this shouldn't be a problem in the general
      * case.
      */
-    if ( gic_events_need_delivery() )
+    if ( vgic_pending_irq() )
         return 1;
 
     if ( !vcpu_info(current, evtchn_upcall_pending) )
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 3b2d0217a6..a23c307c3a 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -238,7 +238,7 @@ int gic_remove_irq_from_guest(struct domain *d, unsigned int virq,
 
 extern void vgic_sync_to_lrs(void);
 extern void gic_clear_pending_irqs(struct vcpu *v);
-extern int gic_events_need_delivery(void);
+extern int vgic_pending_irq(void);
 
 extern void init_maintenance_interrupt(void);
 extern void gic_raise_guest_irq(struct vcpu *v, unsigned int irq,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 11/57] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Andre Przywara]

At the moment vgic_vcpu_inject_irq() is the interface for Xen internal
code and virtual devices to inject IRQs into a guest. This interface has
two shortcomings:
1) It requires a VCPU pointer, which we may not know (and don't need!)
for shared interrupts. A second function (vgic_vcpu_inject_spi()), was
there to work around this issue.
2) This interface only really supports edge triggered IRQs, which is
what the Xen VGIC emulates only anyway. However this needs to and will
change, so we need to add the desired level (high or low) to the
interface.
This replaces the existing injection call (taking a VCPU and an IRQ
parameter) with a new one, taking domain, VCPU, IRQ and level parameters.
The VCPU can be NULL in case we don't know and don't care.
We change all call sites to use this new interface. This still doesn't
give us the missing level IRQ handling, but at least prepares the callers
to do the right thing later automatically.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no change

 xen/arch/arm/domain.c      |  4 ++--
 xen/arch/arm/gic-v3-lpi.c  |  2 +-
 xen/arch/arm/irq.c         |  2 +-
 xen/arch/arm/time.c        |  2 +-
 xen/arch/arm/vgic.c        | 43 +++++++++++++++++++++++++------------------
 xen/arch/arm/vpl011.c      |  2 +-
 xen/arch/arm/vtimer.c      |  4 ++--
 xen/include/asm-arm/vgic.h |  4 ++--
 8 files changed, 35 insertions(+), 28 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 8546443bad..a7bba3ad44 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -952,14 +952,14 @@ void vcpu_mark_events_pending(struct vcpu *v)
     if ( already_pending )
         return;
 
-    vgic_vcpu_inject_irq(v, v->domain->arch.evtchn_irq);
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
 }
 
 /* The ARM spec declares that even if local irqs are masked in
  * the CPSR register, an irq should wake up a cpu from WFI anyway.
  * For this reason we need to check for irqs that need delivery,
  * ignoring the CPSR register, *after* calling SCHEDOP_block to
- * avoid races with vgic_vcpu_inject_irq.
+ * avoid races with vgic_inject_irq.
  */
 void vcpu_block_unless_event_pending(struct vcpu *v)
 {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 84582157b8..efd5cd62fb 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -153,7 +153,7 @@ void vgic_vcpu_inject_lpi(struct domain *d, unsigned int virq)
     if ( vcpu_id >= d->max_vcpus )
           return;
 
-    vgic_vcpu_inject_irq(d->vcpu[vcpu_id], virq);
+    vgic_inject_irq(d, d->vcpu[vcpu_id], virq, true);
 }
 
 /*
diff --git a/xen/arch/arm/irq.c b/xen/arch/arm/irq.c
index 29af10e82c..aa4e832cae 100644
--- a/xen/arch/arm/irq.c
+++ b/xen/arch/arm/irq.c
@@ -225,7 +225,7 @@ void do_IRQ(struct cpu_user_regs *regs, unsigned int irq, int is_fiq)
          * The irq cannot be a PPI, we only support delivery of SPIs to
          * guests.
 	 */
-        vgic_vcpu_inject_spi(info->d, info->virq);
+        vgic_inject_irq(info->d, NULL, info->virq, true);
         goto out_no_end;
     }
 
diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index 36f640f0c1..c11fcfeadd 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -260,7 +260,7 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
 
     current->arch.virt_timer.ctl = READ_SYSREG32(CNTV_CTL_EL0);
     WRITE_SYSREG32(current->arch.virt_timer.ctl | CNTx_CTL_MASK, CNTV_CTL_EL0);
-    vgic_vcpu_inject_irq(current, current->arch.virt_timer.irq);
+    vgic_inject_irq(current->domain, current, current->arch.virt_timer.irq, true);
 }
 
 /*
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index c3fdcebbde..3c77d5fef6 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -291,7 +291,7 @@ bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq)
         vgic_remove_irq_from_queues(old, p);
         irq_set_affinity(p->desc, cpumask_of(new->processor));
         spin_unlock_irqrestore(&old->arch.vgic.lock, flags);
-        vgic_vcpu_inject_irq(new, irq);
+        vgic_inject_irq(new->domain, new, irq, true);
         return true;
     }
     /* if the IRQ is in a GICH_LR register, set GIC_IRQ_GUEST_MIGRATING
@@ -450,7 +450,7 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
                         sgir, target->list);
                 continue;
             }
-            vgic_vcpu_inject_irq(d->vcpu[vcpuid], virq);
+            vgic_inject_irq(d, d->vcpu[vcpuid], virq, true);
         }
         break;
     case SGI_TARGET_OTHERS:
@@ -459,12 +459,12 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
         {
             if ( i != current->vcpu_id && d->vcpu[i] != NULL &&
                  is_vcpu_online(d->vcpu[i]) )
-                vgic_vcpu_inject_irq(d->vcpu[i], virq);
+                vgic_inject_irq(d, d->vcpu[i], virq, true);
         }
         break;
     case SGI_TARGET_SELF:
         perfc_incr(vgic_sgi_self);
-        vgic_vcpu_inject_irq(d->vcpu[current->vcpu_id], virq);
+        vgic_inject_irq(d, current, virq, true);
         break;
     default:
         gprintk(XENLOG_WARNING,
@@ -524,13 +524,29 @@ void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p)
     gic_remove_from_lr_pending(v, p);
 }
 
-void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
+int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                    bool level)
 {
     uint8_t priority;
     struct pending_irq *iter, *n;
     unsigned long flags;
     bool running;
 
+    /*
+     * For edge triggered interrupts we always ignore a "falling edge".
+     * For level triggered interrupts we shouldn't, but do anyways.
+     */
+    if ( !level )
+        return 0;
+
+    if ( !v )
+    {
+        /* The IRQ needs to be an SPI if no vCPU is specified. */
+        ASSERT(virq >= 32 && virq <= vgic_num_irqs(d));
+
+        v = vgic_get_target_vcpu(d->vcpu[0], virq);
+    };
+
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
 
     n = irq_to_pending(v, virq);
@@ -538,14 +554,14 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
     if ( unlikely(!n) )
     {
         spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
-        return;
+        return 0;
     }
 
     /* vcpu offline */
     if ( test_bit(_VPF_down, &v->pause_flags) )
     {
         spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
-        return;
+        return 0;
     }
 
     set_bit(GIC_IRQ_GUEST_QUEUED, &n->status);
@@ -582,22 +598,13 @@ out:
         perfc_incr(vgic_cross_cpu_intr_inject);
         smp_send_event_check_mask(cpumask_of(v->processor));
     }
-}
-
-void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq)
-{
-    struct vcpu *v;
 
-    /* the IRQ needs to be an SPI */
-    ASSERT(virq >= 32 && virq <= vgic_num_irqs(d));
-
-    v = vgic_get_target_vcpu(d->vcpu[0], virq);
-    vgic_vcpu_inject_irq(v, virq);
+    return 0;
 }
 
 void arch_evtchn_inject(struct vcpu *v)
 {
-    vgic_vcpu_inject_irq(v, v->domain->arch.evtchn_irq);
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
 }
 
 bool vgic_evtchn_irq_pending(struct vcpu *v)
diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 7788c2fc32..5dcf4bec18 100644
--- a/xen/arch/arm/vpl011.c
+++ b/xen/arch/arm/vpl011.c
@@ -68,7 +68,7 @@ static void vpl011_update_interrupt_status(struct domain *d)
      * status bit has been set since the last time.
      */
     if ( uartmis & ~vpl011->shadow_uartmis )
-        vgic_vcpu_inject_spi(d, GUEST_VPL011_SPI);
+        vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, true);
 
     vpl011->shadow_uartmis = uartmis;
 }
diff --git a/xen/arch/arm/vtimer.c b/xen/arch/arm/vtimer.c
index f52a723a5f..8164f6c7f1 100644
--- a/xen/arch/arm/vtimer.c
+++ b/xen/arch/arm/vtimer.c
@@ -46,7 +46,7 @@ static void phys_timer_expired(void *data)
     if ( !(t->ctl & CNTx_CTL_MASK) )
     {
         perfc_incr(vtimer_phys_inject);
-        vgic_vcpu_inject_irq(t->v, t->irq);
+        vgic_inject_irq(t->v->domain, t->v, t->irq, true);
     }
     else
         perfc_incr(vtimer_phys_masked);
@@ -56,7 +56,7 @@ static void virt_timer_expired(void *data)
 {
     struct vtimer *t = data;
     t->ctl |= CNTx_CTL_MASK;
-    vgic_vcpu_inject_irq(t->v, t->irq);
+    vgic_inject_irq(t->v->domain, t->v, t->irq, true);
     perfc_incr(vtimer_virt_inject);
 }
 
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index d03298e12c..b75fdeb068 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -202,8 +202,8 @@ extern int domain_vgic_init(struct domain *d, unsigned int nr_spis);
 extern void domain_vgic_free(struct domain *d);
 extern int vcpu_vgic_init(struct vcpu *v);
 extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
-extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
-extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
+extern int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                           bool level);
 extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
 extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 12/57] ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist

[Andre Przywara]

Currently we describe the VGIC specific fields in a structure
*embedded* in struct arch_domain and struct arch_vcpu. These members
there are however related to the current VGIC implementation, and will
be substantially different in the future.
To allow coexistence of two implementations, move the definition of these
embedded structures into vgic.h, and just use the opaque type in the arch
specific structures.
This allows easy switching between different implementations later.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- No changes

 xen/include/asm-arm/domain.h | 85 +-----------------------------------------
 xen/include/asm-arm/vgic.h   | 88 ++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 90 insertions(+), 83 deletions(-)

diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 1dd9683d25..bb7a46c1d0 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -74,57 +74,7 @@ struct arch_domain
         uint64_t offset;
     } virt_timer_base;
 
-    struct {
-        /* Version of the vGIC */
-        enum gic_version version;
-        /* GIC HW version specific vGIC driver handler */
-        const struct vgic_ops *handler;
-        /*
-         * Covers access to other members of this struct _except_ for
-         * shared_irqs where each member contains its own locking.
-         *
-         * If both class of lock is required then this lock must be
-         * taken first. If multiple rank locks are required (including
-         * the per-vcpu private_irqs rank) then they must be taken in
-         * rank order.
-         */
-        spinlock_t lock;
-        uint32_t ctlr;
-        int nr_spis; /* Number of SPIs */
-        unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
-        struct vgic_irq_rank *shared_irqs;
-        /*
-         * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
-         * struct arch_vcpu.
-         */
-        struct pending_irq *pending_irqs;
-        /* Base address for guest GIC */
-        paddr_t dbase; /* Distributor base address */
-#ifdef CONFIG_HAS_GICV3
-        /* GIC V3 addressing */
-        /* List of contiguous occupied by the redistributors */
-        struct vgic_rdist_region {
-            paddr_t base;                   /* Base address */
-            paddr_t size;                   /* Size */
-            unsigned int first_cpu;         /* First CPU handled */
-        } *rdist_regions;
-        int nr_regions;                     /* Number of rdist regions */
-        unsigned long int nr_lpis;
-        uint64_t rdist_propbase;
-        struct rb_root its_devices;         /* Devices mapped to an ITS */
-        spinlock_t its_devices_lock;        /* Protects the its_devices tree */
-        struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
-        rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
-        struct list_head vits_list;         /* List of virtual ITSes */
-        unsigned int intid_bits;
-        /*
-         * TODO: if there are more bool's being added below, consider
-         * a flags variable instead.
-         */
-        bool rdists_enabled;                /* Is any redistributor enabled? */
-        bool has_its;
-#endif
-    } vgic;
+    struct vgic_dist vgic;
 
     struct vuart {
 #define VUART_BUF_SIZE 128
@@ -247,38 +197,7 @@ struct arch_vcpu
     union gic_state_data gic;
     uint64_t lr_mask;
 
-    struct {
-        /*
-         * SGIs and PPIs are per-VCPU, SPIs are domain global and in
-         * struct arch_domain.
-         */
-        struct pending_irq pending_irqs[32];
-        struct vgic_irq_rank *private_irqs;
-
-        /* This list is ordered by IRQ priority and it is used to keep
-         * track of the IRQs that the VGIC injected into the guest.
-         * Depending on the availability of LR registers, the IRQs might
-         * actually be in an LR, and therefore injected into the guest,
-         * or queued in gic.lr_pending.
-         * As soon as an IRQ is EOI'd by the guest and removed from the
-         * corresponding LR it is also removed from this list. */
-        struct list_head inflight_irqs;
-        /* lr_pending is used to queue IRQs (struct pending_irq) that the
-         * vgic tried to inject in the guest (calling gic_set_guest_irq) but
-         * no LRs were available at the time.
-         * As soon as an LR is freed we remove the first IRQ from this
-         * list and write it to the LR register.
-         * lr_pending is a subset of vgic.inflight_irqs. */
-        struct list_head lr_pending;
-        spinlock_t lock;
-
-        /* GICv3: redistributor base and flags for this vCPU */
-        paddr_t rdist_base;
-        uint64_t rdist_pendbase;
-#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
-#define VGIC_V3_LPIS_ENABLED    (1 << 1)
-        uint8_t flags;
-    } vgic;
+    struct vgic_cpu vgic;
 
     /* Timer registers  */
     uint32_t cntkctl;
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index b75fdeb068..4e1c37f091 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -19,6 +19,9 @@
 #define __ASM_ARM_VGIC_H__
 
 #include <xen/bitops.h>
+#include <xen/radix-tree.h>
+#include <xen/rbtree.h>
+#include <asm/gic.h>
 #include <asm/mmio.h>
 #include <asm/vreg.h>
 
@@ -123,6 +126,91 @@ struct vgic_irq_rank {
     uint8_t vcpu[32];
 };
 
+struct vgic_dist {
+    /* Version of the vGIC */
+    enum gic_version version;
+    /* GIC HW version specific vGIC driver handler */
+    const struct vgic_ops *handler;
+    /*
+     * Covers access to other members of this struct _except_ for
+     * shared_irqs where each member contains its own locking.
+     *
+     * If both class of lock is required then this lock must be
+     * taken first. If multiple rank locks are required (including
+     * the per-vcpu private_irqs rank) then they must be taken in
+     * rank order.
+     */
+    spinlock_t lock;
+    uint32_t ctlr;
+    int nr_spis; /* Number of SPIs */
+    unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
+    struct vgic_irq_rank *shared_irqs;
+    /*
+     * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
+     * struct arch_vcpu.
+     */
+    struct pending_irq *pending_irqs;
+    /* Base address for guest GIC */
+    paddr_t dbase; /* Distributor base address */
+#ifdef CONFIG_HAS_GICV3
+    /* GIC V3 addressing */
+    /* List of contiguous occupied by the redistributors */
+    struct vgic_rdist_region {
+        paddr_t base;                   /* Base address */
+        paddr_t size;                   /* Size */
+        unsigned int first_cpu;         /* First CPU handled */
+    } *rdist_regions;
+    int nr_regions;                     /* Number of rdist regions */
+    unsigned long int nr_lpis;
+    uint64_t rdist_propbase;
+    struct rb_root its_devices;         /* Devices mapped to an ITS */
+    spinlock_t its_devices_lock;        /* Protects the its_devices tree */
+    struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
+    rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
+    struct list_head vits_list;         /* List of virtual ITSes */
+    unsigned int intid_bits;
+    /*
+     * TODO: if there are more bool's being added below, consider
+     * a flags variable instead.
+     */
+    bool rdists_enabled;                /* Is any redistributor enabled? */
+    bool has_its;
+#endif
+};
+
+struct vgic_cpu {
+    /*
+     * SGIs and PPIs are per-VCPU, SPIs are domain global and in
+     * struct arch_domain.
+     */
+    struct pending_irq pending_irqs[32];
+    struct vgic_irq_rank *private_irqs;
+
+    /* This list is ordered by IRQ priority and it is used to keep
+     * track of the IRQs that the VGIC injected into the guest.
+     * Depending on the availability of LR registers, the IRQs might
+     * actually be in an LR, and therefore injected into the guest,
+     * or queued in gic.lr_pending.
+     * As soon as an IRQ is EOI'd by the guest and removed from the
+     * corresponding LR it is also removed from this list. */
+    struct list_head inflight_irqs;
+    /* lr_pending is used to queue IRQs (struct pending_irq) that the
+     * vgic tried to inject in the guest (calling gic_set_guest_irq) but
+     * no LRs were available at the time.
+     * As soon as an LR is freed we remove the first IRQ from this
+     * list and write it to the LR register.
+     * lr_pending is a subset of vgic.inflight_irqs. */
+    struct list_head lr_pending;
+    spinlock_t lock;
+
+    /* GICv3: redistributor base and flags for this vCPU */
+    paddr_t rdist_base;
+    uint64_t rdist_pendbase;
+#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
+#define VGIC_V3_LPIS_ENABLED    (1 << 1)
+    uint8_t flags;
+};
+
 struct sgi_target {
     uint8_t aff1;
     uint16_t list;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 13/57] ARM: VGIC: reorder prototypes in vgic.h

[Andre Przywara]

Currently vgic.h both contains prototypes used by Xen arch code outside
of the actual VGIC (for instance vgic_vcpu_inject_irq()), and prototypes
for functions used by the VGIC internally.
Group them to later allow an easy split with one #ifdef.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
Reviewed-by: Julien Grall <julien.grall@arm.com>
---
Changelog RFC ... v1:
- Remove two redundant prototypes
- Add Julien's Reviewed-by:

 xen/include/asm-arm/vgic.h | 54 +++++++++++++++++++++++++---------------------
 1 file changed, 30 insertions(+), 24 deletions(-)

diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 4e1c37f091..84d82e6eb3 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -279,49 +279,34 @@ enum gic_sgi_mode;
  */
 #define REG_RANK_INDEX(b, n, s) ((((n) >> s) & ((b)-1)) % 32)
 
-/*
- * In the moment vgic_num_irqs() just covers SPIs and the private IRQs,
- * as it's mostly used for allocating the pending_irq and irq_desc array,
- * in which LPIs don't participate.
- */
-#define vgic_num_irqs(d)        ((d)->arch.vgic.nr_spis + 32)
 
-extern int domain_vgic_init(struct domain *d, unsigned int nr_spis);
-extern void domain_vgic_free(struct domain *d);
-extern int vcpu_vgic_init(struct vcpu *v);
 extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
-extern int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
-                           bool level);
 extern void vgic_remove_irq_from_queues(struct vcpu *v, struct pending_irq *p);
 extern void gic_remove_from_lr_pending(struct vcpu *v, struct pending_irq *p);
-extern void vgic_clear_pending_irqs(struct vcpu *v);
 extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
 extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
 extern struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq);
 extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v, int b, int n, int s);
 extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
-extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
 extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
 extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
 extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
 int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
 
-bool vgic_evtchn_irq_pending(struct vcpu *v);
-struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
-                                      unsigned int virq);
-int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
-                        struct irq_desc *desc, bool connect);
-
-extern int domain_vgic_register(struct domain *d, int *mmio_count);
-extern int vcpu_vgic_free(struct vcpu *v);
 extern bool vgic_to_sgi(struct vcpu *v, register_t sgir,
                         enum gic_sgi_mode irqmode, int virq,
                         const struct sgi_target *target);
 extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq);
 
-/* Reserve a specific guest vIRQ */
-extern bool vgic_reserve_virq(struct domain *d, unsigned int virq);
+/*** Common VGIC functions used by Xen arch code ****/
+
+/*
+ * In the moment vgic_num_irqs() just covers SPIs and the private IRQs,
+ * as it's mostly used for allocating the pending_irq and irq_desc array,
+ * in which LPIs don't participate.
+ */
+#define vgic_num_irqs(d)        ((d)->arch.vgic.nr_spis + 32)
 
 /*
  * Allocate a guest VIRQ
@@ -329,6 +314,9 @@ extern bool vgic_reserve_virq(struct domain *d, unsigned int virq);
  *  - spi == 1 => allocate an SPI
  */
 extern int vgic_allocate_virq(struct domain *d, bool spi);
+/* Reserve a specific guest vIRQ */
+extern bool vgic_reserve_virq(struct domain *d, unsigned int virq);
+extern void vgic_free_virq(struct domain *d, unsigned int virq);
 
 static inline int vgic_allocate_ppi(struct domain *d)
 {
@@ -340,7 +328,25 @@ static inline int vgic_allocate_spi(struct domain *d)
     return vgic_allocate_virq(d, true /* spi */);
 }
 
-extern void vgic_free_virq(struct domain *d, unsigned int virq);
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq);
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                        struct irq_desc *desc, bool connect);
+
+bool vgic_evtchn_irq_pending(struct vcpu *v);
+
+int domain_vgic_register(struct domain *d, int *mmio_count);
+int domain_vgic_init(struct domain *d, unsigned int nr_spis);
+void domain_vgic_free(struct domain *d);
+int vcpu_vgic_init(struct vcpu *vcpu);
+int vcpu_vgic_free(struct vcpu *vcpu);
+
+int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
+                    bool level);
+
+extern void vgic_clear_pending_irqs(struct vcpu *v);
+
+extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
 
 void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
                       paddr_t vbase, uint32_t aliased_offset);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 14/57] ARM: VGIC: Introduce gic_get_nr_lrs()

[Andre Przywara]

So far the number of list registers (LRs) a GIC implements is only
needed in the hardware facing side of the VGIC code (gic-vgic.c).
The new VGIC will need this information in more and multiple places, so
export a function that returns the number.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- move gic_get_nr_lrs() into gic.h (as a static inline)

 xen/arch/arm/gic-vgic.c   | 10 +++++-----
 xen/include/asm-arm/gic.h |  6 ++++++
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
index 60c6c463e9..93e42739d9 100644
--- a/xen/arch/arm/gic-vgic.c
+++ b/xen/arch/arm/gic-vgic.c
@@ -25,7 +25,7 @@
 #include <asm/gic.h>
 #include <asm/vgic.h>
 
-#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->info->nr_lrs) - 1))
+#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_get_nr_lrs()) - 1))
 
 #undef GIC_DEBUG
 
@@ -110,7 +110,7 @@ static unsigned int gic_find_unused_lr(struct vcpu *v,
                                        struct pending_irq *p,
                                        unsigned int lr)
 {
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
     unsigned long *lr_mask = (unsigned long *) &this_cpu(lr_mask);
     struct gic_lr lr_val;
 
@@ -137,7 +137,7 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq,
         unsigned int priority)
 {
     int i;
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
     struct pending_irq *p = irq_to_pending(v, virtual_irq);
 
     ASSERT(spin_is_locked(&v->arch.vgic.lock));
@@ -251,7 +251,7 @@ void vgic_sync_from_lrs(struct vcpu *v)
 {
     int i = 0;
     unsigned long flags;
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
 
     /* The idle domain has no LRs to be cleared. Since gic_restore_state
      * doesn't write any LR registers for the idle domain they could be
@@ -278,7 +278,7 @@ static void gic_restore_pending_irqs(struct vcpu *v)
     struct pending_irq *p, *t, *p_r;
     struct list_head *inflight_r;
     unsigned long flags;
-    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
+    unsigned int nr_lrs = gic_get_nr_lrs();
     int lrs = nr_lrs;
 
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index a23c307c3a..b3f840ea9a 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -374,6 +374,12 @@ struct gic_hw_operations {
 };
 
 extern const struct gic_hw_operations *gic_hw_ops;
+
+static inline unsigned int gic_get_nr_lrs(void)
+{
+    return gic_hw_ops->info->nr_lrs;
+}
+
 void register_gic_ops(const struct gic_hw_operations *ops);
 int gic_make_hwdom_dt_node(const struct domain *d,
                            const struct dt_device_node *gic,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 15/57] ARM: GICv2: Extend and adjust register definitions

[Andre Przywara]

The new VGIC will shortly use more bits of the various GIC registers, so
add the respective definitions from the manual.
This includes bits from the GICC_CTL register and some minor other bits.
Adjust the usage of ICC_CTL_ENABLE on the way, to be more precise about
which of the two enable bits we actually deal with.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- extend commit message

 xen/arch/arm/gic-v2.c     |  2 +-
 xen/include/asm-arm/gic.h | 18 ++++++++++++++++--
 2 files changed, 17 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 2b271ba322..7938a42591 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -358,7 +358,7 @@ static void gicv2_cpu_init(void)
     /* Finest granularity of priority */
     writel_gicc(0x0, GICC_BPR);
     /* Turn on delivery */
-    writel_gicc(GICC_CTL_ENABLE|GICC_CTL_EOI, GICC_CTLR);
+    writel_gicc(GICC_CTL_ENABLE0|GICC_CTL_EOI, GICC_CTLR);
 }
 
 static void gicv2_cpu_disable(void)
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index b3f840ea9a..8fab458d7f 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -77,6 +77,7 @@
 #define GICC_EOIR       (0x0010)
 #define GICC_RPR        (0x0014)
 #define GICC_HPPIR      (0x0018)
+#define GICC_ABPR       (0x001c)
 #define GICC_APR        (0x00D0)
 #define GICC_NSAPR      (0x00E0)
 #define GICC_IIDR       (0x00FC)
@@ -102,8 +103,18 @@
 #define GICD_TYPE_SEC   0x400
 #define GICD_TYPER_DVIS (1U << 18)
 
-#define GICC_CTL_ENABLE 0x1
-#define GICC_CTL_EOI    (0x1 << 9)
+#define GICC_CTL_ENABLE0_SHIFT  0
+#define GICC_CTL_ENABLE0        (1U << GICC_CTL_ENABLE0_SHIFT)
+#define GICC_CTL_ENABLE1_SHIFT  1
+#define GICC_CTL_ENABLE1        (1U << GICC_CTL_ENABLE1)
+#define GICC_CTL_AC_SHIFT       2
+#define GICC_CTL_AC             (1U << GICC_CTL_AC_SHIFT)
+#define GICC_CTL_FIQEN_SHIFT    3
+#define GICC_CTL_FIQEN          (1U << GICC_CTL_FIQEN_SHIFT)
+#define GICC_CTL_CBPR_SHIFT     4
+#define GICC_CTL_CBPR           (1U << GICC_CTL_CBPR_SHIFT)
+#define GICC_CTL_EOI_SHIFT      9
+#define GICC_CTL_EOI            (1U << GICC_CTL_EOI_SHIFT)
 
 #define GICC_IA_IRQ       0x03ff
 #define GICC_IA_CPU_MASK  0x1c00
@@ -127,6 +138,9 @@
 #define GICH_MISR_VGRP1E  (1 << 6)
 #define GICH_MISR_VGRP1D  (1 << 7)
 
+#define GICV_PMR_PRIORITY_SHIFT		3
+#define GICV_PMR_PRIORITY_MASK		(0x1f << GICV_PMR_PRIORITY_SHIFT)
+
 /*
  * The minimum GICC_BPR is required to be in the range 0-3. We set
  * GICC_BPR to 0 but we must expect that it might be 3. This means we
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 16/57] ARM: GICv3: rename HYP interface definitions to use ICH_ prefix

[Andre Przywara]

On a GICv3 in non-compat mode the hypervisor interface is always
accessed via system registers. Those register names have a "ICH_" prefix
in the manual, to differentiate them from the MMIO registers. Also those
registers are mostly 64-bit (compared to the 32-bit GICv2 registers) and
use different bit assignments.
To make this obvious and to avoid clashes with double definitions using
the same names for actually different bits, lets change all GICv3
hypervisor interface registers to use the "ICH_" prefix from the manual.
This renames the definitions in gic_v3_defs.h and their usage in gic-v3.c
and is needed to allow co-existence of the GICv2 and GICv3 definitions
in the same file.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/gic-v3.c             | 48 +++++++++++++++++++-------------------
 xen/include/asm-arm/gic_v3_defs.h | 49 +++++++++++++++++++--------------------
 2 files changed, 48 insertions(+), 49 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index ea14ab4028..3e381d031b 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -830,14 +830,14 @@ static void gicv3_hyp_init(void)
     uint32_t vtr;
 
     vtr = READ_SYSREG32(ICH_VTR_EL2);
-    gicv3_info.nr_lrs  = (vtr & GICH_VTR_NRLRGS) + 1;
-    gicv3.nr_priorities = ((vtr >> GICH_VTR_PRIBITS_SHIFT) &
-                          GICH_VTR_PRIBITS_MASK) + 1;
+    gicv3_info.nr_lrs  = (vtr & ICH_VTR_NRLRGS) + 1;
+    gicv3.nr_priorities = ((vtr >> ICH_VTR_PRIBITS_SHIFT) &
+                          ICH_VTR_PRIBITS_MASK) + 1;
 
     if ( !((gicv3.nr_priorities > 4) && (gicv3.nr_priorities < 8)) )
         panic("GICv3: Invalid number of priority bits\n");
 
-    WRITE_SYSREG32(GICH_VMCR_EOI | GICH_VMCR_VENG1, ICH_VMCR_EL2);
+    WRITE_SYSREG32(ICH_VMCR_EOI | ICH_VMCR_VENG1, ICH_VMCR_EL2);
     WRITE_SYSREG32(GICH_HCR_EN, ICH_HCR_EL2);
 }
 
@@ -976,21 +976,21 @@ static void gicv3_update_lr(int lr, unsigned int virq, uint8_t priority,
     BUG_ON(lr >= gicv3_info.nr_lrs);
     BUG_ON(lr < 0);
 
-    val =  (((uint64_t)state & 0x3) << GICH_LR_STATE_SHIFT);
+    val =  (((uint64_t)state & 0x3) << ICH_LR_STATE_SHIFT);
 
     /*
      * When the guest is GICv3, all guest IRQs are Group 1, as Group0
      * would result in a FIQ in the guest, which it wouldn't expect
      */
     if ( current->domain->arch.vgic.version == GIC_V3 )
-        val |= GICH_LR_GRP1;
+        val |= ICH_LR_GRP1;
 
-    val |= (uint64_t)priority << GICH_LR_PRIORITY_SHIFT;
-    val |= ((uint64_t)virq & GICH_LR_VIRTUAL_MASK) << GICH_LR_VIRTUAL_SHIFT;
+    val |= (uint64_t)priority << ICH_LR_PRIORITY_SHIFT;
+    val |= ((uint64_t)virq & ICH_LR_VIRTUAL_MASK) << ICH_LR_VIRTUAL_SHIFT;
 
    if ( hw_irq != INVALID_IRQ )
-       val |= GICH_LR_HW | (((uint64_t)hw_irq & GICH_LR_PHYSICAL_MASK)
-                           << GICH_LR_PHYSICAL_SHIFT);
+       val |= ICH_LR_HW | (((uint64_t)hw_irq & ICH_LR_PHYSICAL_MASK)
+                           << ICH_LR_PHYSICAL_SHIFT);
 
     gicv3_ich_write_lr(lr, val);
 }
@@ -1006,25 +1006,25 @@ static void gicv3_read_lr(int lr, struct gic_lr *lr_reg)
 
     lrv = gicv3_ich_read_lr(lr);
 
-    lr_reg->pirq = (lrv >> GICH_LR_PHYSICAL_SHIFT) & GICH_LR_PHYSICAL_MASK;
-    lr_reg->virq = (lrv >> GICH_LR_VIRTUAL_SHIFT) & GICH_LR_VIRTUAL_MASK;
+    lr_reg->pirq = (lrv >> ICH_LR_PHYSICAL_SHIFT) & ICH_LR_PHYSICAL_MASK;
+    lr_reg->virq = (lrv >> ICH_LR_VIRTUAL_SHIFT) & ICH_LR_VIRTUAL_MASK;
 
-    lr_reg->priority  = (lrv >> GICH_LR_PRIORITY_SHIFT) & GICH_LR_PRIORITY_MASK;
-    lr_reg->state     = (lrv >> GICH_LR_STATE_SHIFT) & GICH_LR_STATE_MASK;
-    lr_reg->hw_status = (lrv >> GICH_LR_HW_SHIFT) & GICH_LR_HW_MASK;
-    lr_reg->grp       = (lrv >> GICH_LR_GRP_SHIFT) & GICH_LR_GRP_MASK;
+    lr_reg->priority  = (lrv >> ICH_LR_PRIORITY_SHIFT) & ICH_LR_PRIORITY_MASK;
+    lr_reg->state     = (lrv >> ICH_LR_STATE_SHIFT) & ICH_LR_STATE_MASK;
+    lr_reg->hw_status = (lrv >> ICH_LR_HW_SHIFT) & ICH_LR_HW_MASK;
+    lr_reg->grp       = (lrv >> ICH_LR_GRP_SHIFT) & ICH_LR_GRP_MASK;
 }
 
 static void gicv3_write_lr(int lr_reg, const struct gic_lr *lr)
 {
     uint64_t lrv = 0;
 
-    lrv = ( ((u64)(lr->pirq & GICH_LR_PHYSICAL_MASK) << GICH_LR_PHYSICAL_SHIFT)|
-        ((u64)(lr->virq & GICH_LR_VIRTUAL_MASK)  << GICH_LR_VIRTUAL_SHIFT) |
-        ((u64)(lr->priority & GICH_LR_PRIORITY_MASK) << GICH_LR_PRIORITY_SHIFT)|
-        ((u64)(lr->state & GICH_LR_STATE_MASK) << GICH_LR_STATE_SHIFT) |
-        ((u64)(lr->hw_status & GICH_LR_HW_MASK) << GICH_LR_HW_SHIFT)  |
-        ((u64)(lr->grp & GICH_LR_GRP_MASK) << GICH_LR_GRP_SHIFT) );
+    lrv = ( ((u64)(lr->pirq & ICH_LR_PHYSICAL_MASK) << ICH_LR_PHYSICAL_SHIFT)|
+        ((u64)(lr->virq & ICH_LR_VIRTUAL_MASK)  << ICH_LR_VIRTUAL_SHIFT) |
+        ((u64)(lr->priority & ICH_LR_PRIORITY_MASK) << ICH_LR_PRIORITY_SHIFT)|
+        ((u64)(lr->state & ICH_LR_STATE_MASK) << ICH_LR_STATE_SHIFT) |
+        ((u64)(lr->hw_status & ICH_LR_HW_MASK) << ICH_LR_HW_SHIFT)  |
+        ((u64)(lr->grp & ICH_LR_GRP_MASK) << ICH_LR_GRP_SHIFT) );
 
     gicv3_ich_write_lr(lr_reg, lrv);
 }
@@ -1043,8 +1043,8 @@ static void gicv3_hcr_status(uint32_t flag, bool status)
 
 static unsigned int gicv3_read_vmcr_priority(void)
 {
-   return ((READ_SYSREG32(ICH_VMCR_EL2) >> GICH_VMCR_PRIORITY_SHIFT) &
-            GICH_VMCR_PRIORITY_MASK);
+   return ((READ_SYSREG32(ICH_VMCR_EL2) >> ICH_VMCR_PRIORITY_SHIFT) &
+            ICH_VMCR_PRIORITY_MASK);
 }
 
 /* Only support reading GRP1 APRn registers */
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
index 412e41afed..947de5162d 100644
--- a/xen/include/asm-arm/gic_v3_defs.h
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -160,31 +160,30 @@
 #define LPI_PROP_RES1                (1 << 1)
 #define LPI_PROP_ENABLED             (1 << 0)
 
-#define GICH_VMCR_EOI                (1 << 9)
-#define GICH_VMCR_VENG1              (1 << 1)
-
-#define GICH_LR_VIRTUAL_MASK         0xffff
-#define GICH_LR_VIRTUAL_SHIFT        0
-#define GICH_LR_PHYSICAL_MASK        0x3ff
-#define GICH_LR_PHYSICAL_SHIFT       32
-#define GICH_LR_STATE_MASK           0x3
-#define GICH_LR_STATE_SHIFT          62
-#define GICH_LR_PRIORITY_MASK        0xff
-#define GICH_LR_PRIORITY_SHIFT       48
-#define GICH_LR_HW_MASK              0x1
-#define GICH_LR_HW_SHIFT             61
-#define GICH_LR_GRP_MASK             0x1
-#define GICH_LR_GRP_SHIFT            60
-#define GICH_LR_MAINTENANCE_IRQ      (1UL<<41)
-#define GICH_LR_GRP1                 (1UL<<60)
-#define GICH_LR_HW                   (1UL<<61)
-
-#define GICH_VTR_NRLRGS              0x3f
-#define GICH_VTR_PRIBITS_MASK        0x7
-#define GICH_VTR_PRIBITS_SHIFT       29
-
-#define GICH_VMCR_PRIORITY_MASK      0xff
-#define GICH_VMCR_PRIORITY_SHIFT     24
+#define ICH_VMCR_EOI                 (1 << 9)
+#define ICH_VMCR_VENG1               (1 << 1)
+#define ICH_VMCR_PRIORITY_MASK       0xff
+#define ICH_VMCR_PRIORITY_SHIFT      24
+
+#define ICH_LR_VIRTUAL_MASK          0xffff
+#define ICH_LR_VIRTUAL_SHIFT         0
+#define ICH_LR_PHYSICAL_MASK         0x3ff
+#define ICH_LR_PHYSICAL_SHIFT        32
+#define ICH_LR_STATE_MASK            0x3
+#define ICH_LR_STATE_SHIFT           62
+#define ICH_LR_PRIORITY_MASK         0xff
+#define ICH_LR_PRIORITY_SHIFT        48
+#define ICH_LR_HW_MASK               0x1
+#define ICH_LR_HW_SHIFT              61
+#define ICH_LR_GRP_MASK              0x1
+#define ICH_LR_GRP_SHIFT             60
+#define ICH_LR_MAINTENANCE_IRQ       (1UL<<41)
+#define ICH_LR_GRP1                  (1UL<<60)
+#define ICH_LR_HW                    (1UL<<61)
+
+#define ICH_VTR_NRLRGS               0x3f
+#define ICH_VTR_PRIBITS_MASK         0x7
+#define ICH_VTR_PRIBITS_SHIFT        29
 
 #define ICH_SGI_IRQMODE_SHIFT        40
 #define ICH_SGI_IRQMODE_MASK         0x1
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 17/57] ARM: Introduce kick_vcpu()

[Andre Przywara]

If we change something in a vCPU that affects its runnability or
otherwise needs the vCPU's attention, we might need to tell the scheduler
about it.
We are using this in one place (vIRQ injection) at the moment, but will
need this at more places soon.
So let's factor out this functionality in the new kick_vcpu() function
and make this available to the whole Xen arch code.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/smp.c        | 14 ++++++++++++++
 xen/arch/arm/vgic.c       | 10 ++--------
 xen/include/asm-arm/smp.h |  3 +++
 3 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
index 62f57f0ba2..381a4786a2 100644
--- a/xen/arch/arm/smp.c
+++ b/xen/arch/arm/smp.c
@@ -4,6 +4,8 @@
 #include <asm/page.h>
 #include <asm/gic.h>
 #include <asm/flushtlb.h>
+#include <xen/perfc.h>
+#include <xen/sched.h>
 
 void flush_tlb_mask(const cpumask_t *mask)
 {
@@ -32,6 +34,18 @@ void smp_send_call_function_mask(const cpumask_t *mask)
     }
 }
 
+void kick_vcpu(struct vcpu *vcpu)
+{
+    bool running = vcpu->is_running;
+
+    vcpu_unblock(vcpu);
+    if ( running && vcpu != current )
+    {
+        perfc_incr(vgic_cross_cpu_intr_inject);
+        smp_send_event_check_mask(cpumask_of(vcpu->processor));
+    }
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 3c77d5fef6..e44de9ea95 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -530,7 +530,6 @@ int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
     uint8_t priority;
     struct pending_irq *iter, *n;
     unsigned long flags;
-    bool running;
 
     /*
      * For edge triggered interrupts we always ignore a "falling edge".
@@ -590,14 +589,9 @@ int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
     list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
 out:
     spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
+
     /* we have a new higher priority irq, inject it into the guest */
-    running = v->is_running;
-    vcpu_unblock(v);
-    if ( running && v != current )
-    {
-        perfc_incr(vgic_cross_cpu_intr_inject);
-        smp_send_event_check_mask(cpumask_of(v->processor));
-    }
+    kick_vcpu(v);
 
     return 0;
 }
diff --git a/xen/include/asm-arm/smp.h b/xen/include/asm-arm/smp.h
index 3c122681d7..7c8ef75789 100644
--- a/xen/include/asm-arm/smp.h
+++ b/xen/include/asm-arm/smp.h
@@ -28,6 +28,9 @@ extern void init_secondary(void);
 extern void smp_init_cpus(void);
 extern void smp_clear_cpu_maps (void);
 extern int smp_get_max_cpus (void);
+
+void kick_vcpu(struct vcpu *vcpu);
+
 #endif
 
 /*
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 18/57] ARM: GICv2: introduce gicv2_poke_irq()

[Andre Przywara]

The GICv2 uses bitmaps spanning several MMIO registers for holding some
interrupt state. Similar to GICv3, add a poke helper functions to set a bit
for a given irq_desc in one of those bitmaps.
At the moment there is only one use in gic-v2.c, but there will be more
coming soon.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/gic-v2.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 7938a42591..618dd94120 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -235,6 +235,11 @@ static unsigned int gicv2_read_irq(void)
     return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
 }
 
+static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)
+{
+    writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
+}
+
 static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int type)
 {
     uint32_t cfg, actual, edgebit;
@@ -509,7 +514,6 @@ static unsigned int gicv2_read_apr(int apr_reg)
 static void gicv2_irq_enable(struct irq_desc *desc)
 {
     unsigned long flags;
-    int irq = desc->irq;
 
     ASSERT(spin_is_locked(&desc->lock));
 
@@ -517,20 +521,19 @@ static void gicv2_irq_enable(struct irq_desc *desc)
     clear_bit(_IRQ_DISABLED, &desc->status);
     dsb(sy);
     /* Enable routing */
-    writel_gicd((1u << (irq % 32)), GICD_ISENABLER + (irq / 32) * 4);
+    gicv2_poke_irq(desc, GICD_ISENABLER);
     spin_unlock_irqrestore(&gicv2.lock, flags);
 }
 
 static void gicv2_irq_disable(struct irq_desc *desc)
 {
     unsigned long flags;
-    int irq = desc->irq;
 
     ASSERT(spin_is_locked(&desc->lock));
 
     spin_lock_irqsave(&gicv2.lock, flags);
     /* Disable routing */
-    writel_gicd(1u << (irq % 32), GICD_ICENABLER + (irq / 32) * 4);
+    gicv2_poke_irq(desc, GICD_ICENABLER);
     set_bit(_IRQ_DISABLED, &desc->status);
     spin_unlock_irqrestore(&gicv2.lock, flags);
 }
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 19/57] ARM: GICv3: poke_irq: make RWP optional

[Andre Przywara]

A GICv3 hardware implementation can be implemented in several parts that
communicate with each other (think multi-socket systems).
To make sure that critical settings have arrived at all endpoints, some
bits are tracked using the RWP bit in the GICD_CTLR register, which
signals whether a register write is still in progress.
However this only applies to *some* registers, namely the bits in the
GICD_ICENABLER (disabling interrupts) and some bits in the GICD_CTLR
register (cf. Arm IHI 0069D, 8.9.4: RWP, bit[31]).
But our gicv3_poke_irq() was always polling this bit before returning,
resulting in pointless MMIO reads for many registers.
Add an option to gicv3_poke_irq() to state whether we want to wait for
this bit and use it accordingly to match the spec.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/gic-v3.c | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 3e381d031b..44dfba2267 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -428,9 +428,9 @@ static void gicv3_dump_state(const struct vcpu *v)
     }
 }
 
-static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset)
+static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
 {
-    u32 mask = 1 << (irqd->irq % 32);
+    u32 mask = 1U << (irqd->irq % 32);
     void __iomem *base;
 
     if ( irqd->irq < NR_GIC_LOCAL_IRQS )
@@ -439,17 +439,19 @@ static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset)
         base = GICD;
 
     writel_relaxed(mask, base + offset + (irqd->irq / 32) * 4);
-    gicv3_wait_for_rwp(irqd->irq);
+
+    if ( wait_for_rwp )
+        gicv3_wait_for_rwp(irqd->irq);
 }
 
 static void gicv3_unmask_irq(struct irq_desc *irqd)
 {
-    gicv3_poke_irq(irqd, GICD_ISENABLER);
+    gicv3_poke_irq(irqd, GICD_ISENABLER, false);
 }
 
 static void gicv3_mask_irq(struct irq_desc *irqd)
 {
-    gicv3_poke_irq(irqd, GICD_ICENABLER);
+    gicv3_poke_irq(irqd, GICD_ICENABLER, true);
 }
 
 static void gicv3_eoi_irq(struct irq_desc *irqd)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 20/57] ARM: GICv2: fix GICH_V2_LR definitions

[Andre Przywara]

The bit definition for the CPUID mask in the GICv2 LR register was
wrong, fortunately the current implementation does not use that bit.
Fix it up (it's starting at bit 10, not bit 9) and clean up some
nearby definitions on the way.
This will be used by the new VGIC shortly.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch
 xen/arch/arm/gic-v2.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 618dd94120..031be920cc 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -57,10 +57,11 @@
 #define GICH_V2_LR_HW_MASK         0x1
 #define GICH_V2_LR_GRP_SHIFT       30
 #define GICH_V2_LR_GRP_MASK        0x1
-#define GICH_V2_LR_MAINTENANCE_IRQ (1<<19)
-#define GICH_V2_LR_GRP1            (1<<30)
-#define GICH_V2_LR_HW              (1<<31)
-#define GICH_V2_LR_CPUID_SHIFT     9
+#define GICH_V2_LR_MAINTENANCE_IRQ (1U << 19)
+#define GICH_V2_LR_GRP1            (1U << 30)
+#define GICH_V2_LR_HW              (1U << GICH_V2_LR_HW_SHIFT)
+#define GICH_V2_LR_CPUID_SHIFT     10
+#define GICH_V2_LR_CPUID_MASK      0x7
 #define GICH_V2_VTR_NRLRGS         0x3f
 
 #define GICH_V2_VMCR_PRIORITY_MASK   0x1f
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 21/57] ARM: GICv2: extend LR read/write functions to cover EOI and source

[Andre Przywara]

So far our LR read/write functions do not handle the EOI bit and the
source CPUID bits in an LR, because the current VGIC implementation does
not use them.
Extend the gic_lr data structure to hold these bits of information as
well, packing it on the way to avoid it to grow.
Then extract and assemble those bits from/to an LR.

This allows the new VGIC to use this information.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- new patch

 xen/arch/arm/gic-v2.c     | 7 +++++++
 xen/include/asm-arm/gic.h | 8 +++++---
 2 files changed, 12 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 031be920cc..c5ec0d4d35 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -470,6 +470,9 @@ static void gicv2_read_lr(int lr, struct gic_lr *lr_reg)
     lr_reg->state     = (lrv >> GICH_V2_LR_STATE_SHIFT) & GICH_V2_LR_STATE_MASK;
     lr_reg->hw_status = (lrv >> GICH_V2_LR_HW_SHIFT) & GICH_V2_LR_HW_MASK;
     lr_reg->grp       = (lrv >> GICH_V2_LR_GRP_SHIFT) & GICH_V2_LR_GRP_MASK;
+    lr_reg->eoi       = !!(lrv & GICH_V2_LR_MAINTENANCE_IRQ);
+    if ( lr_reg->virq < NR_GIC_SGI )
+        lr_reg->source = (lrv >> GICH_V2_LR_CPUID_SHIFT) & GICH_V2_LR_CPUID_MASK;
 }
 
 static void gicv2_write_lr(int lr, const struct gic_lr *lr_reg)
@@ -485,6 +488,10 @@ static void gicv2_write_lr(int lr, const struct gic_lr *lr_reg)
           ((uint32_t)(lr_reg->hw_status & GICH_V2_LR_HW_MASK)
                                        << GICH_V2_LR_HW_SHIFT)  |
           ((uint32_t)(lr_reg->grp & GICH_V2_LR_GRP_MASK) << GICH_V2_LR_GRP_SHIFT) );
+    if ( lr_reg->eoi )
+        lrv |= GICH_V2_LR_MAINTENANCE_IRQ;
+    if ( lr_reg->virq < NR_GIC_SGI )
+        lrv |= (uint32_t)lr_reg->source << GICH_V2_LR_CPUID_SHIFT;
 
     writel_gich(lrv, GICH_LR + lr * 4);
 }
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 8fab458d7f..89a07ae6b4 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -223,9 +223,11 @@ struct gic_lr {
    /* Virtual IRQ */
    uint32_t virq;
    uint8_t priority;
-   uint8_t state;
-   uint8_t hw_status;
-   uint8_t grp;
+   uint8_t source;
+   uint8_t state:2;
+   uint8_t hw_status:1;
+   uint8_t grp:1;
+   uint8_t eoi:1;
 };
 
 enum gic_version {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 22/57] ARM: GIC: Allow tweaking the active and pending state of an IRQ

[Andre Przywara]

When playing around with hardware mapped, level triggered virtual IRQs,
there is the need to explicitly set the active or pending state of an
interrupt at some point.
To prepare the GIC for that, we introduce a set_active_state() and a
set_pending_state() function to let the VGIC manipulate the state of
an associated hardware IRQ.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use struct irq_desc* in the interface (instead of just the IRQ number)
- add set_pending_state() (needed later)

 xen/arch/arm/gic-v2.c     | 32 ++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3.c     | 28 ++++++++++++++++++++++++++++
 xen/arch/arm/gic.c        | 10 ++++++++++
 xen/include/asm-arm/gic.h | 10 ++++++++++
 4 files changed, 80 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index c5ec0d4d35..74169b5633 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -241,6 +241,36 @@ static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)
     writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
 }
 
+static void gicv2_set_active_state(struct irq_desc *irqd, bool active)
+{
+    ASSERT(spin_is_locked(&irqd->lock));
+
+    if ( active )
+    {
+        set_bit(_IRQ_INPROGRESS, &irqd->status);
+        gicv2_poke_irq(irqd, GICD_ISACTIVER);
+    }
+    else
+    {
+        gicv2_poke_irq(irqd, GICD_ICACTIVER);
+    }
+}
+
+static void gicv2_set_pending_state(struct irq_desc *irqd, bool pending)
+{
+    ASSERT(spin_is_locked(&irqd->lock));
+
+    if ( pending )
+    {
+        set_bit(_IRQ_INPROGRESS, &irqd->status);
+        gicv2_poke_irq(irqd, GICD_ISPENDR);
+    }
+    else
+    {
+        gicv2_poke_irq(irqd, GICD_ICPENDR);
+    }
+}
+
 static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int type)
 {
     uint32_t cfg, actual, edgebit;
@@ -1251,6 +1281,8 @@ const static struct gic_hw_operations gicv2_ops = {
     .eoi_irq             = gicv2_eoi_irq,
     .deactivate_irq      = gicv2_dir_irq,
     .read_irq            = gicv2_read_irq,
+    .set_active_state    = gicv2_set_active_state,
+    .set_pending_state   = gicv2_set_pending_state,
     .set_irq_type        = gicv2_set_irq_type,
     .set_irq_priority    = gicv2_set_irq_priority,
     .send_SGI            = gicv2_send_SGI,
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 44dfba2267..c96469f09d 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -477,6 +477,32 @@ static unsigned int gicv3_read_irq(void)
     return irq;
 }
 
+static void gicv3_set_active_state(struct irq_desc *irqd, bool active)
+{
+    ASSERT(spin_is_locked(&irqd->lock));
+
+    if ( active )
+    {
+        set_bit(_IRQ_INPROGRESS, &irqd->status);
+        gicv3_poke_irq(irqd, GICD_ISACTIVER, false);
+    }
+    else
+        gicv3_poke_irq(irqd, GICD_ICACTIVER, false);
+}
+
+static void gicv3_set_pending_state(struct irq_desc *irqd, bool pending)
+{
+    ASSERT(spin_is_locked(&irqd->lock));
+
+    if ( pending )
+    {
+        set_bit(_IRQ_INPROGRESS, &irqd->status);
+        gicv3_poke_irq(irqd, GICD_ISPENDR, false);
+    }
+    else
+        gicv3_poke_irq(irqd, GICD_ICPENDR, false);
+}
+
 static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
 {
      uint64_t mpidr = cpu_logical_map(cpu);
@@ -1723,6 +1749,8 @@ static const struct gic_hw_operations gicv3_ops = {
     .eoi_irq             = gicv3_eoi_irq,
     .deactivate_irq      = gicv3_dir_irq,
     .read_irq            = gicv3_read_irq,
+    .set_active_state    = gicv3_set_active_state,
+    .set_pending_state   = gicv3_set_pending_state,
     .set_irq_type        = gicv3_set_irq_type,
     .set_irq_priority    = gicv3_set_irq_priority,
     .send_SGI            = gicv3_send_sgi,
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index 968e46fabb..f1329a630a 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -87,6 +87,16 @@ void gic_restore_state(struct vcpu *v)
     isb();
 }
 
+void gic_set_active_state(struct irq_desc *irqd, bool state)
+{
+    gic_hw_ops->set_active_state(irqd, state);
+}
+
+void gic_set_pending_state(struct irq_desc *irqd, bool state)
+{
+    gic_hw_ops->set_pending_state(irqd, state);
+}
+
 /* desc->irq needs to be disabled before calling this function */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type)
 {
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 89a07ae6b4..46dcb0fe7c 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -239,6 +239,12 @@ DECLARE_PER_CPU(uint64_t, lr_mask);
 
 extern enum gic_version gic_hw_version(void);
 
+/* Force the active state of an IRQ. */
+void gic_set_active_state(struct irq_desc *irqd, bool state);
+
+/* Force the pending state of an IRQ. */
+void gic_set_pending_state(struct irq_desc *irqd, bool state);
+
 /* Program the IRQ type into the GIC */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
 
@@ -348,6 +354,10 @@ struct gic_hw_operations {
     void (*deactivate_irq)(struct irq_desc *irqd);
     /* Read IRQ id and Ack */
     unsigned int (*read_irq)(void);
+    /* Force the active state of an IRQ by accessing the distributor */
+    void (*set_active_state)(struct irq_desc *irqd, bool state);
+    /* Force the pending state of an IRQ by accessing the distributor */
+    void (*set_pending_state)(struct irq_desc *irqd, bool state);
     /* Set IRQ type */
     void (*set_irq_type)(struct irq_desc *desc, unsigned int type);
     /* Set IRQ priority */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 23/57] ARM: GIC: allow reading pending state of a hardware IRQ

[Andre Przywara]

To synchronize level triggered interrupts which are mapped into a guest,
we need to update the virtual line level at certain points in time.
For a hardware mapped interrupt the GIC is the only place where we can
easily access this information.
Implement a gic_hw_operations member to return the pending state of a
particular interrupt. Due to hardware limitations this only works for
private interrupts of the current CPU, so there is no CPU field in the
prototype.
This adds gicv2/3_peek_irq() helper functions, to read a bit in a bitmap
spread over several MMIO registers.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- add gicv2/3_peek_irq() helpers
- use struct irq_desc* in the interface (instead of just the IRQ number)

 xen/arch/arm/gic-v2.c     | 15 +++++++++++++++
 xen/arch/arm/gic-v3.c     | 19 +++++++++++++++++++
 xen/arch/arm/gic.c        |  5 +++++
 xen/include/asm-arm/gic.h |  5 +++++
 4 files changed, 44 insertions(+)

diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
index 74169b5633..48352f6499 100644
--- a/xen/arch/arm/gic-v2.c
+++ b/xen/arch/arm/gic-v2.c
@@ -241,6 +241,15 @@ static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)
     writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
 }
 
+static bool gicv2_peek_irq(struct irq_desc *irqd, uint32_t offset)
+{
+    uint32_t reg;
+
+    reg = readl_gicd(offset + (irqd->irq / 32) * 4) & (1U << (irqd->irq % 32));
+
+    return reg;
+}
+
 static void gicv2_set_active_state(struct irq_desc *irqd, bool active)
 {
     ASSERT(spin_is_locked(&irqd->lock));
@@ -549,6 +558,11 @@ static unsigned int gicv2_read_apr(int apr_reg)
    return readl_gich(GICH_APR);
 }
 
+static bool gicv2_read_pending_state(struct irq_desc *irqd)
+{
+    return gicv2_peek_irq(irqd, GICD_ISPENDR);
+}
+
 static void gicv2_irq_enable(struct irq_desc *desc)
 {
     unsigned long flags;
@@ -1294,6 +1308,7 @@ const static struct gic_hw_operations gicv2_ops = {
     .write_lr            = gicv2_write_lr,
     .read_vmcr_priority  = gicv2_read_vmcr_priority,
     .read_apr            = gicv2_read_apr,
+    .read_pending_state  = gicv2_read_pending_state,
     .make_hwdom_dt_node  = gicv2_make_hwdom_dt_node,
     .make_hwdom_madt     = gicv2_make_hwdom_madt,
     .get_hwdom_extra_madt_size = gicv2_get_hwdom_extra_madt_size,
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index c96469f09d..3e75d06c3b 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -444,6 +444,19 @@ static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
         gicv3_wait_for_rwp(irqd->irq);
 }
 
+static bool gicv3_peek_irq(struct irq_desc *irqd, u32 offset)
+{
+    void __iomem *base;
+    unsigned int irq = irqd->irq;
+
+    if ( irq >= NR_GIC_LOCAL_IRQS)
+        base = GICD + (irq / 32) * 4;
+    else
+        base = GICD_RDIST_SGI_BASE;
+
+    return !!(readl(base + offset) & (1U << (irq % 32)));
+}
+
 static void gicv3_unmask_irq(struct irq_desc *irqd)
 {
     gicv3_poke_irq(irqd, GICD_ISENABLER, false);
@@ -1094,6 +1107,11 @@ static unsigned int gicv3_read_apr(int apr_reg)
     }
 }
 
+static bool gicv3_read_pending_state(struct irq_desc *irqd)
+{
+    return gicv3_peek_irq(irqd, GICD_ISPENDR);
+}
+
 static void gicv3_irq_enable(struct irq_desc *desc)
 {
     unsigned long flags;
@@ -1762,6 +1780,7 @@ static const struct gic_hw_operations gicv3_ops = {
     .write_lr            = gicv3_write_lr,
     .read_vmcr_priority  = gicv3_read_vmcr_priority,
     .read_apr            = gicv3_read_apr,
+    .read_pending_state  = gicv3_read_pending_state,
     .secondary_init      = gicv3_secondary_cpu_init,
     .make_hwdom_dt_node  = gicv3_make_hwdom_dt_node,
     .make_hwdom_madt     = gicv3_make_hwdom_madt,
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index f1329a630a..67c3b4d86d 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -116,6 +116,11 @@ static void gic_set_irq_priority(struct irq_desc *desc, unsigned int priority)
     gic_hw_ops->set_irq_priority(desc, priority);
 }
 
+bool gic_read_pending_state(struct irq_desc *irqd)
+{
+    return gic_hw_ops->read_pending_state(irqd);
+}
+
 /* Program the GIC to route an interrupt to the host (i.e. Xen)
  * - needs to be called with desc.lock held
  */
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 46dcb0fe7c..03667f00cf 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -248,6 +248,9 @@ void gic_set_pending_state(struct irq_desc *irqd, bool state);
 /* Program the IRQ type into the GIC */
 void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
 
+/* Read the pending state of an interrupt from the distributor. */
+bool gic_read_pending_state(struct irq_desc *irqd);
+
 /* Program the GIC to route an interrupt */
 extern void gic_route_irq_to_xen(struct irq_desc *desc, unsigned int priority);
 extern int gic_route_irq_to_guest(struct domain *, unsigned int virq,
@@ -382,6 +385,8 @@ struct gic_hw_operations {
     unsigned int (*read_vmcr_priority)(void);
     /* Read APRn register */
     unsigned int (*read_apr)(int apr_reg);
+    /* Query the pending state of an interrupt at the distributor level. */
+    bool (*read_pending_state)(struct irq_desc *irqd);
     /* Secondary CPU init */
     int (*secondary_init)(void);
     /* Create GIC node for the hardware domain */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 24/57] ARM: timer: Handle level triggered IRQs correctly

[Andre Przywara]

The ARM Generic Timer uses a level-sensitive interrupt semantic. We
easily catch when the line goes high, as this triggers the hardware IRQ.
However we have to sync the state of the interrupt condition at certain
points to catch when the line goes low and we can remove the vtimer vIRQ
from the vGIC (and the LR).
The VGIC in Xen so far only implemented edge triggered vIRQs, really, so
we need to add new functionality to re-sample the interrupt state.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- extend comments
- don't read CNTV_CVAL_EL0
- use symbolic names for constants

 xen/arch/arm/time.c     | 36 ++++++++++++++++++++++++++++++++++++
 xen/arch/arm/traps.c    |  6 ++++++
 xen/include/xen/timer.h |  2 ++
 3 files changed, 44 insertions(+)

diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
index c11fcfeadd..c0ae781ecd 100644
--- a/xen/arch/arm/time.c
+++ b/xen/arch/arm/time.c
@@ -263,6 +263,42 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
     vgic_inject_irq(current->domain, current, current->arch.virt_timer.irq, true);
 }
 
+/**
+ * vtimer_sync() - update the state of the virtual timer after a guest run
+ * @vcpu: The VCPU to sync the arch timer state
+ *
+ * After returning from a guest, update the state of the virtual interrupt
+ * line, to model the level triggered interrupt correctly.
+ * If the guest has handled a timer interrupt, the virtual interrupt line
+ * needs to be lowered explicitly. vgic_inject_irq() takes care of that.
+ */
+void vtimer_sync(struct vcpu *vcpu)
+{
+    struct vtimer *vtimer = &vcpu->arch.virt_timer;
+    uint32_t vtimer_ctl = READ_SYSREG32(CNTV_CTL_EL0);
+    bool level;
+
+    /*
+     * Technically the mask should include the CNTx_CTL_MASK bit here,
+     * to catch if the timer interrupt is masked. However Xen always masks
+     * the timer upon entering the hypervisor, leaving it up to the guest
+     * to un-mask it. So we would always read a "low" level, despite the
+     * condition being actually "high".
+     * Ignoring the mask bit solves this (for now).
+     * Another possible check would be to compare the value of CNTVCT_EL0
+     * against vtimer->cval and derive the interrupt state from that.
+     */
+    vtimer_ctl &= (CNTx_CTL_ENABLE | CNTx_CTL_PENDING);
+    level = (vtimer_ctl == (CNTx_CTL_ENABLE | CNTx_CTL_PENDING));
+
+     /*
+      * TODO: The proper fix for this is to make vtimer vIRQ hardware mapped,
+      * but this requires reworking the arch timer to implement this.
+      */
+
+    vgic_inject_irq(vcpu->domain, vcpu, vtimer->irq, level);
+}
+
 /*
  * Arch timer interrupt really ought to be level triggered, since the
  * design of the timer/comparator mechanism is based around that
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 7411bff7a7..0713723bb7 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2024,6 +2024,12 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
         if ( current->arch.hcr_el2 & HCR_VA )
             current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
 
+        /*
+         * We need to update the state of our emulated devices using level
+         * triggered interrupts before syncing back the VGIC state.
+         */
+        vtimer_sync(current);
+
         vgic_sync_from_lrs(current);
     }
 }
diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
index 4513260b0d..eddbbf3903 100644
--- a/xen/include/xen/timer.h
+++ b/xen/include/xen/timer.h
@@ -94,6 +94,8 @@ DECLARE_PER_CPU(s_time_t, timer_deadline);
 /* Arch-defined function to reprogram timer hardware for new deadline. */
 int reprogram_timer(s_time_t timeout);
 
+void vtimer_sync(struct vcpu *vcpu);
+
 /* Calculate the aligned first tick time for a given periodic timer. */
 s_time_t align_timer(s_time_t firsttick, uint64_t period);
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 25/57] ARM: evtchn: Handle level triggered IRQs correctly

[Andre Przywara]

The event channel IRQ has level triggered semantics, however the current
VGIC treats everything as edge triggered.
To correctly process those IRQs, we have to lower the (virtual) IRQ line
at some point in time, depending on whether ther interrupt condition
still prevails.
Check the per-VCPU evtchn_upcall_pending variable to make the interrupt
line match its status, and call this function upon every hypervisor
entry.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/domain.c       | 7 +++++++
 xen/arch/arm/traps.c        | 1 +
 xen/include/asm-arm/event.h | 1 +
 3 files changed, 9 insertions(+)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index a7bba3ad44..11a46aa27f 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -955,6 +955,13 @@ void vcpu_mark_events_pending(struct vcpu *v)
     vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
 }
 
+void vcpu_update_evtchn_irq(struct vcpu *v)
+{
+    bool pending = vcpu_info(v, evtchn_upcall_pending);
+
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, pending);
+}
+
 /* The ARM spec declares that even if local irqs are masked in
  * the CPSR register, an irq should wake up a cpu from WFI anyway.
  * For this reason we need to check for irqs that need delivery,
diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 0713723bb7..fcf5db50ae 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -2029,6 +2029,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
          * triggered interrupts before syncing back the VGIC state.
          */
         vtimer_sync(current);
+        vcpu_update_evtchn_irq(current);
 
         vgic_sync_from_lrs(current);
     }
diff --git a/xen/include/asm-arm/event.h b/xen/include/asm-arm/event.h
index c4c79fa87d..eafb2e7f3e 100644
--- a/xen/include/asm-arm/event.h
+++ b/xen/include/asm-arm/event.h
@@ -6,6 +6,7 @@
 
 void vcpu_kick(struct vcpu *v);
 void vcpu_mark_events_pending(struct vcpu *v);
+void vcpu_update_evtchn_irq(struct vcpu *v);
 void vcpu_block_unless_event_pending(struct vcpu *v);
 
 static inline int vcpu_event_delivery_is_enabled(struct vcpu *v)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 26/57] ARM: vPL011: Use the VGIC's level triggered IRQs handling if available

[Andre Przywara]

The emulated ARM SBSA UART is using level triggered IRQ semantics,
however the current VGIC can only handle edge triggered IRQs, really.
Disable the existing workaround for this problem in case we have the
new VGIC in place, which can properly handle level triggered IRQs.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/vpl011.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
index 5dcf4bec18..197ece8873 100644
--- a/xen/arch/arm/vpl011.c
+++ b/xen/arch/arm/vpl011.c
@@ -54,6 +54,7 @@ static void vpl011_update_interrupt_status(struct domain *d)
      */
     ASSERT(spin_is_locked(&vpl011->lock));
 
+#ifndef CONFIG_NEW_VGIC
     /*
      * TODO: PL011 interrupts are level triggered which means
      * that interrupt needs to be set/clear instead of being
@@ -71,6 +72,9 @@ static void vpl011_update_interrupt_status(struct domain *d)
         vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, true);
 
     vpl011->shadow_uartmis = uartmis;
+#else
+    vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, !!uartmis);
+#endif
 }
 
 static uint8_t vpl011_read_data(struct domain *d)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 27/57] ARM: new VGIC: Add data structure definitions

[Andre Przywara]

Add a new header file for the new and improved GIC implementation.
The big change is that we now have a struct vgic_irq per IRQ instead
of spreading all the information over various bitmaps in the ranks.

We include this new header conditionally from within the old header
file for the time being to avoid touching all the users.

This is based on Linux commit b18b57787f5e, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- rename header file to new_vgic.h
- drop unneeded data structures (vgic_its, vgic_v<x>_cpu_if)
- reorder members in vgic_irq to avoid padding
- move flags members into bool bitfields
- drop prototypes
- use unsigned and uint<x>_t data types
- keep arch_vcpu member name as "vgic"

 xen/include/asm-arm/new_vgic.h | 198 +++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/vgic.h     |   6 ++
 2 files changed, 204 insertions(+)
 create mode 100644 xen/include/asm-arm/new_vgic.h

diff --git a/xen/include/asm-arm/new_vgic.h b/xen/include/asm-arm/new_vgic.h
new file mode 100644
index 0000000000..54be5aa3eb
--- /dev/null
+++ b/xen/include/asm-arm/new_vgic.h
@@ -0,0 +1,198 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __ASM_ARM_NEW_VGIC_H
+#define __ASM_ARM_NEW_VGIC_H
+
+#include <asm/atomic.h>
+#include <asm/mmio.h>
+#include <xen/list.h>
+#include <xen/mm.h>
+#include <xen/spinlock.h>
+
+#define VGIC_V3_MAX_CPUS        255
+#define VGIC_V2_MAX_CPUS        8
+#define VGIC_NR_SGIS            16
+#define VGIC_NR_PPIS            16
+#define VGIC_NR_PRIVATE_IRQS    (VGIC_NR_SGIS + VGIC_NR_PPIS)
+#define VGIC_MAX_PRIVATE        (VGIC_NR_PRIVATE_IRQS - 1)
+#define VGIC_MAX_SPI            1019
+#define VGIC_MAX_RESERVED       1023
+#define VGIC_MIN_LPI            8192
+
+#define irq_is_ppi(irq) ((irq) >= VGIC_NR_SGIS && (irq) < VGIC_NR_PRIVATE_IRQS)
+#define irq_is_spi(irq) ((irq) >= VGIC_NR_PRIVATE_IRQS && \
+                         (irq) <= VGIC_MAX_SPI)
+
+enum vgic_type {
+    VGIC_V2,        /* Good ol' GICv2 */
+    VGIC_V3,        /* New fancy GICv3 */
+};
+
+#define VGIC_V2_MAX_LRS         (1 << 6)
+#define VGIC_V3_MAX_LRS         16
+#define VGIC_V3_LR_INDEX(lr)    (VGIC_V3_MAX_LRS - 1 - lr)
+
+enum vgic_irq_config {
+    VGIC_CONFIG_EDGE = 0,
+    VGIC_CONFIG_LEVEL
+};
+
+struct vgic_irq {
+    struct list_head ap_list;
+
+    struct vcpu *vcpu;          /*
+                                 * SGIs and PPIs: The VCPU
+                                 * SPIs and LPIs: The VCPU whose ap_list
+                                 * this is queued on.
+                                 */
+
+    struct vcpu *target_vcpu;   /*
+                                 * The VCPU that this interrupt should
+                                 * be sent to, as a result of the
+                                 * targets reg (v2) or the affinity reg (v3).
+                                 */
+
+    spinlock_t irq_lock;        /* Protects the content of the struct */
+    uint32_t intid;             /* Guest visible INTID */
+    atomic_t refcount;          /* Used for LPIs */
+    uint32_t hwintid;           /* HW INTID number */
+    union
+    {
+        struct {
+            uint8_t targets;    /* GICv2 target VCPUs mask */
+            uint8_t source;     /* GICv2 SGIs only */
+        };
+        uint32_t mpidr;         /* GICv3 target VCPU */
+    };
+    uint8_t priority;
+    bool line_level:1;          /* Level only */
+    bool pending_latch:1;       /*
+                                 * The pending latch state used to
+                                 * calculate the pending state for both
+                                 * level and edge triggered IRQs.
+                                 */
+    bool active:1;              /* not used for LPIs */
+    bool enabled:1;
+    bool hw:1;                  /* Tied to HW IRQ */
+    bool config:1;              /* Level or edge */
+    struct list_head lpi_list;  /* Used to link all LPIs together */
+};
+
+struct vgic_register_region;
+
+enum iodev_type {
+    IODEV_DIST,
+    IODEV_REDIST,
+};
+
+struct vgic_io_device {
+    gfn_t base_fn;
+    struct vcpu *redist_vcpu;
+    const struct vgic_register_region *regions;
+    enum iodev_type iodev_type;
+    unsigned int nr_regions;
+};
+
+struct vgic_dist {
+    bool                ready;
+    bool                initialized;
+
+    /* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
+    uint32_t            version;
+
+    /* Do injected MSIs require an additional device ID? */
+    bool                msis_require_devid;
+
+    unsigned int        nr_spis;
+
+    /* base addresses in guest physical address space: */
+    paddr_t             vgic_dist_base;     /* distributor */
+    union
+    {
+        /* either a GICv2 CPU interface */
+        paddr_t         vgic_cpu_base;
+        /* or a number of GICv3 redistributor regions */
+        struct
+        {
+            paddr_t     vgic_redist_base;
+            paddr_t     vgic_redist_free_offset;
+        };
+    };
+
+    /* distributor enabled */
+    bool                enabled;
+
+    struct vgic_irq     *spis;
+    unsigned long       *allocated_irqs; /* bitmap of IRQs allocated */
+
+    struct vgic_io_device   dist_iodev;
+
+    bool                has_its;
+
+    /*
+     * Contains the attributes and gpa of the LPI configuration table.
+     * Since we report GICR_TYPER.CommonLPIAff as 0b00, we can share
+     * one address across all redistributors.
+     * GICv3 spec: 6.1.2 "LPI Configuration tables"
+     */
+    uint64_t            propbaser;
+
+    /* Protects the lpi_list and the count value below. */
+    spinlock_t          lpi_list_lock;
+    struct list_head    lpi_list_head;
+    unsigned int        lpi_list_count;
+};
+
+struct vgic_cpu {
+    struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS];
+
+    struct list_head ap_list_head;
+    spinlock_t ap_list_lock;    /* Protects the ap_list */
+
+    unsigned int used_lrs;
+
+    /*
+     * List of IRQs that this VCPU should consider because they are either
+     * Active or Pending (hence the name; AP list), or because they recently
+     * were one of the two and need to be migrated off this list to another
+     * VCPU.
+     */
+
+    /*
+     * Members below are used with GICv3 emulation only and represent
+     * parts of the redistributor.
+     */
+    struct vgic_io_device   rd_iodev;
+    struct vgic_io_device   sgi_iodev;
+
+    /* Contains the attributes and gpa of the LPI pending tables. */
+    uint64_t pendbaser;
+
+    bool lpis_enabled;
+
+    /* Cache guest priority bits */
+    uint32_t num_pri_bits;
+
+    /* Cache guest interrupt ID bits */
+    uint32_t num_id_bits;
+};
+
+#define vgic_initialized(k) ((k)->arch.vgic.initialized)
+#define vgic_ready(k)       ((k)->arch.vgic.ready)
+#define vgic_valid_spi(k, i)    (((i) >= VGIC_NR_PRIVATE_IRQS) && \
+            ((i) < (k)->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS))
+
+#endif /* __ASM_ARM_NEW_VGIC_H */
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 84d82e6eb3..b28b8f8df7 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -18,6 +18,10 @@
 #ifndef __ASM_ARM_VGIC_H__
 #define __ASM_ARM_VGIC_H__
 
+#ifdef CONFIG_NEW_VGIC
+#include <asm/new_vgic.h>
+#else
+
 #include <xen/bitops.h>
 #include <xen/radix-tree.h>
 #include <xen/rbtree.h>
@@ -299,6 +303,8 @@ extern bool vgic_to_sgi(struct vcpu *v, register_t sgir,
                         const struct sgi_target *target);
 extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq);
 
+#endif /* !CONFIG_NEW_VGIC */
+
 /*** Common VGIC functions used by Xen arch code ****/
 
 /*
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 28/57] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Andre Przywara]

The new VGIC implementation centers around a struct vgic_irq instance
per virtual IRQ.
Provide a function to retrieve the right instance for a given IRQ
number and (in case of private interrupts) the right VCPU.
This also includes the corresponding put function, which does nothing
for private interrupts and SPIs, but handles the ref-counting for LPIs.

This is based on Linux commit 64a959d66e47, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- add kernel-doc comments to exported functions
- adapt to previous changes (new_vgic.h, arch_vcpu member name)
- use ASSERT_UNREACHABLE

 xen/arch/arm/vgic/vgic.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h |  41 ++++++++++++++++
 2 files changed, 165 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic.c
 create mode 100644 xen/arch/arm/vgic/vgic.h

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
new file mode 100644
index 0000000000..ace30f78d0
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic.c
@@ -0,0 +1,124 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/bug.h>
+#include <asm/new_vgic.h>
+#include <xen/sched.h>
+
+#include "vgic.h"
+
+/*
+ * Iterate over the VM's list of mapped LPIs to find the one with a
+ * matching interrupt ID and return a reference to the IRQ structure.
+ */
+static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+    struct vgic_irq *irq = NULL;
+
+    spin_lock(&dist->lpi_list_lock);
+
+    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
+    {
+        if ( irq->intid != intid )
+            continue;
+
+        /*
+         * This increases the refcount, the caller is expected to
+         * call vgic_put_irq() later once it's finished with the IRQ.
+         */
+        vgic_get_irq_kref(irq);
+        goto out_unlock;
+    }
+    irq = NULL;
+
+out_unlock:
+    spin_unlock(&dist->lpi_list_lock);
+
+    return irq;
+}
+
+/**
+ * vgic_get_irq() - obtain a reference to a virtual IRQ
+ * @d:        The domain the virtual IRQ belongs to.
+ * @vcpu:     For private IRQs (SGIs, PPIs) the virtual CPU this IRQ
+ *            is associated with. Will be ignored for SPIs and LPIs.
+ * @intid:    The virtual IRQ number.
+ *
+ * This looks up the virtual interrupt ID to get the corresponding
+ * struct vgic_irq. It also increases the refcount, so any caller is expected
+ * to call vgic_put_irq() once it's finished with this IRQ.
+ *
+ * Return: The pointer to the requested struct vgic_irq.
+ */
+struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
+                              u32 intid)
+{
+    /* SGIs and PPIs */
+    if ( intid <= VGIC_MAX_PRIVATE )
+        return &vcpu->arch.vgic.private_irqs[intid];
+
+    /* SPIs */
+    if ( intid <= VGIC_MAX_SPI )
+        return &d->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS];
+
+    /* LPIs */
+    if ( intid >= VGIC_MIN_LPI )
+        return vgic_get_lpi(d, intid);
+
+    ASSERT_UNREACHABLE();
+}
+
+/**
+ * vgic_put_irq() - drop the reference to a virtual IRQ
+ * @d:        The domain the virtual IRQ belongs to.
+ * @irq:      The pointer to struct vgic_irq, as obtained from vgic_get_irq().
+ *
+ * This drops the reference to a virtual IRQ. It decreases the refcount
+ * of the pointer, so dynamic IRQs can be freed when no longer needed.
+ * This should always be called after a vgic_get_irq(), though the reference
+ * can be deliberately held for longer periods, if needed.
+ */
+void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+
+    if ( irq->intid < VGIC_MIN_LPI )
+        return;
+
+    spin_lock(&dist->lpi_list_lock);
+    if ( !atomic_dec_and_test(&irq->refcount) )
+    {
+        spin_unlock(&dist->lpi_list_lock);
+        return;
+    };
+
+    list_del(&irq->lpi_list);
+    dist->lpi_list_count--;
+    spin_unlock(&dist->lpi_list_lock);
+
+    xfree(irq);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
new file mode 100644
index 0000000000..a3befd386b
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic.h
@@ -0,0 +1,41 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __XEN_ARM_VGIC_VGIC_H__
+#define __XEN_ARM_VGIC_VGIC_H__
+
+struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
+                              u32 intid);
+void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
+
+static inline void vgic_get_irq_kref(struct vgic_irq *irq)
+{
+    if ( irq->intid < VGIC_MIN_LPI )
+        return;
+
+    atomic_inc(&irq->refcount);
+}
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 29/57] ARM: new VGIC: Implement virtual IRQ injection

[Andre Przywara]

Provide a vgic_queue_irq_unlock() function which decides whether a
given IRQ needs to be queued to a VCPU's ap_list.
This should be called whenever an IRQ becomes pending or enabled,
either as a result of a hardware IRQ injection, from devices emulated by
Xen (like the architected timer) or from MMIO accesses to the distributor
emulation.
Also provides the necessary functions to allow to inject an IRQ to a guest.
Since this is the first code that starts using our locking mechanism,
we add some (hopefully) clear documentation of our locking strategy and
requirements along with this patch.

This is based on Linux commit 81eeb95ddbab, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- fix locking order comment
- adapt to former changes
- extend comments
- use kick_vcpu()

 xen/arch/arm/vgic/vgic.c | 227 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h |  10 +++
 2 files changed, 237 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index ace30f78d0..ae922815bd 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -21,6 +21,31 @@
 
 #include "vgic.h"
 
+/*
+ * Locking order is always:
+ *   vgic->lock
+ *     vgic_cpu->ap_list_lock
+ *       vgic->lpi_list_lock
+ *         desc->lock
+ *           vgic_irq->irq_lock
+ *
+ * If you need to take multiple locks, always take the upper lock first,
+ * then the lower ones, e.g. first take the ap_list_lock, then the irq_lock.
+ * If you are already holding a lock and need to take a higher one, you
+ * have to drop the lower ranking lock first and re-acquire it after having
+ * taken the upper one.
+ *
+ * When taking more than one ap_list_lock at the same time, always take the
+ * lowest numbered VCPU's ap_list_lock first, so:
+ *   vcpuX->vcpu_id < vcpuY->vcpu_id:
+ *     spin_lock(vcpuX->arch.vgic.ap_list_lock);
+ *     spin_lock(vcpuY->arch.vgic.ap_list_lock);
+ *
+ * Since the VGIC must support injecting virtual interrupts from ISRs, we have
+ * to use the spin_lock_irqsave/spin_unlock_irqrestore versions of outer
+ * spinlocks for any lock that may be taken while injecting an interrupt.
+ */
+
 /*
  * Iterate over the VM's list of mapped LPIs to find the one with a
  * matching interrupt ID and return a reference to the IRQ structure.
@@ -114,6 +139,208 @@ void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
     xfree(irq);
 }
 
+/**
+ * vgic_target_oracle() - compute the target vcpu for an irq
+ * @irq:    The irq to route. Must be already locked.
+ *
+ * Based on the current state of the interrupt (enabled, pending,
+ * active, vcpu and target_vcpu), compute the next vcpu this should be
+ * given to. Return NULL if this shouldn't be injected at all.
+ *
+ * Requires the IRQ lock to be held.
+ *
+ * Returns: The pointer to the virtual CPU this interrupt should be injected
+ *          to. Will be NULL if this IRQ does not need to be injected.
+ */
+static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
+{
+    ASSERT(spin_is_locked(&irq->irq_lock));
+
+    /* If the interrupt is active, it must stay on the current vcpu */
+    if ( irq->active )
+        return irq->vcpu ? : irq->target_vcpu;
+
+    /*
+     * If the IRQ is not active but enabled and pending, we should direct
+     * it to its configured target VCPU.
+     * If the distributor is disabled, pending interrupts shouldn't be
+     * forwarded.
+     */
+    if ( irq->enabled && irq_is_pending(irq) )
+    {
+        if ( unlikely(irq->target_vcpu &&
+                      !irq->target_vcpu->domain->arch.vgic.enabled) )
+            return NULL;
+
+        return irq->target_vcpu;
+    }
+
+    /*
+     * If neither active nor pending and enabled, then this IRQ should not
+     * be queued to any VCPU.
+     */
+    return NULL;
+}
+
+/*
+ * Only valid injection if changing level for level-triggered IRQs or for a
+ * rising edge.
+ */
+static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
+{
+    if ( irq->config == VGIC_CONFIG_LEVEL )
+        return irq->line_level != level;
+
+    return level;
+}
+
+/**
+ * vgic_queue_irq_unlock() - Queue an IRQ to a VCPU, to be injected to a guest.
+ * @d:        The domain the virtual IRQ belongs to.
+ * @irq:      A pointer to the vgic_irq of the virtual IRQ, with the lock held.
+ * @flags:    The flags used when having grabbed the IRQ lock.
+ *
+ * Check whether an IRQ needs to (and can) be queued to a VCPU's ap list.
+ * Do the queuing if necessary, taking the right locks in the right order.
+ *
+ * Needs to be entered with the IRQ lock already held, but will return
+ * with all locks dropped.
+ *
+ * Returns: True when the IRQ was queued, false otherwise.
+ */
+void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
+                           unsigned long flags)
+{
+    struct vcpu *vcpu;
+
+    ASSERT(spin_is_locked(&irq->irq_lock));
+
+retry:
+    vcpu = vgic_target_oracle(irq);
+    if ( irq->vcpu || !vcpu )
+    {
+        /*
+         * If this IRQ is already on a VCPU's ap_list, then it
+         * cannot be moved or modified and there is no more work for
+         * us to do.
+         *
+         * Otherwise, if the irq is not pending and enabled, it does
+         * not need to be inserted into an ap_list and there is also
+         * no more work for us to do.
+         */
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        /*
+         * We have to kick the VCPU here, because we could be
+         * queueing an edge-triggered interrupt for which we
+         * get no EOI maintenance interrupt. In that case,
+         * while the IRQ is already on the VCPU's AP list, the
+         * VCPU could have EOI'ed the original interrupt and
+         * won't see this one until it exits for some other
+         * reason.
+         */
+        if ( vcpu )
+            kick_vcpu(vcpu);
+
+        return;
+    }
+
+    /*
+     * We must unlock the irq lock to take the ap_list_lock where
+     * we are going to insert this new pending interrupt.
+     */
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+    /* someone can do stuff here, which we re-check below */
+
+    spin_lock_irqsave(&vcpu->arch.vgic.ap_list_lock, flags);
+    spin_lock(&irq->irq_lock);
+
+    /*
+     * Did something change behind our backs?
+     *
+     * There are two cases:
+     * 1) The irq lost its pending state or was disabled behind our
+     *    backs and/or it was queued to another VCPU's ap_list.
+     * 2) Someone changed the affinity on this irq behind our
+     *    backs and we are now holding the wrong ap_list_lock.
+     *
+     * In both cases, drop the locks and retry.
+     */
+
+    if ( unlikely(irq->vcpu || vcpu != vgic_target_oracle(irq)) )
+    {
+        spin_unlock(&irq->irq_lock);
+        spin_unlock_irqrestore(&vcpu->arch.vgic.ap_list_lock, flags);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        goto retry;
+    }
+
+    /*
+     * Grab a reference to the irq to reflect the fact that it is
+     * now in the ap_list.
+     */
+    vgic_get_irq_kref(irq);
+    list_add_tail(&irq->ap_list, &vcpu->arch.vgic.ap_list_head);
+    irq->vcpu = vcpu;
+
+    spin_unlock(&irq->irq_lock);
+    spin_unlock_irqrestore(&vcpu->arch.vgic.ap_list_lock, flags);
+
+    kick_vcpu(vcpu);
+
+    return;
+}
+
+/**
+ * vgic_inject_irq() - Inject an IRQ from a device to the vgic
+ * @d:       The domain pointer
+ * @vcpu:    The vCPU for private IRQs (PPIs, SGIs). Ignored for SPIs and LPIs.
+ * @intid:   The INTID to inject a new state to.
+ * @level:   Edge-triggered:  true:  to trigger the interrupt
+ *                            false: to ignore the call
+ *           Level-sensitive  true:  raise the input signal
+ *                            false: lower the input signal
+ *
+ * Injects an instance of the given virtual IRQ into a domain.
+ * The VGIC is not concerned with devices being active-LOW or active-HIGH for
+ * level-sensitive interrupts.  You can think of the level parameter as 1
+ * being HIGH and 0 being LOW and all devices being active-HIGH.
+ *
+ * Return: A negative error code if the injection failed, or 0 on success.
+ */
+int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int intid,
+                    bool level)
+{
+    struct vgic_irq *irq;
+    unsigned long flags;
+
+    irq = vgic_get_irq(d, vcpu, intid);
+    if ( !irq )
+        return -EINVAL;
+
+    spin_lock_irqsave(&irq->irq_lock, flags);
+
+    if ( !vgic_validate_injection(irq, level) )
+    {
+        /* Nothing to see here, move along... */
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(d, irq);
+        return 0;
+    }
+
+    if ( irq->config == VGIC_CONFIG_LEVEL )
+        irq->line_level = level;
+    else
+        irq->pending_latch = true;
+
+    vgic_queue_irq_unlock(d, irq, flags);
+    vgic_put_irq(d, irq);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index a3befd386b..3430955d9f 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -17,9 +17,19 @@
 #ifndef __XEN_ARM_VGIC_VGIC_H__
 #define __XEN_ARM_VGIC_VGIC_H__
 
+static inline bool irq_is_pending(struct vgic_irq *irq)
+{
+    if ( irq->config == VGIC_CONFIG_EDGE )
+        return irq->pending_latch;
+    else
+        return irq->pending_latch || irq->line_level;
+}
+
 struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
                               u32 intid);
 void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
+void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
+               unsigned long flags);
 
 static inline void vgic_get_irq_kref(struct vgic_irq *irq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 30/57] ARM: new VGIC: Add IRQ sorting

[Andre Przywara]

Adds the sorting function to cover the case where you have more IRQs
to consider than you have LRs. We consider their priorities.
This pulls in Linux' list_sort.c, which is a merge sort implementation
for linked lists. Apart from adding a full featured license header and
adjusting the #include file, nothing has been changed in this code.

This is based on Linux commit 8e4447457965, written by Christoffer Dall.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use Linux coding style for list_sort.c

 xen/arch/arm/vgic/vgic.c    |  59 +++++++++++++++++
 xen/common/list_sort.c      | 157 ++++++++++++++++++++++++++++++++++++++++++++
 xen/include/xen/list_sort.h |  11 ++++
 3 files changed, 227 insertions(+)
 create mode 100644 xen/common/list_sort.c
 create mode 100644 xen/include/xen/list_sort.h

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index ae922815bd..efa6c67cb7 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -17,6 +17,7 @@
 
 #include <asm/bug.h>
 #include <asm/new_vgic.h>
+#include <xen/list_sort.h>
 #include <xen/sched.h>
 
 #include "vgic.h"
@@ -182,6 +183,64 @@ static struct vcpu *vgic_target_oracle(struct vgic_irq *irq)
     return NULL;
 }
 
+/*
+ * The order of items in the ap_lists defines how we'll pack things in LRs as
+ * well, the first items in the list being the first things populated in the
+ * LRs.
+ *
+ * A hard rule is that active interrupts can never be pushed out of the LRs
+ * (and therefore take priority) since we cannot reliably trap on deactivation
+ * of IRQs and therefore they have to be present in the LRs.
+ *
+ * Otherwise things should be sorted by the priority field and the GIC
+ * hardware support will take care of preemption of priority groups etc.
+ *
+ * Return negative if "a" sorts before "b", 0 to preserve order, and positive
+ * to sort "b" before "a".
+ */
+static int vgic_irq_cmp(void *priv, struct list_head *a, struct list_head *b)
+{
+    struct vgic_irq *irqa = container_of(a, struct vgic_irq, ap_list);
+    struct vgic_irq *irqb = container_of(b, struct vgic_irq, ap_list);
+    bool penda, pendb;
+    int ret;
+
+    spin_lock(&irqa->irq_lock);
+    spin_lock(&irqb->irq_lock);
+
+    if ( irqa->active || irqb->active )
+    {
+        ret = (int)irqb->active - (int)irqa->active;
+        goto out;
+    }
+
+    penda = irqa->enabled && irq_is_pending(irqa);
+    pendb = irqb->enabled && irq_is_pending(irqb);
+
+    if ( !penda || !pendb )
+    {
+        ret = (int)pendb - (int)penda;
+        goto out;
+    }
+
+    /* Both pending and enabled, sort by priority */
+    ret = irqa->priority - irqb->priority;
+out:
+    spin_unlock(&irqb->irq_lock);
+    spin_unlock(&irqa->irq_lock);
+    return ret;
+}
+
+/* Must be called with the ap_list_lock held */
+static void vgic_sort_ap_list(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+
+    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
+
+    list_sort(NULL, &vgic_cpu->ap_list_head, vgic_irq_cmp);
+}
+
 /*
  * Only valid injection if changing level for level-triggered IRQs or for a
  * rising edge.
diff --git a/xen/common/list_sort.c b/xen/common/list_sort.c
new file mode 100644
index 0000000000..af2b2f6519
--- /dev/null
+++ b/xen/common/list_sort.c
@@ -0,0 +1,157 @@
+/*
+ * list_sort.c: merge sort implementation for linked lists
+ * Copied from the Linux kernel (lib/list_sort.c)
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms and conditions of the GNU General Public License,
+ * version 2, as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for
+ * more details.
+ *
+ * You should have received a copy of the GNU General Public License along with
+ * this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/lib.h>
+#include <xen/list.h>
+
+#define MAX_LIST_LENGTH_BITS 20
+
+/*
+ * Returns a list organized in an intermediate format suited
+ * to chaining of merge() calls: null-terminated, no reserved or
+ * sentinel head node, "prev" links not maintained.
+ */
+static struct list_head *merge(void *priv,
+				int (*cmp)(void *priv, struct list_head *a,
+					struct list_head *b),
+				struct list_head *a, struct list_head *b)
+{
+	struct list_head head, *tail = &head;
+
+	while (a && b) {
+		/* if equal, take 'a' -- important for sort stability */
+		if ((*cmp)(priv, a, b) <= 0) {
+			tail->next = a;
+			a = a->next;
+		} else {
+			tail->next = b;
+			b = b->next;
+		}
+		tail = tail->next;
+	}
+	tail->next = a?:b;
+	return head.next;
+}
+
+/*
+ * Combine final list merge with restoration of standard doubly-linked
+ * list structure.  This approach duplicates code from merge(), but
+ * runs faster than the tidier alternatives of either a separate final
+ * prev-link restoration pass, or maintaining the prev links
+ * throughout.
+ */
+static void merge_and_restore_back_links(void *priv,
+				int (*cmp)(void *priv, struct list_head *a,
+					struct list_head *b),
+				struct list_head *head,
+				struct list_head *a, struct list_head *b)
+{
+	struct list_head *tail = head;
+	u8 count = 0;
+
+	while (a && b) {
+		/* if equal, take 'a' -- important for sort stability */
+		if ((*cmp)(priv, a, b) <= 0) {
+			tail->next = a;
+			a->prev = tail;
+			a = a->next;
+		} else {
+			tail->next = b;
+			b->prev = tail;
+			b = b->next;
+		}
+		tail = tail->next;
+	}
+	tail->next = a ? : b;
+
+	do {
+		/*
+		 * In worst cases this loop may run many iterations.
+		 * Continue callbacks to the client even though no
+		 * element comparison is needed, so the client's cmp()
+		 * routine can invoke cond_resched() periodically.
+		 */
+		if (unlikely(!(++count)))
+			(*cmp)(priv, tail->next, tail->next);
+
+		tail->next->prev = tail;
+		tail = tail->next;
+	} while (tail->next);
+
+	tail->next = head;
+	head->prev = tail;
+}
+
+/**
+ * list_sort - sort a list
+ * @priv: private data, opaque to list_sort(), passed to @cmp
+ * @head: the list to sort
+ * @cmp: the elements comparison function
+ *
+ * This function implements "merge sort", which has O(nlog(n))
+ * complexity.
+ *
+ * The comparison function @cmp must return a negative value if @a
+ * should sort before @b, and a positive value if @a should sort after
+ * @b. If @a and @b are equivalent, and their original relative
+ * ordering is to be preserved, @cmp must return 0.
+ */
+void list_sort(void *priv, struct list_head *head,
+		int (*cmp)(void *priv, struct list_head *a,
+			struct list_head *b))
+{
+	struct list_head *part[MAX_LIST_LENGTH_BITS+1]; /* sorted partial lists
+						-- last slot is a sentinel */
+	int lev;  /* index into part[] */
+	int max_lev = 0;
+	struct list_head *list;
+
+	if (list_empty(head))
+		return;
+
+	memset(part, 0, sizeof(part));
+
+	head->prev->next = NULL;
+	list = head->next;
+
+	while (list) {
+		struct list_head *cur = list;
+		list = list->next;
+		cur->next = NULL;
+
+		for (lev = 0; part[lev]; lev++) {
+			cur = merge(priv, cmp, part[lev], cur);
+			part[lev] = NULL;
+		}
+		if (lev > max_lev) {
+			if (unlikely(lev >= ARRAY_SIZE(part)-1)) {
+				dprintk(XENLOG_DEBUG,
+					"list too long for efficiency\n");
+				lev--;
+			}
+			max_lev = lev;
+		}
+		part[lev] = cur;
+	}
+
+	for (lev = 0; lev < max_lev; lev++)
+		if (part[lev])
+			list = merge(priv, cmp, part[lev], list);
+
+	merge_and_restore_back_links(priv, cmp, head, part[max_lev], list);
+}
+EXPORT_SYMBOL(list_sort);
diff --git a/xen/include/xen/list_sort.h b/xen/include/xen/list_sort.h
new file mode 100644
index 0000000000..13ce0a55ec
--- /dev/null
+++ b/xen/include/xen/list_sort.h
@@ -0,0 +1,11 @@
+#ifndef _LINUX_LIST_SORT_H
+#define _LINUX_LIST_SORT_H
+
+#include <xen/types.h>
+
+struct list_head;
+
+void list_sort(void *priv, struct list_head *head,
+	       int (*cmp)(void *priv, struct list_head *a,
+			  struct list_head *b));
+#endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 31/57] ARM: new VGIC: Add IRQ sync/flush framework

[Andre Przywara]

Implement the framework for syncing IRQs between our emulation and the
list registers, which represent the guest's view of IRQs.
This is done in kvm_vgic_flush_hwstate and kvm_vgic_sync_hwstate, which
gets called on guest entry and exit.
The code talking to the actual GICv2/v3 hardware is added in the
following patches.

This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- extend comments
- adapt to former changes
- remove gic_clear_lrs()

 xen/arch/arm/vgic/vgic.c | 239 +++++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h |   2 +
 2 files changed, 241 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index efa6c67cb7..8e5215a00d 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -400,6 +400,245 @@ int vgic_inject_irq(struct domain *d, struct vcpu *vcpu, unsigned int intid,
     return 0;
 }
 
+/**
+ * vgic_prune_ap_list() - Remove non-relevant interrupts from the ap_list
+ *
+ * @vcpu:       The VCPU of which the ap_list should be pruned.
+ *
+ * Go over the list of interrupts on a VCPU's ap_list, and prune those that
+ * we won't have to consider in the near future.
+ * This removes interrupts that have been successfully handled by the guest,
+ * or that have otherwise became obsolete (not pending anymore).
+ * Also this moves interrupts between VCPUs, if their affinity has changed.
+ */
+static void vgic_prune_ap_list(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+    struct vgic_irq *irq, *tmp;
+    unsigned long flags;
+
+retry:
+    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
+
+    list_for_each_entry_safe( irq, tmp, &vgic_cpu->ap_list_head, ap_list )
+    {
+        struct vcpu *target_vcpu, *vcpuA, *vcpuB;
+
+        spin_lock(&irq->irq_lock);
+
+        BUG_ON(vcpu != irq->vcpu);
+
+        target_vcpu = vgic_target_oracle(irq);
+
+        if ( !target_vcpu )
+        {
+            /*
+             * We don't need to process this interrupt any
+             * further, move it off the list.
+             */
+            list_del(&irq->ap_list);
+            irq->vcpu = NULL;
+            spin_unlock(&irq->irq_lock);
+
+            /*
+             * This vgic_put_irq call matches the
+             * vgic_get_irq_kref in vgic_queue_irq_unlock,
+             * where we added the LPI to the ap_list. As
+             * we remove the irq from the list, we drop
+             * also drop the refcount.
+             */
+            vgic_put_irq(vcpu->domain, irq);
+            continue;
+        }
+
+        if ( target_vcpu == vcpu )
+        {
+            /* We're on the right CPU */
+            spin_unlock(&irq->irq_lock);
+            continue;
+        }
+
+        /* This interrupt looks like it has to be migrated. */
+
+        spin_unlock(&irq->irq_lock);
+        spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
+
+        /*
+         * Ensure locking order by always locking the smallest
+         * ID first.
+         */
+        if ( vcpu->vcpu_id < target_vcpu->vcpu_id )
+        {
+            vcpuA = vcpu;
+            vcpuB = target_vcpu;
+        }
+        else
+        {
+            vcpuA = target_vcpu;
+            vcpuB = vcpu;
+        }
+
+        spin_lock_irqsave(&vcpuA->arch.vgic.ap_list_lock, flags);
+        spin_lock(&vcpuB->arch.vgic.ap_list_lock);
+        spin_lock(&irq->irq_lock);
+
+        /*
+         * If the affinity has been preserved, move the
+         * interrupt around. Otherwise, it means things have
+         * changed while the interrupt was unlocked, and we
+         * need to replay this.
+         *
+         * In all cases, we cannot trust the list not to have
+         * changed, so we restart from the beginning.
+         */
+        if ( target_vcpu == vgic_target_oracle(irq) )
+        {
+            struct vgic_cpu *new_cpu = &target_vcpu->arch.vgic;
+
+            list_del(&irq->ap_list);
+            irq->vcpu = target_vcpu;
+            list_add_tail(&irq->ap_list, &new_cpu->ap_list_head);
+        }
+
+        spin_unlock(&irq->irq_lock);
+        spin_unlock(&vcpuB->arch.vgic.ap_list_lock);
+        spin_unlock_irqrestore(&vcpuA->arch.vgic.ap_list_lock, flags);
+        goto retry;
+    }
+
+    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
+}
+
+static inline void vgic_fold_lr_state(struct vcpu *vcpu)
+{
+}
+
+/* Requires the irq_lock to be held. */
+static inline void vgic_populate_lr(struct vcpu *vcpu,
+                                    struct vgic_irq *irq, int lr)
+{
+    ASSERT(spin_is_locked(&irq->irq_lock));
+}
+
+static inline void vgic_set_underflow(struct vcpu *vcpu)
+{
+}
+
+/* Requires the ap_list_lock to be held. */
+static int compute_ap_list_depth(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+    struct vgic_irq *irq;
+    int count = 0;
+
+    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
+
+    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
+    {
+        spin_lock(&irq->irq_lock);
+        /* GICv2 SGIs can count for more than one... */
+        if ( vgic_irq_is_sgi(irq->intid) && irq->source )
+            count += hweight8(irq->source);
+        else
+            count++;
+        spin_unlock(&irq->irq_lock);
+    }
+    return count;
+}
+
+/* Requires the VCPU's ap_list_lock to be held. */
+static void vgic_flush_lr_state(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+    struct vgic_irq *irq;
+    int count = 0;
+
+    ASSERT(spin_is_locked(&vgic_cpu->ap_list_lock));
+
+    if ( compute_ap_list_depth(vcpu) > gic_get_nr_lrs() )
+        vgic_sort_ap_list(vcpu);
+
+    list_for_each_entry( irq, &vgic_cpu->ap_list_head, ap_list )
+    {
+        spin_lock(&irq->irq_lock);
+
+        if ( unlikely(vgic_target_oracle(irq) != vcpu) )
+            goto next;
+
+        /*
+         * If we get an SGI with multiple sources, try to get
+         * them in all at once.
+         */
+        do
+        {
+            vgic_populate_lr(vcpu, irq, count++);
+        } while ( irq->source && count < gic_get_nr_lrs() );
+
+next:
+        spin_unlock(&irq->irq_lock);
+
+        if ( count == gic_get_nr_lrs() )
+        {
+            if ( !list_is_last(&irq->ap_list, &vgic_cpu->ap_list_head) )
+                vgic_set_underflow(vcpu);
+            break;
+        }
+    }
+
+    vcpu->arch.vgic.used_lrs = count;
+}
+
+/**
+ * vgic_sync_from_lrs() - Update VGIC state from hardware after a guest's run.
+ * @vcpu: the VCPU for which to transfer from the LRs to the IRQ list.
+ *
+ * Sync back the hardware VGIC state after the guest has run, into our
+ * VGIC emulation structures, It reads the LRs and updates the respective
+ * struct vgic_irq, taking level/edge into account.
+ * This is the high level function which takes care of the conditions,
+ * also bails out early if there were no interrupts queued.
+ * Was: kvm_vgic_sync_hwstate()
+ */
+void vgic_sync_from_lrs(struct vcpu *vcpu)
+{
+    /* An empty ap_list_head implies used_lrs == 0 */
+    if ( list_empty(&vcpu->arch.vgic.ap_list_head) )
+        return;
+
+    vgic_fold_lr_state(vcpu);
+
+    vgic_prune_ap_list(vcpu);
+}
+
+/**
+ * vgic_sync_to_lrs() - flush emulation state into the hardware on guest entry
+ *
+ * Before we enter a guest, we have to translate the virtual GIC state of a
+ * VCPU into the GIC virtualization hardware registers, namely the LRs.
+ * This is the high level function which takes care about the conditions
+ * and the locking, also bails out early if there are no interrupts queued.
+ * Was: kvm_vgic_flush_hwstate()
+ */
+void vgic_sync_to_lrs(void)
+{
+    /*
+     * If there are no virtual interrupts active or pending for this
+     * VCPU, then there is no work to do and we can bail out without
+     * taking any lock.  There is a potential race with someone injecting
+     * interrupts to the VCPU, but it is a benign race as the VCPU will
+     * either observe the new interrupt before or after doing this check,
+     * and introducing additional synchronization mechanism doesn't change
+     * this.
+     */
+    if ( list_empty(&current->arch.vgic.ap_list_head) )
+        return;
+
+    ASSERT(!local_irq_is_enabled());
+
+    spin_lock(&current->arch.vgic.ap_list_lock);
+    vgic_flush_lr_state(current);
+    spin_unlock(&current->arch.vgic.ap_list_lock);
+}
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 3430955d9f..a495116cb7 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -17,6 +17,8 @@
 #ifndef __XEN_ARM_VGIC_VGIC_H__
 #define __XEN_ARM_VGIC_VGIC_H__
 
+#define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
+
 static inline bool irq_is_pending(struct vgic_irq *irq)
 {
     if ( irq->config == VGIC_CONFIG_EDGE )
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 32/57] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Processing maintenance interrupts and accessing the list registers
are dependent on the host's GIC version.
Introduce vgic-v2.c to contain GICv2 specific functions.
Implement the GICv2 specific code for syncing the emulation state
into the VGIC registers.
This also adds the hook to let Xen setup the host GIC addresses.

This is based on Linux commit 140b086dd197, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- extend comments
- adapt to former changes
- use existing Xen LR accessor interface (->write_lr, ->read_lr)
- merge save_state and restore_state into callers
- add vgic_irq_is_mapped_level() helper

 xen/arch/arm/vgic/vgic-v2.c | 231 ++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.c    |   7 ++
 xen/arch/arm/vgic/vgic.h    |   9 ++
 3 files changed, 247 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-v2.c

diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
new file mode 100644
index 0000000000..4e74ebf7f5
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -0,0 +1,231 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <asm/new_vgic.h>
+#include <asm/bug.h>
+#include <asm/gic.h>
+#include <asm/io.h>
+#include <xen/sched.h>
+#include <xen/sizes.h>
+
+#include "vgic.h"
+
+static struct {
+    bool enabled;
+    paddr_t dbase;          /* Distributor interface address */
+    paddr_t cbase;          /* CPU interface address & size */
+    paddr_t csize;
+    paddr_t vbase;          /* Virtual CPU interface address */
+
+    /* Offset to add to get an 8kB contiguous region if GIC is aliased */
+    uint32_t aliased_offset;
+} gic_v2_hw_data;
+
+void vgic_v2_setup_hw(paddr_t dbase, paddr_t cbase, paddr_t csize,
+                      paddr_t vbase, uint32_t aliased_offset)
+{
+    gic_v2_hw_data.enabled = true;
+    gic_v2_hw_data.dbase = dbase;
+    gic_v2_hw_data.cbase = cbase;
+    gic_v2_hw_data.csize = csize;
+    gic_v2_hw_data.vbase = vbase;
+    gic_v2_hw_data.aliased_offset = aliased_offset;
+}
+
+void vgic_v2_set_underflow(struct vcpu *vcpu)
+{
+    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 1);
+}
+
+/*
+ * transfer the content of the LRs back into the corresponding ap_list:
+ * - active bit is transferred as is
+ * - pending bit is
+ *   - transferred as is in case of edge sensitive IRQs
+ *   - set to the line-level (resample time) for level sensitive IRQs
+ */
+void vgic_v2_fold_lr_state(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+    unsigned int used_lrs = vcpu->arch.vgic.used_lrs;
+    unsigned long flags;
+    unsigned int lr;
+
+    if ( !used_lrs )    /* No LRs used, so nothing to sync back here. */
+        return;
+
+    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 0);
+
+    for ( lr = 0; lr < used_lrs; lr++ )
+    {
+        struct gic_lr lr_val;
+        uint32_t intid;
+        struct vgic_irq *irq;
+
+        gic_hw_ops->read_lr(lr, &lr_val);
+
+        /*
+         * TODO: Possible optimization to avoid reading LRs:
+         * Read the ELRSR to find out which of our LRs have been cleared
+         * by the guest. We just need to know the IRQ number for those, which
+         * we could save in an array when populating the LRs.
+         * This trades one MMIO access (ELRSR) for possibly more than one (LRs),
+         * but requires some more code to save the IRQ number and to handle
+         * those finished IRQs according to the algorithm below.
+         * We need some numbers to justify this: chances are that we don't
+         * have many LRs in use most of the time, so we might not save much.
+         */
+        gic_hw_ops->clear_lr(lr);
+
+        intid = lr_val.virq;
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        /* Always preserve the active bit */
+        irq->active = !!(lr_val.state & GICH_LR_ACTIVE);
+
+        /* Edge is the only case where we preserve the pending bit */
+        if ( irq->config == VGIC_CONFIG_EDGE && (lr_val.state & GICH_LR_PENDING) )
+        {
+            irq->pending_latch = true;
+
+            if ( vgic_irq_is_sgi(intid) )
+                irq->source |= (1U << lr_val.source);
+        }
+
+	/*
+	 * Level-triggered mapped IRQs are special because we only
+	 * observe rising edges as input to the VGIC.
+	 *
+	 * If the guest never acked the interrupt we have to sample
+	 * the physical line and set the line level, because the
+	 * device state could have changed or we simply need to
+	 * process the still pending interrupt later.
+	 *
+	 * If this causes us to lower the level, we have to also clear
+	 * the physical active state, since we will otherwise never be
+	 * told when the interrupt becomes asserted again.
+	 */
+        if ( vgic_irq_is_mapped_level(irq) && (lr_val.state & GICH_LR_PENDING) )
+        {
+            struct irq_desc *irqd;
+
+            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
+
+            irqd = irq_to_desc(irq->hwintid);
+            irq->line_level = gic_read_pending_state(irqd);
+
+            if ( !irq->line_level )
+                gic_set_active_state(irqd, false);
+        }
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    gic_hw_ops->update_hcr_status(GICH_HCR_EN, 0);
+    vgic_cpu->used_lrs = 0;
+}
+
+/**
+ * vgic_v2_populate_lr() - Populates an LR with the state of a given IRQ.
+ * @vcpu: The VCPU which the given @irq belongs to.
+ * @irq:  The IRQ to convert into an LR. The irq_lock must be held already.
+ * @lr:   The LR number to transfer the state into.
+ *
+ * This moves a virtual IRQ, represented by its vgic_irq, into a list register.
+ * Apart from translating the logical state into the LR bitfields, it also
+ * changes some state in the vgic_irq.
+ * For an edge sensitive IRQ the pending state is cleared in struct vgic_irq,
+ * for a level sensitive IRQ the pending state value is unchanged, as it is
+ * dictated directly by the input line level.
+ *
+ * If @irq describes an SGI with multiple sources, we choose the
+ * lowest-numbered source VCPU and clear that bit in the source bitmap.
+ *
+ * The irq_lock must be held by the caller.
+ */
+void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)
+{
+    struct gic_lr lr_val = {0};
+
+    lr_val.virq = irq->intid;
+
+    if ( irq_is_pending(irq) )
+    {
+        lr_val.state |= GICH_LR_PENDING;
+
+        if ( irq->config == VGIC_CONFIG_EDGE )
+            irq->pending_latch = false;
+
+        if ( vgic_irq_is_sgi(irq->intid) )
+        {
+            u32 src = ffs(irq->source);
+
+            BUG_ON(!src);
+            lr_val.source = (src - 1);
+            irq->source &= ~(1 << (src - 1));
+            if ( irq->source )
+                irq->pending_latch = true;
+        }
+    }
+
+    if ( irq->active )
+        lr_val.state |= GICH_LR_ACTIVE;
+
+    if ( irq->hw )
+    {
+        lr_val.hw_status = 1;
+        lr_val.pirq = irq->hwintid;
+        /*
+         * Never set pending+active on a HW interrupt, as the
+         * pending state is kept at the physical distributor
+         * level.
+         */
+        if ( irq->active && irq_is_pending(irq) )
+            lr_val.state &= ~GICH_LR_PENDING;
+    }
+    else
+    {
+        if ( irq->config == VGIC_CONFIG_LEVEL )
+            lr_val.eoi = 1;
+    }
+
+    /*
+     * Level-triggered mapped IRQs are special because we only observe
+     * rising edges as input to the VGIC.  We therefore lower the line
+     * level here, so that we can take new virtual IRQs.  See
+     * vgic_v2_fold_lr_state for more info.
+     */
+    if ( vgic_irq_is_mapped_level(irq) && (lr_val.state & GICH_LR_PENDING) )
+        irq->line_level = false;
+
+    /* The GICv2 LR only holds five bits of priority. */
+    lr_val.priority = irq->priority >> 3;
+
+    gic_hw_ops->write_lr(lr, &lr_val);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 8e5215a00d..85e39f6f42 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -511,6 +511,7 @@ retry:
 
 static inline void vgic_fold_lr_state(struct vcpu *vcpu)
 {
+    vgic_v2_fold_lr_state(vcpu);
 }
 
 /* Requires the irq_lock to be held. */
@@ -518,10 +519,13 @@ static inline void vgic_populate_lr(struct vcpu *vcpu,
                                     struct vgic_irq *irq, int lr)
 {
     ASSERT(spin_is_locked(&irq->irq_lock));
+
+    vgic_v2_populate_lr(vcpu, irq, lr);
 }
 
 static inline void vgic_set_underflow(struct vcpu *vcpu)
 {
+    vgic_v2_set_underflow(vcpu);
 }
 
 /* Requires the ap_list_lock to be held. */
@@ -638,7 +642,10 @@ void vgic_sync_to_lrs(void)
     spin_lock(&current->arch.vgic.ap_list_lock);
     vgic_flush_lr_state(current);
     spin_unlock(&current->arch.vgic.ap_list_lock);
+
+    gic_hw_ops->update_hcr_status(GICH_HCR_EN, 1);
 }
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index a495116cb7..116b26544e 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -27,6 +27,11 @@ static inline bool irq_is_pending(struct vgic_irq *irq)
         return irq->pending_latch || irq->line_level;
 }
 
+static inline bool vgic_irq_is_mapped_level(struct vgic_irq *irq)
+{
+    return irq->config == VGIC_CONFIG_LEVEL && irq->hw;
+}
+
 struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
                               u32 intid);
 void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
@@ -41,6 +46,10 @@ static inline void vgic_get_irq_kref(struct vgic_irq *irq)
     atomic_inc(&irq->refcount);
 }
 
+void vgic_v2_fold_lr_state(struct vcpu *vcpu);
+void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
+void vgic_v2_set_underflow(struct vcpu *vcpu);
+
 #endif
 
 /*
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 33/57] ARM: new VGIC: Implement vgic_vcpu_pending_irq

[Andre Przywara]

Tell Xen whether a particular VCPU has an IRQ that needs handling
in the guest. This is used to decide whether a VCPU is runnable or
if a hypercall should be preempted to let the guest handle the IRQ.

This is based on Linux commit 90eee56c5f90, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- extend commit message
- use new function name

 xen/arch/arm/vgic/vgic.c | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 85e39f6f42..66a366176a 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -646,6 +646,43 @@ void vgic_sync_to_lrs(void)
     gic_hw_ops->update_hcr_status(GICH_HCR_EN, 1);
 }
 
+static int vgic_vcpu_pending_irq(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+    struct vgic_irq *irq;
+    bool pending = false;
+    unsigned long flags;
+
+    if ( !vcpu->domain->arch.vgic.enabled )
+        return false;
+
+    spin_lock_irqsave(&vgic_cpu->ap_list_lock, flags);
+
+    list_for_each_entry(irq, &vgic_cpu->ap_list_head, ap_list)
+    {
+        spin_lock(&irq->irq_lock);
+        pending = irq_is_pending(irq) && irq->enabled;
+        spin_unlock(&irq->irq_lock);
+
+        if ( pending )
+            break;
+    }
+
+    spin_unlock_irqrestore(&vgic_cpu->ap_list_lock, flags);
+
+    return pending;
+}
+
+/**
+ * vgic_pending_irq() - determine if interrupts need to be injected
+ *
+ * Returns: 1 if the guest should run to handle interrupts, 0 otherwise.
+ */
+int vgic_pending_irq(void)
+{
+    return vgic_vcpu_pending_irq(current);
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 34/57] ARM: new VGIC: Add MMIO handling framework

[Andre Przywara]

Add an MMIO handling framework to the VGIC emulation:
Each register is described by its offset, size (or number of bits per
IRQ, if applicable) and the read/write handler functions. We provide
initialization macros to describe each GIC register later easily.

Separate dispatch functions for read and write accesses are connected
to Xen's MMIO handling framework and binary-search for the responsible
register handler based on the offset address within the region.

The register handler prototype are courtesy of Christoffer Dall.

This is based on Linux commit 4493b1c4866a, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- adapt to former changes
- whitespace fixes
- use C99 data types
- drop unneeded regions support (ITS, CPU i/f)

 xen/arch/arm/vgic/vgic-mmio.c | 180 ++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h |  99 +++++++++++++++++++++++
 2 files changed, 279 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
 create mode 100644 xen/arch/arm/vgic/vgic-mmio.h

diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
new file mode 100644
index 0000000000..393460d25a
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -0,0 +1,180 @@
+/*
+ * VGIC MMIO handling functions
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/bitops.h>
+#include <xen/lib.h>
+#include <xen/sched.h>
+#include <asm/new_vgic.h>
+#include <asm/byteorder.h>
+
+#include "vgic.h"
+#include "vgic-mmio.h"
+
+unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
+                                 paddr_t addr, unsigned int len)
+{
+    return 0;
+}
+
+unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
+                                 paddr_t addr, unsigned int len)
+{
+    return -1UL;
+}
+
+void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
+                        unsigned int len, unsigned long val)
+{
+    /* Ignore */
+}
+
+static int match_region(const void *key, const void *elt)
+{
+    const unsigned int offset = (unsigned long)key;
+    const struct vgic_register_region *region = elt;
+
+    if ( offset < region->reg_offset )
+        return -1;
+
+    if ( offset >= region->reg_offset + region->len )
+        return 1;
+
+    return 0;
+}
+
+static const struct vgic_register_region *
+vgic_find_mmio_region(const struct vgic_register_region *regions,
+                      int nr_regions, unsigned int offset)
+{
+    return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
+                   sizeof(regions[0]), match_region);
+}
+
+static bool check_region(const struct domain *d,
+                         const struct vgic_register_region *region,
+                         paddr_t addr, int len)
+{
+    int flags, nr_irqs = d->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
+
+    switch ( len )
+    {
+    case sizeof(uint8_t):
+        flags = VGIC_ACCESS_8bit;
+        break;
+    case sizeof(uint32_t):
+        flags = VGIC_ACCESS_32bit;
+        break;
+    case sizeof(uint64_t):
+        flags = VGIC_ACCESS_64bit;
+        break;
+    default:
+        return false;
+    }
+
+    if ( (region->access_flags & flags) && IS_ALIGNED(addr, len) )
+    {
+        if ( !region->bits_per_irq )
+            return true;
+
+        /* Do we access a non-allocated IRQ? */
+        return VGIC_ADDR_TO_INTID(addr, region->bits_per_irq) < nr_irqs;
+    }
+
+    return false;
+}
+
+static const struct vgic_register_region *
+vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
+                     paddr_t addr, unsigned int len)
+{
+    const struct vgic_register_region *region;
+
+    region = vgic_find_mmio_region(iodev->regions, iodev->nr_regions,
+                                   addr - gfn_to_gaddr(iodev->base_fn));
+    if ( !region || !check_region(vcpu->domain, region, addr, len) )
+        return NULL;
+
+    return region;
+}
+
+static int dispatch_mmio_read(struct vcpu *vcpu, mmio_info_t *info,
+                              register_t *r, void *priv)
+{
+    struct vgic_io_device *iodev = priv;
+    const struct vgic_register_region *region;
+    unsigned long data = 0;
+    paddr_t addr = info->gpa;
+    int len = 1U << info->dabt.size;
+
+    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
+    if ( !region )
+    {
+        memset(r, 0, len);
+        return 0;
+    }
+
+    switch (iodev->iodev_type)
+    {
+    case IODEV_DIST:
+        data = region->read(vcpu, addr, len);
+        break;
+    case IODEV_REDIST:
+        data = region->read(iodev->redist_vcpu, addr, len);
+        break;
+    }
+
+    memcpy(r, &data, len);
+
+    return 1;
+}
+
+static int dispatch_mmio_write(struct vcpu *vcpu, mmio_info_t *info,
+                               register_t r, void *priv)
+{
+    struct vgic_io_device *iodev = priv;
+    const struct vgic_register_region *region;
+    unsigned long data = r;
+    paddr_t addr = info->gpa;
+    int len = 1U << info->dabt.size;
+
+    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
+    if ( !region )
+        return 0;
+
+    switch (iodev->iodev_type)
+    {
+    case IODEV_DIST:
+        region->write(vcpu, addr, len, data);
+        break;
+    case IODEV_REDIST:
+        region->write(iodev->redist_vcpu, addr, len, data);
+        break;
+    }
+
+    return 1;
+}
+
+struct mmio_handler_ops vgic_io_ops = {
+    .read = dispatch_mmio_read,
+    .write = dispatch_mmio_write,
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
new file mode 100644
index 0000000000..9219142732
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -0,0 +1,99 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+#ifndef __XEN_ARM_VGIC_VGIC_MMIO_H__
+#define __XEN_ARM_VGIC_VGIC_MMIO_H__
+
+struct vgic_register_region {
+    unsigned int reg_offset;
+    unsigned int len;
+    unsigned int bits_per_irq;
+    unsigned int access_flags;
+    unsigned long (*read)(struct vcpu *vcpu, paddr_t addr,
+                          unsigned int len);
+    void (*write)(struct vcpu *vcpu, paddr_t addr,
+                  unsigned int len, unsigned long val);
+};
+
+extern struct mmio_handler_ops vgic_io_ops;
+
+#define VGIC_ACCESS_8bit    1
+#define VGIC_ACCESS_32bit   2
+#define VGIC_ACCESS_64bit   4
+
+/*
+ * Generate a mask that covers the number of bytes required to address
+ * up to 1024 interrupts, each represented by <bits> bits. This assumes
+ * that <bits> is a power of two.
+ */
+#define VGIC_ADDR_IRQ_MASK(bits) (((bits) * 1024 / 8) - 1)
+
+/*
+ * (addr & mask) gives us the _byte_ offset for the INT ID.
+ * We multiply this by 8 the get the _bit_ offset, then divide this by
+ * the number of bits to learn the actual INT ID.
+ * But instead of a division (which requires a "long long div" implementation),
+ * we shift by the binary logarithm of <bits>.
+ * This assumes that <bits> is a power of two.
+ */
+#define VGIC_ADDR_TO_INTID(addr, bits)  (((addr) & VGIC_ADDR_IRQ_MASK(bits)) * \
+                                         8 >> LOG_2(bits))
+
+/*
+ * Some VGIC registers store per-IRQ information, with a different number
+ * of bits per IRQ. For those registers this macro is used.
+ * The _WITH_LENGTH version instantiates registers with a fixed length
+ * and is mutually exclusive with the _PER_IRQ version.
+ */
+#define REGISTER_DESC_WITH_BITS_PER_IRQ(off, rd, wr, bpi, acc)  \
+    {                           \
+        .reg_offset = off,      \
+        .bits_per_irq = bpi,    \
+        .len = bpi * 1024 / 8,  \
+        .access_flags = acc,    \
+        .read = rd,             \
+        .write = wr,            \
+    }
+
+#define REGISTER_DESC_WITH_LENGTH(off, rd, wr, length, acc)     \
+    {                           \
+        .reg_offset = off,      \
+        .bits_per_irq = 0,      \
+        .len = length,          \
+        .access_flags = acc,    \
+        .read = rd,             \
+        .write = wr,            \
+    }
+
+#define REGISTER_DESC_WITH_LENGTH_UACCESS(off, rd, wr, length, acc) \
+    {                           \
+        .reg_offset = off,      \
+        .bits_per_irq = 0,      \
+        .len = length,          \
+        .access_flags = acc,    \
+        .read = rd,             \
+        .write = wr,            \
+    }
+
+unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
+                                 paddr_t addr, unsigned int len);
+
+unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
+                                 paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
+                        unsigned int len, unsigned long val);
+
+#endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 35/57] ARM: new VGIC: Add GICv2 MMIO handling framework

[Andre Przywara]

Create vgic-mmio-v2.c to describe GICv2 emulation specific handlers
using the initializer macros provided by the VGIC MMIO framework.
Provide a function to register the GICv2 distributor registers to
the Xen MMIO framework.
The actual handler functions are still stubs in this patch.

This is based on Linux commit fb848db39661, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- drop (dummy) user interface entries
- use frame number instead of physical address

 xen/arch/arm/vgic/vgic-mmio-v2.c | 83 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.c    | 25 ++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  2 +
 xen/arch/arm/vgic/vgic.h         |  2 +
 4 files changed, 112 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-mmio-v2.c

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
new file mode 100644
index 0000000000..6f10cf16ca
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -0,0 +1,83 @@
+/*
+ * VGICv2 MMIO handling functions
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <xen/bitops.h>
+#include <xen/sched.h>
+#include <xen/sizes.h>
+#include <asm/new_vgic.h>
+
+#include "vgic.h"
+#include "vgic-mmio.h"
+
+static const struct vgic_register_region vgic_v2_dist_registers[] = {
+    REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
+        vgic_mmio_read_rao, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+    REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 2,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
+        VGIC_ACCESS_32bit),
+    REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+    REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
+        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
+};
+
+unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev)
+{
+    dev->regions = vgic_v2_dist_registers;
+    dev->nr_regions = ARRAY_SIZE(vgic_v2_dist_registers);
+
+    return SZ_4K;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 393460d25a..284a92d288 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -170,6 +170,31 @@ struct mmio_handler_ops vgic_io_ops = {
     .write = dispatch_mmio_write,
 };
 
+int vgic_register_dist_iodev(struct domain *d, gfn_t dist_base_fn,
+                             enum vgic_type type)
+{
+    struct vgic_io_device *io_device = &d->arch.vgic.dist_iodev;
+    unsigned int len;
+
+    switch ( type )
+    {
+    case VGIC_V2:
+        len = vgic_v2_init_dist_iodev(io_device);
+        break;
+    default:
+        BUG();
+    }
+
+    io_device->base_fn = dist_base_fn;
+    io_device->iodev_type = IODEV_DIST;
+    io_device->redist_vcpu = NULL;
+
+    register_mmio_handler(d, &vgic_io_ops, gfn_to_gaddr(dist_base_fn), len,
+                          io_device);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 9219142732..621b9a281c 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -96,4 +96,6 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
 void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
                         unsigned int len, unsigned long val);
 
+unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
+
 #endif
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 116b26544e..632b246e93 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -49,6 +49,8 @@ static inline void vgic_get_irq_kref(struct vgic_irq *irq)
 void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
+int vgic_register_dist_iodev(struct domain *d, gfn_t dist_base_fn,
+                             enum vgic_type);
 
 #endif
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 36/57] ARM: new VGIC: Add CTLR, TYPER and IIDR handlers

[Andre Przywara]

Those three registers are v2 emulation specific, so their implementation
lives entirely in vgic-mmio-v2.c. Also they are handled in one function,
as their implementation is pretty simple.
When the guest enables the distributor, we kick all VCPUs to get
potentially pending interrupts serviced.

This is based on Linux commit 2b0cda878965, written by Marc Zyngier.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use PRODUCT_ID_XEN
- use proper locking on enabling VGIC
- use kick_vcpu()

 xen/arch/arm/vgic/vgic-mmio-v2.c | 54 +++++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/vgic/vgic.c         | 15 +++++++++++
 xen/arch/arm/vgic/vgic.h         |  4 +++
 3 files changed, 72 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 6f10cf16ca..2e015ed0b1 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -20,9 +20,61 @@
 #include "vgic.h"
 #include "vgic-mmio.h"
 
+static unsigned long vgic_mmio_read_v2_misc(struct vcpu *vcpu,
+                                            paddr_t addr, unsigned int len)
+{
+    uint32_t value;
+
+    switch ( addr & 0x0c )      /* filter for the 4 registers handled here */
+    {
+    case GICD_CTLR:
+        value = vcpu->domain->arch.vgic.enabled ? GICD_CTL_ENABLE : 0;
+        break;
+    case GICD_TYPER:
+        value = vcpu->domain->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;
+        value = (value >> 5) - 1;
+        value |= (vcpu->domain->max_vcpus - 1) << 5;
+        break;
+    case GICD_IIDR:
+        value = (PRODUCT_ID_XEN << 24) | (IMPLEMENTER_ARM << 0);
+        break;
+    default:
+        return 0;
+    }
+
+    return value;
+}
+
+static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
+                                    paddr_t addr, unsigned int len,
+                                    unsigned long val)
+{
+    struct vgic_dist *dist = &vcpu->domain->arch.vgic;
+    bool enabled;
+
+    switch ( addr & 0x0c )      /* filter for the 4 registers handled here */
+    {
+    case GICD_CTLR:
+        domain_lock(vcpu->domain);
+        enabled = dist->enabled;
+        dist->enabled = val & GICD_CTL_ENABLE;
+        enabled = !enabled && dist->enabled;
+        domain_unlock(vcpu->domain);
+
+        if (enabled)
+            vgic_kick_vcpus(vcpu->domain);
+
+        break;
+    case GICD_TYPER:
+    case GICD_IIDR:
+        /* read-only, writes ignored */
+        return;
+    }
+}
+
 static const struct vgic_register_region vgic_v2_dist_registers[] = {
     REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 12,
+        vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IGROUPR,
         vgic_mmio_read_rao, vgic_mmio_write_wi, 1,
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 66a366176a..465a95f415 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -683,6 +683,21 @@ int vgic_pending_irq(void)
     return vgic_vcpu_pending_irq(current);
 }
 
+void vgic_kick_vcpus(struct domain *d)
+{
+    struct vcpu *vcpu;
+
+    /*
+     * We've injected an interrupt, time to find out who deserves
+     * a good kick...
+     */
+    for_each_vcpu( d, vcpu )
+    {
+        if ( vgic_vcpu_pending_irq(vcpu) )
+            kick_vcpu(vcpu);
+    }
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 632b246e93..588bd067b7 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -17,6 +17,9 @@
 #ifndef __XEN_ARM_VGIC_VGIC_H__
 #define __XEN_ARM_VGIC_VGIC_H__
 
+#define PRODUCT_ID_XEN      0x58    /* ASCII code X */
+#define IMPLEMENTER_ARM     0x43b
+
 #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
@@ -37,6 +40,7 @@ struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
 void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
 void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
                unsigned long flags);
+void vgic_kick_vcpus(struct domain *d);
 
 static inline void vgic_get_irq_kref(struct vgic_irq *irq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 37/57] ARM: new VGIC: Add ENABLE registers handlers

[Andre Przywara]

As the enable register handlers are shared between the v2 and v3
emulation, their implementation goes into vgic-mmio.c, to be easily
referenced from the v3 emulation as well later.
This introduces a vgic_sync_hardware_irq() function, which updates the
physical side of a hardware mapped virtual IRQ.
Because the existing locking order between vgic_irq->irq_lock and
irq_desc->lock dictates so, we drop the irq_lock and retake them in the
proper order.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- extend and move vgic_sync_hardware_irq()
- do proper locking sequence
- skip already disabled/enabled IRQs

 xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 117 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
 xen/arch/arm/vgic/vgic.c         |  38 +++++++++++++
 xen/arch/arm/vgic/vgic.h         |   3 +
 5 files changed, 171 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 2e015ed0b1..3dd983f885 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -80,10 +80,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_rao, vgic_mmio_write_wi, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        vgic_mmio_read_enable, vgic_mmio_write_senable, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        vgic_mmio_read_enable, vgic_mmio_write_cenable, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 284a92d288..f8f0252eff 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -39,6 +39,123 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
     /* Ignore */
 }
 
+/*
+ * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the value
+ * of the enabled bit, so there is only one function for both here.
+ */
+unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
+                                    paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    uint32_t value = 0;
+    unsigned int i;
+
+    /* Loop over all IRQs affected by this read */
+    for ( i = 0; i < len * 8; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq->enabled )
+            value |= (1U << i);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+void vgic_mmio_write_senable(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    unsigned int i;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        unsigned long flags;
+        irq_desc_t *desc;
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        if ( irq->enabled )            /* skip already enabled IRQs */
+        {
+            spin_unlock_irqrestore(&irq->irq_lock, flags);
+            vgic_put_irq(vcpu->domain, irq);
+            continue;
+        }
+
+        irq->enabled = true;
+        if ( irq->hw )
+        {
+            /*
+             * The irq cannot be a PPI, we only support delivery
+             * of SPIs to guests.
+             */
+            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
+
+            desc = irq_to_desc(irq->hwintid);
+        }
+        else
+            desc = NULL;
+
+        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
+
+        if ( desc )
+            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+void vgic_mmio_write_cenable(struct vcpu *vcpu,
+                 paddr_t addr, unsigned int len,
+                 unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    unsigned int i;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq;
+        unsigned long flags;
+        irq_desc_t *desc;
+
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        if ( !irq->enabled )            /* skip already disabled IRQs */
+        {
+            spin_unlock_irqrestore(&irq->irq_lock, flags);
+            vgic_put_irq(vcpu->domain, irq);
+            continue;
+        }
+
+        irq->enabled = false;
+
+        if ( irq->hw )
+        {
+            /*
+             * The irq cannot be a PPI, we only support delivery
+             * of SPIs to guests.
+             */
+            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
+
+            desc = irq_to_desc(irq->hwintid);
+        }
+        else
+            desc = NULL;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        if ( desc )
+            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 621b9a281c..2ddcbbf58d 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -96,6 +96,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
 void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
                         unsigned int len, unsigned long val);
 
+unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
+                                    paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_senable(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val);
+
+void vgic_mmio_write_cenable(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 #endif
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 465a95f415..5246d7c2e7 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -698,6 +698,44 @@ void vgic_kick_vcpus(struct domain *d)
     }
 }
 
+static unsigned int translate_irq_type(bool is_level)
+{
+    return is_level ? IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING;
+}
+
+void vgic_sync_hardware_irq(struct domain *d,
+                            irq_desc_t *desc, struct vgic_irq *irq)
+{
+    unsigned long flags;
+
+    spin_lock_irqsave(&desc->lock, flags);
+    spin_lock(&irq->irq_lock);
+
+    /* Is that association actually still valid? (we entered with no locks) */
+    if ( desc->irq == irq->hwintid )
+    {
+        if ( irq->enabled )
+        {
+            /*
+             * We might end up from various callers, so check that the
+             * interrrupt is disabled before trying to change the config.
+             */
+            if ( irq_type_set_by_domain(d) &&
+                 test_bit(_IRQ_DISABLED, &desc->status) )
+                gic_set_irq_type(desc, translate_irq_type(irq->config));
+
+            if ( irq->target_vcpu )
+                irq_set_affinity(desc, cpumask_of(irq->target_vcpu->processor));
+            desc->handler->enable(desc);
+        }
+        else
+            desc->handler->disable(desc);
+    }
+
+    spin_unlock(&irq->irq_lock);
+    spin_unlock_irqrestore(&desc->lock, flags);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 588bd067b7..68e205d10a 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -50,6 +50,9 @@ static inline void vgic_get_irq_kref(struct vgic_irq *irq)
     atomic_inc(&irq->refcount);
 }
 
+void vgic_sync_hardware_irq(struct domain *d,
+                            irq_desc_t *desc, struct vgic_irq *irq);
+
 void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 38/57] ARM: new VGIC: Add PENDING registers handlers

[Andre Przywara]

The pending register handlers are shared between the v2 and v3
emulation, so their implementation goes into vgic-mmio.c, to be easily
referenced from the v3 emulation as well later.
For level triggered interrupts the real line level is unaffected by
this write, so we keep this state separate and combine it with the
device's level to get the actual pending state.
Hardware mapped IRQs need some special handling, as their hardware state
has to be coordinated with the virtual pending bit to avoid hanging
or masked interrupts.

This is based on Linux commit 96b298000db4, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- propagate SET/CLEAR_PENDING requests to hardware

 xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 125 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
 3 files changed, 138 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 3dd983f885..efdd73301d 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -86,10 +86,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_enable, vgic_mmio_write_cenable, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        vgic_mmio_read_pending, vgic_mmio_write_spending, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        vgic_mmio_read_pending, vgic_mmio_write_cpending, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index f8f0252eff..2e939d5e39 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -156,6 +156,131 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
     }
 }
 
+unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
+                                     paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    uint32_t value = 0;
+    unsigned int i;
+
+    /* Loop over all IRQs affected by this read */
+    for ( i = 0; i < len * 8; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq_is_pending(irq) )
+            value |= (1U << i);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+void vgic_mmio_write_spending(struct vcpu *vcpu,
+                              paddr_t addr, unsigned int len,
+                              unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    unsigned int i;
+    unsigned long flags;
+    irq_desc_t *desc;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        irq->pending_latch = true;
+
+        /* To observe the locking order, just take the irq_desc pointer here. */
+        if ( irq->hw )
+            desc = irq_to_desc(irq->hwintid);
+        else
+            desc = NULL;
+
+        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
+
+        /*
+         * When the VM sets the pending state for a HW interrupt on the virtual
+         * distributor we set the active state on the physical distributor,
+         * because the virtual interrupt can become active and then the guest
+         * can deactivate it.
+         */
+        if ( desc )
+        {
+            spin_lock_irqsave(&desc->lock, flags);
+            spin_lock(&irq->irq_lock);
+
+            /* Is this h/w IRQ still assigned to the virtual IRQ? */
+            if ( irq->hw && desc->irq == irq->hwintid )
+                gic_set_active_state(desc, true);
+
+            spin_unlock(&irq->irq_lock);
+            spin_unlock_irqrestore(&desc->lock, flags);
+        }
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+void vgic_mmio_write_cpending(struct vcpu *vcpu,
+                              paddr_t addr, unsigned int len,
+                              unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    unsigned int i;
+    unsigned long flags;
+    irq_desc_t *desc;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        irq->pending_latch = false;
+
+        /* To observe the locking order, just take the irq_desc pointer here. */
+        if ( irq->hw )
+            desc = irq_to_desc(irq->hwintid);
+        else
+            desc = NULL;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        /*
+         * We don't want the guest to effectively mask the physical
+         * interrupt by doing a write to SPENDR followed by a write to
+         * CPENDR for HW interrupts, so we clear the active state on
+         * the physical side if the virtual interrupt is not active.
+         * This may lead to taking an additional interrupt on the
+         * host, but that should not be a problem as the worst that
+         * can happen is an additional vgic injection.  We also clear
+         * the pending state to maintain proper semantics for edge HW
+         * interrupts.
+         */
+        if ( desc )
+        {
+            spin_lock_irqsave(&desc->lock, flags);
+            spin_lock(&irq->irq_lock);
+
+            /* Is this h/w IRQ still assigned to the virtual IRQ? */
+            if ( irq->hw && desc->irq == irq->hwintid )
+            {
+                gic_set_pending_state(desc, false);
+                if (!irq->active)
+                    gic_set_active_state(desc, false);
+            }
+
+            spin_unlock(&irq->irq_lock);
+            spin_unlock_irqrestore(&desc->lock, flags);
+        }
+
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 2ddcbbf58d..4465f3b7e5 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -107,6 +107,17 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
                              paddr_t addr, unsigned int len,
                              unsigned long val);
 
+unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
+                                     paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_spending(struct vcpu *vcpu,
+                              paddr_t addr, unsigned int len,
+                              unsigned long val);
+
+void vgic_mmio_write_cpending(struct vcpu *vcpu,
+                              paddr_t addr, unsigned int len,
+                              unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 #endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Andre Przywara]

The active register handlers are shared between the v2 and v3 emulation,
so their implementation goes into vgic-mmio.c, to be easily referenced
from the v3 emulation as well later.
Since activation/deactivation of an interrupt may happen entirely in the
guest without it ever exiting, we need some extra logic to properly track
the active state.
For clearing the active state, we would basically have to halt the guest
to make sure this is properly propagated into the respective VCPUs.
This is not yet implemented in Xen.
Fortunately this feature is mostly used to reset a just in initialised
GIC, so chances are we are tasked to clear bits that are already zero.
Add some simple check to avoid a pointless warning in this case.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- remove premature "proper ACTIVE" handler stub
- avoid unnecessary warnings on NO-OP register writes
- extend comments

 xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 103 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  11 +++++
 3 files changed, 116 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index efdd73301d..c93455fbb2 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -92,10 +92,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_pending, vgic_mmio_write_cpending, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        vgic_mmio_read_active, vgic_mmio_write_sactive, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
+        vgic_mmio_read_active, vgic_mmio_write_cactive, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 2e939d5e39..c44d67082f 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -281,6 +281,109 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
     }
 }
 
+/*
+ * The actual active bit for a virtual IRQ is held in the LR. Our shadow
+ * copy in struct vgic_irq is only synced when needed and may not be
+ * up-to-date all of the time.
+ * Returning the actual active state is quite costly (stopping all
+ * VCPUs processing any affected vIRQs), so we use a simple implementation
+ * to get the best possible answer.
+ */
+unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
+                                    paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    uint32_t value = 0;
+    unsigned int i;
+
+    /* Loop over all IRQs affected by this read */
+    for ( i = 0; i < len * 8; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq->active )
+            value |= (1U << i);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+/*
+ * We don't actually support clearing the active state of an IRQ (yet).
+ * However there is a chance that most guests use this for initialization.
+ * We check whether this MMIO access would actually affect any active IRQ,
+ * and only print our warning in this case. So clearing already non-active
+ * IRQs would not be moaned about in the logs.
+ */
+void vgic_mmio_write_cactive(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    unsigned int i;
+    bool bail_out = false;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        /*
+         * If we know that the IRQ is active or we can't be sure about
+         * it (because it is currently in a CPU), log the not properly
+         * emulated MMIO access.
+         */
+        if ( irq->active || irq->vcpu )
+        {
+            gdprintk(XENLOG_ERR,
+                     "%pv: vGICD: IRQ%d: clearing active state not supported\n",
+                     vcpu, irq->intid);
+            bail_out = true;
+        }
+
+        vgic_put_irq(vcpu->domain, irq);
+        if ( bail_out )
+            return;
+    }
+}
+
+/*
+ * We don't actually support setting the active state of an IRQ (yet).
+ * We check whether this MMIO access would actually affect any non-active IRQ,
+ * and only print our warning in this case.
+ */
+void vgic_mmio_write_sactive(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
+    unsigned int i;
+    bool bail_out = false;
+
+    for_each_set_bit( i, &val, len * 8 )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        /*
+         * If we know that the IRQ is not active or we can't be sure about
+         * it (because it is currently in a CPU), log the not properly
+         * emulated MMIO access.
+         */
+        if ( !irq->active || irq->vcpu )
+        {
+            gdprintk(XENLOG_ERR,
+                     "%pv: vGICD: IRQ%d: setting active state not supported\n",
+                     vcpu, irq->intid);
+            bail_out = true;
+        }
+
+        vgic_put_irq(vcpu->domain, irq);
+        if ( bail_out )
+            return;
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 4465f3b7e5..8604720628 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -118,6 +118,17 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
                               paddr_t addr, unsigned int len,
                               unsigned long val);
 
+unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
+                                    paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_cactive(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val);
+
+void vgic_mmio_write_sactive(struct vcpu *vcpu,
+                             paddr_t addr, unsigned int len,
+                             unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 #endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 40/57] ARM: new VGIC: Add PRIORITY registers handlers

[Andre Przywara]

The priority register handlers are shared between the v2 and v3 emulation,
so their implementation goes into vgic-mmio.c, to be easily referenced
from the v3 emulation as well later.
There is a corner case when we change the priority of a pending
interrupt which we don't handle at the moment.

This is based on Linux commit dd238ec2b87b, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use 32 bit register types

 xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 47 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
 xen/arch/arm/vgic/vgic.h         |  2 ++
 4 files changed, 57 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index c93455fbb2..29db9dec6f 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -98,7 +98,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_active, vgic_mmio_write_cactive, 1,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
+        vgic_mmio_read_priority, vgic_mmio_write_priority, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index c44d67082f..538f08bc66 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -384,6 +384,53 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
     }
 }
 
+unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
+                                      paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    unsigned int i;
+    uint32_t val = 0;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        val |= (uint32_t)irq->priority << (i * 8);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return val;
+}
+
+/*
+ * We currently don't handle changing the priority of an interrupt that
+ * is already pending on a VCPU. If there is a need for this, we would
+ * need to make this VCPU exit and re-evaluate the priorities, potentially
+ * leading to this interrupt getting presented now to the guest (if it has
+ * been masked by the priority mask before).
+ */
+void vgic_mmio_write_priority(struct vcpu *vcpu,
+                              paddr_t addr, unsigned int len,
+                              unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    unsigned int i;
+    unsigned long flags;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+        /* Narrow the priority range to what we actually support */
+        irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index 8604720628..e3f9029344 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -129,6 +129,13 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
                              paddr_t addr, unsigned int len,
                              unsigned long val);
 
+unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
+                      paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_priority(struct vcpu *vcpu,
+                  paddr_t addr, unsigned int len,
+                  unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 #endif
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 68e205d10a..b294b04391 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -20,6 +20,8 @@
 #define PRODUCT_ID_XEN      0x58    /* ASCII code X */
 #define IMPLEMENTER_ARM     0x43b
 
+#define VGIC_PRI_BITS       5
+
 #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
 
 static inline bool irq_is_pending(struct vgic_irq *irq)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 41/57] ARM: new VGIC: Add CONFIG registers handlers

[Andre Przywara]

The config register handlers are shared between the v2 and v3 emulation,
so their implementation goes into vgic-mmio.c, to be easily referenced
from the v3 emulation as well later.

This is based on Linux commit 79717e4ac09c, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use C99 data types

 xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
 xen/arch/arm/vgic/vgic-mmio.c    | 54 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
 3 files changed, 62 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 29db9dec6f..d19ddd3f1e 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -104,7 +104,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 2,
+        vgic_mmio_read_config, vgic_mmio_write_config, 2,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
index 538f08bc66..31f7cf706b 100644
--- a/xen/arch/arm/vgic/vgic-mmio.c
+++ b/xen/arch/arm/vgic/vgic-mmio.c
@@ -431,6 +431,60 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
     }
 }
 
+unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
+                                    paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 2);
+    uint32_t value = 0;
+    int i;
+
+    for ( i = 0; i < len * 4; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        if ( irq->config == VGIC_CONFIG_EDGE )
+            value |= (2U << (i * 2));
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return value;
+}
+
+void vgic_mmio_write_config(struct vcpu *vcpu,
+                            paddr_t addr, unsigned int len,
+                            unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 2);
+    int i;
+    unsigned long flags;
+
+    for ( i = 0; i < len * 4; i++ )
+    {
+        struct vgic_irq *irq;
+
+        /*
+         * The configuration cannot be changed for SGIs in general,
+         * for PPIs this is IMPLEMENTATION DEFINED. The arch timer
+         * code relies on PPIs being level triggered, so we also
+         * make them read-only here.
+         */
+        if ( intid + i < VGIC_NR_PRIVATE_IRQS )
+            continue;
+
+        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        if ( test_bit(i * 2 + 1, &val) )
+            irq->config = VGIC_CONFIG_EDGE;
+        else
+            irq->config = VGIC_CONFIG_LEVEL;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static int match_region(const void *key, const void *elt)
 {
     const unsigned int offset = (unsigned long)key;
diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index e3f9029344..bbf0d181ae 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -136,6 +136,13 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
                   paddr_t addr, unsigned int len,
                   unsigned long val);
 
+unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
+                    paddr_t addr, unsigned int len);
+
+void vgic_mmio_write_config(struct vcpu *vcpu,
+                paddr_t addr, unsigned int len,
+                unsigned long val);
+
 unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
 
 #endif
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 42/57] ARM: new VGIC: Add TARGET registers handlers

[Andre Przywara]

The target register handlers are v2 emulation specific, so their
implementation lives entirely in vgic-mmio-v2.c.
We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
set in the target mask instead of making it possibly pending on
multiple VCPUs.
We update the physical affinity of a hardware mapped vIRQ on the way.

This is based on Linux commit 2c234d6f1826, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- propagate affinity changes to hardware mapped IRQs

 xen/arch/arm/vgic/vgic-mmio-v2.c | 64 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 63 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index d19ddd3f1e..01c6a7198c 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -72,6 +72,68 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
     }
 }
 
+static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
+                                           paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    uint32_t val = 0;
+    unsigned int i;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        val |= (uint32_t)irq->targets << (i * 8);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return val;
+}
+
+static void vgic_mmio_write_target(struct vcpu *vcpu,
+                                   paddr_t addr, unsigned int len,
+                                   unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    uint8_t cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);
+    unsigned int i;
+    unsigned long flags;
+
+    /* GICD_ITARGETSR[0-7] are read-only */
+    if ( intid < VGIC_NR_PRIVATE_IRQS )
+        return;
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, NULL, intid + i);
+        struct irq_desc *desc;
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->targets = (val >> (i * 8)) & cpu_mask;
+        if ( irq->targets )
+        {
+            irq->target_vcpu = vcpu->domain->vcpu[ffs(irq->targets) - 1];
+            if ( irq->hw )
+                desc = irq_to_desc(irq->hwintid);
+            else
+                desc = NULL;
+        }
+        else {
+            irq->target_vcpu = NULL;
+            desc = NULL;
+        }
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        if ( desc )
+            vgic_update_hardware_irq(desc, irq);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static const struct vgic_register_region vgic_v2_dist_registers[] = {
     REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
         vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
@@ -101,7 +163,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_priority, vgic_mmio_write_priority, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
+        vgic_mmio_read_target, vgic_mmio_write_target, 8,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
         vgic_mmio_read_config, vgic_mmio_write_config, 2,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 43/57] ARM: new VGIC: Add SGIR register handler

[Andre Przywara]

Triggering an IPI via this register is v2 specific, so the
implementation lives entirely in vgic-mmio-v2.c.

This is based on Linux commit 55cc01fb9004, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use symbolic names instead of magic values
- iterates over set bits instead of every VCPU

 xen/arch/arm/vgic/vgic-mmio-v2.c | 47 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 45 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 01c6a7198c..5f1fdb9a70 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -72,6 +72,49 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
     }
 }
 
+static void vgic_mmio_write_sgir(struct vcpu *source_vcpu,
+                                 paddr_t addr, unsigned int len,
+                                 unsigned long val)
+{
+    struct domain *d = source_vcpu->domain;
+    unsigned int nr_vcpus = d->max_vcpus;
+    unsigned int intid = val & GICD_SGI_INTID_MASK;
+    unsigned long targets = (val & GICD_SGI_TARGET_MASK) >>
+                            GICD_SGI_TARGET_SHIFT;
+    unsigned int vcpu_id;
+
+    switch ( val & GICD_SGI_TARGET_LIST_MASK )
+    {
+    case GICD_SGI_TARGET_LIST:                    /* as specified by targets */
+        targets &= GENMASK(nr_vcpus, 0);          /* limit to existing VCPUs */
+        break;
+    case GICD_SGI_TARGET_OTHERS:
+        targets = GENMASK(nr_vcpus, 0);           /* all, ...   */
+        targets &= ~(1U << source_vcpu->vcpu_id); /*   but self */
+        break;
+    case GICD_SGI_TARGET_SELF:                    /* this very vCPU only */
+        targets = (1U << source_vcpu->vcpu_id);
+        break;
+    case 0x3:                                     /* reserved */
+        return;
+    }
+
+    for_each_set_bit( vcpu_id, &targets, 8 )
+    {
+        struct vcpu *vcpu = d->vcpu[vcpu_id];
+        struct vgic_irq *irq = vgic_get_irq(d, vcpu, intid);
+        unsigned long flags;
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->pending_latch = true;
+        irq->source |= 1U << source_vcpu->vcpu_id;
+
+        vgic_queue_irq_unlock(d, irq, flags);
+        vgic_put_irq(d, irq);
+    }
+}
+
 static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
                                            paddr_t addr, unsigned int len)
 {
@@ -128,7 +171,7 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
         spin_unlock_irqrestore(&irq->irq_lock, flags);
 
         if ( desc )
-            vgic_update_hardware_irq(desc, irq);
+            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
 
         vgic_put_irq(vcpu->domain, irq);
     }
@@ -169,7 +212,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_config, vgic_mmio_write_config, 2,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
+        vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
         vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 44/57] ARM: new VGIC: Add SGIPENDR register handlers

[Andre Przywara]

As this register is v2 specific, its implementation lives entirely
in vgic-mmio-v2.c.
This register allows setting the source mask of an IPI.

This is based on Linux commit ed40213ef9b0, written by Andre Przywara.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use C99 and unsigned data types

 xen/arch/arm/vgic/vgic-mmio-v2.c | 81 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 79 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
index 5f1fdb9a70..dd9857e8a6 100644
--- a/xen/arch/arm/vgic/vgic-mmio-v2.c
+++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
@@ -177,6 +177,83 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
     }
 }
 
+static unsigned long vgic_mmio_read_sgipend(struct vcpu *vcpu,
+                                            paddr_t addr, unsigned int len)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    uint32_t val = 0;
+    unsigned int i;
+
+    ASSERT(intid < VGIC_NR_SGIS);
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        val |= (uint32_t)irq->source << (i * 8);
+
+        vgic_put_irq(vcpu->domain, irq);
+    }
+
+    return val;
+}
+
+static void vgic_mmio_write_sgipendc(struct vcpu *vcpu,
+                                     paddr_t addr, unsigned int len,
+                                     unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    unsigned int i;
+    unsigned long flags;
+
+    ASSERT(intid < VGIC_NR_SGIS);
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->source &= ~((val >> (i * 8)) & 0xff);
+        if ( !irq->source )
+            irq->pending_latch = false;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
+static void vgic_mmio_write_sgipends(struct vcpu *vcpu,
+                                     paddr_t addr, unsigned int len,
+                                     unsigned long val)
+{
+    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
+    unsigned int i;
+    unsigned long flags;
+
+    ASSERT(intid < VGIC_NR_SGIS);
+
+    for ( i = 0; i < len; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        irq->source |= (val >> (i * 8)) & 0xff;
+
+        if ( irq->source )
+        {
+            irq->pending_latch = true;
+            vgic_queue_irq_unlock(vcpu->domain, irq, flags);
+        }
+        else
+        {
+            spin_unlock_irqrestore(&irq->irq_lock, flags);
+        }
+        vgic_put_irq(vcpu->domain, irq);
+    }
+}
+
 static const struct vgic_register_region vgic_v2_dist_registers[] = {
     REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
         vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
@@ -215,10 +292,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
         vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
         VGIC_ACCESS_32bit),
     REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        vgic_mmio_read_sgipend, vgic_mmio_write_sgipendc, 16,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
     REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
-        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
+        vgic_mmio_read_sgipend, vgic_mmio_write_sgipends, 16,
         VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
 };
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 45/57] ARM: new VGIC: Handle hardware mapped IRQs

[Andre Przywara]

The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
when a guest EOIs the virtual interrupt, it affects the state of that
corresponding interrupt on the hardware side at the same time.
Implement the interface that the Xen arch/core code expects to connect
the virtual and the physical world.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- add ASSERT for hardware mapped IRQs being SPI only
- check h/w IRQ matches before disconnecting

 xen/arch/arm/vgic/vgic.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 71 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 5246d7c2e7..5bbf55da21 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -698,6 +698,77 @@ void vgic_kick_vcpus(struct domain *d)
     }
 }
 
+struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
+                                      unsigned int virq)
+{
+    struct irq_desc *desc = NULL;
+    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
+    unsigned long flags;
+
+    if ( !irq )
+        return NULL;
+
+    spin_lock_irqsave(&irq->irq_lock, flags);
+    if ( irq->hw )
+    {
+        ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
+        desc = irq_to_desc(irq->hwintid);
+    }
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+    vgic_put_irq(d, irq);
+
+    return desc;
+}
+
+/*
+ * was:
+ *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
+ *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int virt_irq)
+ */
+int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
+                        unsigned int virt_irq, struct irq_desc *desc,
+                        bool connect)
+{
+    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
+    unsigned long flags;
+    int ret = 0;
+
+    if ( !irq )
+        return -EINVAL;
+
+    spin_lock_irqsave(&irq->irq_lock, flags);
+
+    if ( connect )                      /* assign a mapped IRQ */
+    {
+        /* The VIRQ should not be already enabled by the guest */
+        if ( !irq->hw && !irq->enabled )
+        {
+            irq->hw = true;
+            irq->hwintid = desc->irq;
+        }
+        else
+            ret = -EBUSY;
+    }
+    else                                /* remove a mapped IRQ */
+    {
+        if ( desc && irq->hwintid != desc->irq )
+        {
+            ret = -EINVAL;
+        }
+        else
+        {
+            irq->hw = false;
+            irq->hwintid = 0;
+        }
+    }
+
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+    vgic_put_irq(d, irq);
+
+    return ret;
+}
+
 static unsigned int translate_irq_type(bool is_level)
 {
     return is_level ? IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING;
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 46/57] ARM: new VGIC: Add event channel IRQ handling

[Andre Przywara]

The Xen core/arch code relies on two abstracted functions to inject an
event channel IRQ and to query its pending state.
Implement those to query the state of the new VGIC implementation.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- add locking

 xen/arch/arm/vgic/vgic.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 5bbf55da21..0bf257c865 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -698,6 +698,29 @@ void vgic_kick_vcpus(struct domain *d)
     }
 }
 
+void arch_evtchn_inject(struct vcpu *v)
+{
+    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
+}
+
+bool vgic_evtchn_irq_pending(struct vcpu *v)
+{
+    struct vgic_irq *irq;
+    unsigned long flags;
+    bool pending;
+
+    /* Does not work for LPIs. */
+    ASSERT(!is_lpi(v->domain->arch.evtchn_irq));
+
+    irq = vgic_get_irq(v->domain, v, v->domain->arch.evtchn_irq);
+    spin_lock_irqsave(&irq->irq_lock, flags);
+    pending = irq_is_pending(irq);
+    spin_unlock_irqrestore(&irq->irq_lock, flags);
+    vgic_put_irq(v->domain, irq);
+
+    return pending;
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 47/57] ARM: new VGIC: Handle virtual IRQ allocation/reservation

[Andre Przywara]

To find an unused virtual IRQ number Xen uses a scheme to track used
virtual IRQs.
Implement this interface in the new VGIC to make the Xen core/arch code
happy.
This is actually somewhat VGIC agnostic, so is mostly a copy of the code
from the old VGIC. But it has to live in the VGIC files, so we can't
easily reuse the existing implementation.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/vgic/vgic.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 0bf257c865..e9ef992e1e 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -721,6 +721,50 @@ bool vgic_evtchn_irq_pending(struct vcpu *v)
     return pending;
 }
 
+bool vgic_reserve_virq(struct domain *d, unsigned int virq)
+{
+    if ( virq >= vgic_num_irqs(d) )
+        return false;
+
+    return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
+}
+
+int vgic_allocate_virq(struct domain *d, bool spi)
+{
+    int first, end;
+    unsigned int virq;
+
+    if ( !spi )
+    {
+        /* We only allocate PPIs. SGIs are all reserved */
+        first = 16;
+        end = 32;
+    }
+    else
+    {
+        first = 32;
+        end = vgic_num_irqs(d);
+    }
+
+    /*
+     * There is no spinlock to protect allocated_irqs, therefore
+     * test_and_set_bit may fail. If so retry it.
+     */
+    do
+    {
+        virq = find_next_zero_bit(d->arch.vgic.allocated_irqs, end, first);
+        if ( virq >= end )
+            return -1;
+    } while ( test_and_set_bit(virq, d->arch.vgic.allocated_irqs) );
+
+    return virq;
+}
+
+void vgic_free_virq(struct domain *d, unsigned int virq)
+{
+    clear_bit(virq, d->arch.vgic.allocated_irqs);
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 48/57] ARM: new VGIC: Dump virtual IRQ info

[Andre Przywara]

When we dump guest state on the Xen console, we also print the state of
IRQs that are on a VCPU.
Add the code to dump the state of an IRQ handled by the new VGIC.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- use proper locking
- use one header line to announce active or pending IRQs

 xen/arch/arm/vgic/vgic.c | 25 +++++++++++++++++++++++++
 1 file changed, 25 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index e9ef992e1e..2a2b8fd1eb 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -765,6 +765,31 @@ void vgic_free_virq(struct domain *d, unsigned int virq)
     clear_bit(virq, d->arch.vgic.allocated_irqs);
 }
 
+void gic_dump_vgic_info(struct vcpu *v)
+{
+    struct vgic_cpu *vgic_cpu = &v->arch.vgic;
+    struct vgic_irq *irq;
+    unsigned long flags;
+
+    spin_lock_irqsave(&v->arch.vgic.ap_list_lock, flags);
+
+    if ( !list_empty(&vgic_cpu->ap_list_head) )
+        printk("   active or pending interrupts queued:\n");
+
+    list_for_each_entry ( irq, &vgic_cpu->ap_list_head, ap_list )
+    {
+        spin_lock(&irq->irq_lock);
+        printk("     %s %s irq %u: %spending, %sactive, %senabled\n",
+               irq->hw ? "hardware" : "virtual",
+               irq->config == VGIC_CONFIG_LEVEL ? "level" : "edge",
+               irq->intid, irq_is_pending(irq) ? "" : "not ",
+               irq->active ? "" : "not ", irq->enabled ? "" : "not ");
+        spin_unlock(&irq->irq_lock);
+    }
+
+    spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 49/57] ARM: new VGIC: provide system register emulation stub

[Andre Przywara]

The Xen arch code traps system registers writes from the guest and will
relay anything GIC related to the VGIC.
Since this affects only GICv3 (which we don't yet emulate), provide a
stub implementation of vgic_emulate() for now.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/vgic/vgic.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 2a2b8fd1eb..e1952c872d 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -813,6 +813,13 @@ struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
     return desc;
 }
 
+bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr)
+{
+    ASSERT(current->domain->arch.vgic.version == GIC_V3);
+
+    return false;
+}
+
 /*
  * was:
  *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 50/57] ARM: new VGIC: Implement arch_move_irqs()

[Andre Przywara]

When a VCPU moves to another CPU, we need to adjust the target affinity
of any hardware mapped vIRQs, to observe our "physical-follows-virtual"
policy.
Implement arch_move_irqs() to adjust the physical affinity of all hardware
mapped vIRQs targetting this VCPU.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- actually implement arch_move_irqs() (instead of just stubbing it)

 xen/arch/arm/vgic/vgic.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 42 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index e1952c872d..5e767927c0 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -790,6 +790,48 @@ void gic_dump_vgic_info(struct vcpu *v)
     spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
 }
 
+/**
+ * arch_move_irqs() - migrate the physical affinity of hardware mapped vIRQs
+ * @v:  the vCPU, already assigned to the new pCPU
+ *
+ * arch_move_irqs() updates the physical affinity of all virtual IRQs
+ * targetting this given vCPU. This only affects hardware mapped IRQs. The
+ * new pCPU to target is already set in v->processor.
+ * This is called by the core code after a vCPU has been migrated to a new
+ * physical CPU.
+ */
+void arch_move_irqs(struct vcpu *v)
+{
+    struct domain *d = v->domain;
+    unsigned int i;
+
+    /* We only target SPIs with this function */
+    for ( i = 0; i < d->arch.vgic.nr_spis; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(d, NULL, i + VGIC_NR_PRIVATE_IRQS);
+        unsigned long flags;
+        irq_desc_t *desc;
+
+        if ( !irq )
+            continue;
+
+        spin_lock_irqsave(&irq->irq_lock, flags);
+
+        /* only vIRQs that are not on a vCPU yet , but targetting this vCPU */
+        if ( irq->hw && !irq->vcpu && irq->target_vcpu == v)
+            desc = irq_to_desc(irq->hwintid);
+        else
+            desc = NULL;
+
+        spin_unlock_irqrestore(&irq->irq_lock, flags);
+
+        if ( desc )
+            vgic_sync_hardware_irq(d, desc, irq);
+
+        vgic_put_irq(d, irq);
+    }
+}
+
 struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
                                       unsigned int virq)
 {
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 51/57] ARM: new VGIC: Add preliminary stub implementation

[Andre Przywara]

The ARM arch code requires an interrupt controller emulation to implement
vgic_clear_pending_irqs(), although it is suspected that it is actually
not necessary. Go with a stub for now to make the linker happy.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- split off from former patch, otherwise unchanged

 xen/arch/arm/vgic/vgic.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 5e767927c0..5d84a4d81a 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -790,6 +790,14 @@ void gic_dump_vgic_info(struct vcpu *v)
     spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
 }
 
+void vgic_clear_pending_irqs(struct vcpu *v)
+{
+    /*
+     * TODO: It is unclear whether we really need this, so we might instead
+     * remove it on the caller site.
+     */
+}
+
 /**
  * arch_move_irqs() - migrate the physical affinity of hardware mapped vIRQs
  * @v:  the vCPU, already assigned to the new pCPU
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 52/57] ARM: new VGIC: vgic-init: register VGIC

[Andre Przywara]

This patch implements the function which is called by Xen when it wants
to register the virtual GIC.
This also implements domain_max_vcpus() for the new VGIC, which reports
back the maximum number of VCPUs a certain GIC model supports.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- split off definition of domain_max_vcpus()

 xen/arch/arm/vgic/vgic-init.c | 60 +++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.c      | 22 ++++++++++++++++
 xen/arch/arm/vgic/vgic.h      |  3 +++
 3 files changed, 85 insertions(+)
 create mode 100644 xen/arch/arm/vgic/vgic-init.c

diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
new file mode 100644
index 0000000000..d091c92ed0
--- /dev/null
+++ b/xen/arch/arm/vgic/vgic-init.c
@@ -0,0 +1,60 @@
+/*
+ * Copyright (C) 2015, 2016 ARM Ltd.
+ * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/sched.h>
+#include <asm/new_vgic.h>
+
+#include "vgic.h"
+
+/* CREATION */
+
+/**
+ * domain_vgic_register: create a virtual GIC
+ * @d: domain pointer
+ * @mmio_count: pointer to add number of required MMIO regions
+ *
+ * was: kvm_vgic_create
+ */
+int domain_vgic_register(struct domain *d, int *mmio_count)
+{
+    switch ( d->arch.vgic.version )
+    {
+    case GIC_V2:
+        *mmio_count = 1;
+        break;
+    default:
+        BUG();
+    }
+
+    if ( d->max_vcpus > domain_max_vcpus(d) )
+        return -E2BIG;
+
+    d->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
+    d->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
+    d->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index 5d84a4d81a..f42092fec3 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -956,6 +956,28 @@ void vgic_sync_hardware_irq(struct domain *d,
     spin_unlock_irqrestore(&desc->lock, flags);
 }
 
+unsigned int domain_max_vcpus(const struct domain *d)
+{
+    unsigned int vgic_vcpu_limit;
+
+    switch ( d->arch.vgic.version )
+    {
+#ifdef CONFIG_HAS_GICV3
+    case GIC_V3:
+        vgic_vcpu_limit = VGIC_V3_MAX_CPUS;
+        break;
+#endif
+    case GIC_V2:
+        vgic_vcpu_limit = VGIC_V2_MAX_CPUS;
+        break;
+    default:
+        vgic_vcpu_limit = MAX_VIRT_CPUS;
+        break;
+    }
+
+    return min_t(unsigned int, MAX_VIRT_CPUS, vgic_vcpu_limit);
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index b294b04391..f19dc9502f 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -20,6 +20,9 @@
 #define PRODUCT_ID_XEN      0x58    /* ASCII code X */
 #define IMPLEMENTER_ARM     0x43b
 
+#define VGIC_ADDR_UNDEF     INVALID_PADDR
+#define IS_VGIC_ADDR_UNDEF(_x)  ((_x) == VGIC_ADDR_UNDEF)
+
 #define VGIC_PRI_BITS       5
 
 #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 53/57] ARM: new VGIC: vgic-init: implement vgic_init

[Andre Przywara]

This patch allocates and initializes the data structures used to model
the vgic distributor and virtual cpu interfaces. At that stage the
number of IRQs and number of virtual CPUs is frozen.
Implement the various functions that the Xen arch code is expecting to
call during domain and VCPU setup to initialize the VGIC.
Their prototypes are already in existing header files.

This is based on Linux commit ad275b8bb1e6, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- adapt to former changes
- add missing comment line
- extend commit message

 xen/arch/arm/vgic/vgic-init.c | 196 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 196 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
index d091c92ed0..8bc83f677b 100644
--- a/xen/arch/arm/vgic/vgic-init.c
+++ b/xen/arch/arm/vgic/vgic-init.c
@@ -20,6 +20,77 @@
 
 #include "vgic.h"
 
+/*
+ * Initialization rules: there are multiple stages to the vgic
+ * initialization, both for the distributor and the CPU interfaces.  The basic
+ * idea is that even though the VGIC is not functional or not requested from
+ * user space, the critical path of the run loop can still call VGIC functions
+ * that just won't do anything, without them having to check additional
+ * initialization flags to ensure they don't look at uninitialized data
+ * structures.
+ *
+ * Distributor:
+ *
+ * - vgic_early_init(): initialization of static data that doesn't
+ *   depend on any sizing information or emulation type. No allocation
+ *   is allowed there.
+ *
+ * - vgic_init(): allocation and initialization of the generic data
+ *   structures that depend on sizing information (number of CPUs,
+ *   number of interrupts). Also initializes the vcpu specific data
+ *   structures. Can be executed lazily for GICv2.
+ *
+ * CPU Interface:
+ *
+ * - kvm_vgic_vcpu_early_init(): initialization of static data that
+ *   doesn't depend on any sizing information or emulation type. No
+ *   allocation is allowed there.
+ */
+
+/**
+ * vgic_vcpu_early_init() - Initialize static VGIC VCPU data structures
+ * @vcpu: The VCPU whose VGIC data structures whould be initialized
+ *
+ * Only do initialization, but do not actually enable the VGIC CPU interface
+ * yet.
+ */
+static void vgic_vcpu_early_init(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+    int i;
+
+    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
+    spin_lock_init(&vgic_cpu->ap_list_lock);
+
+    /*
+     * Enable and configure all SGIs to be edge-triggered and
+     * configure all PPIs as level-triggered.
+     */
+    for ( i = 0; i < VGIC_NR_PRIVATE_IRQS; i++ )
+    {
+        struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
+
+        INIT_LIST_HEAD(&irq->ap_list);
+        spin_lock_init(&irq->irq_lock);
+        irq->intid = i;
+        irq->vcpu = NULL;
+        irq->target_vcpu = vcpu;
+        irq->targets = 1U << vcpu->vcpu_id;
+        atomic_set(&irq->refcount, 0);
+        if ( vgic_irq_is_sgi(i) )
+        {
+            /* SGIs */
+            irq->enabled = 1;
+            irq->config = VGIC_CONFIG_EDGE;
+        }
+        else
+        {
+            /* PPIs */
+            irq->config = VGIC_CONFIG_LEVEL;
+        }
+    }
+}
+
 /* CREATION */
 
 /**
@@ -50,6 +121,131 @@ int domain_vgic_register(struct domain *d, int *mmio_count)
     return 0;
 }
 
+/* INIT/DESTROY */
+
+/**
+ * domain_vgic_init: initialize the dist data structures
+ * @d: domain pointer
+ * @nr_spis: number of SPIs
+ */
+int domain_vgic_init(struct domain *d, unsigned int nr_spis)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+    int i, ret;
+
+    /* Limit the number of virtual SPIs supported to (1020 - 32) = 988  */
+    if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
+        return -EINVAL;
+
+    dist->nr_spis = nr_spis;
+    dist->spis = xzalloc_array(struct vgic_irq, nr_spis);
+    if ( !dist->spis )
+        return  -ENOMEM;
+
+    /*
+     * In the following code we do not take the irq struct lock since
+     * no other action on irq structs can happen while the VGIC is
+     * not initialized yet:
+     * If someone wants to inject an interrupt or does a MMIO access, we
+     * require prior initialization in case of a virtual GICv3 or trigger
+     * initialization when using a virtual GICv2.
+     */
+    for ( i = 0; i < nr_spis; i++ )
+    {
+        struct vgic_irq *irq = &dist->spis[i];
+
+        irq->intid = i + VGIC_NR_PRIVATE_IRQS;
+        INIT_LIST_HEAD(&irq->ap_list);
+        spin_lock_init(&irq->irq_lock);
+        irq->vcpu = NULL;
+        irq->target_vcpu = NULL;
+        atomic_set(&irq->refcount, 0);
+        if ( dist->version == GIC_V2 )
+            irq->targets = 0;
+        else
+            irq->mpidr = 0;
+    }
+
+    INIT_LIST_HEAD(&dist->lpi_list_head);
+    spin_lock_init(&dist->lpi_list_lock);
+
+    if ( dist->version == GIC_V2 )
+        ret = vgic_v2_map_resources(d);
+    else
+        ret = -ENXIO;
+
+    if ( ret )
+        return ret;
+
+    /* allocated_irqs() is used by Xen to find available vIRQs */
+    d->arch.vgic.allocated_irqs =
+        xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
+    if ( !d->arch.vgic.allocated_irqs )
+        return -ENOMEM;
+
+    /* vIRQ0-15 (SGIs) are reserved */
+    for ( i = 0; i < NR_GIC_SGI; i++ )
+        set_bit(i, d->arch.vgic.allocated_irqs);
+
+    return 0;
+}
+
+/**
+ * vcpu_vgic_init() - Register VCPU-specific KVM iodevs
+ * was: kvm_vgic_vcpu_init()
+ * Xen: adding vgic_vx_enable() call
+ * @vcpu: pointer to the VCPU being created and initialized
+ */
+int vcpu_vgic_init(struct vcpu *vcpu)
+{
+    int ret = 0;
+
+    vgic_vcpu_early_init(vcpu);
+
+    if ( gic_hw_version() == GIC_V2 )
+        vgic_v2_enable(vcpu);
+    else
+        ret = -ENXIO;
+
+    return ret;
+}
+
+void domain_vgic_free(struct domain *d)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+        int i, ret;
+
+    for ( i = 0; i < dist->nr_spis; i++ )
+    {
+        struct vgic_irq *irq = vgic_get_irq(d, NULL, 32 + i);
+
+        if ( !irq->hw )
+            continue;
+
+        ret = release_guest_irq(d, irq->hwintid);
+        if ( ret )
+            dprintk(XENLOG_G_WARNING,
+                "d%u: Failed to release virq %u ret = %d\n",
+                d->domain_id, 32 + i, ret);
+    }
+
+    dist->ready = false;
+    dist->initialized = false;
+
+    xfree(dist->spis);
+    xfree(dist->allocated_irqs);
+    dist->nr_spis = 0;
+}
+
+int vcpu_vgic_free(struct vcpu *vcpu)
+{
+    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
+
+    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 54/57] ARM: new VGIC: vgic-init: implement map_resources

[Andre Przywara]

map_resources is the last initialization step needed before the first
VCPU is run. At that stage the code stores the MMIO base addresses used.
Also it registers the respective register frames with the MMIO framework.

This is based on Linux commit cbae53e663ea, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- adapting to previous changes

 xen/arch/arm/vgic/vgic-v2.c | 66 +++++++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic/vgic.h    |  1 +
 2 files changed, 67 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
index 4e74ebf7f5..da64b4758c 100644
--- a/xen/arch/arm/vgic/vgic-v2.c
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -221,6 +221,72 @@ void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)
     gic_hw_ops->write_lr(lr, &lr_val);
 }
 
+int vgic_v2_map_resources(struct domain *d)
+{
+    struct vgic_dist *dist = &d->arch.vgic;
+    paddr_t cbase, csize;
+    paddr_t vbase;
+    int ret;
+
+    /*
+     * The hardware domain gets the hardware address.
+     * Guests get the virtual platform layout.
+     */
+    if ( is_hardware_domain(d) )
+    {
+        d->arch.vgic.vgic_dist_base = gic_v2_hw_data.dbase;
+        /*
+         * For the hardware domain, we always map the whole HW CPU
+         * interface region in order to match the device tree (the "reg"
+         * properties is copied as it is).
+         * Note that we assume the size of the CPU interface is always
+         * aligned to PAGE_SIZE.
+         */
+        cbase = gic_v2_hw_data.cbase;   /* was: dist->vgic_cpu_base */
+        csize = gic_v2_hw_data.csize;
+        vbase = gic_v2_hw_data.vbase; /* was: kvm_vgic_global_state.vcpu_base */
+    }
+    else
+    {
+        d->arch.vgic.vgic_dist_base = GUEST_GICD_BASE;
+        /*
+         * The CPU interface exposed to the guest is always 8kB. We may
+         * need to add an offset to the virtual CPU interface base
+         * address when in the GIC is aliased to get a 8kB contiguous
+         * region.
+         */
+        BUILD_BUG_ON(GUEST_GICC_SIZE != SZ_8K);
+        cbase = GUEST_GICC_BASE;
+        csize = GUEST_GICC_SIZE;
+        vbase = gic_v2_hw_data.vbase + gic_v2_hw_data.aliased_offset;
+    }
+
+
+    ret = vgic_register_dist_iodev(d, gaddr_to_gfn(dist->vgic_dist_base),
+                                   VGIC_V2);
+    if ( ret )
+    {
+        gdprintk(XENLOG_ERR, "Unable to register VGIC MMIO regions\n");
+        return ret;
+    }
+
+    /*
+     * Map the gic virtual cpu interface in the gic cpu interface
+     * region of the guest.
+     */
+    ret = map_mmio_regions(d, gaddr_to_gfn(cbase), csize / PAGE_SIZE,
+                   maddr_to_mfn(vbase));
+    if ( ret )
+    {
+        gdprintk(XENLOG_ERR, "Unable to remap VGIC CPU to VCPU\n");
+        return ret;
+    }
+
+    dist->ready = true;
+
+        return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index f19dc9502f..6fab994b9c 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -61,6 +61,7 @@ void vgic_sync_hardware_irq(struct domain *d,
 void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
+int vgic_v2_map_resources(struct domain *d);
 int vgic_register_dist_iodev(struct domain *d, gfn_t dist_base_fn,
                              enum vgic_type);
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 55/57] ARM: new VGIC: Add vgic_v2_enable

[Andre Przywara]

Enable the VGIC operation by properly initialising the registers
in the hypervisor GIC interface.

This is based on Linux commit f7b6985cc3d0, written by Eric Auger.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- drop unneeded vgic_vmcr initialization
- use update_hcr_status wrapper

 xen/arch/arm/vgic/vgic-v2.c | 6 ++++++
 xen/arch/arm/vgic/vgic.h    | 1 +
 2 files changed, 7 insertions(+)

diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
index da64b4758c..b7d6493e5a 100644
--- a/xen/arch/arm/vgic/vgic-v2.c
+++ b/xen/arch/arm/vgic/vgic-v2.c
@@ -221,6 +221,12 @@ void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)
     gic_hw_ops->write_lr(lr, &lr_val);
 }
 
+void vgic_v2_enable(struct vcpu *vcpu)
+{
+    /* Get the show on the road... */
+    gic_hw_ops->update_hcr_status(GICH_HCR_EN, 1);
+}
+
 int vgic_v2_map_resources(struct domain *d)
 {
     struct vgic_dist *dist = &d->arch.vgic;
diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
index 6fab994b9c..bd0c3fe5ab 100644
--- a/xen/arch/arm/vgic/vgic.h
+++ b/xen/arch/arm/vgic/vgic.h
@@ -61,6 +61,7 @@ void vgic_sync_hardware_irq(struct domain *d,
 void vgic_v2_fold_lr_state(struct vcpu *vcpu);
 void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
 void vgic_v2_set_underflow(struct vcpu *vcpu);
+void vgic_v2_enable(struct vcpu *vcpu);
 int vgic_v2_map_resources(struct domain *d);
 int vgic_register_dist_iodev(struct domain *d, gfn_t dist_base_fn,
                              enum vgic_type);
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 56/57] ARM: allocate two pages for struct vcpu

[Andre Przywara]

At the moment we allocate exactly one page for struct vcpu on ARM, also
have a check in place to prevent it growing beyond 4KB.
As the struct includes the state of all 32 private (per-VCPU) interrupts,
we are at 3840 bytes on arm64 at the moment already. Growing the per-IRQ
VGIC structure even slightly makes the VCPU quickly exceed the 4K limit.
The new VGIC will need more space per virtual IRQ. I spent a few hours
trying to trim this down, but couldn't get it below 4KB, even with the
nasty hacks piling up to save some bytes here and there.
It turns out that beyond efficiency, maybe, there is no real technical
reason this struct has to fit in one page, so lifting the limit to two
pages seems like the most pragmatic solution.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/domain.c | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
index 11a46aa27f..0bec6aad17 100644
--- a/xen/arch/arm/domain.c
+++ b/xen/arch/arm/domain.c
@@ -502,10 +502,13 @@ void dump_pageframe_info(struct domain *d)
 struct vcpu *alloc_vcpu_struct(void)
 {
     struct vcpu *v;
-    BUILD_BUG_ON(sizeof(*v) > PAGE_SIZE);
-    v = alloc_xenheap_pages(0, 0);
-    if ( v != NULL )
+
+    BUILD_BUG_ON(sizeof(*v) > 2 * PAGE_SIZE);
+    v = alloc_xenheap_pages(1, 0);
+    if ( v != NULL ) {
         clear_page(v);
+        clear_page((void *)v + PAGE_SIZE);
+    }
     return v;
 }
 
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[PATCH 57/57] ARM: VGIC: wire new VGIC(-v2) files into Xen build system

[Andre Przywara]

Now that we have both the old VGIC prepared to cope with a sibling and
the code for the new VGIC in place, lets add a Kconfig option to enable
the new code and wire it into the Xen build system.
This will add a compile time option to use either the "old" or the "new"
VGIC.
In the moment this is restricted to a vGIC-v2. To make the build system
happy, we provide a temporary dummy implementation of
vgic_v3_setup_hw() to allow building for now.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Changelog RFC ... v1:
- no changes

 xen/arch/arm/Kconfig     |  6 +++++-
 xen/arch/arm/Makefile    | 10 +++++++++-
 xen/arch/arm/vgic/vgic.c |  8 ++++++++
 xen/common/Makefile      |  1 +
 4 files changed, 23 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 2782ee6589..aad19927ce 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -48,7 +48,11 @@ config HAS_GICV3
 config HAS_ITS
         bool
         prompt "GICv3 ITS MSI controller support" if EXPERT = "y"
-        depends on HAS_GICV3
+        depends on HAS_GICV3 && !NEW_VGIC
+
+config NEW_VGIC
+        bool
+        prompt "Use new VGIC implementation"
 
 config SBSA_VUART_CONSOLE
 	bool "Emulated SBSA UART console support"
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 41d7366527..2a3ec94a18 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -16,7 +16,6 @@ obj-y += domain_build.o
 obj-y += domctl.o
 obj-$(EARLY_PRINTK) += early_printk.o
 obj-y += gic.o
-obj-y += gic-vgic.o
 obj-y += gic-v2.o
 obj-$(CONFIG_HAS_GICV3) += gic-v3.o
 obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
@@ -47,10 +46,19 @@ obj-y += sysctl.o
 obj-y += time.o
 obj-y += traps.o
 obj-y += vcpreg.o
+ifeq ($(CONFIG_NEW_VGIC),y)
+obj-y += vgic/vgic.o
+obj-y += vgic/vgic-v2.o
+obj-y += vgic/vgic-mmio.o
+obj-y += vgic/vgic-mmio-v2.o
+obj-y += vgic/vgic-init.o
+else
+obj-y += gic-vgic.o
 obj-y += vgic.o
 obj-y += vgic-v2.o
 obj-$(CONFIG_HAS_GICV3) += vgic-v3.o
 obj-$(CONFIG_HAS_ITS) += vgic-v3-its.o
+endif
 obj-y += vm_event.o
 obj-y += vtimer.o
 obj-$(CONFIG_SBSA_VUART_CONSOLE) += vpl011.o
diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
index f42092fec3..f7b4779a71 100644
--- a/xen/arch/arm/vgic/vgic.c
+++ b/xen/arch/arm/vgic/vgic.c
@@ -978,6 +978,14 @@ unsigned int domain_max_vcpus(const struct domain *d)
     return min_t(unsigned int, MAX_VIRT_CPUS, vgic_vcpu_limit);
 }
 
+void vgic_v3_setup_hw(paddr_t dbase,
+                      unsigned int nr_rdist_regions,
+                      const struct rdist_region *regions,
+                      unsigned int intid_bits)
+{
+    /* Dummy implementation to allow building without actual vGICv3 support. */
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/common/Makefile b/xen/common/Makefile
index 3a349f478b..92a1d1fa58 100644
--- a/xen/common/Makefile
+++ b/xen/common/Makefile
@@ -19,6 +19,7 @@ obj-y += keyhandler.o
 obj-$(CONFIG_KEXEC) += kexec.o
 obj-$(CONFIG_KEXEC) += kimage.o
 obj-y += lib.o
+obj-$(CONFIG_NEW_VGIC) += list_sort.o
 obj-$(CONFIG_LIVEPATCH) += livepatch.o livepatch_elf.o
 obj-y += lzo.o
 obj-$(CONFIG_HAS_MEM_ACCESS) += mem_access.o
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/57] tools: ARM: vGICv3: Avoid inserting optional DT properties

[Julien Grall]

(+ tools maintainers)

Hi Andre,

Please don't forget to CC the relevant maintainers.

On 05/03/18 16:03, Andre Przywara wrote:
> When creating a GICv3 devicetree node, we currently insert the
> redistributor-stride and #redistributor-regions properties, with fixed
> values which are actually the architected ones. Since those properties are
> optional, and in the case of the stride only needed to cover for broken
> platforms, we don't need to describe them if they don't differ from the
> default values. This will always be the case for our constructed
> DomU memory map.
> So we drop those properties altogether and provide a clean and architected
> GICv3 DT node for DomUs.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - improve commit message
> 
>   tools/libxl/libxl_arm.c | 8 --------
>   1 file changed, 8 deletions(-)
> 
> diff --git a/tools/libxl/libxl_arm.c b/tools/libxl/libxl_arm.c
> index 86f59c0d80..906fd0dcdf 100644
> --- a/tools/libxl/libxl_arm.c
> +++ b/tools/libxl/libxl_arm.c
> @@ -525,14 +525,6 @@ static int make_gicv3_node(libxl__gc *gc, void *fdt)
>       res = fdt_property(fdt, "interrupt-controller", NULL, 0);
>       if (res) return res;
>   
> -    res = fdt_property_cell(fdt, "redistributor-stride",
> -                            GUEST_GICV3_RDIST_STRIDE);
> -    if (res) return res;
> -
> -    res = fdt_property_cell(fdt, "#redistributor-regions",
> -                            GUEST_GICV3_RDIST_REGIONS);
> -    if (res) return res;
> -
>       res = fdt_property_regs(gc, fdt, ROOT_ADDRESS_CELLS, ROOT_SIZE_CELLS,
>                               2,
>                               gicd_base, gicd_size,
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 02/57] ARM: vGICv3: clarify on GUEST_GICV3_RDIST_REGIONS symbol

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> Normally there is only one GICv3 redistributor region, and we use
> that for DomU guests using a GICv3.
> Explain the background in a comment and why we need to keep the number
> of hardware regions for Dom0.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - Keep GUEST_GICV3_RDIST_REGIONS symbol around, just extend comments
> 
>   xen/arch/arm/vgic-v3.c | 12 +++++++++++-
>   1 file changed, 11 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index 2ad8a6be62..d5b34a7d0f 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -1632,6 +1632,16 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
>   
>   static inline unsigned int vgic_v3_rdist_count(struct domain *d)
>   {
> +    /*
> +     * Normally there is only one GICv3 redistributor region.
> +     * The GICv3 DT binding provisions for multiple regions, since there are
> +     * platforms out there which need those (multi-socket systems).
> +     * For Dom0 we have to live with the MMIO layout the hardware provides,
> +     * so we have to copy the multiple regions - as the first region may not
> +     * provide enough space to hold all redistributors we need.
> +     * However DomU get a constructed memory map, so we can go with
> +     * the architected single redistributor region.
> +     */
>       return is_hardware_domain(d) ? vgic_v3_hw.nr_rdist_regions :
>                  GUEST_GICV3_RDIST_REGIONS;
>   }
> @@ -1692,7 +1702,7 @@ static int vgic_v3_domain_init(struct domain *d)
>       {
>           d->arch.vgic.dbase = GUEST_GICV3_GICD_BASE;
>   
> -        /* XXX: Only one Re-distributor region mapped for the guest */
> +        /* A single Re-distributor region is mapped for the guest. */
>           BUILD_BUG_ON(GUEST_GICV3_RDIST_REGIONS != 1);
>   
>           d->arch.vgic.rdist_stride = GUEST_GICV3_RDIST_STRIDE;
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 03/57] ARM: GICv3: use hardware GICv3 redistributor values for Dom0

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The code to generate the DT node or MADT table for Dom0 reaches into the
> domain's VGIC structure to learn the number of redistributor regions and

Mega NIT: Can you either use VGIC or vGIC within the same patch? :)

> their base addresses.
> Since those values are copied from the hardware, we can as well use
> those hardware values directly when setting up the hardware domain.
> 
> This avoids the hardware GIC code to reference vGIC data structures.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - Use GIC hardware values consistently in this function.
> 
>   xen/arch/arm/gic-v3.c | 17 +++++++----------
>   1 file changed, 7 insertions(+), 10 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 25c30bb9ea..b1f8a86409 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -1162,13 +1162,11 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
>       if ( res )
>           return res;
>   
> -    res = fdt_property_cell(fdt, "redistributor-stride",
> -                            d->arch.vgic.rdist_stride);
> +    res = fdt_property_cell(fdt, "redistributor-stride", gicv3.rdist_stride);
>       if ( res )
>           return res;
>   
> -    res = fdt_property_cell(fdt, "#redistributor-regions",
> -                            d->arch.vgic.nr_regions);
> +    res = fdt_property_cell(fdt, "#redistributor-regions", gicv3.rdist_count);
>       if ( res )
>           return res;
>   
> @@ -1178,7 +1176,7 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
>        * CPU interface and virtual cpu interfaces accessesed as System registers
>        * So cells are created only for Distributor and rdist regions
>        */
> -    new_len = new_len * (d->arch.vgic.nr_regions + 1);
> +    new_len = new_len * (gicv3.rdist_count + 1);
>   
>       hw_reg = dt_get_property(gic, "reg", &len);
>       if ( !hw_reg )
> @@ -1406,13 +1404,13 @@ static int gicv3_make_hwdom_madt(const struct domain *d, u32 offset)
>   
>       /* Add Generic Redistributor */
>       size = sizeof(struct acpi_madt_generic_redistributor);
> -    for ( i = 0; i < d->arch.vgic.nr_regions; i++ )
> +    for ( i = 0; i < gicv3.rdist_count; i++ )
>       {
>           gicr = (struct acpi_madt_generic_redistributor *)(base_ptr + table_len);
>           gicr->header.type = ACPI_MADT_TYPE_GENERIC_REDISTRIBUTOR;
>           gicr->header.length = size;
> -        gicr->base_address = d->arch.vgic.rdist_regions[i].base;
> -        gicr->length = d->arch.vgic.rdist_regions[i].size;
> +        gicr->base_address = gicv3.rdist_regions[i].base;
> +        gicr->length = gicv3.rdist_regions[i].size;
>           table_len += size;
>       }
>   
> @@ -1425,8 +1423,7 @@ static unsigned long gicv3_get_hwdom_extra_madt_size(const struct domain *d)
>   {
>       unsigned long size;
>   
> -    size = sizeof(struct acpi_madt_generic_redistributor)
> -           * d->arch.vgic.nr_regions;
> +    size = sizeof(struct acpi_madt_generic_redistributor) * gicv3.rdist_count;
>   
>       size += sizeof(struct acpi_madt_generic_translator)
>               * vgic_v3_its_count(d);
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[FIXUP] replace LOG_2 with ilog2

[Andre Przywara]

The macro formerly named LOG_2 has been recently renamed to ilog2,
so adjust the name in vgic-mmio.h.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
Hi,

I somehow rebased without actually trying to re-compile (since the rebase
itself went through smoothly). So I missed this subtle renaming in the tree.
Please apply this patch (or merge it into
"[PATCH 34/57] ARM: new VGIC: Add MMIO handling framework") to get actually
something that compiles.
Apologies for that mishap!

Cheers,
Andre

 xen/arch/arm/vgic/vgic-mmio.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
index bbf0d181ae..5e6cff9eb2 100644
--- a/xen/arch/arm/vgic/vgic-mmio.h
+++ b/xen/arch/arm/vgic/vgic-mmio.h
@@ -49,7 +49,7 @@ extern struct mmio_handler_ops vgic_io_ops;
  * This assumes that <bits> is a power of two.
  */
 #define VGIC_ADDR_TO_INTID(addr, bits)  (((addr) & VGIC_ADDR_IRQ_MASK(bits)) * \
-                                         8 >> LOG_2(bits))
+                                         8 >> ilog2(bits))
 
 /*
  * Some VGIC registers store per-IRQ information, with a different number
-- 
2.14.1


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/57] ARM: GICv3: simplify GICv3 redistributor stride handling

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> Instead of hard coding the architected redistributor stride into the
> code, lets use a clear #define to the two values for GICv3 and GICv4 and
> clarify the algorithm to determine the needed stride value.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - no changes
> 
>   xen/arch/arm/gic-v3.c             | 18 ++++++++++--------
>   xen/include/asm-arm/gic_v3_defs.h |  5 +++++
>   2 files changed, 15 insertions(+), 8 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index b1f8a86409..be1787b39a 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -690,6 +690,15 @@ static int __init gicv3_populate_rdist(void)
>           do {
>               typer = readq_relaxed(ptr + GICR_TYPER);
>   
> +            /* Set the architectural redist size if not overridden by DT. */
> +            if ( !gicv3.rdist_stride )
> +            {
> +                if ( typer & GICR_TYPER_VLPIS )

Is there anything in the spec promising you that *all* the 
redistributors will support vLPIs?

But I am not fully convinced of having this patch in Xen. I feel it 
makes the code more confusing to read. The current code shows that the 
next re-distributor start is based on the current TYPER values. So I 
think it would make sense to just rework the current code:

if ( gicv3.rdist_stride )
{
    ptr += ...
}
else
{
   if ( typer & GICR_TYPER_VLPIS )
     ptr += GICV4_GICR_SIZE;
   else
     ptr += GICV3_GICR_SIZE;
}

> +                    gicv3.rdist_stride = GICV4_GICR_SIZE;
> +                else
> +                    gicv3.rdist_stride = GICV3_GICR_SIZE;
> +            } > +
>               if ( (typer >> 32) == aff )
>               {
>                   this_cpu(rbase) = ptr;
> @@ -732,14 +741,7 @@ static int __init gicv3_populate_rdist(void)
>               if ( gicv3.rdist_regions[i].single_rdist )
>                   break;
>   
> -            if ( gicv3.rdist_stride )
> -                ptr += gicv3.rdist_stride;
> -            else
> -            {
> -                ptr += SZ_64K * 2; /* Skip RD_base + SGI_base */
> -                if ( typer & GICR_TYPER_VLPIS )
> -                    ptr += SZ_64K * 2; /* Skip VLPI_base + reserved page */
> -            }
> +            ptr += gicv3.rdist_stride;
>   
>           } while ( !(typer & GICR_TYPER_LAST) );
>       }
> diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
> index 65c9dc47cf..412e41afed 100644
> --- a/xen/include/asm-arm/gic_v3_defs.h
> +++ b/xen/include/asm-arm/gic_v3_defs.h
> @@ -18,6 +18,8 @@
>   #ifndef __ASM_ARM_GIC_V3_DEFS_H__
>   #define __ASM_ARM_GIC_V3_DEFS_H__
>   
> +#include <xen/sizes.h>
> +
>   /*
>    * Additional registers defined in GIC v3.
>    * Common GICD registers are defined in gic.h
> @@ -68,6 +70,9 @@
>   #define GICV3_GICD_IIDR_VAL          0x34c
>   #define GICV3_GICR_IIDR_VAL          GICV3_GICD_IIDR_VAL
>   
> +#define GICV3_GICR_SIZE              (2 * SZ_64K)
> +#define GICV4_GICR_SIZE              (4 * SZ_64K)

Do you mind adding a comment either linking to the spec (Section 8.10 
ARM IHI 0069D) or explain the 2/4?

> +
>   #define GICR_CTLR                    (0x0000)
>   #define GICR_IIDR                    (0x0004)
>   #define GICR_TYPER                   (0x0008)
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 07/57] ARM: VGIC: rename gic_inject() and gic_clear_lrs()

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The two central functions to synchronise our emulated VGIC state with
> the GIC hardware (the LRs, really), are named somewhat confusingly.
> Rename them from gic_inject() to vgic_sync_to_lrs() and from
> gic_clear_lrs() to vgic_sync_from_lrs(), to make the code more readable.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - new patch
> 
>   xen/arch/arm/gic-vgic.c   | 4 ++--
>   xen/arch/arm/traps.c      | 4 ++--
>   xen/include/asm-arm/gic.h | 4 ++--
>   3 files changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
> index d273863556..c0fe38fd37 100644
> --- a/xen/arch/arm/gic-vgic.c
> +++ b/xen/arch/arm/gic-vgic.c
> @@ -247,7 +247,7 @@ static void gic_update_one_lr(struct vcpu *v, int i)
>       }
>   }
>   
> -void gic_clear_lrs(struct vcpu *v)
> +void vgic_sync_from_lrs(struct vcpu *v)
>   {
>       int i = 0;
>       unsigned long flags;
> @@ -377,7 +377,7 @@ out:
>       return rc;
>   }
>   
> -void gic_inject(void)
> +void vgic_sync_to_lrs(void)
>   {
>       ASSERT(!local_irq_is_enabled());
>   
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 1cba7e584d..7411bff7a7 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2024,7 +2024,7 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
>           if ( current->arch.hcr_el2 & HCR_VA )
>               current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
>   
> -        gic_clear_lrs(current);
> +        vgic_sync_from_lrs(current);
>       }
>   }
>   
> @@ -2234,7 +2234,7 @@ void leave_hypervisor_tail(void)
>       {
>           local_irq_disable();
>           if (!softirq_pending(smp_processor_id())) {
> -            gic_inject();
> +            vgic_sync_to_lrs();
>   
>               /*
>                * If the SErrors handle option is "DIVERSE", we have to prevent
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 497f195bc1..e2ae4254ed 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -237,7 +237,7 @@ extern int gic_route_irq_to_guest(struct domain *, unsigned int virq,
>   int gic_remove_irq_from_guest(struct domain *d, unsigned int virq,
>                                 struct irq_desc *desc);
>   
> -extern void gic_inject(void);
> +extern void vgic_sync_to_lrs(void);
>   extern void gic_clear_pending_irqs(struct vcpu *v);
>   extern int gic_events_need_delivery(void);
>   
> @@ -295,7 +295,7 @@ extern unsigned int gic_number_lines(void);
>   /* IRQ translation function for the device tree */
>   int gic_irq_xlate(const u32 *intspec, unsigned int intsize,
>                     unsigned int *out_hwirq, unsigned int *out_type);
> -void gic_clear_lrs(struct vcpu *v);
> +void vgic_sync_from_lrs(struct vcpu *v);
>   
>   struct gic_info {
>       /* GIC version */
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 09/57] ARM: VGIC: Move domain_max_vcpus() to be VGIC specific

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> domain_max_vcpus(), which is used by generic Xen code, returns the
> maximum number of VCPUs for a domain, which on ARM is mostly limited by
> the VGIC model emulated (a (v)GICv2 can only handle 8 CPUs).
> Our current implementation lives in arch/arm/domain.c, but reaches into
> VGIC internal data structures.
> Move this function into vgic.c, to keep this VGIC internal.

While I understand the reason behind moving the code in vGIC, it feels a 
bit strange to have a domain function in vgic.c code.

How about renaming the function with a dummy helper in domain.h to 
accommodate common code? Maybe with a comment on top?

Cheers,

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - dump previous approach, move function to VGIC specific file instead
> 
>   xen/arch/arm/domain.c | 14 --------------
>   xen/arch/arm/vgic.c   | 14 ++++++++++++++
>   2 files changed, 14 insertions(+), 14 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index a010443bfd..8546443bad 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -968,20 +968,6 @@ void vcpu_block_unless_event_pending(struct vcpu *v)
>           vcpu_unblock(current);
>   }
>   
> -unsigned int domain_max_vcpus(const struct domain *d)
> -{
> -    /*
> -     * Since evtchn_init would call domain_max_vcpus for poll_mask
> -     * allocation when the vgic_ops haven't been initialised yet,
> -     * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
> -     */
> -    if ( !d->arch.vgic.handler )
> -        return MAX_VIRT_CPUS;
> -    else
> -        return min_t(unsigned int, MAX_VIRT_CPUS,
> -                     d->arch.vgic.handler->max_vcpus);
> -}
> -
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 34269bcf27..c3fdcebbde 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -665,6 +665,20 @@ void vgic_free_virq(struct domain *d, unsigned int virq)
>       clear_bit(virq, d->arch.vgic.allocated_irqs);
>   }
>   
> +unsigned int domain_max_vcpus(const struct domain *d)
> +{
> +    /*
> +     * Since evtchn_init would call domain_max_vcpus for poll_mask
> +     * allocation when the vgic_ops haven't been initialised yet,
> +     * we return MAX_VIRT_CPUS if d->arch.vgic.handler is null.
> +     */
> +    if ( !d->arch.vgic.handler )
> +        return MAX_VIRT_CPUS;
> +    else
> +        return min_t(unsigned int, MAX_VIRT_CPUS,
> +                     d->arch.vgic.handler->max_vcpus);
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/57] tools: ARM: vGICv3: Avoid inserting optional DT properties

[Wei Liu]

On Mon, Mar 05, 2018 at 04:39:09PM +0000, Julien Grall wrote:
> (+ tools maintainers)
> 
> Hi Andre,
> 
> Please don't forget to CC the relevant maintainers.
> 
> On 05/03/18 16:03, Andre Przywara wrote:
> > When creating a GICv3 devicetree node, we currently insert the
> > redistributor-stride and #redistributor-regions properties, with fixed
> > values which are actually the architected ones. Since those properties are
> > optional, and in the case of the stride only needed to cover for broken
> > platforms, we don't need to describe them if they don't differ from the
> > default values. This will always be the case for our constructed
> > DomU memory map.
> > So we drop those properties altogether and provide a clean and architected
> > GICv3 DT node for DomUs.
> > 
> > Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> 
> Reviewed-by: Julien Grall <julien.grall@arm.com>
> 

Acked-by: Wei Liu <wei.liu2@citrix.com>

Julien please commit this yourself.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 00/57] New VGIC(-v2) implementation

[Andre Przywara]

Hi,

On 05/03/18 16:03, Andre Przywara wrote:
> tl;dr: Coarse changelog below, individual patches have changelogs as
> well.
> 
> This is an updated version of the new VGIC-v2 implementation.

For the brave amongst you who actually want to try it, there is a git
repo with these patches on top of current origin/staging:
http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/vgic-new/v1
git://linux-arm.org/xen-ap.git branch vgic-new/v1

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 01/57] tools: ARM: vGICv3: Avoid inserting optional DT properties

[Julien Grall]

On 05/03/18 17:18, Wei Liu wrote:
> On Mon, Mar 05, 2018 at 04:39:09PM +0000, Julien Grall wrote:
>> (+ tools maintainers)
>>
>> Hi Andre,
>>
>> Please don't forget to CC the relevant maintainers.
>>
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> When creating a GICv3 devicetree node, we currently insert the
>>> redistributor-stride and #redistributor-regions properties, with fixed
>>> values which are actually the architected ones. Since those properties are
>>> optional, and in the case of the stride only needed to cover for broken
>>> platforms, we don't need to describe them if they don't differ from the
>>> default values. This will always be the case for our constructed
>>> DomU memory map.
>>> So we drop those properties altogether and provide a clean and architected
>>> GICv3 DT node for DomUs.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>
>> Reviewed-by: Julien Grall <julien.grall@arm.com>
>>
> 
> Acked-by: Wei Liu <wei.liu2@citrix.com>
> 
> Julien please commit this yourself.

Done.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 10/57] ARM: VGIC: rename gic_event_needs_delivery()

[Julien Grall]

Hi,

On 05/03/18 16:03, Andre Przywara wrote:
> gic_event_needs_delivery() is not named very intuitively, especially
> the gic_ prefix is somewhat misleading.
> Rename it to vgic_pending_irq(), which makes it clear that this relates
> to the virtual GIC and is about interrupts.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - new patch
> 
>   xen/arch/arm/gic-vgic.c     | 2 +-
>   xen/include/asm-arm/event.h | 2 +-
>   xen/include/asm-arm/gic.h   | 2 +-
>   3 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
> index c0fe38fd37..60c6c463e9 100644
> --- a/xen/arch/arm/gic-vgic.c
> +++ b/xen/arch/arm/gic-vgic.c
> @@ -339,7 +339,7 @@ void gic_clear_pending_irqs(struct vcpu *v)
>           gic_remove_from_lr_pending(v, p);
>   }
>   
> -int gic_events_need_delivery(void)
> +int vgic_pending_irq(void)
>   {
>       struct vcpu *v = current;
>       struct pending_irq *p;
> diff --git a/xen/include/asm-arm/event.h b/xen/include/asm-arm/event.h
> index e8c2a6cb44..c4c79fa87d 100644
> --- a/xen/include/asm-arm/event.h
> +++ b/xen/include/asm-arm/event.h
> @@ -24,7 +24,7 @@ static inline int local_events_need_delivery_nomask(void)
>        * interrupts disabled so this shouldn't be a problem in the general
>        * case.
>        */
> -    if ( gic_events_need_delivery() )
> +    if ( vgic_pending_irq() )

I am not a big fan of this name either. I would much prefer to follow 
the suggestion I made in your RFC patch #26.

By that I mean renaming the function vgic_vcpu_pending_irq and take a 
vCPU in parameter.

For the old vGIC implementation, add an ASSERT(vcpu == current) to catch 
misuse for the function.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 11/57] ARM: VGIC: change to level-IRQ compatible IRQ injection interface

[Julien Grall]

Hi,

On 05/03/18 16:03, Andre Przywara wrote:
> At the moment vgic_vcpu_inject_irq() is the interface for Xen internal
> code and virtual devices to inject IRQs into a guest. This interface has
> two shortcomings:
> 1) It requires a VCPU pointer, which we may not know (and don't need!)
> for shared interrupts. A second function (vgic_vcpu_inject_spi()), was
> there to work around this issue.
> 2) This interface only really supports edge triggered IRQs, which is
> what the Xen VGIC emulates only anyway. However this needs to and will
> change, so we need to add the desired level (high or low) to the
> interface.
> This replaces the existing injection call (taking a VCPU and an IRQ
> parameter) with a new one, taking domain, VCPU, IRQ and level parameters.
> The VCPU can be NULL in case we don't know and don't care.
> We change all call sites to use this new interface. This still doesn't
> give us the missing level IRQ handling, but at least prepares the callers
> to do the right thing later automatically.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - no change

Hmmm, on the RFC we agreed that vgic_inject_irq should return void. And 
printing some debug messages if the vgic_get_irq() fails.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 12/57] ARM: VGIC: carve out struct vgic_cpu and struct vgic_dist

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> Currently we describe the VGIC specific fields in a structure
> *embedded* in struct arch_domain and struct arch_vcpu. These members
> there are however related to the current VGIC implementation, and will
> be substantially different in the future.
> To allow coexistence of two implementations, move the definition of these
> embedded structures into vgic.h, and just use the opaque type in the arch
> specific structures.
> This allows easy switching between different implementations later.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Please try to collect the tags for patch that was reviewed. I don't 
fancy to review them again ;).

FIY that was a:

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - No changes
> 
>   xen/include/asm-arm/domain.h | 85 +-----------------------------------------
>   xen/include/asm-arm/vgic.h   | 88 ++++++++++++++++++++++++++++++++++++++++++++
>   2 files changed, 90 insertions(+), 83 deletions(-)
> 
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 1dd9683d25..bb7a46c1d0 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -74,57 +74,7 @@ struct arch_domain
>           uint64_t offset;
>       } virt_timer_base;
>   
> -    struct {
> -        /* Version of the vGIC */
> -        enum gic_version version;
> -        /* GIC HW version specific vGIC driver handler */
> -        const struct vgic_ops *handler;
> -        /*
> -         * Covers access to other members of this struct _except_ for
> -         * shared_irqs where each member contains its own locking.
> -         *
> -         * If both class of lock is required then this lock must be
> -         * taken first. If multiple rank locks are required (including
> -         * the per-vcpu private_irqs rank) then they must be taken in
> -         * rank order.
> -         */
> -        spinlock_t lock;
> -        uint32_t ctlr;
> -        int nr_spis; /* Number of SPIs */
> -        unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
> -        struct vgic_irq_rank *shared_irqs;
> -        /*
> -         * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> -         * struct arch_vcpu.
> -         */
> -        struct pending_irq *pending_irqs;
> -        /* Base address for guest GIC */
> -        paddr_t dbase; /* Distributor base address */
> -#ifdef CONFIG_HAS_GICV3
> -        /* GIC V3 addressing */
> -        /* List of contiguous occupied by the redistributors */
> -        struct vgic_rdist_region {
> -            paddr_t base;                   /* Base address */
> -            paddr_t size;                   /* Size */
> -            unsigned int first_cpu;         /* First CPU handled */
> -        } *rdist_regions;
> -        int nr_regions;                     /* Number of rdist regions */
> -        unsigned long int nr_lpis;
> -        uint64_t rdist_propbase;
> -        struct rb_root its_devices;         /* Devices mapped to an ITS */
> -        spinlock_t its_devices_lock;        /* Protects the its_devices tree */
> -        struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
> -        rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
> -        struct list_head vits_list;         /* List of virtual ITSes */
> -        unsigned int intid_bits;
> -        /*
> -         * TODO: if there are more bool's being added below, consider
> -         * a flags variable instead.
> -         */
> -        bool rdists_enabled;                /* Is any redistributor enabled? */
> -        bool has_its;
> -#endif
> -    } vgic;
> +    struct vgic_dist vgic;
>   
>       struct vuart {
>   #define VUART_BUF_SIZE 128
> @@ -247,38 +197,7 @@ struct arch_vcpu
>       union gic_state_data gic;
>       uint64_t lr_mask;
>   
> -    struct {
> -        /*
> -         * SGIs and PPIs are per-VCPU, SPIs are domain global and in
> -         * struct arch_domain.
> -         */
> -        struct pending_irq pending_irqs[32];
> -        struct vgic_irq_rank *private_irqs;
> -
> -        /* This list is ordered by IRQ priority and it is used to keep
> -         * track of the IRQs that the VGIC injected into the guest.
> -         * Depending on the availability of LR registers, the IRQs might
> -         * actually be in an LR, and therefore injected into the guest,
> -         * or queued in gic.lr_pending.
> -         * As soon as an IRQ is EOI'd by the guest and removed from the
> -         * corresponding LR it is also removed from this list. */
> -        struct list_head inflight_irqs;
> -        /* lr_pending is used to queue IRQs (struct pending_irq) that the
> -         * vgic tried to inject in the guest (calling gic_set_guest_irq) but
> -         * no LRs were available at the time.
> -         * As soon as an LR is freed we remove the first IRQ from this
> -         * list and write it to the LR register.
> -         * lr_pending is a subset of vgic.inflight_irqs. */
> -        struct list_head lr_pending;
> -        spinlock_t lock;
> -
> -        /* GICv3: redistributor base and flags for this vCPU */
> -        paddr_t rdist_base;
> -        uint64_t rdist_pendbase;
> -#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
> -#define VGIC_V3_LPIS_ENABLED    (1 << 1)
> -        uint8_t flags;
> -    } vgic;
> +    struct vgic_cpu vgic;
>   
>       /* Timer registers  */
>       uint32_t cntkctl;
> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index b75fdeb068..4e1c37f091 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -19,6 +19,9 @@
>   #define __ASM_ARM_VGIC_H__
>   
>   #include <xen/bitops.h>
> +#include <xen/radix-tree.h>
> +#include <xen/rbtree.h>
> +#include <asm/gic.h>
>   #include <asm/mmio.h>
>   #include <asm/vreg.h>
>   
> @@ -123,6 +126,91 @@ struct vgic_irq_rank {
>       uint8_t vcpu[32];
>   };
>   
> +struct vgic_dist {
> +    /* Version of the vGIC */
> +    enum gic_version version;
> +    /* GIC HW version specific vGIC driver handler */
> +    const struct vgic_ops *handler;
> +    /*
> +     * Covers access to other members of this struct _except_ for
> +     * shared_irqs where each member contains its own locking.
> +     *
> +     * If both class of lock is required then this lock must be
> +     * taken first. If multiple rank locks are required (including
> +     * the per-vcpu private_irqs rank) then they must be taken in
> +     * rank order.
> +     */
> +    spinlock_t lock;
> +    uint32_t ctlr;
> +    int nr_spis; /* Number of SPIs */
> +    unsigned long *allocated_irqs; /* bitmap of IRQs allocated */
> +    struct vgic_irq_rank *shared_irqs;
> +    /*
> +     * SPIs are domain global, SGIs and PPIs are per-VCPU and stored in
> +     * struct arch_vcpu.
> +     */
> +    struct pending_irq *pending_irqs;
> +    /* Base address for guest GIC */
> +    paddr_t dbase; /* Distributor base address */
> +#ifdef CONFIG_HAS_GICV3
> +    /* GIC V3 addressing */
> +    /* List of contiguous occupied by the redistributors */
> +    struct vgic_rdist_region {
> +        paddr_t base;                   /* Base address */
> +        paddr_t size;                   /* Size */
> +        unsigned int first_cpu;         /* First CPU handled */
> +    } *rdist_regions;
> +    int nr_regions;                     /* Number of rdist regions */
> +    unsigned long int nr_lpis;
> +    uint64_t rdist_propbase;
> +    struct rb_root its_devices;         /* Devices mapped to an ITS */
> +    spinlock_t its_devices_lock;        /* Protects the its_devices tree */
> +    struct radix_tree_root pend_lpi_tree; /* Stores struct pending_irq's */
> +    rwlock_t pend_lpi_tree_lock;        /* Protects the pend_lpi_tree */
> +    struct list_head vits_list;         /* List of virtual ITSes */
> +    unsigned int intid_bits;
> +    /*
> +     * TODO: if there are more bool's being added below, consider
> +     * a flags variable instead.
> +     */
> +    bool rdists_enabled;                /* Is any redistributor enabled? */
> +    bool has_its;
> +#endif
> +};
> +
> +struct vgic_cpu {
> +    /*
> +     * SGIs and PPIs are per-VCPU, SPIs are domain global and in
> +     * struct arch_domain.
> +     */
> +    struct pending_irq pending_irqs[32];
> +    struct vgic_irq_rank *private_irqs;
> +
> +    /* This list is ordered by IRQ priority and it is used to keep
> +     * track of the IRQs that the VGIC injected into the guest.
> +     * Depending on the availability of LR registers, the IRQs might
> +     * actually be in an LR, and therefore injected into the guest,
> +     * or queued in gic.lr_pending.
> +     * As soon as an IRQ is EOI'd by the guest and removed from the
> +     * corresponding LR it is also removed from this list. */
> +    struct list_head inflight_irqs;
> +    /* lr_pending is used to queue IRQs (struct pending_irq) that the
> +     * vgic tried to inject in the guest (calling gic_set_guest_irq) but
> +     * no LRs were available at the time.
> +     * As soon as an LR is freed we remove the first IRQ from this
> +     * list and write it to the LR register.
> +     * lr_pending is a subset of vgic.inflight_irqs. */
> +    struct list_head lr_pending;
> +    spinlock_t lock;
> +
> +    /* GICv3: redistributor base and flags for this vCPU */
> +    paddr_t rdist_base;
> +    uint64_t rdist_pendbase;
> +#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
> +#define VGIC_V3_LPIS_ENABLED    (1 << 1)
> +    uint8_t flags;
> +};
> +
>   struct sgi_target {
>       uint8_t aff1;
>       uint16_t list;
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/57] ARM: GICv3: simplify GICv3 redistributor stride handling

[Julien Grall]

On 05/03/18 17:08, Julien Grall wrote:
> On 05/03/18 16:03, Andre Przywara wrote:
>> Instead of hard coding the architected redistributor stride into the
>> code, lets use a clear #define to the two values for GICv3 and GICv4 and
>> clarify the algorithm to determine the needed stride value.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - no changes
>>
>>   xen/arch/arm/gic-v3.c             | 18 ++++++++++--------
>>   xen/include/asm-arm/gic_v3_defs.h |  5 +++++
>>   2 files changed, 15 insertions(+), 8 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index b1f8a86409..be1787b39a 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -690,6 +690,15 @@ static int __init gicv3_populate_rdist(void)
>>           do {
>>               typer = readq_relaxed(ptr + GICR_TYPER);
>> +            /* Set the architectural redist size if not overridden by 
>> DT. */
>> +            if ( !gicv3.rdist_stride )
>> +            {
>> +                if ( typer & GICR_TYPER_VLPIS )
> 
> Is there anything in the spec promising you that *all* the 
> redistributors will support vLPIs?

Answering to myself, Marc pointed out that nothing was promising all 
redistributors will support vLPIs. This is confirmed by the section 9.7 
"Mixing GICv3 and GICv4" in "GICv3 and GICv4 Software Overview" DAI 0492B.

So I would prefer to drop that patch and move GICV*_GICR_SIZE definition 
in patch #5. I am assuming we will never expose a mix of GICv3/GICv4 on Xen.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 14/57] ARM: VGIC: Introduce gic_get_nr_lrs()

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> So far the number of list registers (LRs) a GIC implements is only
> needed in the hardware facing side of the VGIC code (gic-vgic.c).
> The new VGIC will need this information in more and multiple places, so
> export a function that returns the number.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - move gic_get_nr_lrs() into gic.h (as a static inline)
> 
>   xen/arch/arm/gic-vgic.c   | 10 +++++-----
>   xen/include/asm-arm/gic.h |  6 ++++++
>   2 files changed, 11 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-vgic.c b/xen/arch/arm/gic-vgic.c
> index 60c6c463e9..93e42739d9 100644
> --- a/xen/arch/arm/gic-vgic.c
> +++ b/xen/arch/arm/gic-vgic.c
> @@ -25,7 +25,7 @@
>   #include <asm/gic.h>
>   #include <asm/vgic.h>
>   
> -#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_hw_ops->info->nr_lrs) - 1))
> +#define lr_all_full() (this_cpu(lr_mask) == ((1 << gic_get_nr_lrs()) - 1))
>   
>   #undef GIC_DEBUG
>   
> @@ -110,7 +110,7 @@ static unsigned int gic_find_unused_lr(struct vcpu *v,
>                                          struct pending_irq *p,
>                                          unsigned int lr)
>   {
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>       unsigned long *lr_mask = (unsigned long *) &this_cpu(lr_mask);
>       struct gic_lr lr_val;
>   
> @@ -137,7 +137,7 @@ void gic_raise_guest_irq(struct vcpu *v, unsigned int virtual_irq,
>           unsigned int priority)
>   {
>       int i;
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>       struct pending_irq *p = irq_to_pending(v, virtual_irq);
>   
>       ASSERT(spin_is_locked(&v->arch.vgic.lock));
> @@ -251,7 +251,7 @@ void vgic_sync_from_lrs(struct vcpu *v)
>   {
>       int i = 0;
>       unsigned long flags;
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>   
>       /* The idle domain has no LRs to be cleared. Since gic_restore_state
>        * doesn't write any LR registers for the idle domain they could be
> @@ -278,7 +278,7 @@ static void gic_restore_pending_irqs(struct vcpu *v)
>       struct pending_irq *p, *t, *p_r;
>       struct list_head *inflight_r;
>       unsigned long flags;
> -    unsigned int nr_lrs = gic_hw_ops->info->nr_lrs;
> +    unsigned int nr_lrs = gic_get_nr_lrs();
>       int lrs = nr_lrs;
>   
>       spin_lock_irqsave(&v->arch.vgic.lock, flags);
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index a23c307c3a..b3f840ea9a 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -374,6 +374,12 @@ struct gic_hw_operations {
>   };
>   
>   extern const struct gic_hw_operations *gic_hw_ops;
> +
> +static inline unsigned int gic_get_nr_lrs(void)
> +{
> +    return gic_hw_ops->info->nr_lrs;
> +}
> +
>   void register_gic_ops(const struct gic_hw_operations *ops);
>   int gic_make_hwdom_dt_node(const struct domain *d,
>                              const struct dt_device_node *gic,
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 15/57] ARM: GICv2: Extend and adjust register definitions

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The new VGIC will shortly use more bits of the various GIC registers, so
> add the respective definitions from the manual.
This series does not seem to use any of the new value you added. Did I 
miss anything?

Note that I am not against this patch, but the commit message should be 
updated.

> This includes bits from the GICC_CTL register and some minor other bits.

s/GICC_CTL/GICC_CLTR/ and same with the update you do below as there are 
only limited use in the code. So should not much impact the rest of the 
series.

> Adjust the usage of ICC_CTL_ENABLE on the way, to be more precise about

Did you mean GICC_CTLR_ENABLE?

> which of the two enable bits we actually deal with.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - extend commit message
> 
>   xen/arch/arm/gic-v2.c     |  2 +-
>   xen/include/asm-arm/gic.h | 18 ++++++++++++++++--
>   2 files changed, 17 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 2b271ba322..7938a42591 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -358,7 +358,7 @@ static void gicv2_cpu_init(void)
>       /* Finest granularity of priority */
>       writel_gicc(0x0, GICC_BPR);
>       /* Turn on delivery */
> -    writel_gicc(GICC_CTL_ENABLE|GICC_CTL_EOI, GICC_CTLR);
> +    writel_gicc(GICC_CTL_ENABLE0|GICC_CTL_EOI, GICC_CTLR);
>   }
>   
>   static void gicv2_cpu_disable(void)
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index b3f840ea9a..8fab458d7f 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -77,6 +77,7 @@
>   #define GICC_EOIR       (0x0010)
>   #define GICC_RPR        (0x0014)
>   #define GICC_HPPIR      (0x0018)
> +#define GICC_ABPR       (0x001c)
>   #define GICC_APR        (0x00D0)
>   #define GICC_NSAPR      (0x00E0)
>   #define GICC_IIDR       (0x00FC)
> @@ -102,8 +103,18 @@
>   #define GICD_TYPE_SEC   0x400
>   #define GICD_TYPER_DVIS (1U << 18)
>   
> -#define GICC_CTL_ENABLE 0x1
> -#define GICC_CTL_EOI    (0x1 << 9)
> +#define GICC_CTL_ENABLE0_SHIFT  0
> +#define GICC_CTL_ENABLE0        (1U << GICC_CTL_ENABLE0_SHIFT)
> +#define GICC_CTL_ENABLE1_SHIFT  1
> +#define GICC_CTL_ENABLE1        (1U << GICC_CTL_ENABLE1)
> +#define GICC_CTL_AC_SHIFT       2
> +#define GICC_CTL_AC             (1U << GICC_CTL_AC_SHIFT)
> +#define GICC_CTL_FIQEN_SHIFT    3
> +#define GICC_CTL_FIQEN          (1U << GICC_CTL_FIQEN_SHIFT)
> +#define GICC_CTL_CBPR_SHIFT     4
> +#define GICC_CTL_CBPR           (1U << GICC_CTL_CBPR_SHIFT)
> +#define GICC_CTL_EOI_SHIFT      9
> +#define GICC_CTL_EOI            (1U << GICC_CTL_EOI_SHIFT)
>   
>   #define GICC_IA_IRQ       0x03ff
>   #define GICC_IA_CPU_MASK  0x1c00
> @@ -127,6 +138,9 @@
>   #define GICH_MISR_VGRP1E  (1 << 6)
>   #define GICH_MISR_VGRP1D  (1 << 7)
>   
> +#define GICV_PMR_PRIORITY_SHIFT		3
> +#define GICV_PMR_PRIORITY_MASK		(0x1f << GICV_PMR_PRIORITY_SHIFT)
> +
>   /*
>    * The minimum GICC_BPR is required to be in the range 0-3. We set
>    * GICC_BPR to 0 but we must expect that it might be 3. This means we
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 16/57] ARM: GICv3: rename HYP interface definitions to use ICH_ prefix

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> On a GICv3 in non-compat mode the hypervisor interface is always
> accessed via system registers. Those register names have a "ICH_" prefix
> in the manual, to differentiate them from the MMIO registers. Also those
> registers are mostly 64-bit (compared to the 32-bit GICv2 registers) and
> use different bit assignments.
> To make this obvious and to avoid clashes with double definitions using
> the same names for actually different bits, lets change all GICv3
> hypervisor interface registers to use the "ICH_" prefix from the manual.
> This renames the definitions in gic_v3_defs.h and their usage in gic-v3.c
> and is needed to allow co-existence of the GICv2 and GICv3 definitions
> in the same file.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

I guess this was compiled test in standalone? On that a basis:

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 17/57] ARM: Introduce kick_vcpu()

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> If we change something in a vCPU that affects its runnability or
> otherwise needs the vCPU's attention, we might need to tell the scheduler
> about it.
> We are using this in one place (vIRQ injection) at the moment, but will
> need this at more places soon.
> So let's factor out this functionality in the new kick_vcpu() function
> and make this available to the whole Xen arch code.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - new patch
> 
>   xen/arch/arm/smp.c        | 14 ++++++++++++++
>   xen/arch/arm/vgic.c       | 10 ++--------
>   xen/include/asm-arm/smp.h |  3 +++
>   3 files changed, 19 insertions(+), 8 deletions(-)
> 
> diff --git a/xen/arch/arm/smp.c b/xen/arch/arm/smp.c
> index 62f57f0ba2..381a4786a2 100644
> --- a/xen/arch/arm/smp.c
> +++ b/xen/arch/arm/smp.c
> @@ -4,6 +4,8 @@
>   #include <asm/page.h>
>   #include <asm/gic.h>
>   #include <asm/flushtlb.h>
> +#include <xen/perfc.h>
> +#include <xen/sched.h>
>   
>   void flush_tlb_mask(const cpumask_t *mask)
>   {
> @@ -32,6 +34,18 @@ void smp_send_call_function_mask(const cpumask_t *mask)
>       }
>   }
>   
> +void kick_vcpu(struct vcpu *vcpu)

Can we name it vcpu_kick? This is to match the x86 side and it seems we 
have a prototype in events.h.

Also, IHMO this belongs to domain.c as this deal with vCPU. smp.c is 
more for dealing with pCPU.

> +{
> +    bool running = vcpu->is_running;
> +
> +    vcpu_unblock(vcpu);
> +    if ( running && vcpu != current )
> +    {
> +        perfc_incr(vgic_cross_cpu_intr_inject);
> +        smp_send_event_check_mask(cpumask_of(vcpu->processor));
> +    }
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 3c77d5fef6..e44de9ea95 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -530,7 +530,6 @@ int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
>       uint8_t priority;
>       struct pending_irq *iter, *n;
>       unsigned long flags;
> -    bool running;
>   
>       /*
>        * For edge triggered interrupts we always ignore a "falling edge".
> @@ -590,14 +589,9 @@ int vgic_inject_irq(struct domain *d, struct vcpu *v, unsigned int virq,
>       list_add_tail(&n->inflight, &v->arch.vgic.inflight_irqs);
>   out:
>       spin_unlock_irqrestore(&v->arch.vgic.lock, flags);
> +
>       /* we have a new higher priority irq, inject it into the guest */
> -    running = v->is_running;
> -    vcpu_unblock(v);
> -    if ( running && v != current )
> -    {
> -        perfc_incr(vgic_cross_cpu_intr_inject);
> -        smp_send_event_check_mask(cpumask_of(v->processor));
> -    }
> +    kick_vcpu(v);
>   
>       return 0;
>   }
> diff --git a/xen/include/asm-arm/smp.h b/xen/include/asm-arm/smp.h
> index 3c122681d7..7c8ef75789 100644
> --- a/xen/include/asm-arm/smp.h
> +++ b/xen/include/asm-arm/smp.h
> @@ -28,6 +28,9 @@ extern void init_secondary(void);
>   extern void smp_init_cpus(void);
>   extern void smp_clear_cpu_maps (void);
>   extern int smp_get_max_cpus (void);
> +
> +void kick_vcpu(struct vcpu *vcpu);
> +
>   #endif
>   
>   /*
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 18/57] ARM: GICv2: introduce gicv2_poke_irq()

[Julien Grall]

Hi,

On 05/03/18 16:03, Andre Przywara wrote:
> The GICv2 uses bitmaps spanning several MMIO registers for holding some
> interrupt state. Similar to GICv3, add a poke helper functions to set a bit
> for a given irq_desc in one of those bitmaps.
> At the moment there is only one use in gic-v2.c, but there will be more
> coming soon.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - new patch
> 
>   xen/arch/arm/gic-v2.c | 11 +++++++----
>   1 file changed, 7 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 7938a42591..618dd94120 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -235,6 +235,11 @@ static unsigned int gicv2_read_irq(void)
>       return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>   }
>   
> +static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)

NIT: s/irqd/desc/ to match the naming in Xen.

With that:

Reviewed-by: Julien Grall <julien.grall@arm.com>

> +{
> +    writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
> +}
> +
>   static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int type)
>   {
>       uint32_t cfg, actual, edgebit;
> @@ -509,7 +514,6 @@ static unsigned int gicv2_read_apr(int apr_reg)
>   static void gicv2_irq_enable(struct irq_desc *desc)
>   {
>       unsigned long flags;
> -    int irq = desc->irq;
>   
>       ASSERT(spin_is_locked(&desc->lock));
>   
> @@ -517,20 +521,19 @@ static void gicv2_irq_enable(struct irq_desc *desc)
>       clear_bit(_IRQ_DISABLED, &desc->status);
>       dsb(sy);
>       /* Enable routing */
> -    writel_gicd((1u << (irq % 32)), GICD_ISENABLER + (irq / 32) * 4);
> +    gicv2_poke_irq(desc, GICD_ISENABLER);
>       spin_unlock_irqrestore(&gicv2.lock, flags);
>   }
>   
>   static void gicv2_irq_disable(struct irq_desc *desc)
>   {
>       unsigned long flags;
> -    int irq = desc->irq;
>   
>       ASSERT(spin_is_locked(&desc->lock));
>   
>       spin_lock_irqsave(&gicv2.lock, flags);
>       /* Disable routing */
> -    writel_gicd(1u << (irq % 32), GICD_ICENABLER + (irq / 32) * 4);
> +    gicv2_poke_irq(desc, GICD_ICENABLER);
>       set_bit(_IRQ_DISABLED, &desc->status);
>       spin_unlock_irqrestore(&gicv2.lock, flags);
>   }
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 18/57] ARM: GICv2: introduce gicv2_poke_irq()

[Julien Grall]

Hi,

On 06/03/18 15:23, Julien Grall wrote:
> Hi,
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> The GICv2 uses bitmaps spanning several MMIO registers for holding some
>> interrupt state. Similar to GICv3, add a poke helper functions to set 
>> a bit
>> for a given irq_desc in one of those bitmaps.
>> At the moment there is only one use in gic-v2.c, but there will be more
>> coming soon.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - new patch
>>
>>   xen/arch/arm/gic-v2.c | 11 +++++++----
>>   1 file changed, 7 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>> index 7938a42591..618dd94120 100644
>> --- a/xen/arch/arm/gic-v2.c
>> +++ b/xen/arch/arm/gic-v2.c
>> @@ -235,6 +235,11 @@ static unsigned int gicv2_read_irq(void)
>>       return (readl_gicc(GICC_IAR) & GICC_IA_IRQ);
>>   }
>> +static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)
> 
> NIT: s/irqd/desc/ to match the naming in Xen.

Scratch that. It looks like we already have quite a few irqd in the code.

Cheers,

>  > With that:
> 
> Reviewed-by: Julien Grall <julien.grall@arm.com>
> 
>> +{
>> +    writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
>> +}
>> +
>>   static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int 
>> type)
>>   {
>>       uint32_t cfg, actual, edgebit;
>> @@ -509,7 +514,6 @@ static unsigned int gicv2_read_apr(int apr_reg)
>>   static void gicv2_irq_enable(struct irq_desc *desc)
>>   {
>>       unsigned long flags;
>> -    int irq = desc->irq;
>>       ASSERT(spin_is_locked(&desc->lock));
>> @@ -517,20 +521,19 @@ static void gicv2_irq_enable(struct irq_desc *desc)
>>       clear_bit(_IRQ_DISABLED, &desc->status);
>>       dsb(sy);
>>       /* Enable routing */
>> -    writel_gicd((1u << (irq % 32)), GICD_ISENABLER + (irq / 32) * 4);
>> +    gicv2_poke_irq(desc, GICD_ISENABLER);
>>       spin_unlock_irqrestore(&gicv2.lock, flags);
>>   }
>>   static void gicv2_irq_disable(struct irq_desc *desc)
>>   {
>>       unsigned long flags;
>> -    int irq = desc->irq;
>>       ASSERT(spin_is_locked(&desc->lock));
>>       spin_lock_irqsave(&gicv2.lock, flags);
>>       /* Disable routing */
>> -    writel_gicd(1u << (irq % 32), GICD_ICENABLER + (irq / 32) * 4);
>> +    gicv2_poke_irq(desc, GICD_ICENABLER);
>>       set_bit(_IRQ_DISABLED, &desc->status);
>>       spin_unlock_irqrestore(&gicv2.lock, flags);
>>   }
>>
> 
> Cheers,
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 19/57] ARM: GICv3: poke_irq: make RWP optional

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> A GICv3 hardware implementation can be implemented in several parts that
> communicate with each other (think multi-socket systems).
> To make sure that critical settings have arrived at all endpoints, some
> bits are tracked using the RWP bit in the GICD_CTLR register, which
> signals whether a register write is still in progress.
> However this only applies to *some* registers, namely the bits in the
> GICD_ICENABLER (disabling interrupts) and some bits in the GICD_CTLR
> register (cf. Arm IHI 0069D, 8.9.4: RWP, bit[31]).
> But our gicv3_poke_irq() was always polling this bit before returning,
> resulting in pointless MMIO reads for many registers.
> Add an option to gicv3_poke_irq() to state whether we want to wait for
> this bit and use it accordingly to match the spec.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - new patch
> 
>   xen/arch/arm/gic-v3.c | 12 +++++++-----
>   1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 3e381d031b..44dfba2267 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -428,9 +428,9 @@ static void gicv3_dump_state(const struct vcpu *v)
>       }
>   }
>   
> -static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset)
> +static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
>   {
> -    u32 mask = 1 << (irqd->irq % 32);
> +    u32 mask = 1U << (irqd->irq % 32);

Do you mind adding a word about 1U in the commit message?

With that:

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 20/57] ARM: GICv2: fix GICH_V2_LR definitions

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The bit definition for the CPUID mask in the GICv2 LR register was
> wrong, fortunately the current implementation does not use that bit.
> Fix it up (it's starting at bit 10, not bit 9) and clean up some
> nearby definitions on the way.
> This will be used by the new VGIC shortly.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

> ---
> Changelog RFC ... v1:
> - new patch
>   xen/arch/arm/gic-v2.c | 9 +++++----
>   1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 618dd94120..031be920cc 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -57,10 +57,11 @@
>   #define GICH_V2_LR_HW_MASK         0x1
>   #define GICH_V2_LR_GRP_SHIFT       30
>   #define GICH_V2_LR_GRP_MASK        0x1
> -#define GICH_V2_LR_MAINTENANCE_IRQ (1<<19)
> -#define GICH_V2_LR_GRP1            (1<<30)
> -#define GICH_V2_LR_HW              (1<<31)
> -#define GICH_V2_LR_CPUID_SHIFT     9
> +#define GICH_V2_LR_MAINTENANCE_IRQ (1U << 19)
> +#define GICH_V2_LR_GRP1            (1U << 30)
> +#define GICH_V2_LR_HW              (1U << GICH_V2_LR_HW_SHIFT)

I think we want this patch to get backported as 1 << 31 is an undefined 
behavior.

> +#define GICH_V2_LR_CPUID_SHIFT     10
> +#define GICH_V2_LR_CPUID_MASK      0x7
>   #define GICH_V2_VTR_NRLRGS         0x3f
>   
>   #define GICH_V2_VMCR_PRIORITY_MASK   0x1f
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 20/57] ARM: GICv2: fix GICH_V2_LR definitions

[Andre Przywara]

Hi,

On 06/03/18 15:46, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> The bit definition for the CPUID mask in the GICv2 LR register was
>> wrong, fortunately the current implementation does not use that bit.
>> Fix it up (it's starting at bit 10, not bit 9) and clean up some
>> nearby definitions on the way.
>> This will be used by the new VGIC shortly.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> 
> Reviewed-by: Julien Grall <julien.grall@arm.com>

Thanks!

> 
>> ---
>> Changelog RFC ... v1:
>> - new patch
>>   xen/arch/arm/gic-v2.c | 9 +++++----
>>   1 file changed, 5 insertions(+), 4 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>> index 618dd94120..031be920cc 100644
>> --- a/xen/arch/arm/gic-v2.c
>> +++ b/xen/arch/arm/gic-v2.c
>> @@ -57,10 +57,11 @@
>>   #define GICH_V2_LR_HW_MASK         0x1
>>   #define GICH_V2_LR_GRP_SHIFT       30
>>   #define GICH_V2_LR_GRP_MASK        0x1
>> -#define GICH_V2_LR_MAINTENANCE_IRQ (1<<19)
>> -#define GICH_V2_LR_GRP1            (1<<30)
>> -#define GICH_V2_LR_HW              (1<<31)
>> -#define GICH_V2_LR_CPUID_SHIFT     9
>> +#define GICH_V2_LR_MAINTENANCE_IRQ (1U << 19)
>> +#define GICH_V2_LR_GRP1            (1U << 30)
>> +#define GICH_V2_LR_HW              (1U << GICH_V2_LR_HW_SHIFT)
> 
> I think we want this patch to get backported as 1 << 31 is an undefined
> behavior.

I don't think this is necessary. While I agree that 1<<31 is bad (thus
the fix), there is only one user of that macro (down below in that very
file), and the result type is unsigned. If I understand this issue
correctly, the undefined behaviour is about a signed *result* type.

Cheers,
Andre.

>> +#define GICH_V2_LR_CPUID_SHIFT     10
>> +#define GICH_V2_LR_CPUID_MASK      0x7
>>   #define GICH_V2_VTR_NRLRGS         0x3f
>>     #define GICH_V2_VMCR_PRIORITY_MASK   0x1f
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 21/57] ARM: GICv2: extend LR read/write functions to cover EOI and source

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> So far our LR read/write functions do not handle the EOI bit and the
> source CPUID bits in an LR, because the current VGIC implementation does
> not use them.
> Extend the gic_lr data structure to hold these bits of information as
> well, packing it on the way to avoid it to grow.

Not sure if it matter that much as you will always allocate gic_lr on 
the stack.

> Then extract and assemble those bits from/to an LR.
> 
> This allows the new VGIC to use this information.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - new patch
> 
>   xen/arch/arm/gic-v2.c     | 7 +++++++
>   xen/include/asm-arm/gic.h | 8 +++++---
>   2 files changed, 12 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 031be920cc..c5ec0d4d35 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -470,6 +470,9 @@ static void gicv2_read_lr(int lr, struct gic_lr *lr_reg)
>       lr_reg->state     = (lrv >> GICH_V2_LR_STATE_SHIFT) & GICH_V2_LR_STATE_MASK;
>       lr_reg->hw_status = (lrv >> GICH_V2_LR_HW_SHIFT) & GICH_V2_LR_HW_MASK;
>       lr_reg->grp       = (lrv >> GICH_V2_LR_GRP_SHIFT) & GICH_V2_LR_GRP_MASK;
> +    lr_reg->eoi       = !!(lrv & GICH_V2_LR_MAINTENANCE_IRQ);

If you turn eoi to bool eoi:1, you can drop !!.

> +    if ( lr_reg->virq < NR_GIC_SGI )
> +        lr_reg->source = (lrv >> GICH_V2_LR_CPUID_SHIFT) & GICH_V2_LR_CPUID_MASK;
>   }
>   
>   static void gicv2_write_lr(int lr, const struct gic_lr *lr_reg)
> @@ -485,6 +488,10 @@ static void gicv2_write_lr(int lr, const struct gic_lr *lr_reg)
>             ((uint32_t)(lr_reg->hw_status & GICH_V2_LR_HW_MASK)
>                                          << GICH_V2_LR_HW_SHIFT)  |
>             ((uint32_t)(lr_reg->grp & GICH_V2_LR_GRP_MASK) << GICH_V2_LR_GRP_SHIFT) );
> +    if ( lr_reg->eoi )
> +        lrv |= GICH_V2_LR_MAINTENANCE_IRQ;
> +    if ( lr_reg->virq < NR_GIC_SGI )
> +        lrv |= (uint32_t)lr_reg->source << GICH_V2_LR_CPUID_SHIFT;
>   
>       writel_gich(lrv, GICH_LR + lr * 4);
>   }
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 8fab458d7f..89a07ae6b4 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -223,9 +223,11 @@ struct gic_lr {
>      /* Virtual IRQ */
>      uint32_t virq;
>      uint8_t priority;
> -   uint8_t state;
> -   uint8_t hw_status;
> -   uint8_t grp;
> +   uint8_t source;
> +   uint8_t state:2;
> +   uint8_t hw_status:1;
> +   uint8_t grp:1;
> +   uint8_t eoi:1;

I would much prefer to introduce an union with specific information for 
the physical interrupt (i.e GICH_LR.HW = 1) in one side and purely 
virtual in the other side.

That would also help to understand the purpose of each field without 
looking at the write_lr callback details.

Also, please mention that source is GICv2 only.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 20/57] ARM: GICv2: fix GICH_V2_LR definitions

[Julien Grall]

Hi,

On 06/03/18 15:58, Andre Przywara wrote:
> Hi,
> 
> On 06/03/18 15:46, Julien Grall wrote:
>> Hi Andre,
>>
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> The bit definition for the CPUID mask in the GICv2 LR register was
>>> wrong, fortunately the current implementation does not use that bit.
>>> Fix it up (it's starting at bit 10, not bit 9) and clean up some
>>> nearby definitions on the way.
>>> This will be used by the new VGIC shortly.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>
>> Reviewed-by: Julien Grall <julien.grall@arm.com>
> 
> Thanks!
> 
>>
>>> ---
>>> Changelog RFC ... v1:
>>> - new patch
>>>    xen/arch/arm/gic-v2.c | 9 +++++----
>>>    1 file changed, 5 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>>> index 618dd94120..031be920cc 100644
>>> --- a/xen/arch/arm/gic-v2.c
>>> +++ b/xen/arch/arm/gic-v2.c
>>> @@ -57,10 +57,11 @@
>>>    #define GICH_V2_LR_HW_MASK         0x1
>>>    #define GICH_V2_LR_GRP_SHIFT       30
>>>    #define GICH_V2_LR_GRP_MASK        0x1
>>> -#define GICH_V2_LR_MAINTENANCE_IRQ (1<<19)
>>> -#define GICH_V2_LR_GRP1            (1<<30)
>>> -#define GICH_V2_LR_HW              (1<<31)
>>> -#define GICH_V2_LR_CPUID_SHIFT     9
>>> +#define GICH_V2_LR_MAINTENANCE_IRQ (1U << 19)
>>> +#define GICH_V2_LR_GRP1            (1U << 30)
>>> +#define GICH_V2_LR_HW              (1U << GICH_V2_LR_HW_SHIFT)
>>
>> I think we want this patch to get backported as 1 << 31 is an undefined
>> behavior.
> 
> I don't think this is necessary. While I agree that 1<<31 is bad (thus
> the fix), there is only one user of that macro (down below in that very
> file), and the result type is unsigned. If I understand this issue
> correctly, the undefined behaviour is about a signed *result* type.

Here a small example showing the problem:

#include <stdio.h>
#include <stdint.h>

int foo(uint32_t foo)
{
     foo &= 1 << 31;
}

int main(void)
{
     foo(-1);

     return 0;
}

42sh> ./a.out
test.c:6:14: runtime error: left shift of 1 by 31 places cannot be 
represented in type 'int'

I also remember ubsan complaining on some of the GICv2 shift. I still 
need to send those patches :/.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 22/57] ARM: GIC: Allow tweaking the active and pending state of an IRQ

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> When playing around with hardware mapped, level triggered virtual IRQs,
> there is the need to explicitly set the active or pending state of an
> interrupt at some point.
> To prepare the GIC for that, we introduce a set_active_state() and a
> set_pending_state() function to let the VGIC manipulate the state of
> an associated hardware IRQ.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - use struct irq_desc* in the interface (instead of just the IRQ number)
> - add set_pending_state() (needed later)
> 
>   xen/arch/arm/gic-v2.c     | 32 ++++++++++++++++++++++++++++++++
>   xen/arch/arm/gic-v3.c     | 28 ++++++++++++++++++++++++++++
>   xen/arch/arm/gic.c        | 10 ++++++++++
>   xen/include/asm-arm/gic.h | 10 ++++++++++
>   4 files changed, 80 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index c5ec0d4d35..74169b5633 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -241,6 +241,36 @@ static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)
>       writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
>   }
>   
> +static void gicv2_set_active_state(struct irq_desc *irqd, bool active)
> +{
> +    ASSERT(spin_is_locked(&irqd->lock));
> +
> +    if ( active )
> +    {
> +        set_bit(_IRQ_INPROGRESS, &irqd->status); > +        gicv2_poke_irq(irqd, GICD_ISACTIVER);
> +    }
> +    else
> +    {

Why don't you clear _IRQ_INPROGRESS here?

> +        gicv2_poke_irq(irqd, GICD_ICACTIVER);
> +    }
> +}
> +
> +static void gicv2_set_pending_state(struct irq_desc *irqd, bool pending)
> +{
> +    ASSERT(spin_is_locked(&irqd->lock));
> +
> +    if ( pending )
> +    {
> +        set_bit(_IRQ_INPROGRESS, &irqd->status);

Why do you set _IRQ_INPROGRESS here? If you set the hardware interrupt 
pending, it will fire and then set this bit for you.

> +        gicv2_poke_irq(irqd, GICD_ISPENDR);
> +    }
> +    else
> +    {
> +        gicv2_poke_irq(irqd, GICD_ICPENDR);
> +    }
> +}
> +
>   static void gicv2_set_irq_type(struct irq_desc *desc, unsigned int type)
>   {
>       uint32_t cfg, actual, edgebit;
> @@ -1251,6 +1281,8 @@ const static struct gic_hw_operations gicv2_ops = {
>       .eoi_irq             = gicv2_eoi_irq,
>       .deactivate_irq      = gicv2_dir_irq,
>       .read_irq            = gicv2_read_irq,
> +    .set_active_state    = gicv2_set_active_state,
> +    .set_pending_state   = gicv2_set_pending_state,
>       .set_irq_type        = gicv2_set_irq_type,
>       .set_irq_priority    = gicv2_set_irq_priority,
>       .send_SGI            = gicv2_send_SGI,
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 44dfba2267..c96469f09d 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c

My remark are the same as GICv2.

> @@ -477,6 +477,32 @@ static unsigned int gicv3_read_irq(void)
>       return irq;
>   }
>   
> +static void gicv3_set_active_state(struct irq_desc *irqd, bool active)
> +{
> +    ASSERT(spin_is_locked(&irqd->lock));
> +
> +    if ( active )
> +    {
> +        set_bit(_IRQ_INPROGRESS, &irqd->status);
> +        gicv3_poke_irq(irqd, GICD_ISACTIVER, false);
> +    }
> +    else
> +        gicv3_poke_irq(irqd, GICD_ICACTIVER, false);
> +}
> +
> +static void gicv3_set_pending_state(struct irq_desc *irqd, bool pending)
> +{
> +    ASSERT(spin_is_locked(&irqd->lock));
> +
> +    if ( pending )
> +    {
> +        set_bit(_IRQ_INPROGRESS, &irqd->status);
> +        gicv3_poke_irq(irqd, GICD_ISPENDR, false);
> +    }
> +    else
> +        gicv3_poke_irq(irqd, GICD_ICPENDR, false);
> +}
> +
>   static inline uint64_t gicv3_mpidr_to_affinity(int cpu)
>   {
>        uint64_t mpidr = cpu_logical_map(cpu);
> @@ -1723,6 +1749,8 @@ static const struct gic_hw_operations gicv3_ops = {
>       .eoi_irq             = gicv3_eoi_irq,
>       .deactivate_irq      = gicv3_dir_irq,
>       .read_irq            = gicv3_read_irq,
> +    .set_active_state    = gicv3_set_active_state,
> +    .set_pending_state   = gicv3_set_pending_state,
>       .set_irq_type        = gicv3_set_irq_type,
>       .set_irq_priority    = gicv3_set_irq_priority,
>       .send_SGI            = gicv3_send_sgi,
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index 968e46fabb..f1329a630a 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -87,6 +87,16 @@ void gic_restore_state(struct vcpu *v)
>       isb();
>   }
>   
> +void gic_set_active_state(struct irq_desc *irqd, bool state)
> +{
> +    gic_hw_ops->set_active_state(irqd, state);
> +}
> +
> +void gic_set_pending_state(struct irq_desc *irqd, bool state)
> +{
> +    gic_hw_ops->set_pending_state(irqd, state);
> +}

This possibly can be static inline in gic.h?

> +
>   /* desc->irq needs to be disabled before calling this function */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type)
>   {
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 89a07ae6b4..46dcb0fe7c 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -239,6 +239,12 @@ DECLARE_PER_CPU(uint64_t, lr_mask);
>   
>   extern enum gic_version gic_hw_version(void);
>   
> +/* Force the active state of an IRQ. */
> +void gic_set_active_state(struct irq_desc *irqd, bool state);
> +
> +/* Force the pending state of an IRQ. */
> +void gic_set_pending_state(struct irq_desc *irqd, bool state);
> +
>   /* Program the IRQ type into the GIC */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
>   
> @@ -348,6 +354,10 @@ struct gic_hw_operations {
>       void (*deactivate_irq)(struct irq_desc *irqd);
>       /* Read IRQ id and Ack */
>       unsigned int (*read_irq)(void);
> +    /* Force the active state of an IRQ by accessing the distributor */
> +    void (*set_active_state)(struct irq_desc *irqd, bool state);
> +    /* Force the pending state of an IRQ by accessing the distributor */
> +    void (*set_pending_state)(struct irq_desc *irqd, bool state);

Based on the discussion we had today, could expand the comment saying 
that anyone who wants to use those 2 helpers need to carefully think 
before calling them?

>       /* Set IRQ type */
>       void (*set_irq_type)(struct irq_desc *desc, unsigned int type);
>       /* Set IRQ priority */ >

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 23/57] ARM: GIC: allow reading pending state of a hardware IRQ

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> To synchronize level triggered interrupts which are mapped into a guest,
> we need to update the virtual line level at certain points in time.
> For a hardware mapped interrupt the GIC is the only place where we can
> easily access this information.
> Implement a gic_hw_operations member to return the pending state of a
> particular interrupt. Due to hardware limitations this only works for
> private interrupts of the current CPU, so there is no CPU field in the
> prototype.

Would it be worth adding a comment on top of the callback?

> This adds gicv2/3_peek_irq() helper functions, to read a bit in a bitmap
> spread over several MMIO registers.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - add gicv2/3_peek_irq() helpers
> - use struct irq_desc* in the interface (instead of just the IRQ number)
> 
>   xen/arch/arm/gic-v2.c     | 15 +++++++++++++++
>   xen/arch/arm/gic-v3.c     | 19 +++++++++++++++++++
>   xen/arch/arm/gic.c        |  5 +++++
>   xen/include/asm-arm/gic.h |  5 +++++
>   4 files changed, 44 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
> index 74169b5633..48352f6499 100644
> --- a/xen/arch/arm/gic-v2.c
> +++ b/xen/arch/arm/gic-v2.c
> @@ -241,6 +241,15 @@ static void gicv2_poke_irq(struct irq_desc *irqd, uint32_t offset)
>       writel_gicd(1U << (irqd->irq % 32), offset + (irqd->irq / 32) * 4);
>   }
>   
> +static bool gicv2_peek_irq(struct irq_desc *irqd, uint32_t offset)
> +{
> +    uint32_t reg;
> +
> +    reg = readl_gicd(offset + (irqd->irq / 32) * 4) & (1U << (irqd->irq % 32));
> +
> +    return reg;
> +}
> +
>   static void gicv2_set_active_state(struct irq_desc *irqd, bool active)
>   {
>       ASSERT(spin_is_locked(&irqd->lock));
> @@ -549,6 +558,11 @@ static unsigned int gicv2_read_apr(int apr_reg)
>      return readl_gich(GICH_APR);
>   }
>   
> +static bool gicv2_read_pending_state(struct irq_desc *irqd)
> +{
> +    return gicv2_peek_irq(irqd, GICD_ISPENDR);
> +}
> +
>   static void gicv2_irq_enable(struct irq_desc *desc)
>   {
>       unsigned long flags;
> @@ -1294,6 +1308,7 @@ const static struct gic_hw_operations gicv2_ops = {
>       .write_lr            = gicv2_write_lr,
>       .read_vmcr_priority  = gicv2_read_vmcr_priority,
>       .read_apr            = gicv2_read_apr,
> +    .read_pending_state  = gicv2_read_pending_state,
>       .make_hwdom_dt_node  = gicv2_make_hwdom_dt_node,
>       .make_hwdom_madt     = gicv2_make_hwdom_madt,
>       .get_hwdom_extra_madt_size = gicv2_get_hwdom_extra_madt_size,
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index c96469f09d..3e75d06c3b 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -444,6 +444,19 @@ static void gicv3_poke_irq(struct irq_desc *irqd, u32 offset, bool wait_for_rwp)
>           gicv3_wait_for_rwp(irqd->irq);
>   }
>   
> +static bool gicv3_peek_irq(struct irq_desc *irqd, u32 offset)
> +{
> +    void __iomem *base;
> +    unsigned int irq = irqd->irq;
> +
> +    if ( irq >= NR_GIC_LOCAL_IRQS)
> +        base = GICD + (irq / 32) * 4;
> +    else
> +        base = GICD_RDIST_SGI_BASE;
> +
> +    return !!(readl(base + offset) & (1U << (irq % 32)));
> +}
> +
>   static void gicv3_unmask_irq(struct irq_desc *irqd)
>   {
>       gicv3_poke_irq(irqd, GICD_ISENABLER, false);
> @@ -1094,6 +1107,11 @@ static unsigned int gicv3_read_apr(int apr_reg)
>       }
>   }
>   
> +static bool gicv3_read_pending_state(struct irq_desc *irqd)
> +{
> +    return gicv3_peek_irq(irqd, GICD_ISPENDR);
> +}
> +
>   static void gicv3_irq_enable(struct irq_desc *desc)
>   {
>       unsigned long flags;
> @@ -1762,6 +1780,7 @@ static const struct gic_hw_operations gicv3_ops = {
>       .write_lr            = gicv3_write_lr,
>       .read_vmcr_priority  = gicv3_read_vmcr_priority,
>       .read_apr            = gicv3_read_apr,
> +    .read_pending_state  = gicv3_read_pending_state,
>       .secondary_init      = gicv3_secondary_cpu_init,
>       .make_hwdom_dt_node  = gicv3_make_hwdom_dt_node,
>       .make_hwdom_madt     = gicv3_make_hwdom_madt,
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index f1329a630a..67c3b4d86d 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -116,6 +116,11 @@ static void gic_set_irq_priority(struct irq_desc *desc, unsigned int priority)
>       gic_hw_ops->set_irq_priority(desc, priority);
>   }
>   
> +bool gic_read_pending_state(struct irq_desc *irqd)
> +{
> +    return gic_hw_ops->read_pending_state(irqd);
> +}

How about providing a static inline helper in gic.h?

> +
>   /* Program the GIC to route an interrupt to the host (i.e. Xen)
>    * - needs to be called with desc.lock held
>    */
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 46dcb0fe7c..03667f00cf 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -248,6 +248,9 @@ void gic_set_pending_state(struct irq_desc *irqd, bool state);
>   /* Program the IRQ type into the GIC */
>   void gic_set_irq_type(struct irq_desc *desc, unsigned int type);
>   
> +/* Read the pending state of an interrupt from the distributor. */
> +bool gic_read_pending_state(struct irq_desc *irqd);
> +
>   /* Program the GIC to route an interrupt */
>   extern void gic_route_irq_to_xen(struct irq_desc *desc, unsigned int priority);
>   extern int gic_route_irq_to_guest(struct domain *, unsigned int virq,
> @@ -382,6 +385,8 @@ struct gic_hw_operations {
>       unsigned int (*read_vmcr_priority)(void);
>       /* Read APRn register */
>       unsigned int (*read_apr)(int apr_reg);
> +    /* Query the pending state of an interrupt at the distributor level. */
> +    bool (*read_pending_state)(struct irq_desc *irqd);
>       /* Secondary CPU init */
>       int (*secondary_init)(void);
>       /* Create GIC node for the hardware domain */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 24/57] ARM: timer: Handle level triggered IRQs correctly

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The ARM Generic Timer uses a level-sensitive interrupt semantic. We
> easily catch when the line goes high, as this triggers the hardware IRQ.
> However we have to sync the state of the interrupt condition at certain
> points to catch when the line goes low and we can remove the vtimer vIRQ
> from the vGIC (and the LR).
> The VGIC in Xen so far only implemented edge triggered vIRQs, really, so
> we need to add new functionality to re-sample the interrupt state.

As requested on the previous series, can you please explain in the 
commit message/code why sampling is necessary?

Also, do we need to do that for the emulated physical timer?

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - extend comments
> - don't read CNTV_CVAL_EL0
> - use symbolic names for constants
> 
>   xen/arch/arm/time.c     | 36 ++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/traps.c    |  6 ++++++
>   xen/include/xen/timer.h |  2 ++
>   3 files changed, 44 insertions(+)
> 
> diff --git a/xen/arch/arm/time.c b/xen/arch/arm/time.c
> index c11fcfeadd..c0ae781ecd 100644
> --- a/xen/arch/arm/time.c
> +++ b/xen/arch/arm/time.c
> @@ -263,6 +263,42 @@ static void vtimer_interrupt(int irq, void *dev_id, struct cpu_user_regs *regs)
>       vgic_inject_irq(current->domain, current, current->arch.virt_timer.irq, true);
>   }
>   
> +/**
> + * vtimer_sync() - update the state of the virtual timer after a guest run
> + * @vcpu: The VCPU to sync the arch timer state
> + *
> + * After returning from a guest, update the state of the virtual interrupt
> + * line, to model the level triggered interrupt correctly.
> + * If the guest has handled a timer interrupt, the virtual interrupt line
> + * needs to be lowered explicitly. vgic_inject_irq() takes care of that.
> + */
> +void vtimer_sync(struct vcpu *vcpu)
> +{
> +    struct vtimer *vtimer = &vcpu->arch.virt_timer;
> +    uint32_t vtimer_ctl = READ_SYSREG32(CNTV_CTL_EL0);
> +    bool level;
> +
> +    /*
> +     * Technically the mask should include the CNTx_CTL_MASK bit here,
> +     * to catch if the timer interrupt is masked. However Xen always masks
> +     * the timer upon entering the hypervisor, leaving it up to the guest
> +     * to un-mask it. So we would always read a "low" level, despite the
> +     * condition being actually "high".
> +     * Ignoring the mask bit solves this (for now).
> +     * Another possible check would be to compare the value of CNTVCT_EL0
> +     * against vtimer->cval and derive the interrupt state from that.
> +     */
> +    vtimer_ctl &= (CNTx_CTL_ENABLE | CNTx_CTL_PENDING);
> +    level = (vtimer_ctl == (CNTx_CTL_ENABLE | CNTx_CTL_PENDING));
> +
> +     /*
> +      * TODO: The proper fix for this is to make vtimer vIRQ hardware mapped,
> +      * but this requires reworking the arch timer to implement this.
> +      */

The comment seems to be offset by one space.

> +
> +    vgic_inject_irq(vcpu->domain, vcpu, vtimer->irq, level);
> +}
> +
>   /*
>    * Arch timer interrupt really ought to be level triggered, since the
>    * design of the timer/comparator mechanism is based around that
> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
> index 7411bff7a7..0713723bb7 100644
> --- a/xen/arch/arm/traps.c
> +++ b/xen/arch/arm/traps.c
> @@ -2024,6 +2024,12 @@ static void enter_hypervisor_head(struct cpu_user_regs *regs)
>           if ( current->arch.hcr_el2 & HCR_VA )
>               current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
>   
> +        /*
> +         * We need to update the state of our emulated devices using level
> +         * triggered interrupts before syncing back the VGIC state.
> +         */
> +        vtimer_sync(current);

Am I wondering if it would be worth to #ifdef the code so it is only 
used for the new vGIC? After all, this will be a nop for it, right?

> +
>           vgic_sync_from_lrs(current);
>       }
>   }
> diff --git a/xen/include/xen/timer.h b/xen/include/xen/timer.h
> index 4513260b0d..eddbbf3903 100644
> --- a/xen/include/xen/timer.h
> +++ b/xen/include/xen/timer.h
> @@ -94,6 +94,8 @@ DECLARE_PER_CPU(s_time_t, timer_deadline);
>   /* Arch-defined function to reprogram timer hardware for new deadline. */
>   int reprogram_timer(s_time_t timeout);
>   
> +void vtimer_sync(struct vcpu *vcpu);
> +
>   /* Calculate the aligned first tick time for a given periodic timer. */
>   s_time_t align_timer(s_time_t firsttick, uint64_t period);
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 24/57] ARM: timer: Handle level triggered IRQs correctly

[Julien Grall]

On 06/03/18 17:15, Julien Grall wrote:
> On 05/03/18 16:03, Andre Przywara wrote:
>>   /*
>>    * Arch timer interrupt really ought to be level triggered, since the
>>    * design of the timer/comparator mechanism is based around that
>> diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
>> index 7411bff7a7..0713723bb7 100644
>> --- a/xen/arch/arm/traps.c
>> +++ b/xen/arch/arm/traps.c
>> @@ -2024,6 +2024,12 @@ static void enter_hypervisor_head(struct 
>> cpu_user_regs *regs)
>>           if ( current->arch.hcr_el2 & HCR_VA )
>>               current->arch.hcr_el2 = READ_SYSREG(HCR_EL2);
>> +        /*
>> +         * We need to update the state of our emulated devices using 
>> level
>> +         * triggered interrupts before syncing back the VGIC state.
>> +         */
>> +        vtimer_sync(current);
> 
> Am I wondering if it would be worth to #ifdef the code so it is only 
> used for the new vGIC? After all, this will be a nop for it, right?

Also, I would like to see a TODO in the code and the cover letter about 
optimizing this (see RFC patch #17):

"I am a bit worry about re-sampling the virtual interrupt state at every 
traps. It might be worth thinking to do the re-sample when syncing the 
LRs (as you do for HW level interrupt in patch #25). Probably once we 
get the new vGIC merged."

It is not a deal breaker right now. But I don't want see to be forgotten 
once after we merge it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 26/57] ARM: vPL011: Use the VGIC's level triggered IRQs handling if available

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The emulated ARM SBSA UART is using level triggered IRQ semantics,
> however the current VGIC can only handle edge triggered IRQs, really.
> Disable the existing workaround for this problem in case we have the
> new VGIC in place, which can properly handle level triggered IRQs.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - no changes
> 
>   xen/arch/arm/vpl011.c | 4 ++++
>   1 file changed, 4 insertions(+)
> 
> diff --git a/xen/arch/arm/vpl011.c b/xen/arch/arm/vpl011.c
> index 5dcf4bec18..197ece8873 100644
> --- a/xen/arch/arm/vpl011.c
> +++ b/xen/arch/arm/vpl011.c
> @@ -54,6 +54,7 @@ static void vpl011_update_interrupt_status(struct domain *d)
>        */
>       ASSERT(spin_is_locked(&vpl011->lock));
>   
> +#ifndef CONFIG_NEW_VGIC
>       /*
>        * TODO: PL011 interrupts are level triggered which means
>        * that interrupt needs to be set/clear instead of being
> @@ -71,6 +72,9 @@ static void vpl011_update_interrupt_status(struct domain *d)
>           vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, true);
>   
>       vpl011->shadow_uartmis = uartmis;
> +#else
> +    vgic_inject_irq(d, NULL, GUEST_VPL011_SPI, !!uartmis);

The !! is not necessary as the parameter is bool.


> +#endif
>   }
>   
>   static uint8_t vpl011_read_data(struct domain *d)
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 27/57] ARM: new VGIC: Add data structure definitions

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> Add a new header file for the new and improved GIC implementation.
> The big change is that we now have a struct vgic_irq per IRQ instead
> of spreading all the information over various bitmaps in the ranks.
> 
> We include this new header conditionally from within the old header
> file for the time being to avoid touching all the users.
> 
> This is based on Linux commit b18b57787f5e, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - rename header file to new_vgic.h
> - drop unneeded data structures (vgic_its, vgic_v<x>_cpu_if)
> - reorder members in vgic_irq to avoid padding
> - move flags members into bool bitfields
> - drop prototypes
> - use unsigned and uint<x>_t data types
> - keep arch_vcpu member name as "vgic"
> 
>   xen/include/asm-arm/new_vgic.h | 198 +++++++++++++++++++++++++++++++++++++++++
>   xen/include/asm-arm/vgic.h     |   6 ++
>   2 files changed, 204 insertions(+)
>   create mode 100644 xen/include/asm-arm/new_vgic.h
> 
> diff --git a/xen/include/asm-arm/new_vgic.h b/xen/include/asm-arm/new_vgic.h
> new file mode 100644
> index 0000000000..54be5aa3eb
> --- /dev/null
> +++ b/xen/include/asm-arm/new_vgic.h
> @@ -0,0 +1,198 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#ifndef __ASM_ARM_NEW_VGIC_H
> +#define __ASM_ARM_NEW_VGIC_H
> +
> +#include <asm/atomic.h>
> +#include <asm/mmio.h>
> +#include <xen/list.h>
> +#include <xen/mm.h>
> +#include <xen/spinlock.h>
> +
> +#define VGIC_V3_MAX_CPUS        255
> +#define VGIC_V2_MAX_CPUS        8
> +#define VGIC_NR_SGIS            16
> +#define VGIC_NR_PPIS            16
> +#define VGIC_NR_PRIVATE_IRQS    (VGIC_NR_SGIS + VGIC_NR_PPIS)
> +#define VGIC_MAX_PRIVATE        (VGIC_NR_PRIVATE_IRQS - 1)
> +#define VGIC_MAX_SPI            1019
> +#define VGIC_MAX_RESERVED       1023
> +#define VGIC_MIN_LPI            8192
> +
> +#define irq_is_ppi(irq) ((irq) >= VGIC_NR_SGIS && (irq) < VGIC_NR_PRIVATE_IRQS)
> +#define irq_is_spi(irq) ((irq) >= VGIC_NR_PRIVATE_IRQS && \
> +                         (irq) <= VGIC_MAX_SPI)
> +
> +enum vgic_type {
> +    VGIC_V2,        /* Good ol' GICv2 */
> +    VGIC_V3,        /* New fancy GICv3 */
> +};
> +
> +#define VGIC_V2_MAX_LRS         (1 << 6)
> +#define VGIC_V3_MAX_LRS         16
> +#define VGIC_V3_LR_INDEX(lr)    (VGIC_V3_MAX_LRS - 1 - lr)
> +
> +enum vgic_irq_config {
> +    VGIC_CONFIG_EDGE = 0,

Again, I don't think it is necessary to set to 0 here as IIRC an enum 
always start at 0 if not specified.

Also, you might want to add a comment that this enum can only contain 
two values because of the way you store it (see bool config:1).

> +    VGIC_CONFIG_LEVEL
> +};
> +
> +struct vgic_irq {
> +    struct list_head ap_list;
> +
> +    struct vcpu *vcpu;          /*
> +                                 * SGIs and PPIs: The VCPU
> +                                 * SPIs and LPIs: The VCPU whose ap_list
> +                                 * this is queued on.
> +                                 */
> +
> +    struct vcpu *target_vcpu;   /*
> +                                 * The VCPU that this interrupt should
> +                                 * be sent to, as a result of the
> +                                 * targets reg (v2) or the affinity reg (v3).
> +                                 */
> +
> +    spinlock_t irq_lock;        /* Protects the content of the struct */
> +    uint32_t intid;             /* Guest visible INTID */
> +    atomic_t refcount;          /* Used for LPIs */
> +    uint32_t hwintid;           /* HW INTID number */
> +    union
> +    {
> +        struct {
> +            uint8_t targets;    /* GICv2 target VCPUs mask */
> +            uint8_t source;     /* GICv2 SGIs only */
> +        };
> +        uint32_t mpidr;         /* GICv3 target VCPU */
> +    };
> +    uint8_t priority;
> +    bool line_level:1;          /* Level only */
> +    bool pending_latch:1;       /*
> +                                 * The pending latch state used to
> +                                 * calculate the pending state for both
> +                                 * level and edge triggered IRQs.
> +                                 */
> +    bool active:1;              /* not used for LPIs */
> +    bool enabled:1;
> +    bool hw:1;                  /* Tied to HW IRQ */
> +    bool config:1;              /* Level or edge */
> +    struct list_head lpi_list;  /* Used to link all LPIs together */
> +};
> +
> +struct vgic_register_region;

Again, do we really need the forward declaration here?

> +
> +enum iodev_type {
> +    IODEV_DIST,
> +    IODEV_REDIST,
> +};
> +
> +struct vgic_io_device {
> +    gfn_t base_fn;
> +    struct vcpu *redist_vcpu;
> +    const struct vgic_register_region *regions;
> +    enum iodev_type iodev_type;
> +    unsigned int nr_regions;
> +};
> +
> +struct vgic_dist {
> +    bool                ready;
> +    bool                initialized;
> +
> +    /* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
> +    uint32_t            version;
> +
> +    /* Do injected MSIs require an additional device ID? */
> +    bool                msis_require_devid;
> +
> +    unsigned int        nr_spis;
> +
> +    /* base addresses in guest physical address space: */
> +    paddr_t             vgic_dist_base;     /* distributor */
> +    union
> +    {
> +        /* either a GICv2 CPU interface */
> +        paddr_t         vgic_cpu_base;
> +        /* or a number of GICv3 redistributor regions */
> +        struct
> +        {
> +            paddr_t     vgic_redist_base;
> +            paddr_t     vgic_redist_free_offset;
> +        };
> +    };
> +
> +    /* distributor enabled */
> +    bool                enabled;
> +
> +    struct vgic_irq     *spis;
> +    unsigned long       *allocated_irqs; /* bitmap of IRQs allocated */
> +
> +    struct vgic_io_device   dist_iodev;
> +
> +    bool                has_its;
> +
> +    /*
> +     * Contains the attributes and gpa of the LPI configuration table.
> +     * Since we report GICR_TYPER.CommonLPIAff as 0b00, we can share
> +     * one address across all redistributors.
> +     * GICv3 spec: 6.1.2 "LPI Configuration tables"
> +     */
> +    uint64_t            propbaser;
> +
> +    /* Protects the lpi_list and the count value below. */
> +    spinlock_t          lpi_list_lock;
> +    struct list_head    lpi_list_head;
> +    unsigned int        lpi_list_count;
> +};
> +
> +struct vgic_cpu {
> +    struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS];
> +
> +    struct list_head ap_list_head;
> +    spinlock_t ap_list_lock;    /* Protects the ap_list */
> +
> +    unsigned int used_lrs;
> +
> +    /*
> +     * List of IRQs that this VCPU should consider because they are either
> +     * Active or Pending (hence the name; AP list), or because they recently
> +     * were one of the two and need to be migrated off this list to another
> +     * VCPU.
> +     */
> +
> +    /*
> +     * Members below are used with GICv3 emulation only and represent
> +     * parts of the redistributor.
> +     */
> +    struct vgic_io_device   rd_iodev;
> +    struct vgic_io_device   sgi_iodev;
> +
> +    /* Contains the attributes and gpa of the LPI pending tables. */
> +    uint64_t pendbaser;
> +
> +    bool lpis_enabled;
> +
> +    /* Cache guest priority bits */
> +    uint32_t num_pri_bits;
> +
> +    /* Cache guest interrupt ID bits */
> +    uint32_t num_id_bits;
> +};
> +
> +#define vgic_initialized(k) ((k)->arch.vgic.initialized)
> +#define vgic_ready(k)       ((k)->arch.vgic.ready)
> +#define vgic_valid_spi(k, i)    (((i) >= VGIC_NR_PRIVATE_IRQS) && \
> +            ((i) < (k)->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS))

What does k stands for? Shouldn't it be 'd' for domain?

> +
> +#endif /* __ASM_ARM_NEW_VGIC_H */

Missing emacs magic.

> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index 84d82e6eb3..b28b8f8df7 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -18,6 +18,10 @@
>   #ifndef __ASM_ARM_VGIC_H__
>   #define __ASM_ARM_VGIC_H__
>   
> +#ifdef CONFIG_NEW_VGIC
> +#include <asm/new_vgic.h>
> +#else
> +
>   #include <xen/bitops.h>
>   #include <xen/radix-tree.h>
>   #include <xen/rbtree.h>
> @@ -299,6 +303,8 @@ extern bool vgic_to_sgi(struct vcpu *v, register_t sgir,
>                           const struct sgi_target *target);
>   extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new, unsigned int irq);
>   
> +#endif /* !CONFIG_NEW_VGIC */
> +
>   /*** Common VGIC functions used by Xen arch code ****/
>   
>   /*
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 27/57] ARM: new VGIC: Add data structure definitions

[Andre Przywara]

Hi,

On 06/03/18 17:46, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> Add a new header file for the new and improved GIC implementation.
>> The big change is that we now have a struct vgic_irq per IRQ instead
>> of spreading all the information over various bitmaps in the ranks.
>>
>> We include this new header conditionally from within the old header
>> file for the time being to avoid touching all the users.
>>
>> This is based on Linux commit b18b57787f5e, written by Christoffer Dall.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - rename header file to new_vgic.h
>> - drop unneeded data structures (vgic_its, vgic_v<x>_cpu_if)
>> - reorder members in vgic_irq to avoid padding
>> - move flags members into bool bitfields
>> - drop prototypes
>> - use unsigned and uint<x>_t data types
>> - keep arch_vcpu member name as "vgic"
>>
>>   xen/include/asm-arm/new_vgic.h | 198
>> +++++++++++++++++++++++++++++++++++++++++
>>   xen/include/asm-arm/vgic.h     |   6 ++
>>   2 files changed, 204 insertions(+)
>>   create mode 100644 xen/include/asm-arm/new_vgic.h
>>
>> diff --git a/xen/include/asm-arm/new_vgic.h
>> b/xen/include/asm-arm/new_vgic.h
>> new file mode 100644
>> index 0000000000..54be5aa3eb
>> --- /dev/null
>> +++ b/xen/include/asm-arm/new_vgic.h
>> @@ -0,0 +1,198 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +#ifndef __ASM_ARM_NEW_VGIC_H
>> +#define __ASM_ARM_NEW_VGIC_H
>> +
>> +#include <asm/atomic.h>
>> +#include <asm/mmio.h>
>> +#include <xen/list.h>
>> +#include <xen/mm.h>
>> +#include <xen/spinlock.h>
>> +
>> +#define VGIC_V3_MAX_CPUS        255
>> +#define VGIC_V2_MAX_CPUS        8
>> +#define VGIC_NR_SGIS            16
>> +#define VGIC_NR_PPIS            16
>> +#define VGIC_NR_PRIVATE_IRQS    (VGIC_NR_SGIS + VGIC_NR_PPIS)
>> +#define VGIC_MAX_PRIVATE        (VGIC_NR_PRIVATE_IRQS - 1)
>> +#define VGIC_MAX_SPI            1019
>> +#define VGIC_MAX_RESERVED       1023
>> +#define VGIC_MIN_LPI            8192
>> +
>> +#define irq_is_ppi(irq) ((irq) >= VGIC_NR_SGIS && (irq) <
>> VGIC_NR_PRIVATE_IRQS)
>> +#define irq_is_spi(irq) ((irq) >= VGIC_NR_PRIVATE_IRQS && \
>> +                         (irq) <= VGIC_MAX_SPI)
>> +
>> +enum vgic_type {
>> +    VGIC_V2,        /* Good ol' GICv2 */
>> +    VGIC_V3,        /* New fancy GICv3 */
>> +};
>> +
>> +#define VGIC_V2_MAX_LRS         (1 << 6)
>> +#define VGIC_V3_MAX_LRS         16
>> +#define VGIC_V3_LR_INDEX(lr)    (VGIC_V3_MAX_LRS - 1 - lr)
>> +
>> +enum vgic_irq_config {
>> +    VGIC_CONFIG_EDGE = 0,
> 
> Again, I don't think it is necessary to set to 0 here as IIRC an enum
> always start at 0 if not specified.

But I consider this more readable, especially since this is a bool now.
So it's immediately obvious that EDGE is 0.
Shall I replace this with:
#define VGIC_CONFIG_EDGE false
#define VGIC_CONFIG_LEVEL true
now? Or using 0 and 1?

The enum vgic_irq_config type is now not used anywhere.

> Also, you might want to add a comment that this enum can only contain
> two values because of the way you store it (see bool config:1).
> 
>> +    VGIC_CONFIG_LEVEL
>> +};
>> +
>> +struct vgic_irq {
>> +    struct list_head ap_list;
>> +
>> +    struct vcpu *vcpu;          /*
>> +                                 * SGIs and PPIs: The VCPU
>> +                                 * SPIs and LPIs: The VCPU whose ap_list
>> +                                 * this is queued on.
>> +                                 */
>> +
>> +    struct vcpu *target_vcpu;   /*
>> +                                 * The VCPU that this interrupt should
>> +                                 * be sent to, as a result of the
>> +                                 * targets reg (v2) or the affinity
>> reg (v3).
>> +                                 */
>> +
>> +    spinlock_t irq_lock;        /* Protects the content of the struct */
>> +    uint32_t intid;             /* Guest visible INTID */
>> +    atomic_t refcount;          /* Used for LPIs */
>> +    uint32_t hwintid;           /* HW INTID number */
>> +    union
>> +    {
>> +        struct {
>> +            uint8_t targets;    /* GICv2 target VCPUs mask */
>> +            uint8_t source;     /* GICv2 SGIs only */
>> +        };
>> +        uint32_t mpidr;         /* GICv3 target VCPU */
>> +    };
>> +    uint8_t priority;
>> +    bool line_level:1;          /* Level only */
>> +    bool pending_latch:1;       /*
>> +                                 * The pending latch state used to
>> +                                 * calculate the pending state for both
>> +                                 * level and edge triggered IRQs.
>> +                                 */
>> +    bool active:1;              /* not used for LPIs */
>> +    bool enabled:1;
>> +    bool hw:1;                  /* Tied to HW IRQ */
>> +    bool config:1;              /* Level or edge */
>> +    struct list_head lpi_list;  /* Used to link all LPIs together */
>> +};
>> +
>> +struct vgic_register_region;
> 
> Again, do we really need the forward declaration here?

No, we don't. I think I removed this, but it got reintroduced over a rebase.

>> +
>> +enum iodev_type {
>> +    IODEV_DIST,
>> +    IODEV_REDIST,
>> +};
>> +
>> +struct vgic_io_device {
>> +    gfn_t base_fn;
>> +    struct vcpu *redist_vcpu;
>> +    const struct vgic_register_region *regions;
>> +    enum iodev_type iodev_type;
>> +    unsigned int nr_regions;
>> +};
>> +
>> +struct vgic_dist {
>> +    bool                ready;
>> +    bool                initialized;
>> +
>> +    /* vGIC model the kernel emulates for the guest (GICv2 or GICv3) */
>> +    uint32_t            version;
>> +
>> +    /* Do injected MSIs require an additional device ID? */
>> +    bool                msis_require_devid;
>> +
>> +    unsigned int        nr_spis;
>> +
>> +    /* base addresses in guest physical address space: */
>> +    paddr_t             vgic_dist_base;     /* distributor */
>> +    union
>> +    {
>> +        /* either a GICv2 CPU interface */
>> +        paddr_t         vgic_cpu_base;
>> +        /* or a number of GICv3 redistributor regions */
>> +        struct
>> +        {
>> +            paddr_t     vgic_redist_base;
>> +            paddr_t     vgic_redist_free_offset;
>> +        };
>> +    };
>> +
>> +    /* distributor enabled */
>> +    bool                enabled;
>> +
>> +    struct vgic_irq     *spis;
>> +    unsigned long       *allocated_irqs; /* bitmap of IRQs allocated */
>> +
>> +    struct vgic_io_device   dist_iodev;
>> +
>> +    bool                has_its;
>> +
>> +    /*
>> +     * Contains the attributes and gpa of the LPI configuration table.
>> +     * Since we report GICR_TYPER.CommonLPIAff as 0b00, we can share
>> +     * one address across all redistributors.
>> +     * GICv3 spec: 6.1.2 "LPI Configuration tables"
>> +     */
>> +    uint64_t            propbaser;
>> +
>> +    /* Protects the lpi_list and the count value below. */
>> +    spinlock_t          lpi_list_lock;
>> +    struct list_head    lpi_list_head;
>> +    unsigned int        lpi_list_count;
>> +};
>> +
>> +struct vgic_cpu {
>> +    struct vgic_irq private_irqs[VGIC_NR_PRIVATE_IRQS];
>> +
>> +    struct list_head ap_list_head;
>> +    spinlock_t ap_list_lock;    /* Protects the ap_list */
>> +
>> +    unsigned int used_lrs;
>> +
>> +    /*
>> +     * List of IRQs that this VCPU should consider because they are
>> either
>> +     * Active or Pending (hence the name; AP list), or because they
>> recently
>> +     * were one of the two and need to be migrated off this list to
>> another
>> +     * VCPU.
>> +     */
>> +
>> +    /*
>> +     * Members below are used with GICv3 emulation only and represent
>> +     * parts of the redistributor.
>> +     */
>> +    struct vgic_io_device   rd_iodev;
>> +    struct vgic_io_device   sgi_iodev;
>> +
>> +    /* Contains the attributes and gpa of the LPI pending tables. */
>> +    uint64_t pendbaser;
>> +
>> +    bool lpis_enabled;
>> +
>> +    /* Cache guest priority bits */
>> +    uint32_t num_pri_bits;
>> +
>> +    /* Cache guest interrupt ID bits */
>> +    uint32_t num_id_bits;
>> +};
>> +
>> +#define vgic_initialized(k) ((k)->arch.vgic.initialized)
>> +#define vgic_ready(k)       ((k)->arch.vgic.ready)
>> +#define vgic_valid_spi(k, i)    (((i) >= VGIC_NR_PRIVATE_IRQS) && \
>> +            ((i) < (k)->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS))
> 
> What does k stands for? Shouldn't it be 'd' for domain?

Indeed, good catch.
K is the new D ;-)
But as we don't use those macros anyway, I will just remove them.

Cheers,
Andre.

>> +
>> +#endif /* __ASM_ARM_NEW_VGIC_H */
> 
> Missing emacs magic.
> 
>> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
>> index 84d82e6eb3..b28b8f8df7 100644
>> --- a/xen/include/asm-arm/vgic.h
>> +++ b/xen/include/asm-arm/vgic.h
>> @@ -18,6 +18,10 @@
>>   #ifndef __ASM_ARM_VGIC_H__
>>   #define __ASM_ARM_VGIC_H__
>>   +#ifdef CONFIG_NEW_VGIC
>> +#include <asm/new_vgic.h>
>> +#else
>> +
>>   #include <xen/bitops.h>
>>   #include <xen/radix-tree.h>
>>   #include <xen/rbtree.h>
>> @@ -299,6 +303,8 @@ extern bool vgic_to_sgi(struct vcpu *v, register_t
>> sgir,
>>                           const struct sgi_target *target);
>>   extern bool vgic_migrate_irq(struct vcpu *old, struct vcpu *new,
>> unsigned int irq);
>>   +#endif /* !CONFIG_NEW_VGIC */
>> +
>>   /*** Common VGIC functions used by Xen arch code ****/
>>     /*
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 28/57] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The new VGIC implementation centers around a struct vgic_irq instance
> per virtual IRQ.
> Provide a function to retrieve the right instance for a given IRQ
> number and (in case of private interrupts) the right VCPU.
> This also includes the corresponding put function, which does nothing
> for private interrupts and SPIs, but handles the ref-counting for LPIs.
> 
> This is based on Linux commit 64a959d66e47, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - add kernel-doc comments to exported functions
> - adapt to previous changes (new_vgic.h, arch_vcpu member name)
> - use ASSERT_UNREACHABLE
> 
>   xen/arch/arm/vgic/vgic.c | 124 +++++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h |  41 ++++++++++++++++
>   2 files changed, 165 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic.c
>   create mode 100644 xen/arch/arm/vgic/vgic.h
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> new file mode 100644
> index 0000000000..ace30f78d0
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -0,0 +1,124 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <asm/bug.h>
> +#include <asm/new_vgic.h>
> +#include <xen/sched.h>
> +
> +#include "vgic.h"
> +
> +/*
> + * Iterate over the VM's list of mapped LPIs to find the one with a
> + * matching interrupt ID and return a reference to the IRQ structure.
> + */
> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +    struct vgic_irq *irq = NULL;
> +
> +    spin_lock(&dist->lpi_list_lock);
> +
> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )

I am still not a big fan of the list solution. Strictly speaking nobody 
is populating that list and likely going to be too slow in Xen case (I 
am thinking for the hardware domain). So I think I would prefer to see 
the LPI related code disappear for this cut. This could easily be added 
back as they are standalone.

> +    {
> +        if ( irq->intid != intid )
> +            continue;
> +
> +        /*
> +         * This increases the refcount, the caller is expected to
> +         * call vgic_put_irq() later once it's finished with the IRQ.
> +         */
> +        vgic_get_irq_kref(irq);
> +        goto out_unlock;
> +    }
> +    irq = NULL;
> +
> +out_unlock:
> +    spin_unlock(&dist->lpi_list_lock);
> +
> +    return irq;
> +}
> +
> +/**
> + * vgic_get_irq() - obtain a reference to a virtual IRQ
> + * @d:        The domain the virtual IRQ belongs to.
> + * @vcpu:     For private IRQs (SGIs, PPIs) the virtual CPU this IRQ
> + *            is associated with. Will be ignored for SPIs and LPIs.
> + * @intid:    The virtual IRQ number.
> + *
> + * This looks up the virtual interrupt ID to get the corresponding
> + * struct vgic_irq. It also increases the refcount, so any caller is expected
> + * to call vgic_put_irq() once it's finished with this IRQ.
> + *
> + * Return: The pointer to the requested struct vgic_irq.
> + */
> +struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
> +                              u32 intid)
> +{
> +    /* SGIs and PPIs */
> +    if ( intid <= VGIC_MAX_PRIVATE )
> +        return &vcpu->arch.vgic.private_irqs[intid];
> +
> +    /* SPIs */
> +    if ( intid <= VGIC_MAX_SPI )
> +        return &d->arch.vgic.spis[intid - VGIC_NR_PRIVATE_IRQS];
> +
> +    /* LPIs */
> +    if ( intid >= VGIC_MIN_LPI )
> +        return vgic_get_lpi(d, intid);
> +
> +    ASSERT_UNREACHABLE();

You need to keep the return NULL here as ASSERT_UNREACHABLE() is turned 
to a nop on non-debug build.

My point with ASSERT is to avoid potential flaw in the code and a way to 
flood Xen console with the WARN in non-debug build.

> +}
> +
> +/**
> + * vgic_put_irq() - drop the reference to a virtual IRQ
> + * @d:        The domain the virtual IRQ belongs to.
> + * @irq:      The pointer to struct vgic_irq, as obtained from vgic_get_irq().
> + *
> + * This drops the reference to a virtual IRQ. It decreases the refcount
> + * of the pointer, so dynamic IRQs can be freed when no longer needed.
> + * This should always be called after a vgic_get_irq(), though the reference
> + * can be deliberately held for longer periods, if needed.
> + */
> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +
> +    if ( irq->intid < VGIC_MIN_LPI )
> +        return;
> +
> +    spin_lock(&dist->lpi_list_lock);
> +    if ( !atomic_dec_and_test(&irq->refcount) )
> +    {
> +        spin_unlock(&dist->lpi_list_lock);
> +        return;
> +    };
> +
> +    list_del(&irq->lpi_list);
> +    dist->lpi_list_count--;
> +    spin_unlock(&dist->lpi_list_lock);
> +
> +    xfree(irq);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> new file mode 100644
> index 0000000000..a3befd386b
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -0,0 +1,41 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#ifndef __XEN_ARM_VGIC_VGIC_H__
> +#define __XEN_ARM_VGIC_VGIC_H__
> +
> +struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
> +                              u32 intid);
> +void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
> +
> +static inline void vgic_get_irq_kref(struct vgic_irq *irq)
> +{
> +    if ( irq->intid < VGIC_MIN_LPI )
> +        return;
> +
> +    atomic_inc(&irq->refcount);
> +}
> +
> +#endif
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 27/57] ARM: new VGIC: Add data structure definitions

[Julien Grall]

Hi Andre,

On 03/06/2018 06:01 PM, Andre Przywara wrote:
> On 06/03/18 17:46, Julien Grall wrote:
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> Add a new header file for the new and improved GIC implementation.
>>> The big change is that we now have a struct vgic_irq per IRQ instead
>>> of spreading all the information over various bitmaps in the ranks.
>>>
>>> We include this new header conditionally from within the old header
>>> file for the time being to avoid touching all the users.
>>>
>>> This is based on Linux commit b18b57787f5e, written by Christoffer Dall.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>> Changelog RFC ... v1:
>>> - rename header file to new_vgic.h
>>> - drop unneeded data structures (vgic_its, vgic_v<x>_cpu_if)
>>> - reorder members in vgic_irq to avoid padding
>>> - move flags members into bool bitfields
>>> - drop prototypes
>>> - use unsigned and uint<x>_t data types
>>> - keep arch_vcpu member name as "vgic"
>>>
>>>    xen/include/asm-arm/new_vgic.h | 198
>>> +++++++++++++++++++++++++++++++++++++++++
>>>    xen/include/asm-arm/vgic.h     |   6 ++
>>>    2 files changed, 204 insertions(+)
>>>    create mode 100644 xen/include/asm-arm/new_vgic.h
>>>
>>> diff --git a/xen/include/asm-arm/new_vgic.h
>>> b/xen/include/asm-arm/new_vgic.h
>>> new file mode 100644
>>> index 0000000000..54be5aa3eb
>>> --- /dev/null
>>> +++ b/xen/include/asm-arm/new_vgic.h
>>> @@ -0,0 +1,198 @@
>>> +/*
>>> + * Copyright (C) 2015, 2016 ARM Ltd.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +#ifndef __ASM_ARM_NEW_VGIC_H
>>> +#define __ASM_ARM_NEW_VGIC_H
>>> +
>>> +#include <asm/atomic.h>
>>> +#include <asm/mmio.h>
>>> +#include <xen/list.h>
>>> +#include <xen/mm.h>
>>> +#include <xen/spinlock.h>
>>> +
>>> +#define VGIC_V3_MAX_CPUS        255
>>> +#define VGIC_V2_MAX_CPUS        8
>>> +#define VGIC_NR_SGIS            16
>>> +#define VGIC_NR_PPIS            16
>>> +#define VGIC_NR_PRIVATE_IRQS    (VGIC_NR_SGIS + VGIC_NR_PPIS)
>>> +#define VGIC_MAX_PRIVATE        (VGIC_NR_PRIVATE_IRQS - 1)
>>> +#define VGIC_MAX_SPI            1019
>>> +#define VGIC_MAX_RESERVED       1023
>>> +#define VGIC_MIN_LPI            8192
>>> +
>>> +#define irq_is_ppi(irq) ((irq) >= VGIC_NR_SGIS && (irq) <
>>> VGIC_NR_PRIVATE_IRQS)
>>> +#define irq_is_spi(irq) ((irq) >= VGIC_NR_PRIVATE_IRQS && \
>>> +                         (irq) <= VGIC_MAX_SPI)
>>> +
>>> +enum vgic_type {
>>> +    VGIC_V2,        /* Good ol' GICv2 */
>>> +    VGIC_V3,        /* New fancy GICv3 */
>>> +};
>>> +
>>> +#define VGIC_V2_MAX_LRS         (1 << 6)
>>> +#define VGIC_V3_MAX_LRS         16
>>> +#define VGIC_V3_LR_INDEX(lr)    (VGIC_V3_MAX_LRS - 1 - lr)
>>> +
>>> +enum vgic_irq_config {
>>> +    VGIC_CONFIG_EDGE = 0,
>>
>> Again, I don't think it is necessary to set to 0 here as IIRC an enum
>> always start at 0 if not specified.
> 
> But I consider this more readable, especially since this is a bool now.
> So it's immediately obvious that EDGE is 0.

Fair enough, I probably know to much the C spec :).

> Shall I replace this with:
> #define VGIC_CONFIG_EDGE false
> #define VGIC_CONFIG_LEVEL true
> now? Or using 0 and 1?
> 
> The enum vgic_irq_config type is now not used anywhere.

I don't mind to keep the enum with a comment above. If you switch to 
define, then I would prefer to use false/true.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 29/57] ARM: new VGIC: Implement virtual IRQ injection

[Julien Grall]

Hi Andre,

Overall this patch looks good. Few comments below.

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> +/*
> + * Only valid injection if changing level for level-triggered IRQs or for a
> + * rising edge.
> + */
> +static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
> +{
> +    if ( irq->config == VGIC_CONFIG_LEVEL )
> +        return irq->line_level != level;
> +
> +    return level;

TBH, I would have preferred to keep the switch here. It was much clearer 
the second case is for edge. I would be ok with the if providing comment 
explaining "return level" is for edge.

> +}
> +
> +/**
> + * vgic_queue_irq_unlock() - Queue an IRQ to a VCPU, to be injected to a guest.
> + * @d:        The domain the virtual IRQ belongs to.
> + * @irq:      A pointer to the vgic_irq of the virtual IRQ, with the lock held.
> + * @flags:    The flags used when having grabbed the IRQ lock.
> + *
> + * Check whether an IRQ needs to (and can) be queued to a VCPU's ap list.
> + * Do the queuing if necessary, taking the right locks in the right order.
> + *
> + * Needs to be entered with the IRQ lock already held, but will return
> + * with all locks dropped.
> + *
> + * Returns: True when the IRQ was queued, false otherwise.

The function is now returning void. It sounds like a left-over from the 
previous version?

> + */
> +void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
> +                           unsigned long flags)
> +{

[...]

> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index a3befd386b..3430955d9f 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -17,9 +17,19 @@
>   #ifndef __XEN_ARM_VGIC_VGIC_H__
>   #define __XEN_ARM_VGIC_VGIC_H__
>   
> +static inline bool irq_is_pending(struct vgic_irq *irq)
> +{
> +    if ( irq->config == VGIC_CONFIG_EDGE )
> +        return irq->pending_latch;
> +    else
> +        return irq->pending_latch || irq->line_level;
> +}
> +
>   struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>                                 u32 intid);
>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
> +void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
> +               unsigned long flags);

The indentation looks wrong here.

>   
>   static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>   {
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 30/57] ARM: new VGIC: Add IRQ sorting

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> Adds the sorting function to cover the case where you have more IRQs
> to consider than you have LRs. We consider their priorities.
> This pulls in Linux' list_sort.c, which is a merge sort implementation
> for linked lists. Apart from adding a full featured license header and
> adjusting the #include file, nothing has been changed in this code.
> 
> This is based on Linux commit 8e4447457965, written by Christoffer Dall.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Again, you need to CC "THE REST" maintainers for this code. It would 
also make sense to have a separate patch for adding list_sort.c.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 29/57] ARM: new VGIC: Implement virtual IRQ injection

[Andre Przywara]

Hi,

On 07/03/18 11:02, Julien Grall wrote:
> Hi Andre,
> 
> Overall this patch looks good. Few comments below.
> 
> On 03/05/2018 04:03 PM, Andre Przywara wrote:
>> +/*
>> + * Only valid injection if changing level for level-triggered IRQs or
>> for a
>> + * rising edge.
>> + */
>> +static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
>> +{
>> +    if ( irq->config == VGIC_CONFIG_LEVEL )
>> +        return irq->line_level != level;
>> +
>> +    return level;
> 
> TBH, I would have preferred to keep the switch here. It was much clearer
> the second case is for edge. I would be ok with the if providing comment
> explaining "return level" is for edge.

I see what you mean, and actually had it first this way, but:

vgic/vgic.c:250:14: error: switch condition has boolean value
[-Werror=switch-bool]
     switch ( irq->config )

I will add comments to document both cases.

>> +}
>> +
>> +/**
>> + * vgic_queue_irq_unlock() - Queue an IRQ to a VCPU, to be injected
>> to a guest.
>> + * @d:        The domain the virtual IRQ belongs to.
>> + * @irq:      A pointer to the vgic_irq of the virtual IRQ, with the
>> lock held.
>> + * @flags:    The flags used when having grabbed the IRQ lock.
>> + *
>> + * Check whether an IRQ needs to (and can) be queued to a VCPU's ap
>> list.
>> + * Do the queuing if necessary, taking the right locks in the right
>> order.
>> + *
>> + * Needs to be entered with the IRQ lock already held, but will return
>> + * with all locks dropped.
>> + *
>> + * Returns: True when the IRQ was queued, false otherwise.
> 
> The function is now returning void. It sounds like a left-over from the
> previous version?

True, thanks for catching this.

>> + */
>> +void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>> +                           unsigned long flags)
>> +{
> 
> [...]
> 
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index a3befd386b..3430955d9f 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -17,9 +17,19 @@
>>   #ifndef __XEN_ARM_VGIC_VGIC_H__
>>   #define __XEN_ARM_VGIC_VGIC_H__
>>   +static inline bool irq_is_pending(struct vgic_irq *irq)
>> +{
>> +    if ( irq->config == VGIC_CONFIG_EDGE )
>> +        return irq->pending_latch;
>> +    else
>> +        return irq->pending_latch || irq->line_level;
>> +}
>> +
>>   struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>>                                 u32 intid);
>>   void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
>> +void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>> +               unsigned long flags);
> 
> The indentation looks wrong here.

Hah, you found the one ;-)
It's a shame we don't have checkpatch, as those things are caught with
checkpatch --strict in Linux.

Cheers,
Andre.

> 
>>     static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>>   {
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 29/57] ARM: new VGIC: Implement virtual IRQ injection

[Julien Grall]

Hi Andre,

On 03/07/2018 11:22 AM, Andre Przywara wrote:
> On 07/03/18 11:02, Julien Grall wrote:
>> Overall this patch looks good. Few comments below.
>>
>> On 03/05/2018 04:03 PM, Andre Przywara wrote:
>>> +/*
>>> + * Only valid injection if changing level for level-triggered IRQs or
>>> for a
>>> + * rising edge.
>>> + */
>>> +static bool vgic_validate_injection(struct vgic_irq *irq, bool level)
>>> +{
>>> +    if ( irq->config == VGIC_CONFIG_LEVEL )
>>> +        return irq->line_level != level;
>>> +
>>> +    return level;
>>
>> TBH, I would have preferred to keep the switch here. It was much clearer
>> the second case is for edge. I would be ok with the if providing comment
>> explaining "return level" is for edge.
> 
> I see what you mean, and actually had it first this way, but:
> 
> vgic/vgic.c:250:14: error: switch condition has boolean value
> [-Werror=switch-bool]
>       switch ( irq->config )

What is your compiler version? I am using GCC 6.3 (x86 from Debian) and 
wasn't able to get the error message in this small snippet:

#include <stdio.h>
#include <stdint.h>
#include <stdbool.h>

int foo(bool f)
{
     switch (f)
     {
     case true:
         return 1;
     case false:
         return 0;
     }
}

int main(void)
{
     foo(true);

     return 0;
}

42sh> gcc -Wall -Werror -Wswitch-bool -std=c99 test.c

This sounds a bit stupid to me to forbid switch boolean. It sometimes 
help when using define and more expressive. Sounds like, there was 
similar discussion on the kernel ML (see [1]).

> I will add comments to document both cases.

>>> + */
>>> +void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>>> +                           unsigned long flags)
>>> +{
>>
>> [...]
>>
>>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>>> index a3befd386b..3430955d9f 100644
>>> --- a/xen/arch/arm/vgic/vgic.h
>>> +++ b/xen/arch/arm/vgic/vgic.h
>>> @@ -17,9 +17,19 @@
>>>    #ifndef __XEN_ARM_VGIC_VGIC_H__
>>>    #define __XEN_ARM_VGIC_VGIC_H__
>>>    +static inline bool irq_is_pending(struct vgic_irq *irq)
>>> +{
>>> +    if ( irq->config == VGIC_CONFIG_EDGE )
>>> +        return irq->pending_latch;
>>> +    else
>>> +        return irq->pending_latch || irq->line_level;
>>> +}
>>> +
>>>    struct vgic_irq *vgic_get_irq(struct domain *d, struct vcpu *vcpu,
>>>                                  u32 intid);
>>>    void vgic_put_irq(struct domain *d, struct vgic_irq *irq);
>>> +void vgic_queue_irq_unlock(struct domain *d, struct vgic_irq *irq,
>>> +               unsigned long flags);
>>
>> The indentation looks wrong here.
> 
> Hah, you found the one ;-)
> It's a shame we don't have checkpatch, as those things are caught with
> checkpatch --strict in Linux.

AFAIK, someone is working on a checkpatch for Xen. Thought I haven't 
seen anything on xen-devel so far. Hopefully, it will appear soon.

> 
> Cheers,
> Andre.
> 
>>
>>>      static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>>>    {
>>>
>>
>> Cheers,
>>

[1] https://lkml.org/lkml/2015/5/27/941

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 31/57] ARM: new VGIC: Add IRQ sync/flush framework

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> Implement the framework for syncing IRQs between our emulation and the
> list registers, which represent the guest's view of IRQs.
> This is done in kvm_vgic_flush_hwstate and kvm_vgic_sync_hwstate, which

You probably want to update the names here.

> gets called on guest entry and exit.
> The code talking to the actual GICv2/v3 hardware is added in the
> following patches.
> 
> This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

[...]

> +/**
> + * vgic_sync_from_lrs() - Update VGIC state from hardware after a guest's run.
> + * @vcpu: the VCPU for which to transfer from the LRs to the IRQ list.
> + *
> + * Sync back the hardware VGIC state after the guest has run, into our
> + * VGIC emulation structures, It reads the LRs and updates the respective
> + * struct vgic_irq, taking level/edge into account.
> + * This is the high level function which takes care of the conditions,
> + * also bails out early if there were no interrupts queued.
> + * Was: kvm_vgic_sync_hwstate()
> + */
> +void vgic_sync_from_lrs(struct vcpu *vcpu)
> +{
> +    /* An empty ap_list_head implies used_lrs == 0 */
> +    if ( list_empty(&vcpu->arch.vgic.ap_list_head) )
> +        return;
> +
> +    vgic_fold_lr_state(vcpu);

OOI, why did you drop the "if (vgic_cpu->used_lrs)" here? It looks like 
to me you want to avoid calling that function when no LRs are been used.

> +
> +    vgic_prune_ap_list(vcpu);
> +}

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 32/57] ARM: new VGIC: Add GICv2 world switch backend

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> +void vgic_v2_fold_lr_state(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
> +    unsigned int used_lrs = vcpu->arch.vgic.used_lrs;
> +    unsigned long flags;
> +    unsigned int lr;
> +
> +    if ( !used_lrs )    /* No LRs used, so nothing to sync back here. */
> +        return;
> +
> +    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 0);
> +
> +    for ( lr = 0; lr < used_lrs; lr++ )
> +    {
> +        struct gic_lr lr_val;
> +        uint32_t intid;
> +        struct vgic_irq *irq;
> +
> +        gic_hw_ops->read_lr(lr, &lr_val);
> +
> +        /*
> +         * TODO: Possible optimization to avoid reading LRs:
> +         * Read the ELRSR to find out which of our LRs have been cleared
> +         * by the guest. We just need to know the IRQ number for those, which
> +         * we could save in an array when populating the LRs.
> +         * This trades one MMIO access (ELRSR) for possibly more than one (LRs),
> +         * but requires some more code to save the IRQ number and to handle
> +         * those finished IRQs according to the algorithm below.
> +         * We need some numbers to justify this: chances are that we don't
> +         * have many LRs in use most of the time, so we might not save much.
> +         */
> +        gic_hw_ops->clear_lr(lr);
> +
> +        intid = lr_val.virq;
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        /* Always preserve the active bit */
> +        irq->active = !!(lr_val.state & GICH_LR_ACTIVE);
> +
> +        /* Edge is the only case where we preserve the pending bit */
> +        if ( irq->config == VGIC_CONFIG_EDGE && (lr_val.state & GICH_LR_PENDING) )
> +        {
> +            irq->pending_latch = true;
> +
> +            if ( vgic_irq_is_sgi(intid) )
> +                irq->source |= (1U << lr_val.source);
> +        }

KVM is clearing pending_latch for level IRQ. Why this is not done in Xen?

> +
> +	/*
> +	 * Level-triggered mapped IRQs are special because we only
> +	 * observe rising edges as input to the VGIC.
> +	 *
> +	 * If the guest never acked the interrupt we have to sample
> +	 * the physical line and set the line level, because the
> +	 * device state could have changed or we simply need to
> +	 * process the still pending interrupt later.
> +	 *
> +	 * If this causes us to lower the level, we have to also clear
> +	 * the physical active state, since we will otherwise never be
> +	 * told when the interrupt becomes asserted again.
> +	 */

The indentation of the comment looks wrong.

> +        if ( vgic_irq_is_mapped_level(irq) && (lr_val.state & GICH_LR_PENDING) )
> +        {
> +            struct irq_desc *irqd;
> +
> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
> +
> +            irqd = irq_to_desc(irq->hwintid);
> +            irq->line_level = gic_read_pending_state(irqd);
> +
> +            if ( !irq->line_level )
> +                gic_set_active_state(irqd, false);
> +        }
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    gic_hw_ops->update_hcr_status(GICH_HCR_EN, 0);
> +    vgic_cpu->used_lrs = 0;
> +}

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 31/57] ARM: new VGIC: Add IRQ sync/flush framework

[Andre Przywara]

Hi,

On 07/03/18 11:47, Julien Grall wrote:
> Hi Andre,
> 
> On 03/05/2018 04:03 PM, Andre Przywara wrote:
>> Implement the framework for syncing IRQs between our emulation and the
>> list registers, which represent the guest's view of IRQs.
>> This is done in kvm_vgic_flush_hwstate and kvm_vgic_sync_hwstate, which
> 
> You probably want to update the names here.
> 
>> gets called on guest entry and exit.
>> The code talking to the actual GICv2/v3 hardware is added in the
>> following patches.
>>
>> This is based on Linux commit 0919e84c0fc1, written by Marc Zyngier.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> 
> [...]
> 
>> +/**
>> + * vgic_sync_from_lrs() - Update VGIC state from hardware after a
>> guest's run.
>> + * @vcpu: the VCPU for which to transfer from the LRs to the IRQ list.
>> + *
>> + * Sync back the hardware VGIC state after the guest has run, into our
>> + * VGIC emulation structures, It reads the LRs and updates the
>> respective
>> + * struct vgic_irq, taking level/edge into account.
>> + * This is the high level function which takes care of the conditions,
>> + * also bails out early if there were no interrupts queued.
>> + * Was: kvm_vgic_sync_hwstate()
>> + */
>> +void vgic_sync_from_lrs(struct vcpu *vcpu)
>> +{
>> +    /* An empty ap_list_head implies used_lrs == 0 */
>> +    if ( list_empty(&vcpu->arch.vgic.ap_list_head) )
>> +        return;
>> +
>> +    vgic_fold_lr_state(vcpu);
> 
> OOI, why did you drop the "if (vgic_cpu->used_lrs)" here? It looks like
> to me you want to avoid calling that function when no LRs are been used.

This is now done in the function itself. I found it cleaner this way. I
believe we will see the actual implementation of vgic_fold_lr_state()
move into this very file here, so it shouldn't make a difference.

Cheers,
Andre.

>> +
>> +    vgic_prune_ap_list(vcpu);
>> +}
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 32/57] ARM: new VGIC: Add GICv2 world switch backend

[Andre Przywara]

Hi,

On 07/03/18 12:10, Julien Grall wrote:
> Hi Andre,
> 
> On 03/05/2018 04:03 PM, Andre Przywara wrote:
>> +void vgic_v2_fold_lr_state(struct vcpu *vcpu)
>> +{
>> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
>> +    unsigned int used_lrs = vcpu->arch.vgic.used_lrs;
>> +    unsigned long flags;
>> +    unsigned int lr;
>> +
>> +    if ( !used_lrs )    /* No LRs used, so nothing to sync back here. */
>> +        return;
>> +
>> +    gic_hw_ops->update_hcr_status(GICH_HCR_UIE, 0);
>> +
>> +    for ( lr = 0; lr < used_lrs; lr++ )
>> +    {
>> +        struct gic_lr lr_val;
>> +        uint32_t intid;
>> +        struct vgic_irq *irq;
>> +
>> +        gic_hw_ops->read_lr(lr, &lr_val);
>> +
>> +        /*
>> +         * TODO: Possible optimization to avoid reading LRs:
>> +         * Read the ELRSR to find out which of our LRs have been cleared
>> +         * by the guest. We just need to know the IRQ number for
>> those, which
>> +         * we could save in an array when populating the LRs.
>> +         * This trades one MMIO access (ELRSR) for possibly more than
>> one (LRs),
>> +         * but requires some more code to save the IRQ number and to
>> handle
>> +         * those finished IRQs according to the algorithm below.
>> +         * We need some numbers to justify this: chances are that we
>> don't
>> +         * have many LRs in use most of the time, so we might not
>> save much.
>> +         */
>> +        gic_hw_ops->clear_lr(lr);
>> +
>> +        intid = lr_val.virq;
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        /* Always preserve the active bit */
>> +        irq->active = !!(lr_val.state & GICH_LR_ACTIVE);
>> +
>> +        /* Edge is the only case where we preserve the pending bit */
>> +        if ( irq->config == VGIC_CONFIG_EDGE && (lr_val.state &
>> GICH_LR_PENDING) )
>> +        {
>> +            irq->pending_latch = true;
>> +
>> +            if ( vgic_irq_is_sgi(intid) )
>> +                irq->source |= (1U << lr_val.source);
>> +        }
> 
> KVM is clearing pending_latch for level IRQ. Why this is not done in Xen?

Good question. I spotted this myself on Monday when adding vGICv3 support.
I checked an old branch, I accidentally removed it when merging in some
later KVM changes.
So it's already back in my tree.

Cheers,
Andre.

> 
>> +
>> +    /*
>> +     * Level-triggered mapped IRQs are special because we only
>> +     * observe rising edges as input to the VGIC.
>> +     *
>> +     * If the guest never acked the interrupt we have to sample
>> +     * the physical line and set the line level, because the
>> +     * device state could have changed or we simply need to
>> +     * process the still pending interrupt later.
>> +     *
>> +     * If this causes us to lower the level, we have to also clear
>> +     * the physical active state, since we will otherwise never be
>> +     * told when the interrupt becomes asserted again.
>> +     */
> 
> The indentation of the comment looks wrong.
> 
>> +        if ( vgic_irq_is_mapped_level(irq) && (lr_val.state &
>> GICH_LR_PENDING) )
>> +        {
>> +            struct irq_desc *irqd;
>> +
>> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
>> +
>> +            irqd = irq_to_desc(irq->hwintid);
>> +            irq->line_level = gic_read_pending_state(irqd);
>> +
>> +            if ( !irq->line_level )
>> +                gic_set_active_state(irqd, false);
>> +        }
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    gic_hw_ops->update_hcr_status(GICH_HCR_EN, 0);
>> +    vgic_cpu->used_lrs = 0;
>> +}
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 34/57] ARM: new VGIC: Add MMIO handling framework

[Julien Grall]

Hi,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> Add an MMIO handling framework to the VGIC emulation:
> Each register is described by its offset, size (or number of bits per
> IRQ, if applicable) and the read/write handler functions. We provide
> initialization macros to describe each GIC register later easily.
> 
> Separate dispatch functions for read and write accesses are connected
> to Xen's MMIO handling framework and binary-search for the responsible
> register handler based on the offset address within the region.
> 
> The register handler prototype are courtesy of Christoffer Dall.
> 
> This is based on Linux commit 4493b1c4866a, written by Marc Zyngier.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - adapt to former changes
> - whitespace fixes
> - use C99 data types
> - drop unneeded regions support (ITS, CPU i/f)
> 
>   xen/arch/arm/vgic/vgic-mmio.c | 180 ++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h |  99 +++++++++++++++++++++++
>   2 files changed, 279 insertions(+)
>   create mode 100644 xen/arch/arm/vgic/vgic-mmio.c
>   create mode 100644 xen/arch/arm/vgic/vgic-mmio.h
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> new file mode 100644
> index 0000000000..393460d25a
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -0,0 +1,180 @@
> +/*
> + * VGIC MMIO handling functions
> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <xen/bitops.h>
> +#include <xen/lib.h>
> +#include <xen/sched.h>
> +#include <asm/new_vgic.h>
> +#include <asm/byteorder.h>
> +
> +#include "vgic.h"
> +#include "vgic-mmio.h"
> +
> +unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
> +                                 paddr_t addr, unsigned int len)
> +{
> +    return 0;
> +}
> +
> +unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
> +                                 paddr_t addr, unsigned int len)
> +{
> +    return -1UL;
> +}
> +
> +void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
> +                        unsigned int len, unsigned long val)
> +{
> +    /* Ignore */
> +}
> +
> +static int match_region(const void *key, const void *elt)
> +{
> +    const unsigned int offset = (unsigned long)key;
> +    const struct vgic_register_region *region = elt;
> +
> +    if ( offset < region->reg_offset )
> +        return -1;
> +
> +    if ( offset >= region->reg_offset + region->len )
> +        return 1;
> +
> +    return 0;
> +}
> +
> +static const struct vgic_register_region *
> +vgic_find_mmio_region(const struct vgic_register_region *regions,
> +                      int nr_regions, unsigned int offset)
> +{
> +    return bsearch((void *)(uintptr_t)offset, regions, nr_regions,
> +                   sizeof(regions[0]), match_region);
> +}
> +
> +static bool check_region(const struct domain *d,
> +                         const struct vgic_register_region *region,
> +                         paddr_t addr, int len)
> +{
> +    int flags, nr_irqs = d->arch.vgic.nr_spis + VGIC_NR_PRIVATE_IRQS;

Likely both should be unsigned.

> +
> +    switch ( len )
> +    {
> +    case sizeof(uint8_t):
> +        flags = VGIC_ACCESS_8bit;
> +        break;
> +    case sizeof(uint32_t):
> +        flags = VGIC_ACCESS_32bit;
> +        break;
> +    case sizeof(uint64_t):
> +        flags = VGIC_ACCESS_64bit;
> +        break;
> +    default:
> +        return false;
> +    }
> +
> +    if ( (region->access_flags & flags) && IS_ALIGNED(addr, len) )
> +    {
> +        if ( !region->bits_per_irq )
> +            return true;
> +
> +        /* Do we access a non-allocated IRQ? */
> +        return VGIC_ADDR_TO_INTID(addr, region->bits_per_irq) < nr_irqs;
> +    }
> +
> +    return false;
> +}
> +
> +static const struct vgic_register_region *
> +vgic_get_mmio_region(struct vcpu *vcpu, struct vgic_io_device *iodev,
> +                     paddr_t addr, unsigned int len)
> +{
> +    const struct vgic_register_region *region;
> +
> +    region = vgic_find_mmio_region(iodev->regions, iodev->nr_regions,
> +                                   addr - gfn_to_gaddr(iodev->base_fn));
> +    if ( !region || !check_region(vcpu->domain, region, addr, len) )
> +        return NULL;
> +
> +    return region;
> +}
> +
> +static int dispatch_mmio_read(struct vcpu *vcpu, mmio_info_t *info,
> +                              register_t *r, void *priv)
> +{
> +    struct vgic_io_device *iodev = priv;
> +    const struct vgic_register_region *region;
> +    unsigned long data = 0;
> +    paddr_t addr = info->gpa;
> +    int len = 1U << info->dabt.size;
> +
> +    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
> +    if ( !region )
> +    {
> +        memset(r, 0, len);
> +        return 0;
> +    }
> +
> +    switch (iodev->iodev_type)
> +    {
> +    case IODEV_DIST:
> +        data = region->read(vcpu, addr, len);
> +        break;
> +    case IODEV_REDIST:
> +        data = region->read(iodev->redist_vcpu, addr, len);
> +        break;
> +    }
> +
> +    memcpy(r, &data, len);
> +
> +    return 1;
> +}
> +
> +static int dispatch_mmio_write(struct vcpu *vcpu, mmio_info_t *info,
> +                               register_t r, void *priv)
> +{
> +    struct vgic_io_device *iodev = priv;
> +    const struct vgic_register_region *region;
> +    unsigned long data = r;
> +    paddr_t addr = info->gpa;
> +    int len = 1U << info->dabt.size;
> +
> +    region = vgic_get_mmio_region(vcpu, iodev, addr, len);
> +    if ( !region )
> +        return 0;
> +
> +    switch (iodev->iodev_type)
> +    {
> +    case IODEV_DIST:
> +        region->write(vcpu, addr, len, data);
> +        break;
> +    case IODEV_REDIST:
> +        region->write(iodev->redist_vcpu, addr, len, data);
> +        break;
> +    }
> +
> +    return 1;
> +}
> +
> +struct mmio_handler_ops vgic_io_ops = {
> +    .read = dispatch_mmio_read,
> +    .write = dispatch_mmio_write,
> +};
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> new file mode 100644
> index 0000000000..9219142732
> --- /dev/null
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -0,0 +1,99 @@
> +/*
> + * Copyright (C) 2015, 2016 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
> + */
> +#ifndef __XEN_ARM_VGIC_VGIC_MMIO_H__
> +#define __XEN_ARM_VGIC_VGIC_MMIO_H__
> +
> +struct vgic_register_region {
> +    unsigned int reg_offset;
> +    unsigned int len;
> +    unsigned int bits_per_irq;
> +    unsigned int access_flags;
> +    unsigned long (*read)(struct vcpu *vcpu, paddr_t addr,
> +                          unsigned int len);
> +    void (*write)(struct vcpu *vcpu, paddr_t addr,
> +                  unsigned int len, unsigned long val);
> +};
> +
> +extern struct mmio_handler_ops vgic_io_ops;
> +
> +#define VGIC_ACCESS_8bit    1
> +#define VGIC_ACCESS_32bit   2
> +#define VGIC_ACCESS_64bit   4
> +
> +/*
> + * Generate a mask that covers the number of bytes required to address
> + * up to 1024 interrupts, each represented by <bits> bits. This assumes
> + * that <bits> is a power of two.
> + */
> +#define VGIC_ADDR_IRQ_MASK(bits) (((bits) * 1024 / 8) - 1)
> +
> +/*
> + * (addr & mask) gives us the _byte_ offset for the INT ID.
> + * We multiply this by 8 the get the _bit_ offset, then divide this by
> + * the number of bits to learn the actual INT ID.
> + * But instead of a division (which requires a "long long div" implementation),
> + * we shift by the binary logarithm of <bits>.
> + * This assumes that <bits> is a power of two.
> + */
> +#define VGIC_ADDR_TO_INTID(addr, bits)  (((addr) & VGIC_ADDR_IRQ_MASK(bits)) * \
> +                                         8 >> LOG_2(bits))
> +
> +/*
> + * Some VGIC registers store per-IRQ information, with a different number
> + * of bits per IRQ. For those registers this macro is used.
> + * The _WITH_LENGTH version instantiates registers with a fixed length
> + * and is mutually exclusive with the _PER_IRQ version.
> + */
> +#define REGISTER_DESC_WITH_BITS_PER_IRQ(off, rd, wr, bpi, acc)  \
> +    {                           \
> +        .reg_offset = off,      \
> +        .bits_per_irq = bpi,    \
> +        .len = bpi * 1024 / 8,  \
> +        .access_flags = acc,    \
> +        .read = rd,             \
> +        .write = wr,            \
> +    }

Please keep all the \ indented as the first one.

> +
> +#define REGISTER_DESC_WITH_LENGTH(off, rd, wr, length, acc)     \
> +    {                           \
> +        .reg_offset = off,      \
> +        .bits_per_irq = 0,      \
> +        .len = length,          \
> +        .access_flags = acc,    \
> +        .read = rd,             \
> +        .write = wr,            \

Ditto.

> +    }
> +
> +#define REGISTER_DESC_WITH_LENGTH_UACCESS(off, rd, wr, length, acc) \

You dropped all uaccess callback, so this one should disappear too.

> +    {                           \
> +        .reg_offset = off,      \
> +        .bits_per_irq = 0,      \
> +        .len = length,          \
> +        .access_flags = acc,    \
> +        .read = rd,             \
> +        .write = wr,            \
> +    }
> +
> +unsigned long vgic_mmio_read_raz(struct vcpu *vcpu,
> +                                 paddr_t addr, unsigned int len);
> +
> +unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
> +                                 paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
> +                        unsigned int len, unsigned long val);
> +
> +#endif
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 35/57] ARM: new VGIC: Add GICv2 MMIO handling framework

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> Create vgic-mmio-v2.c to describe GICv2 emulation specific handlers
> using the initializer macros provided by the VGIC MMIO framework.
> Provide a function to register the GICv2 distributor registers to
> the Xen MMIO framework.
> The actual handler functions are still stubs in this patch.
> 
> This is based on Linux commit fb848db39661, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 36/57] ARM: new VGIC: Add CTLR, TYPER and IIDR handlers

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> +static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
> +                                    paddr_t addr, unsigned int len,
> +                                    unsigned long val)
> +{
> +    struct vgic_dist *dist = &vcpu->domain->arch.vgic;
> +    bool enabled;
> +
> +    switch ( addr & 0x0c )      /* filter for the 4 registers handled here */
> +    {
> +    case GICD_CTLR:
> +        domain_lock(vcpu->domain);
> +        enabled = dist->enabled;
> +        dist->enabled = val & GICD_CTL_ENABLE;
> +        enabled = !enabled && dist->enabled;
> +        domain_unlock(vcpu->domain);
> +
> +        if (enabled)

if ( ... )

Also, it took me a bit of time to understand what is the behavior of 
enable here. If I am not mistake it will be true only when toggle from 
disable -> enable, right? If so, I can you please either rework the code 
to make it more obvious (maybe a new variable) or add more comment?

> +            vgic_kick_vcpus(vcpu->domain);
> +
> +        break;
> +    case GICD_TYPER:
> +    case GICD_IIDR:
> +        /* read-only, writes ignored */
> +        return;
> +    }
> +}
> +

Cheers,


-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 37/57] ARM: new VGIC: Add ENABLE registers handlers

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> As the enable register handlers are shared between the v2 and v3
> emulation, their implementation goes into vgic-mmio.c, to be easily
> referenced from the v3 emulation as well later.
> This introduces a vgic_sync_hardware_irq() function, which updates the
> physical side of a hardware mapped virtual IRQ.
> Because the existing locking order between vgic_irq->irq_lock and
> irq_desc->lock dictates so, we dropu the irq_lock and retake them in the
> proper order.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - extend and move vgic_sync_hardware_irq()
> - do proper locking sequence
> - skip already disabled/enabled IRQs
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 117 +++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
>   xen/arch/arm/vgic/vgic.c         |  38 +++++++++++++
>   xen/arch/arm/vgic/vgic.h         |   3 +
>   5 files changed, 171 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 2e015ed0b1..3dd983f885 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -80,10 +80,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_rao, vgic_mmio_write_wi, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> +        vgic_mmio_read_enable, vgic_mmio_write_senable, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> +        vgic_mmio_read_enable, vgic_mmio_write_cenable, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 284a92d288..f8f0252eff 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -39,6 +39,123 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>       /* Ignore */
>   }
>   
> +/*
> + * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the value
> + * of the enabled bit, so there is only one function for both here.
> + */
> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
> +                                    paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    uint32_t value = 0;
> +    unsigned int i;
> +
> +    /* Loop over all IRQs affected by this read */
> +    for ( i = 0; i < len * 8; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq->enabled )
> +            value |= (1U << i);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +void vgic_mmio_write_senable(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    unsigned int i;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +        unsigned long flags;
> +        irq_desc_t *desc;
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        if ( irq->enabled )            /* skip already enabled IRQs */
> +        {
> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
> +            vgic_put_irq(vcpu->domain, irq);
> +            continue;
> +        }
> +
> +        irq->enabled = true;
> +        if ( irq->hw )
> +        {
> +            /*
> +             * The irq cannot be a PPI, we only support delivery
> +             * of SPIs to guests.
> +             */
> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
> +
> +            desc = irq_to_desc(irq->hwintid);
> +        }
> +        else
> +            desc = NULL;

You could just initialize desc to NULL at the declaration time and drop 
the else part.

> +
> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> +
> +        if ( desc )
> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);

A comment explaining why desc is done outside the locking would be 
useful. This would avoid to loose time using git blame.

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
> +                 paddr_t addr, unsigned int len,
> +                 unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    unsigned int i;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq;
> +        unsigned long flags;
> +        irq_desc_t *desc;
> +
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        if ( !irq->enabled )            /* skip already disabled IRQs */
> +        {
> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
> +            vgic_put_irq(vcpu->domain, irq);
> +            continue;
> +        }
> +
> +        irq->enabled = false;
> +
> +        if ( irq->hw )
> +        {
> +            /*
> +             * The irq cannot be a PPI, we only support delivery
> +             * of SPIs to guests.
> +             */
> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
> +
> +            desc = irq_to_desc(irq->hwintid);
> +        }
> +        else
> +            desc = NULL;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        if ( desc )
> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);

Ditto.

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 621b9a281c..2ddcbbf58d 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -96,6 +96,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>                           unsigned int len, unsigned long val);
>   
> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
> +                                    paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_senable(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val);
> +
> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   #endif
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 465a95f415..5246d7c2e7 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -698,6 +698,44 @@ void vgic_kick_vcpus(struct domain *d)
>       }
>   }
>   
> +static unsigned int translate_irq_type(bool is_level)
> +{
> +    return is_level ? IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING;
> +}
> +
> +void vgic_sync_hardware_irq(struct domain *d,
> +                            irq_desc_t *desc, struct vgic_irq *irq)
> +{
> +    unsigned long flags;
> +
> +    spin_lock_irqsave(&desc->lock, flags);
> +    spin_lock(&irq->irq_lock);
> +
> +    /* Is that association actually still valid? (we entered with no locks) */

If the association is not valid, then you need to fetch the new desc. Right?

> +    if ( desc->irq == irq->hwintid )
> +    {
> +        if ( irq->enabled )
> +        {
> +            /*
> +             * We might end up from various callers, so check that the
> +             * interrrupt is disabled before trying to change the config.
> +             */
> +            if ( irq_type_set_by_domain(d) &&
> +                 test_bit(_IRQ_DISABLED, &desc->status) )
> +                gic_set_irq_type(desc, translate_irq_type(irq->config));
> +
> +            if ( irq->target_vcpu )
> +                irq_set_affinity(desc, cpumask_of(irq->target_vcpu->processor));
> +            desc->handler->enable(desc);
> +        }
> +        else
> +            desc->handler->disable(desc);
> +    }
> +
> +    spin_unlock(&irq->irq_lock);
> +    spin_unlock_irqrestore(&desc->lock, flags);
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 588bd067b7..68e205d10a 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -50,6 +50,9 @@ static inline void vgic_get_irq_kref(struct vgic_irq *irq)
>       atomic_inc(&irq->refcount);
>   }
>   
> +void vgic_sync_hardware_irq(struct domain *d,
> +                            irq_desc_t *desc, struct vgic_irq *irq);
> +
>   void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
>   void vgic_v2_set_underflow(struct vcpu *vcpu);
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 38/57] ARM: new VGIC: Add PENDING registers handlers

[Julien Grall]

Hi Andre,

On 03/05/2018 04:03 PM, Andre Przywara wrote:
> The pending register handlers are shared between the v2 and v3
> emulation, so their implementation goes into vgic-mmio.c, to be easily
> referenced from the v3 emulation as well later.
> For level triggered interrupts the real line level is unaffected by
> this write, so we keep this state separate and combine it with the
> device's level to get the actual pending state.
> Hardware mapped IRQs need some special handling, as their hardware state
> has to be coordinated with the virtual pending bit to avoid hanging
> or masked interrupts.
> 
> This is based on Linux commit 96b298000db4, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - propagate SET/CLEAR_PENDING requests to hardware
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 125 +++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
>   3 files changed, 138 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 3dd983f885..efdd73301d 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -86,10 +86,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_enable, vgic_mmio_write_cenable, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> +        vgic_mmio_read_pending, vgic_mmio_write_spending, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICPENDR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> +        vgic_mmio_read_pending, vgic_mmio_write_cpending, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index f8f0252eff..2e939d5e39 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -156,6 +156,131 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
>       }
>   }
>   
> +unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
> +                                     paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    uint32_t value = 0;
> +    unsigned int i;
> +
> +    /* Loop over all IRQs affected by this read */
> +    for ( i = 0; i < len * 8; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq_is_pending(irq) )
> +            value |= (1U << i);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +void vgic_mmio_write_spending(struct vcpu *vcpu,
> +                              paddr_t addr, unsigned int len,
> +                              unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    unsigned int i;
> +    unsigned long flags;
> +    irq_desc_t *desc;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        irq->pending_latch = true;
> +
> +        /* To observe the locking order, just take the irq_desc pointer here. */
> +        if ( irq->hw )
> +            desc = irq_to_desc(irq->hwintid);
> +        else
> +            desc = NULL;
> +
> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> +
> +        /*
> +         * When the VM sets the pending state for a HW interrupt on the virtual
> +         * distributor we set the active state on the physical distributor,
> +         * because the virtual interrupt can become active and then the guest
> +         * can deactivate it.
> +         */
> +        if ( desc )
> +        {
> +            spin_lock_irqsave(&desc->lock, flags);
> +            spin_lock(&irq->irq_lock);
> +
> +            /* Is this h/w IRQ still assigned to the virtual IRQ? */

Same remark as for the enable in patch #37. What if the h/w IRQ has changed?

I am trying to think in potential use case where a physical interrupt 
would be removed from a domain. The only one I can think is a interrupt 
routed back to the hardware domain after a guest is destroyed.

But I am not entirely convinced this would work correctly today. Mostly 
because we don't sync the vIRQ state back to the pIRQ state (e.g enable, 
pending, active) in vgic_connect_hw_irq.

This could lead to interesting issue depending on the vIRQ state. So may 
we should just enforce that a physical interrupt is routed to a domain 
for its full life. Any opinions?

> +            if ( irq->hw && desc->irq == irq->hwintid )
> +                gic_set_active_state(desc, true);
> +
> +            spin_unlock(&irq->irq_lock);
> +            spin_unlock_irqrestore(&desc->lock, flags);
> +        }
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +void vgic_mmio_write_cpending(struct vcpu *vcpu,
> +                              paddr_t addr, unsigned int len,
> +                              unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    unsigned int i;
> +    unsigned long flags;
> +    irq_desc_t *desc;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        irq->pending_latch = false;
> +
> +        /* To observe the locking order, just take the irq_desc pointer here. */
> +        if ( irq->hw )
> +            desc = irq_to_desc(irq->hwintid);
> +        else
> +            desc = NULL;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        /*
> +         * We don't want the guest to effectively mask the physical
> +         * interrupt by doing a write to SPENDR followed by a write to
> +         * CPENDR for HW interrupts, so we clear the active state on
> +         * the physical side if the virtual interrupt is not active.
> +         * This may lead to taking an additional interrupt on the
> +         * host, but that should not be a problem as the worst that
> +         * can happen is an additional vgic injection.  We also clear
> +         * the pending state to maintain proper semantics for edge HW
> +         * interrupts.
> +         */
> +        if ( desc )
> +        {
> +            spin_lock_irqsave(&desc->lock, flags);
> +            spin_lock(&irq->irq_lock);
> +
> +            /* Is this h/w IRQ still assigned to the virtual IRQ? */
> +            if ( irq->hw && desc->irq == irq->hwintid )
> +            {
> +                gic_set_pending_state(desc, false);
> +                if (!irq->active)
> +                    gic_set_active_state(desc, false);
> +            }
> +
> +            spin_unlock(&irq->irq_lock);
> +            spin_unlock_irqrestore(&desc->lock, flags);
> +        }
> +
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 2ddcbbf58d..4465f3b7e5 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -107,6 +107,17 @@ void vgic_mmio_write_cenable(struct vcpu *vcpu,
>                                paddr_t addr, unsigned int len,
>                                unsigned long val);
>   
> +unsigned long vgic_mmio_read_pending(struct vcpu *vcpu,
> +                                     paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_spending(struct vcpu *vcpu,
> +                              paddr_t addr, unsigned int len,
> +                              unsigned long val);
> +
> +void vgic_mmio_write_cpending(struct vcpu *vcpu,
> +                              paddr_t addr, unsigned int len,
> +                              unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   #endif
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 37/57] ARM: new VGIC: Add ENABLE registers handlers

[Andre Przywara]

Hi,

On 07/03/18 17:01, Julien Grall wrote:
> Hi Andre,
> 
> On 03/05/2018 04:03 PM, Andre Przywara wrote:
>> As the enable register handlers are shared between the v2 and v3
>> emulation, their implementation goes into vgic-mmio.c, to be easily
>> referenced from the v3 emulation as well later.
>> This introduces a vgic_sync_hardware_irq() function, which updates the
>> physical side of a hardware mapped virtual IRQ.
>> Because the existing locking order between vgic_irq->irq_lock and
>> irq_desc->lock dictates so, we dropu the irq_lock and retake them in the
>> proper order.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - extend and move vgic_sync_hardware_irq()
>> - do proper locking sequence
>> - skip already disabled/enabled IRQs
>>
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>>   xen/arch/arm/vgic/vgic-mmio.c    | 117
>> +++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
>>   xen/arch/arm/vgic/vgic.c         |  38 +++++++++++++
>>   xen/arch/arm/vgic/vgic.h         |   3 +
>>   5 files changed, 171 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index 2e015ed0b1..3dd983f885 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -80,10 +80,10 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_rao, vgic_mmio_write_wi, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
>> +        vgic_mmio_read_enable, vgic_mmio_write_senable, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
>> +        vgic_mmio_read_enable, vgic_mmio_write_cenable, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index 284a92d288..f8f0252eff 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -39,6 +39,123 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t
>> addr,
>>       /* Ignore */
>>   }
>>   +/*
>> + * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the
>> value
>> + * of the enabled bit, so there is only one function for both here.
>> + */
>> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
>> +                                    paddr_t addr, unsigned int len)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    uint32_t value = 0;
>> +    unsigned int i;
>> +
>> +    /* Loop over all IRQs affected by this read */
>> +    for ( i = 0; i < len * 8; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        if ( irq->enabled )
>> +            value |= (1U << i);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return value;
>> +}
>> +
>> +void vgic_mmio_write_senable(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    unsigned int i;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +        unsigned long flags;
>> +        irq_desc_t *desc;
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        if ( irq->enabled )            /* skip already enabled IRQs */
>> +        {
>> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +            vgic_put_irq(vcpu->domain, irq);
>> +            continue;
>> +        }
>> +
>> +        irq->enabled = true;
>> +        if ( irq->hw )
>> +        {
>> +            /*
>> +             * The irq cannot be a PPI, we only support delivery
>> +             * of SPIs to guests.
>> +             */
>> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
>> +
>> +            desc = irq_to_desc(irq->hwintid);
>> +        }
>> +        else
>> +            desc = NULL;
> 
> You could just initialize desc to NULL at the declaration time and drop
> the else part.

Can we rely on the initializer to be called on every loop iteration? I
wasn't sure about this and what the standard has to say about this.

>> +
>> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
>> +
>> +        if ( desc )
>> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
> 
> A comment explaining why desc is done outside the locking would be
> useful. This would avoid to loose time using git blame.
> 
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
>> +                 paddr_t addr, unsigned int len,
>> +                 unsigned long val)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    unsigned int i;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq;
>> +        unsigned long flags;
>> +        irq_desc_t *desc;
>> +
>> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        if ( !irq->enabled )            /* skip already disabled IRQs */
>> +        {
>> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +            vgic_put_irq(vcpu->domain, irq);
>> +            continue;
>> +        }
>> +
>> +        irq->enabled = false;
>> +
>> +        if ( irq->hw )
>> +        {
>> +            /*
>> +             * The irq cannot be a PPI, we only support delivery
>> +             * of SPIs to guests.
>> +             */
>> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
>> +
>> +            desc = irq_to_desc(irq->hwintid);
>> +        }
>> +        else
>> +            desc = NULL;
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +        if ( desc )
>> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
> 
> Ditto.
> 
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 621b9a281c..2ddcbbf58d 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -96,6 +96,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
>>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
>>                           unsigned int len, unsigned long val);
>>   +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
>> +                                    paddr_t addr, unsigned int len);
>> +
>> +void vgic_mmio_write_senable(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val);
>> +
>> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val);
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     #endif
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index 465a95f415..5246d7c2e7 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -698,6 +698,44 @@ void vgic_kick_vcpus(struct domain *d)
>>       }
>>   }
>>   +static unsigned int translate_irq_type(bool is_level)
>> +{
>> +    return is_level ? IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING;
>> +}
>> +
>> +void vgic_sync_hardware_irq(struct domain *d,
>> +                            irq_desc_t *desc, struct vgic_irq *irq)
>> +{
>> +    unsigned long flags;
>> +
>> +    spin_lock_irqsave(&desc->lock, flags);
>> +    spin_lock(&irq->irq_lock);
>> +
>> +    /* Is that association actually still valid? (we entered with no
>> locks) */
> 
> If the association is not valid, then you need to fetch the new desc.
> Right?

I am not so sure it's that easy. If the association changed, then the
whole reason of this call might have become invalid. So I rather bail
out here and do nothing. The check is just to prevent doing the wrong
thing, not necessarily to always do the right thing.
To be honest this whole "lock drop dance" is just to cope with the
locking order, which I consider wrong, according to my gut feeling.

This function here is called from several places, so it seems a bit
fragile to assume a way how to fix a broken association here. I can go
back and check every existing caller in this respect, but to be honest
I'd rather change the locking order, so we don't need to worry about
this. But I feel like we should do this as a fixup on top later.

Cheers,
Andre.


> 
>> +    if ( desc->irq == irq->hwintid )
>> +    {
>> +        if ( irq->enabled )
>> +        {
>> +            /*
>> +             * We might end up from various callers, so check that the
>> +             * interrrupt is disabled before trying to change the
>> config.
>> +             */
>> +            if ( irq_type_set_by_domain(d) &&
>> +                 test_bit(_IRQ_DISABLED, &desc->status) )
>> +                gic_set_irq_type(desc, translate_irq_type(irq->config));
>> +
>> +            if ( irq->target_vcpu )
>> +                irq_set_affinity(desc,
>> cpumask_of(irq->target_vcpu->processor));
>> +            desc->handler->enable(desc);
>> +        }
>> +        else
>> +            desc->handler->disable(desc);
>> +    }
>> +
>> +    spin_unlock(&irq->irq_lock);
>> +    spin_unlock_irqrestore(&desc->lock, flags);
>> +}
>> +
>>   /*
>>    * Local variables:
>>    * mode: C
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 588bd067b7..68e205d10a 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -50,6 +50,9 @@ static inline void vgic_get_irq_kref(struct vgic_irq
>> *irq)
>>       atomic_inc(&irq->refcount);
>>   }
>>   +void vgic_sync_hardware_irq(struct domain *d,
>> +                            irq_desc_t *desc, struct vgic_irq *irq);
>> +
>>   void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq,
>> int lr);
>>   void vgic_v2_set_underflow(struct vcpu *vcpu);
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 37/57] ARM: new VGIC: Add ENABLE registers handlers

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 11926 bytes --]

(sorry for the formatting)

On Wed, 7 Mar 2018, 18:23 Andre Przywara, <andre.przywara@linaro.org> wrote:

> Hi,
>
> On 07/03/18 17:01, Julien Grall wrote:
> > Hi Andre,
> >
> > On 03/05/2018 04:03 PM, Andre Przywara wrote:
> >> As the enable register handlers are shared between the v2 and v3
> >> emulation, their implementation goes into vgic-mmio.c, to be easily
> >> referenced from the v3 emulation as well later.
> >> This introduces a vgic_sync_hardware_irq() function, which updates the
> >> physical side of a hardware mapped virtual IRQ.
> >> Because the existing locking order between vgic_irq->irq_lock and
> >> irq_desc->lock dictates so, we dropu the irq_lock and retake them in the
> >> proper order.
> >>
> >> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> >> ---
> >> Changelog RFC ... v1:
> >> - extend and move vgic_sync_hardware_irq()
> >> - do proper locking sequence
> >> - skip already disabled/enabled IRQs
> >>
> >>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
> >>   xen/arch/arm/vgic/vgic-mmio.c    | 117
> >> +++++++++++++++++++++++++++++++++++++++
> >>   xen/arch/arm/vgic/vgic-mmio.h    |  11 ++++
> >>   xen/arch/arm/vgic/vgic.c         |  38 +++++++++++++
> >>   xen/arch/arm/vgic/vgic.h         |   3 +
> >>   5 files changed, 171 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
> >> b/xen/arch/arm/vgic/vgic-mmio-v2.c
> >> index 2e015ed0b1..3dd983f885 100644
> >> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> >> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> >> @@ -80,10 +80,10 @@ static const struct vgic_register_region
> >> vgic_v2_dist_registers[] = {
> >>           vgic_mmio_read_rao, vgic_mmio_write_wi, 1,
> >>           VGIC_ACCESS_32bit),
> >>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISENABLER,
> >> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> >> +        vgic_mmio_read_enable, vgic_mmio_write_senable, 1,
> >>           VGIC_ACCESS_32bit),
> >>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICENABLER,
> >> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> >> +        vgic_mmio_read_enable, vgic_mmio_write_cenable, 1,
> >>           VGIC_ACCESS_32bit),
> >>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISPENDR,
> >>           vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> >> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
> >> b/xen/arch/arm/vgic/vgic-mmio.c
> >> index 284a92d288..f8f0252eff 100644
> >> --- a/xen/arch/arm/vgic/vgic-mmio.c
> >> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> >> @@ -39,6 +39,123 @@ void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t
> >> addr,
> >>       /* Ignore */
> >>   }
> >>   +/*
> >> + * Read accesses to both GICD_ICENABLER and GICD_ISENABLER return the
> >> value
> >> + * of the enabled bit, so there is only one function for both here.
> >> + */
> >> +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
> >> +                                    paddr_t addr, unsigned int len)
> >> +{
> >> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> >> +    uint32_t value = 0;
> >> +    unsigned int i;
> >> +
> >> +    /* Loop over all IRQs affected by this read */
> >> +    for ( i = 0; i < len * 8; i++ )
> >> +    {
> >> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
> >> + i);
> >> +
> >> +        if ( irq->enabled )
> >> +            value |= (1U << i);
> >> +
> >> +        vgic_put_irq(vcpu->domain, irq);
> >> +    }
> >> +
> >> +    return value;
> >> +}
> >> +
> >> +void vgic_mmio_write_senable(struct vcpu *vcpu,
> >> +                             paddr_t addr, unsigned int len,
> >> +                             unsigned long val)
> >> +{
> >> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> >> +    unsigned int i;
> >> +
> >> +    for_each_set_bit( i, &val, len * 8 )
> >> +    {
> >> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
> >> + i);
> >> +        unsigned long flags;
> >> +        irq_desc_t *desc;
> >> +
> >> +        spin_lock_irqsave(&irq->irq_lock, flags);
> >> +
> >> +        if ( irq->enabled )            /* skip already enabled IRQs */
> >> +        {
> >> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
> >> +            vgic_put_irq(vcpu->domain, irq);
> >> +            continue;
> >> +        }
> >> +
> >> +        irq->enabled = true;
> >> +        if ( irq->hw )
> >> +        {
> >> +            /*
> >> +             * The irq cannot be a PPI, we only support delivery
> >> +             * of SPIs to guests.
> >> +             */
> >> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
> >> +
> >> +            desc = irq_to_desc(irq->hwintid);
> >> +        }
> >> +        else
> >> +            desc = NULL;
> >
> > You could just initialize desc to NULL at the declaration time and drop
> > the else part.
>
> Can we rely on the initializer to be called on every loop iteration? I
> wasn't sure about this and what the standard has to say about this.
>

Every loop is a new scope. So everything declared within that scope is
initialized again. We do use that extensively in Xen.



> >> +
> >> +        vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> >> +
> >> +        if ( desc )
> >> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
> >
> > A comment explaining why desc is done outside the locking would be
> > useful. This would avoid to loose time using git blame.
> >
> >> +
> >> +        vgic_put_irq(vcpu->domain, irq);
> >> +    }
> >> +}
> >> +
> >> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
> >> +                 paddr_t addr, unsigned int len,
> >> +                 unsigned long val)
> >> +{
> >> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> >> +    unsigned int i;
> >> +
> >> +    for_each_set_bit( i, &val, len * 8 )
> >> +    {
> >> +        struct vgic_irq *irq;
> >> +        unsigned long flags;
> >> +        irq_desc_t *desc;
> >> +
> >> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> >> +        spin_lock_irqsave(&irq->irq_lock, flags);
> >> +
> >> +        if ( !irq->enabled )            /* skip already disabled IRQs
> */
> >> +        {
> >> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
> >> +            vgic_put_irq(vcpu->domain, irq);
> >> +            continue;
> >> +        }
> >> +
> >> +        irq->enabled = false;
> >> +
> >> +        if ( irq->hw )
> >> +        {
> >> +            /*
> >> +             * The irq cannot be a PPI, we only support delivery
> >> +             * of SPIs to guests.
> >> +             */
> >> +            ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
> >> +
> >> +            desc = irq_to_desc(irq->hwintid);
> >> +        }
> >> +        else
> >> +            desc = NULL;
> >> +
> >> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> >> +
> >> +        if ( desc )
> >> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);
> >
> > Ditto.
> >
> >> +
> >> +        vgic_put_irq(vcpu->domain, irq);
> >> +    }
> >> +}
> >> +
> >>   static int match_region(const void *key, const void *elt)
> >>   {
> >>       const unsigned int offset = (unsigned long)key;
> >> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
> >> b/xen/arch/arm/vgic/vgic-mmio.h
> >> index 621b9a281c..2ddcbbf58d 100644
> >> --- a/xen/arch/arm/vgic/vgic-mmio.h
> >> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> >> @@ -96,6 +96,17 @@ unsigned long vgic_mmio_read_rao(struct vcpu *vcpu,
> >>   void vgic_mmio_write_wi(struct vcpu *vcpu, paddr_t addr,
> >>                           unsigned int len, unsigned long val);
> >>   +unsigned long vgic_mmio_read_enable(struct vcpu *vcpu,
> >> +                                    paddr_t addr, unsigned int len);
> >> +
> >> +void vgic_mmio_write_senable(struct vcpu *vcpu,
> >> +                             paddr_t addr, unsigned int len,
> >> +                             unsigned long val);
> >> +
> >> +void vgic_mmio_write_cenable(struct vcpu *vcpu,
> >> +                             paddr_t addr, unsigned int len,
> >> +                             unsigned long val);
> >> +
> >>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
> >>     #endif
> >> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> >> index 465a95f415..5246d7c2e7 100644
> >> --- a/xen/arch/arm/vgic/vgic.c
> >> +++ b/xen/arch/arm/vgic/vgic.c
> >> @@ -698,6 +698,44 @@ void vgic_kick_vcpus(struct domain *d)
> >>       }
> >>   }
> >>   +static unsigned int translate_irq_type(bool is_level)
> >> +{
> >> +    return is_level ? IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING;
> >> +}
> >> +
> >> +void vgic_sync_hardware_irq(struct domain *d,
> >> +                            irq_desc_t *desc, struct vgic_irq *irq)
> >> +{
> >> +    unsigned long flags;
> >> +
> >> +    spin_lock_irqsave(&desc->lock, flags);
> >> +    spin_lock(&irq->irq_lock);
> >> +
> >> +    /* Is that association actually still valid? (we entered with no
> >> locks) */
> >
> > If the association is not valid, then you need to fetch the new desc.
> > Right?
>
> I am not so sure it's that easy. If the association changed, then the
> whole reason of this call might have become invalid. So I rather bail
> out here and do nothing. The check is just to prevent doing the wrong
> thing, not necessarily to always do the right thing.
> To be honest this whole "lock drop dance" is just to cope with the
> locking order, which I consider wrong, according to my gut feeling.
>

If you don't do the dance here, you would have to do in other place. I
still think taking the desc->lock first is the right thing to do as Xen
deal with physical first then it might be a virtual (so second) or handled
by a driver.



> This function here is called from several places, so it seems a bit
> fragile to assume a way how to fix a broken association here. I can go
> back and check every existing caller in this respect, but to be honest
> I'd rather change the locking order, so we don't need to worry about
> this. But I feel like we should do this as a fixup on top later.
>

See some thought in the next patch. We might be able to simplify the whole
logic by forbidding the interrupt to be removed.



> Cheers,
> Andre.
>
>
> >
> >> +    if ( desc->irq == irq->hwintid )
> >> +    {
> >> +        if ( irq->enabled )
> >> +        {
> >> +            /*
> >> +             * We might end up from various callers, so check that the
> >> +             * interrrupt is disabled before trying to change the
> >> config.
> >> +             */
> >> +            if ( irq_type_set_by_domain(d) &&
> >> +                 test_bit(_IRQ_DISABLED, &desc->status) )
> >> +                gic_set_irq_type(desc,
> translate_irq_type(irq->config));
> >> +
> >> +            if ( irq->target_vcpu )
> >> +                irq_set_affinity(desc,
> >> cpumask_of(irq->target_vcpu->processor));
> >> +            desc->handler->enable(desc);
> >> +        }
> >> +        else
> >> +            desc->handler->disable(desc);
> >> +    }
> >> +
> >> +    spin_unlock(&irq->irq_lock);
> >> +    spin_unlock_irqrestore(&desc->lock, flags);
> >> +}
> >> +
> >>   /*
> >>    * Local variables:
> >>    * mode: C
> >> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> >> index 588bd067b7..68e205d10a 100644
> >> --- a/xen/arch/arm/vgic/vgic.h
> >> +++ b/xen/arch/arm/vgic/vgic.h
> >> @@ -50,6 +50,9 @@ static inline void vgic_get_irq_kref(struct vgic_irq
> >> *irq)
> >>       atomic_inc(&irq->refcount);
> >>   }
> >>   +void vgic_sync_hardware_irq(struct domain *d,
> >> +                            irq_desc_t *desc, struct vgic_irq *irq);
> >> +
> >>   void vgic_v2_fold_lr_state(struct vcpu *vcpu);
> >>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq,
> >> int lr);
> >>   void vgic_v2_set_underflow(struct vcpu *vcpu);
> >>
> >
> > Cheers,
> >
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xenproject.org
> https://lists.xenproject.org/mailman/listinfo/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 17102 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/57] ARM: GICv3: simplify GICv3 redistributor stride handling

[Andre Przywara]

Hi,

On 06/03/18 13:49, Julien Grall wrote:
> 
> 
> On 05/03/18 17:08, Julien Grall wrote:
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> Instead of hard coding the architected redistributor stride into the
>>> code, lets use a clear #define to the two values for GICv3 and GICv4 and
>>> clarify the algorithm to determine the needed stride value.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>> Changelog RFC ... v1:
>>> - no changes
>>>
>>>   xen/arch/arm/gic-v3.c             | 18 ++++++++++--------
>>>   xen/include/asm-arm/gic_v3_defs.h |  5 +++++
>>>   2 files changed, 15 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>>> index b1f8a86409..be1787b39a 100644
>>> --- a/xen/arch/arm/gic-v3.c
>>> +++ b/xen/arch/arm/gic-v3.c
>>> @@ -690,6 +690,15 @@ static int __init gicv3_populate_rdist(void)
>>>           do {
>>>               typer = readq_relaxed(ptr + GICR_TYPER);
>>> +            /* Set the architectural redist size if not overridden
>>> by DT. */
>>> +            if ( !gicv3.rdist_stride )
>>> +            {
>>> +                if ( typer & GICR_TYPER_VLPIS )
>>
>> Is there anything in the spec promising you that *all* the
>> redistributors will support vLPIs?
> 
> Answering to myself, Marc pointed out that nothing was promising all
> redistributors will support vLPIs. This is confirmed by the section 9.7
> "Mixing GICv3 and GICv4" in "GICv3 and GICv4 Software Overview" DAI 0492B.
> 
> So I would prefer to drop that patch and move GICV*_GICR_SIZE definition
> in patch #5.

So I can surely do that, but then we have a definition of
GICV*_GICR_SIZE, but still the hardcoded values in the code.
So what about I keep this as a patch, but remove the actual code change
and just replace it with using the symbol names instead of the "SZ_64K *
2" plus comment?

Cheers,
Andre.


 I am assuming we will never expose a mix of GICv3/GICv4 on
> Xen.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 04/57] ARM: GICv3: simplify GICv3 redistributor stride handling

[Julien Grall]

On 08/03/18 12:40, Andre Przywara wrote:
> Hi,

Hi,

> 
> On 06/03/18 13:49, Julien Grall wrote:
>>
>>
>> On 05/03/18 17:08, Julien Grall wrote:
>>> On 05/03/18 16:03, Andre Przywara wrote:
>>>> Instead of hard coding the architected redistributor stride into the
>>>> code, lets use a clear #define to the two values for GICv3 and GICv4 and
>>>> clarify the algorithm to determine the needed stride value.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>> ---
>>>> Changelog RFC ... v1:
>>>> - no changes
>>>>
>>>>    xen/arch/arm/gic-v3.c             | 18 ++++++++++--------
>>>>    xen/include/asm-arm/gic_v3_defs.h |  5 +++++
>>>>    2 files changed, 15 insertions(+), 8 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>>>> index b1f8a86409..be1787b39a 100644
>>>> --- a/xen/arch/arm/gic-v3.c
>>>> +++ b/xen/arch/arm/gic-v3.c
>>>> @@ -690,6 +690,15 @@ static int __init gicv3_populate_rdist(void)
>>>>            do {
>>>>                typer = readq_relaxed(ptr + GICR_TYPER);
>>>> +            /* Set the architectural redist size if not overridden
>>>> by DT. */
>>>> +            if ( !gicv3.rdist_stride )
>>>> +            {
>>>> +                if ( typer & GICR_TYPER_VLPIS )
>>>
>>> Is there anything in the spec promising you that *all* the
>>> redistributors will support vLPIs?
>>
>> Answering to myself, Marc pointed out that nothing was promising all
>> redistributors will support vLPIs. This is confirmed by the section 9.7
>> "Mixing GICv3 and GICv4" in "GICv3 and GICv4 Software Overview" DAI 0492B.
>>
>> So I would prefer to drop that patch and move GICV*_GICR_SIZE definition
>> in patch #5.
> 
> So I can surely do that, but then we have a definition of
> GICV*_GICR_SIZE, but still the hardcoded values in the code.
> So what about I keep this as a patch, but remove the actual code change
> and just replace it with using the symbol names instead of the "SZ_64K *
> 2" plus comment?

GICV*_GICR_SIZE feels a bit weird to read. More that reading the spec 
gives the impression that it is possible to have GICv4 without vLPIS 
(see the description of GICR_TYPER.vLPIS).

So I think I would prefer to keep the hardcoding values here.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The active register handlers are shared between the v2 and v3 emulation,
> so their implementation goes into vgic-mmio.c, to be easily referenced
> from the v3 emulation as well later.
> Since activation/deactivation of an interrupt may happen entirely in the
> guest without it ever exiting, we need some extra logic to properly track
> the active state.
> For clearing the active state, we would basically have to halt the guest
> to make sure this is properly propagated into the respective VCPUs.
> This is not yet implemented in Xen.
> Fortunately this feature is mostly used to reset a just in initialised
> GIC, so chances are we are tasked to clear bits that are already zero.
> Add some simple check to avoid a pointless warning in this case.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - remove premature "proper ACTIVE" handler stub
> - avoid unnecessary warnings on NO-OP register writes
> - extend comments
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 103 +++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  11 +++++
>   3 files changed, 116 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index efdd73301d..c93455fbb2 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -92,10 +92,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_pending, vgic_mmio_write_cpending, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> +        vgic_mmio_read_active, vgic_mmio_write_sactive, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
> +        vgic_mmio_read_active, vgic_mmio_write_cactive, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 2e939d5e39..c44d67082f 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -281,6 +281,109 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
>       }
>   }
>   
> +/*
> + * The actual active bit for a virtual IRQ is held in the LR. Our shadow
> + * copy in struct vgic_irq is only synced when needed and may not be
> + * up-to-date all of the time.
> + * Returning the actual active state is quite costly (stopping all
> + * VCPUs processing any affected vIRQs), so we use a simple implementation
> + * to get the best possible answer.
> + */
> +unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
> +                                    paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    uint32_t value = 0;
> +    unsigned int i;
> +
> +    /* Loop over all IRQs affected by this read */
> +    for ( i = 0; i < len * 8; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq->active )
> +            value |= (1U << i);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +/*
> + * We don't actually support clearing the active state of an IRQ (yet).
> + * However there is a chance that most guests use this for initialization.
> + * We check whether this MMIO access would actually affect any active IRQ,
> + * and only print our warning in this case. So clearing already non-active
> + * IRQs would not be moaned about in the logs.
> + */
> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    unsigned int i;
> +    bool bail_out = false;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        /*
> +         * If we know that the IRQ is active or we can't be sure about
> +         * it (because it is currently in a CPU), log the not properly
> +         * emulated MMIO access.
> +         */
> +        if ( irq->active || irq->vcpu )
> +        {
> +            gdprintk(XENLOG_ERR,
> +                     "%pv: vGICD: IRQ%d: clearing active state not supported\n",

s/%d/%u/

> +                     vcpu, irq->intid);

gdprintk will always print the vCPU. Thought it is the current which 
might be different from vcpu (mostly in the re-dist case).

So I would use dprintk(XENLOG_G_ERR, "%pv: ..."). I would even be tempt 
to use printk(....) so we can spot potential issue on non-debug build.

> +            bail_out = true;

I admit the bailout is a bit weird here. You would only print the 
warning for the first activated IRQ and give the impression it is fine 
for the rest. So maybe you want to drop IRQ%d?

> +        }
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +        if ( bail_out )
> +            return;
> +    }
> +}
> +
> +/*
> + * We don't actually support setting the active state of an IRQ (yet).
> + * We check whether this MMIO access would actually affect any non-active IRQ,
> + * and only print our warning in this case.
> + */
> +void vgic_mmio_write_sactive(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val)

See my comments on cactive.

> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
> +    unsigned int i;
> +    bool bail_out = false;
> +
> +    for_each_set_bit( i, &val, len * 8 )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        /*
> +         * If we know that the IRQ is not active or we can't be sure about
> +         * it (because it is currently in a CPU), log the not properly
> +         * emulated MMIO access.
> +         */
> +        if ( !irq->active || irq->vcpu )
> +        {
> +            gdprintk(XENLOG_ERR,
> +                     "%pv: vGICD: IRQ%d: setting active state not supported\n",
> +                     vcpu, irq->intid);
> +            bail_out = true;
> +        }
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +        if ( bail_out )
> +            return;
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 4465f3b7e5..8604720628 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -118,6 +118,17 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
>                                 paddr_t addr, unsigned int len,
>                                 unsigned long val);
>   
> +unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
> +                                    paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val);
> +
> +void vgic_mmio_write_sactive(struct vcpu *vcpu,
> +                             paddr_t addr, unsigned int len,
> +                             unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   #endif
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 40/57] ARM: new VGIC: Add PRIORITY registers handlers

[Julien Grall]

On 05/03/18 16:03, Andre Przywara wrote:
> The priority register handlers are shared between the v2 and v3 emulation,
> so their implementation goes into vgic-mmio.c, to be easily referenced
> from the v3 emulation as well later.
> There is a corner case when we change the priority of a pending
> interrupt which we don't handle at the moment.

I don't believe it is a corner case. The spec (8.9.12 ARM IHI 0069d) says:

"Implementations must ensure that an interrupt that is pending at the 
time of the write uses either the old value or the new value and must 
ensure that the interrupt is neither lost nor handled more than once. 
The effect of the change must be visible in finite time."

So the current implementation looks compliant to the spec.

> 
> This is based on Linux commit dd238ec2b87b, written by Andre Przywara.

Using short commit ID is usually a pretty bad idea because it may not be 
uniq. For instance, a git show on my Linux tree will not be able to find it.

The full commit ID can feel a bit too long, so usually I give the short 
commit ID and the title.

But in that context, the commit message looks wrong. From my tree, it 
seems it is 055658bf48fcc6afdf90810e7e8f4e98f486c0d2.

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - use 32 bit register types
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 47 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
>   xen/arch/arm/vgic/vgic.h         |  2 ++
>   4 files changed, 57 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index c93455fbb2..29db9dec6f 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -98,7 +98,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_active, vgic_mmio_write_cactive, 1,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
> +        vgic_mmio_read_priority, vgic_mmio_write_priority, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index c44d67082f..538f08bc66 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -384,6 +384,53 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>       }
>   }
>   
> +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
> +                                      paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    unsigned int i;
> +    uint32_t val = 0;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        val |= (uint32_t)irq->priority << (i * 8);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return val;
> +}
> +
> +/*
> + * We currently don't handle changing the priority of an interrupt that
> + * is already pending on a VCPU. If there is a need for this, we would
> + * need to make this VCPU exit and re-evaluate the priorities, potentially
> + * leading to this interrupt getting presented now to the guest (if it has
> + * been masked by the priority mask before).
> + */
> +void vgic_mmio_write_priority(struct vcpu *vcpu,
> +                              paddr_t addr, unsigned int len,
> +                              unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    unsigned int i;
> +    unsigned long flags;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +        /* Narrow the priority range to what we actually support */
> +        irq->priority = (val >> (i * 8)) & GENMASK(7, 8 - VGIC_PRI_BITS);
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index 8604720628..e3f9029344 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -129,6 +129,13 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>                                paddr_t addr, unsigned int len,
>                                unsigned long val);
>   
> +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
> +                      paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_priority(struct vcpu *vcpu,
> +                  paddr_t addr, unsigned int len,
> +                  unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   #endif
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 68e205d10a..b294b04391 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -20,6 +20,8 @@
>   #define PRODUCT_ID_XEN      0x58    /* ASCII code X */
>   #define IMPLEMENTER_ARM     0x43b
>   
> +#define VGIC_PRI_BITS       5
> +
>   #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>   
>   static inline bool irq_is_pending(struct vgic_irq *irq)
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 41/57] ARM: new VGIC: Add CONFIG registers handlers

[Julien Grall]

Hi Andre,

On 05/03/18 16:03, Andre Przywara wrote:
> The config register handlers are shared between the v2 and v3 emulation,
> so their implementation goes into vgic-mmio.c, to be easily referenced
> from the v3 emulation as well later.
> 
> This is based on Linux commit 79717e4ac09c, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - use C99 data types
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
>   xen/arch/arm/vgic/vgic-mmio.c    | 54 ++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
>   3 files changed, 62 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 29db9dec6f..d19ddd3f1e 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -104,7 +104,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 2,
> +        vgic_mmio_read_config, vgic_mmio_write_config, 2,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
> diff --git a/xen/arch/arm/vgic/vgic-mmio.c b/xen/arch/arm/vgic/vgic-mmio.c
> index 538f08bc66..31f7cf706b 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.c
> +++ b/xen/arch/arm/vgic/vgic-mmio.c
> @@ -431,6 +431,60 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
>       }
>   }
>   
> +unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
> +                                    paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 2);
> +    uint32_t value = 0;
> +    int i;
> +
> +    for ( i = 0; i < len * 4; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        if ( irq->config == VGIC_CONFIG_EDGE )
> +            value |= (2U << (i * 2));
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return value;
> +}
> +
> +void vgic_mmio_write_config(struct vcpu *vcpu,
> +                            paddr_t addr, unsigned int len,
> +                            unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 2);
> +    int i;
> +    unsigned long flags;
> +
> +    for ( i = 0; i < len * 4; i++ )
> +    {
> +        struct vgic_irq *irq;
> +
> +        /*
> +         * The configuration cannot be changed for SGIs in general,
> +         * for PPIs this is IMPLEMENTATION DEFINED. The arch timer
> +         * code relies on PPIs being level triggered, so we also
> +         * make them read-only here.
> +         */
> +        if ( intid + i < VGIC_NR_PRIVATE_IRQS )
> +            continue;
> +
> +        irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        if ( test_bit(i * 2 + 1, &val) )
> +            irq->config = VGIC_CONFIG_EDGE;
> +        else
> +            irq->config = VGIC_CONFIG_LEVEL;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static int match_region(const void *key, const void *elt)
>   {
>       const unsigned int offset = (unsigned long)key;
> diff --git a/xen/arch/arm/vgic/vgic-mmio.h b/xen/arch/arm/vgic/vgic-mmio.h
> index e3f9029344..bbf0d181ae 100644
> --- a/xen/arch/arm/vgic/vgic-mmio.h
> +++ b/xen/arch/arm/vgic/vgic-mmio.h
> @@ -136,6 +136,13 @@ void vgic_mmio_write_priority(struct vcpu *vcpu,
>                     paddr_t addr, unsigned int len,
>                     unsigned long val);
>   
> +unsigned long vgic_mmio_read_config(struct vcpu *vcpu,
> +                    paddr_t addr, unsigned int len);
> +
> +void vgic_mmio_write_config(struct vcpu *vcpu,
> +                paddr_t addr, unsigned int len,
> +                unsigned long val);
> +
>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>   
>   #endif
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 42/57] ARM: new VGIC: Add TARGET registers handlers

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> The target register handlers are v2 emulation specific, so their
> implementation lives entirely in vgic-mmio-v2.c.
> We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
> set in the target mask instead of making it possibly pending on
> multiple VCPUs.
> We update the physical affinity of a hardware mapped vIRQ on the way.
> 
> This is based on Linux commit 2c234d6f1826, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - propagate affinity changes to hardware mapped IRQs
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 64 +++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 63 insertions(+), 1 deletion(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index d19ddd3f1e..01c6a7198c 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -72,6 +72,68 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>       }
>   }
>   
> +static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
> +                                           paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    uint32_t val = 0;
> +    unsigned int i;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        val |= (uint32_t)irq->targets << (i * 8);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return val;
> +}
> +
> +static void vgic_mmio_write_target(struct vcpu *vcpu,
> +                                   paddr_t addr, unsigned int len,
> +                                   unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    uint8_t cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);
> +    unsigned int i;
> +    unsigned long flags;
> +
> +    /* GICD_ITARGETSR[0-7] are read-only */
> +    if ( intid < VGIC_NR_PRIVATE_IRQS )
> +        return;
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, NULL, intid + i);
> +        struct irq_desc *desc;
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->targets = (val >> (i * 8)) & cpu_mask;
> +        if ( irq->targets )
> +        {
> +            irq->target_vcpu = vcpu->domain->vcpu[ffs(irq->targets) - 1];
> +            if ( irq->hw )
> +                desc = irq_to_desc(irq->hwintid);
> +            else
> +                desc = NULL;

If you initialized desc to NULL during the declaration, then the else is 
not necessary.

> +        }
> +        else {
> +            irq->target_vcpu = NULL;
> +            desc = NULL;
> +        }
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        if ( desc )
> +            vgic_update_hardware_irq(desc, irq);

I can't find this function in the tree.

> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
> @@ -101,7 +163,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_priority, vgic_mmio_write_priority, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
> +        vgic_mmio_read_target, vgic_mmio_write_target, 8,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
>           vgic_mmio_read_config, vgic_mmio_write_config, 2,
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 40/57] ARM: new VGIC: Add PRIORITY registers handlers

[Andre Przywara]

Hi,

On 08/03/18 15:48, Julien Grall wrote:
> 
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> The priority register handlers are shared between the v2 and v3
>> emulation,
>> so their implementation goes into vgic-mmio.c, to be easily referenced
>> from the v3 emulation as well later.
>> There is a corner case when we change the priority of a pending
>> interrupt which we don't handle at the moment.
> 
> I don't believe it is a corner case. The spec (8.9.12 ARM IHI 0069d) says:
> 
> "Implementations must ensure that an interrupt that is pending at the
> time of the write uses either the old value or the new value and must
> ensure that the interrupt is neither lost nor handled more than once.
> The effect of the change must be visible in finite time."
> 
> So the current implementation looks compliant to the spec.

Interestingly the GICv2 spec doesn't mention anything at all related to
that, at least not in the IPRIORITYR description.

Anyway, are you saying that I should just change the commit message?

>>
>> This is based on Linux commit dd238ec2b87b, written by Andre Przywara.
> 
> Using short commit ID is usually a pretty bad idea because it may not be
> uniq. For instance, a git show on my Linux tree will not be able to find
> it.

Indeed somehow the commit ID is wrong.
Linux recommends (at least) 12 digits when space is important, plus the
commit title, if possible. But this is just for the records, so I just
went with the shortened SHA. Even if that clashes in the future, the
first hit should be the one. The commit message stem should be the same
as in Linux for those commits.

Cheers,
Andre.

> The full commit ID can feel a bit too long, so usually I give the short
> commit ID and the title.
> 
> But in that context, the commit message looks wrong. From my tree, it
> seems it is 055658bf48fcc6afdf90810e7e8f4e98f486c0d2.
> 
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - use 32 bit register types
>>
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |  2 +-
>>   xen/arch/arm/vgic/vgic-mmio.c    | 47
>> ++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  7 ++++++
>>   xen/arch/arm/vgic/vgic.h         |  2 ++
>>   4 files changed, 57 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index c93455fbb2..29db9dec6f 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -98,7 +98,7 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_active, vgic_mmio_write_cactive, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
>> +        vgic_mmio_read_priority, vgic_mmio_write_priority, 8,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index c44d67082f..538f08bc66 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -384,6 +384,53 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>>       }
>>   }
>>   +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
>> +                                      paddr_t addr, unsigned int len)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
>> +    unsigned int i;
>> +    uint32_t val = 0;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        val |= (uint32_t)irq->priority << (i * 8);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return val;
>> +}
>> +
>> +/*
>> + * We currently don't handle changing the priority of an interrupt that
>> + * is already pending on a VCPU. If there is a need for this, we would
>> + * need to make this VCPU exit and re-evaluate the priorities,
>> potentially
>> + * leading to this interrupt getting presented now to the guest (if
>> it has
>> + * been masked by the priority mask before).
>> + */
>> +void vgic_mmio_write_priority(struct vcpu *vcpu,
>> +                              paddr_t addr, unsigned int len,
>> +                              unsigned long val)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
>> +    unsigned int i;
>> +    unsigned long flags;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +        /* Narrow the priority range to what we actually support */
>> +        irq->priority = (val >> (i * 8)) & GENMASK(7, 8 -
>> VGIC_PRI_BITS);
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 8604720628..e3f9029344 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -129,6 +129,13 @@ void vgic_mmio_write_sactive(struct vcpu *vcpu,
>>                                paddr_t addr, unsigned int len,
>>                                unsigned long val);
>>   +unsigned long vgic_mmio_read_priority(struct vcpu *vcpu,
>> +                      paddr_t addr, unsigned int len);
>> +
>> +void vgic_mmio_write_priority(struct vcpu *vcpu,
>> +                  paddr_t addr, unsigned int len,
>> +                  unsigned long val);
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     #endif
>> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
>> index 68e205d10a..b294b04391 100644
>> --- a/xen/arch/arm/vgic/vgic.h
>> +++ b/xen/arch/arm/vgic/vgic.h
>> @@ -20,6 +20,8 @@
>>   #define PRODUCT_ID_XEN      0x58    /* ASCII code X */
>>   #define IMPLEMENTER_ARM     0x43b
>>   +#define VGIC_PRI_BITS       5
>> +
>>   #define vgic_irq_is_sgi(intid) ((intid) < VGIC_NR_SGIS)
>>     static inline bool irq_is_pending(struct vgic_irq *irq)
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 40/57] ARM: new VGIC: Add PRIORITY registers handlers

[Julien Grall]

On 08/03/18 16:21, Andre Przywara wrote:
> Hi,
> 
> On 08/03/18 15:48, Julien Grall wrote:
>>
>>
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> The priority register handlers are shared between the v2 and v3
>>> emulation,
>>> so their implementation goes into vgic-mmio.c, to be easily referenced
>>> from the v3 emulation as well later.
>>> There is a corner case when we change the priority of a pending
>>> interrupt which we don't handle at the moment.
>>
>> I don't believe it is a corner case. The spec (8.9.12 ARM IHI 0069d) says:
>>
>> "Implementations must ensure that an interrupt that is pending at the
>> time of the write uses either the old value or the new value and must
>> ensure that the interrupt is neither lost nor handled more than once.
>> The effect of the change must be visible in finite time."
>>
>> So the current implementation looks compliant to the spec.
> 
> Interestingly the GICv2 spec doesn't mention anything at all related to
> that, at least not in the IPRIORITYR description.
> 
> Anyway, are you saying that I should just change the commit message?

Yes.

> 
>>>
>>> This is based on Linux commit dd238ec2b87b, written by Andre Przywara.
>>
>> Using short commit ID is usually a pretty bad idea because it may not be
>> uniq. For instance, a git show on my Linux tree will not be able to find
>> it.
> 
> Indeed somehow the commit ID is wrong.
> Linux recommends (at least) 12 digits when space is important, plus the
> commit title, if possible. But this is just for the records, so I just
> went with the shortened SHA.

Well, it is not only just for the records. It is also a way for us to 
know where it comes from so we can find for the 
signed-off-by/authorship. So we need to easily find the exact commit and 
therefore the e-mail addresses.

> Even if that clashes in the future, the
> first hit should be the one. The commit message stem should be the same
> as in Linux for those commits.

Why the first hit should be the one? It likely means another commit have 
similar sha1 and likely going to be the new one. And usually git show 
will tell you "ambiguous".

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 21/57] ARM: GICv2: extend LR read/write functions to cover EOI and source

[Andre Przywara]

Hi,

On 06/03/18 16:06, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> So far our LR read/write functions do not handle the EOI bit and the
>> source CPUID bits in an LR, because the current VGIC implementation does
>> not use them.
>> Extend the gic_lr data structure to hold these bits of information as
>> well, packing it on the way to avoid it to grow.
> 
> Not sure if it matter that much as you will always allocate gic_lr on
> the stack.

Well, someone was complaining the other day about memory usage, so I
figured I shouldn't grow this structure needlessly. Using the bitfields
was a very low hanging fruit.

> 
>> Then extract and assemble those bits from/to an LR.
>>
>> This allows the new VGIC to use this information.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - new patch
>>
>>   xen/arch/arm/gic-v2.c     | 7 +++++++
>>   xen/include/asm-arm/gic.h | 8 +++++---
>>   2 files changed, 12 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v2.c b/xen/arch/arm/gic-v2.c
>> index 031be920cc..c5ec0d4d35 100644
>> --- a/xen/arch/arm/gic-v2.c
>> +++ b/xen/arch/arm/gic-v2.c
>> @@ -470,6 +470,9 @@ static void gicv2_read_lr(int lr, struct gic_lr
>> *lr_reg)
>>       lr_reg->state     = (lrv >> GICH_V2_LR_STATE_SHIFT) &
>> GICH_V2_LR_STATE_MASK;
>>       lr_reg->hw_status = (lrv >> GICH_V2_LR_HW_SHIFT) &
>> GICH_V2_LR_HW_MASK;
>>       lr_reg->grp       = (lrv >> GICH_V2_LR_GRP_SHIFT) &
>> GICH_V2_LR_GRP_MASK;
>> +    lr_reg->eoi       = !!(lrv & GICH_V2_LR_MAINTENANCE_IRQ);
> 
> If you turn eoi to bool eoi:1, you can drop !!.
> 
>> +    if ( lr_reg->virq < NR_GIC_SGI )
>> +        lr_reg->source = (lrv >> GICH_V2_LR_CPUID_SHIFT) &
>> GICH_V2_LR_CPUID_MASK;
>>   }
>>     static void gicv2_write_lr(int lr, const struct gic_lr *lr_reg)
>> @@ -485,6 +488,10 @@ static void gicv2_write_lr(int lr, const struct
>> gic_lr *lr_reg)
>>             ((uint32_t)(lr_reg->hw_status & GICH_V2_LR_HW_MASK)
>>                                          << GICH_V2_LR_HW_SHIFT)  |
>>             ((uint32_t)(lr_reg->grp & GICH_V2_LR_GRP_MASK) <<
>> GICH_V2_LR_GRP_SHIFT) );
>> +    if ( lr_reg->eoi )
>> +        lrv |= GICH_V2_LR_MAINTENANCE_IRQ;
>> +    if ( lr_reg->virq < NR_GIC_SGI )
>> +        lrv |= (uint32_t)lr_reg->source << GICH_V2_LR_CPUID_SHIFT;
>>         writel_gich(lrv, GICH_LR + lr * 4);
>>   }
>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>> index 8fab458d7f..89a07ae6b4 100644
>> --- a/xen/include/asm-arm/gic.h
>> +++ b/xen/include/asm-arm/gic.h
>> @@ -223,9 +223,11 @@ struct gic_lr {
>>      /* Virtual IRQ */
>>      uint32_t virq;
>>      uint8_t priority;
>> -   uint8_t state;
>> -   uint8_t hw_status;
>> -   uint8_t grp;
>> +   uint8_t source;
>> +   uint8_t state:2;
>> +   uint8_t hw_status:1;
>> +   uint8_t grp:1;
>> +   uint8_t eoi:1;
> 
> I would much prefer to introduce an union with specific information for
> the physical interrupt (i.e GICH_LR.HW = 1) in one side and purely
> virtual in the other side.

Feel free to send a patch ;-)
For my part I just wanted to add the two bits I need, without
introducing any fragile assumptions and code changes about which fields
are used exclusively with others and which not.
I figured that I can just be a *user* of this existing interface,
extending it as needed, but not tinkering too much. Which was what
Stefano asked for.
To that extent one could argue to just reuse the existing GICv3 LR
format directly, which should be a superset of the GICv2 format. Not
sure that this is useful, though.

> That would also help to understand the purpose of each field without
> looking at the write_lr callback details.

> Also, please mention that source is GICv2 only.

I am not sure the purpose of the Xen code is to educate people about how
the GIC works. Since you seem like to spec references, I can add a
comment to that regard, if you are OK with that.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 43/57] ARM: new VGIC: Add SGIR register handler

[Julien Grall]

Hi,

On 05/03/18 16:04, Andre Przywara wrote:
> Triggering an IPI via this register is v2 specific, so the
> implementation lives entirely in vgic-mmio-v2.c.
> 
> This is based on Linux commit 55cc01fb9004, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - use symbolic names instead of magic values
> - iterates over set bits instead of every VCPU
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 47 ++++++++++++++++++++++++++++++++++++++--
>   1 file changed, 45 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 01c6a7198c..5f1fdb9a70 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -72,6 +72,49 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>       }
>   }
>   
> +static void vgic_mmio_write_sgir(struct vcpu *source_vcpu,
> +                                 paddr_t addr, unsigned int len,
> +                                 unsigned long val)
> +{
> +    struct domain *d = source_vcpu->domain;
> +    unsigned int nr_vcpus = d->max_vcpus;
> +    unsigned int intid = val & GICD_SGI_INTID_MASK;
> +    unsigned long targets = (val & GICD_SGI_TARGET_MASK) >>
> +                            GICD_SGI_TARGET_SHIFT;
> +    unsigned int vcpu_id;
> +
> +    switch ( val & GICD_SGI_TARGET_LIST_MASK )
> +    {
> +    case GICD_SGI_TARGET_LIST:                    /* as specified by targets */
> +        targets &= GENMASK(nr_vcpus, 0);          /* limit to existing VCPUs */
> +        break;
> +    case GICD_SGI_TARGET_OTHERS:
> +        targets = GENMASK(nr_vcpus, 0);           /* all, ...   */
> +        targets &= ~(1U << source_vcpu->vcpu_id); /*   but self */
> +        break;
> +    case GICD_SGI_TARGET_SELF:                    /* this very vCPU only */
> +        targets = (1U << source_vcpu->vcpu_id);
> +        break;
> +    case 0x3:                                     /* reserved */
> +        return;
> +    }
> +
> +    for_each_set_bit( vcpu_id, &targets, 8 )
> +    {
> +        struct vcpu *vcpu = d->vcpu[vcpu_id];
> +        struct vgic_irq *irq = vgic_get_irq(d, vcpu, intid);
> +        unsigned long flags;
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->pending_latch = true;
> +        irq->source |= 1U << source_vcpu->vcpu_id;
> +
> +        vgic_queue_irq_unlock(d, irq, flags);
> +        vgic_put_irq(d, irq);
> +    }
> +}
> +
>   static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
>                                              paddr_t addr, unsigned int len)
>   {
> @@ -128,7 +171,7 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
>           spin_unlock_irqrestore(&irq->irq_lock, flags);
>   
>           if ( desc )
> -            vgic_update_hardware_irq(desc, irq);
> +            vgic_sync_hardware_irq(vcpu->domain, desc, irq);

This belongs to the previous patch. But calling vgic_sync_hardware_irq 
is pretty inefficient. You will end up to call ->enable everytime you 
route an interrupt.

Also, you don't really need to take the irq->desc_lock for the ITARGET 
case as you will not modify it. So you could just call irq_set_affinity 
directly.

>   
>           vgic_put_irq(vcpu->domain, irq);
>       }
> @@ -169,7 +212,7 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_config, vgic_mmio_write_config, 2,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_SGIR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 4,
> +        vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
>           vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 42/57] ARM: new VGIC: Add TARGET registers handlers

[Andre Przywara]

Hi,

On 08/03/18 16:18, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:04, Andre Przywara wrote:
>> The target register handlers are v2 emulation specific, so their
>> implementation lives entirely in vgic-mmio-v2.c.
>> We copy the old VGIC behaviour of assigning an IRQ to the first VCPU
>> set in the target mask instead of making it possibly pending on
>> multiple VCPUs.
>> We update the physical affinity of a hardware mapped vIRQ on the way.
>>
>> This is based on Linux commit 2c234d6f1826, written by Andre Przywara.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - propagate affinity changes to hardware mapped IRQs
>>
>>   xen/arch/arm/vgic/vgic-mmio-v2.c | 64
>> +++++++++++++++++++++++++++++++++++++++-
>>   1 file changed, 63 insertions(+), 1 deletion(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index d19ddd3f1e..01c6a7198c 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -72,6 +72,68 @@ static void vgic_mmio_write_v2_misc(struct vcpu *vcpu,
>>       }
>>   }
>>   +static unsigned long vgic_mmio_read_target(struct vcpu *vcpu,
>> +                                           paddr_t addr, unsigned int
>> len)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
>> +    uint32_t val = 0;
>> +    unsigned int i;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        val |= (uint32_t)irq->targets << (i * 8);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return val;
>> +}
>> +
>> +static void vgic_mmio_write_target(struct vcpu *vcpu,
>> +                                   paddr_t addr, unsigned int len,
>> +                                   unsigned long val)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
>> +    uint8_t cpu_mask = GENMASK(vcpu->domain->max_vcpus - 1, 0);
>> +    unsigned int i;
>> +    unsigned long flags;
>> +
>> +    /* GICD_ITARGETSR[0-7] are read-only */
>> +    if ( intid < VGIC_NR_PRIVATE_IRQS )
>> +        return;
>> +
>> +    for ( i = 0; i < len; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, NULL, intid
>> + i);
>> +        struct irq_desc *desc;
>> +
>> +        spin_lock_irqsave(&irq->irq_lock, flags);
>> +
>> +        irq->targets = (val >> (i * 8)) & cpu_mask;
>> +        if ( irq->targets )
>> +        {
>> +            irq->target_vcpu = vcpu->domain->vcpu[ffs(irq->targets) -
>> 1];
>> +            if ( irq->hw )
>> +                desc = irq_to_desc(irq->hwintid);
>> +            else
>> +                desc = NULL;
> 
> If you initialized desc to NULL during the declaration, then the else is
> not necessary.
> 
>> +        }
>> +        else {
>> +            irq->target_vcpu = NULL;
>> +            desc = NULL;
>> +        }
>> +
>> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
>> +
>> +        if ( desc )
>> +            vgic_update_hardware_irq(desc, irq);
> 
> I can't find this function in the tree.

Ah, good catch, that's a rebase artefact. It gets fixed in the next patch.

Cheers,
Andre.

> 
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +}
>> +
>>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
>> @@ -101,7 +163,7 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_priority, vgic_mmio_write_priority, 8,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ITARGETSR,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
>> +        vgic_mmio_read_target, vgic_mmio_write_target, 8,
>>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICFGR,
>>           vgic_mmio_read_config, vgic_mmio_write_config, 2,
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 44/57] ARM: new VGIC: Add SGIPENDR register handlers

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> As this register is v2 specific, its implementation lives entirely
> in vgic-mmio-v2.c.
> This register allows setting the source mask of an IPI.
> 
> This is based on Linux commit ed40213ef9b0, written by Andre Przywara.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - use C99 and unsigned data types
> 
>   xen/arch/arm/vgic/vgic-mmio-v2.c | 81 +++++++++++++++++++++++++++++++++++++++-
>   1 file changed, 79 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c b/xen/arch/arm/vgic/vgic-mmio-v2.c
> index 5f1fdb9a70..dd9857e8a6 100644
> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
> @@ -177,6 +177,83 @@ static void vgic_mmio_write_target(struct vcpu *vcpu,
>       }
>   }
>   
> +static unsigned long vgic_mmio_read_sgipend(struct vcpu *vcpu,
> +                                            paddr_t addr, unsigned int len)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    uint32_t val = 0;
> +    unsigned int i;
> +
> +    ASSERT(intid < VGIC_NR_SGIS);
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        val |= (uint32_t)irq->source << (i * 8);
> +
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +
> +    return val;
> +}
> +
> +static void vgic_mmio_write_sgipendc(struct vcpu *vcpu,
> +                                     paddr_t addr, unsigned int len,
> +                                     unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    unsigned int i;
> +    unsigned long flags;
> +
> +    ASSERT(intid < VGIC_NR_SGIS);
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->source &= ~((val >> (i * 8)) & 0xff);
> +        if ( !irq->source )
> +            irq->pending_latch = false;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
> +static void vgic_mmio_write_sgipends(struct vcpu *vcpu,
> +                                     paddr_t addr, unsigned int len,
> +                                     unsigned long val)
> +{
> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 8);
> +    unsigned int i;
> +    unsigned long flags;
> +
> +    ASSERT(intid < VGIC_NR_SGIS);
> +
> +    for ( i = 0; i < len; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid + i);
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        irq->source |= (val >> (i * 8)) & 0xff;
> +
> +        if ( irq->source )
> +        {
> +            irq->pending_latch = true;
> +            vgic_queue_irq_unlock(vcpu->domain, irq, flags);
> +        }
> +        else
> +        {
> +            spin_unlock_irqrestore(&irq->irq_lock, flags);
> +        }
> +        vgic_put_irq(vcpu->domain, irq);
> +    }
> +}
> +
>   static const struct vgic_register_region vgic_v2_dist_registers[] = {
>       REGISTER_DESC_WITH_LENGTH(GICD_CTLR,
>           vgic_mmio_read_v2_misc, vgic_mmio_write_v2_misc, 12,
> @@ -215,10 +292,10 @@ static const struct vgic_register_region vgic_v2_dist_registers[] = {
>           vgic_mmio_read_raz, vgic_mmio_write_sgir, 4,
>           VGIC_ACCESS_32bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_CPENDSGIR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> +        vgic_mmio_read_sgipend, vgic_mmio_write_sgipendc, 16,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>       REGISTER_DESC_WITH_LENGTH(GICD_SPENDSGIR,
> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 16,
> +        vgic_mmio_read_sgipend, vgic_mmio_write_sgipends, 16,
>           VGIC_ACCESS_32bit | VGIC_ACCESS_8bit),
>   };
>   
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 21/57] ARM: GICv2: extend LR read/write functions to cover EOI and source

[Julien Grall]

Hi,

On 08/03/18 16:25, Andre Przywara wrote:
> On 06/03/18 16:06, Julien Grall wrote:
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>>> index 8fab458d7f..89a07ae6b4 100644
>>> --- a/xen/include/asm-arm/gic.h
>>> +++ b/xen/include/asm-arm/gic.h
>>> @@ -223,9 +223,11 @@ struct gic_lr {
>>>       /* Virtual IRQ */
>>>       uint32_t virq;
>>>       uint8_t priority;
>>> -   uint8_t state;
>>> -   uint8_t hw_status;
>>> -   uint8_t grp;
>>> +   uint8_t source;
>>> +   uint8_t state:2;
>>> +   uint8_t hw_status:1;
>>> +   uint8_t grp:1;
>>> +   uint8_t eoi:1;
>>
>> I would much prefer to introduce an union with specific information for
>> the physical interrupt (i.e GICH_LR.HW = 1) in one side and purely
>> virtual in the other side.
> 
> Feel free to send a patch ;-)

Feel free to find someone else acking your patch ;).

> For my part I just wanted to add the two bits I need, without
> introducing any fragile assumptions and code changes about which fields
> are used exclusively with others and which not.
> I figured that I can just be a *user* of this existing interface,
> extending it as needed, but not tinkering too much. Which was what
> Stefano asked for.
> To that extent one could argue to just reuse the existing GICv3 LR
> format directly, which should be a superset of the GICv2 format. Not
> sure that this is useful, though.

While I agree that interface should not be changed too much, this new 
change just does not make sense. It makes more complicate for a reader 
(even after having read the spec...) to understand why some fields are 
only set in certain circumstance. So from my side, this is a Nack.

> 
>> That would also help to understand the purpose of each field without
>> looking at the write_lr callback details.
> 
>> Also, please mention that source is GICv2 only.
> 
> I am not sure the purpose of the Xen code is to educate people about how
> the GIC works. Since you seem like to spec references, I can add a
> comment to that regard, if you are OK with that.

Well, it does not mean you need to have dry code... If you write GICv2 
only the user will have some tips where to look in the spec. I don't 
even ask a specific section.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 21/57] ARM: GICv2: extend LR read/write functions to cover EOI and source

[Julien Grall]

On 03/08/2018 04:41 PM, Julien Grall wrote:
> Hi,
> 
> On 08/03/18 16:25, Andre Przywara wrote:
>> On 06/03/18 16:06, Julien Grall wrote:
>>> On 05/03/18 16:03, Andre Przywara wrote:
>>>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>>>> index 8fab458d7f..89a07ae6b4 100644
>>>> --- a/xen/include/asm-arm/gic.h
>>>> +++ b/xen/include/asm-arm/gic.h
>>>> @@ -223,9 +223,11 @@ struct gic_lr {
>>>>       /* Virtual IRQ */
>>>>       uint32_t virq;
>>>>       uint8_t priority;
>>>> -   uint8_t state;
>>>> -   uint8_t hw_status;
>>>> -   uint8_t grp;
>>>> +   uint8_t source;
>>>> +   uint8_t state:2;
>>>> +   uint8_t hw_status:1;
>>>> +   uint8_t grp:1;
>>>> +   uint8_t eoi:1;
>>>
>>> I would much prefer to introduce an union with specific information for
>>> the physical interrupt (i.e GICH_LR.HW = 1) in one side and purely
>>> virtual in the other side.
>>
>> Feel free to send a patch ;-)
> 
> Feel free to find someone else acking your patch ;).
> 
>> For my part I just wanted to add the two bits I need, without
>> introducing any fragile assumptions and code changes about which fields
>> are used exclusively with others and which not.
>> I figured that I can just be a *user* of this existing interface,
>> extending it as needed, but not tinkering too much. Which was what
>> Stefano asked for.
>> To that extent one could argue to just reuse the existing GICv3 LR
>> format directly, which should be a superset of the GICv2 format. Not
>> sure that this is useful, though.
> 
> While I agree that interface should not be changed too much, this new 
> change just does not make sense. It makes more complicate for a reader 
> (even after having read the spec...) to understand why some fields are 
> only set in certain circumstance. So from my side, this is a Nack.

For clarity:

Nacked-by: Julien Grall <julien.grall@arm.com>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 45/57] ARM: new VGIC: Handle hardware mapped IRQs

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> The VGIC supports virtual IRQs to be connected to a hardware IRQ, so
> when a guest EOIs the virtual interrupt, it affects the state of that
> corresponding interrupt on the hardware side at the same time.
> Implement the interface that the Xen arch/core code expects to connect
> the virtual and the physical world.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - add ASSERT for hardware mapped IRQs being SPI only
> - check h/w IRQ matches before disconnecting
> 
>   xen/arch/arm/vgic/vgic.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 71 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 5246d7c2e7..5bbf55da21 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -698,6 +698,77 @@ void vgic_kick_vcpus(struct domain *d)
>       }
>   }
>   
> +struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
> +                                      unsigned int virq)
> +{
> +    struct irq_desc *desc = NULL;
> +    struct vgic_irq *irq = vgic_get_irq(d, v, virq);
> +    unsigned long flags;
> +
> +    if ( !irq )
> +        return NULL;
> +
> +    spin_lock_irqsave(&irq->irq_lock, flags);
> +    if ( irq->hw )
> +    {
> +        ASSERT(irq->hwintid >= VGIC_NR_PRIVATE_IRQS);
> +        desc = irq_to_desc(irq->hwintid);
> +    }
> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +    vgic_put_irq(d, irq);
> +
> +    return desc;
> +}
> +
> +/*
> + * was:
> + *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
> + *      int kvm_vgic_unmap_phys_irq(struct vcpu *vcpu, unsigned int virt_irq)
> + */
> +int vgic_connect_hw_irq(struct domain *d, struct vcpu *vcpu,
> +                        unsigned int virt_irq, struct irq_desc *desc,
> +                        bool connect)
> +{
> +    struct vgic_irq *irq = vgic_get_irq(d, vcpu, virt_irq);
> +    unsigned long flags;
> +    int ret = 0;
> +
> +    if ( !irq )
> +        return -EINVAL;
> +
> +    spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +    if ( connect )                      /* assign a mapped IRQ */
> +    {
> +        /* The VIRQ should not be already enabled by the guest */
> +        if ( !irq->hw && !irq->enabled )
> +        {
> +            irq->hw = true;
> +            irq->hwintid = desc->irq;
> +        }
> +        else
> +            ret = -EBUSY;
> +    }
> +    else                                /* remove a mapped IRQ */
> +    {
> +        if ( desc && irq->hwintid != desc->irq )
> +        {
> +            ret = -EINVAL;
> +        }
> +        else
> +        {
> +            irq->hw = false;
> +            irq->hwintid = 0;
> +        }
> +    }
> +
> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
> +    vgic_put_irq(d, irq);
> +
> +    return ret;
> +}
> +
>   static unsigned int translate_irq_type(bool is_level)
>   {
>       return is_level ? IRQ_TYPE_LEVEL_HIGH : IRQ_TYPE_EDGE_RISING;
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 46/57] ARM: new VGIC: Add event channel IRQ handling

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> The Xen core/arch code relies on two abstracted functions to inject an
> event channel IRQ and to query its pending state.
> Implement those to query the state of the new VGIC implementation.

The code looks good, but I am wondering why we ever need to call 
vgic_evtchn_irq_pending in local_event_needs_delivery_nomask.

After all, the event channel is an interrupt. So it should already get 
caught by vgic_pending_irq(). If not, then there are no point to wake up 
the vCPU for nothing (it will not get handled). Stefano, do you have 
insight why the current implementation?

Anyway, the code as it is fine so:

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - add locking
> 
>   xen/arch/arm/vgic/vgic.c | 23 +++++++++++++++++++++++
>   1 file changed, 23 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 5bbf55da21..0bf257c865 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -698,6 +698,29 @@ void vgic_kick_vcpus(struct domain *d)
>       }
>   }
>   
> +void arch_evtchn_inject(struct vcpu *v)
> +{
> +    vgic_inject_irq(v->domain, v, v->domain->arch.evtchn_irq, true);
> +}
> +
> +bool vgic_evtchn_irq_pending(struct vcpu *v)
> +{
> +    struct vgic_irq *irq;
> +    unsigned long flags;
> +    bool pending;
> +
> +    /* Does not work for LPIs. */
> +    ASSERT(!is_lpi(v->domain->arch.evtchn_irq));
> +
> +    irq = vgic_get_irq(v->domain, v, v->domain->arch.evtchn_irq);
> +    spin_lock_irqsave(&irq->irq_lock, flags);
> +    pending = irq_is_pending(irq);
> +    spin_unlock_irqrestore(&irq->irq_lock, flags);
> +    vgic_put_irq(v->domain, irq);
> +
> +    return pending;
> +}
> +
>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>                                         unsigned int virq)
>   {
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 47/57] ARM: new VGIC: Handle virtual IRQ allocation/reservation

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> To find an unused virtual IRQ number Xen uses a scheme to track used
> virtual IRQs.
> Implement this interface in the new VGIC to make the Xen core/arch code
> happy.
> This is actually somewhat VGIC agnostic, so is mostly a copy of the code
> from the old VGIC. But it has to live in the VGIC files, so we can't
> easily reuse the existing implementation.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - no changes
> 
>   xen/arch/arm/vgic/vgic.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 44 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 0bf257c865..e9ef992e1e 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -721,6 +721,50 @@ bool vgic_evtchn_irq_pending(struct vcpu *v)
>       return pending;
>   }
>   
> +bool vgic_reserve_virq(struct domain *d, unsigned int virq)
> +{
> +    if ( virq >= vgic_num_irqs(d) )
> +        return false;
> +
> +    return !test_and_set_bit(virq, d->arch.vgic.allocated_irqs);
> +}
> +
> +int vgic_allocate_virq(struct domain *d, bool spi)
> +{
> +    int first, end;
> +    unsigned int virq;
> +
> +    if ( !spi )
> +    {
> +        /* We only allocate PPIs. SGIs are all reserved */
> +        first = 16;
> +        end = 32;
> +    }
> +    else
> +    {
> +        first = 32;
> +        end = vgic_num_irqs(d);
> +    }
> +
> +    /*
> +     * There is no spinlock to protect allocated_irqs, therefore
> +     * test_and_set_bit may fail. If so retry it.
> +     */
> +    do
> +    {
> +        virq = find_next_zero_bit(d->arch.vgic.allocated_irqs, end, first);
> +        if ( virq >= end )
> +            return -1;
> +    } while ( test_and_set_bit(virq, d->arch.vgic.allocated_irqs) );
> +
> +    return virq;
> +}
> +
> +void vgic_free_virq(struct domain *d, unsigned int virq)
> +{
> +    clear_bit(virq, d->arch.vgic.allocated_irqs);
> +}
> +
>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>                                         unsigned int virq)
>   {
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 48/57] ARM: new VGIC: Dump virtual IRQ info

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> When we dump guest state on the Xen console, we also print the state of
> IRQs that are on a VCPU.
> Add the code to dump the state of an IRQ handled by the new VGIC.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - use proper locking
> - use one header line to announce active or pending IRQs
> 
>   xen/arch/arm/vgic/vgic.c | 25 +++++++++++++++++++++++++
>   1 file changed, 25 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index e9ef992e1e..2a2b8fd1eb 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -765,6 +765,31 @@ void vgic_free_virq(struct domain *d, unsigned int virq)
>       clear_bit(virq, d->arch.vgic.allocated_irqs);
>   }
>   
> +void gic_dump_vgic_info(struct vcpu *v)
> +{
> +    struct vgic_cpu *vgic_cpu = &v->arch.vgic;
> +    struct vgic_irq *irq;
> +    unsigned long flags;
> +
> +    spin_lock_irqsave(&v->arch.vgic.ap_list_lock, flags);
> +
> +    if ( !list_empty(&vgic_cpu->ap_list_head) )
> +        printk("   active or pending interrupts queued:\n");
> +
> +    list_for_each_entry ( irq, &vgic_cpu->ap_list_head, ap_list )
> +    {
> +        spin_lock(&irq->irq_lock);
> +        printk("     %s %s irq %u: %spending, %sactive, %senabled\n",
> +               irq->hw ? "hardware" : "virtual",
> +               irq->config == VGIC_CONFIG_LEVEL ? "level" : "edge",
> +               irq->intid, irq_is_pending(irq) ? "" : "not ",
> +               irq->active ? "" : "not ", irq->enabled ? "" : "not ");
> +        spin_unlock(&irq->irq_lock);
> +    }
> +
> +    spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
> +}
> +
>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>                                         unsigned int virq)
>   {
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 49/57] ARM: new VGIC: provide system register emulation stub

[Julien Grall]

On 05/03/18 16:04, Andre Przywara wrote:
> The Xen arch code traps system registers writes from the guest and will
> relay anything GIC related to the VGIC.
> Since this affects only GICv3 (which we don't yet emulate), provide a
> stub implementation of vgic_emulate() for now.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Acked-by: Julien Grall <julien.grall@arm.com>

Cheers,

> ---
> Changelog RFC ... v1:
> - no changes
> 
>   xen/arch/arm/vgic/vgic.c | 7 +++++++
>   1 file changed, 7 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 2a2b8fd1eb..e1952c872d 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -813,6 +813,13 @@ struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>       return desc;
>   }
>   
> +bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr)
> +{
> +    ASSERT(current->domain->arch.vgic.version == GIC_V3);
> +
> +    return false;
> +}
> +
>   /*
>    * was:
>    *      int kvm_vgic_map_phys_irq(struct vcpu *vcpu, u32 virt_irq, u32 phys_irq)
> 

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 50/57] ARM: new VGIC: Implement arch_move_irqs()

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> When a VCPU moves to another CPU, we need to adjust the target affinity
> of any hardware mapped vIRQs, to observe our "physical-follows-virtual"
> policy.
> Implement arch_move_irqs() to adjust the physical affinity of all hardware
> mapped vIRQs targetting this VCPU.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - actually implement arch_move_irqs() (instead of just stubbing it)
> 
>   xen/arch/arm/vgic/vgic.c | 42 ++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 42 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index e1952c872d..5e767927c0 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -790,6 +790,48 @@ void gic_dump_vgic_info(struct vcpu *v)
>       spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
>   }
>   
> +/**
> + * arch_move_irqs() - migrate the physical affinity of hardware mapped vIRQs
> + * @v:  the vCPU, already assigned to the new pCPU
> + *
> + * arch_move_irqs() updates the physical affinity of all virtual IRQs
> + * targetting this given vCPU. This only affects hardware mapped IRQs. The
> + * new pCPU to target is already set in v->processor.
> + * This is called by the core code after a vCPU has been migrated to a new
> + * physical CPU.
> + */
> +void arch_move_irqs(struct vcpu *v)
> +{
> +    struct domain *d = v->domain;
> +    unsigned int i;
> +
> +    /* We only target SPIs with this function */
> +    for ( i = 0; i < d->arch.vgic.nr_spis; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(d, NULL, i + VGIC_NR_PRIVATE_IRQS);
> +        unsigned long flags;
> +        irq_desc_t *desc;
> +
> +        if ( !irq )
> +            continue;
> +
> +        spin_lock_irqsave(&irq->irq_lock, flags);
> +
> +        /* only vIRQs that are not on a vCPU yet , but targetting this vCPU */
> +        if ( irq->hw && !irq->vcpu && irq->target_vcpu == v)
> +            desc = irq_to_desc(irq->hwintid);
> +        else
> +            desc = NULL;
> +
> +        spin_unlock_irqrestore(&irq->irq_lock, flags);
> +
> +        if ( desc )
> +            vgic_sync_hardware_irq(d, desc, irq);

You want to look at my comment about using vgic_sync_hardware_irq for 
routing on patch #43.

> +
> +        vgic_put_irq(d, irq);
> +    }
> +}
> +
>   struct irq_desc *vgic_get_hw_irq_desc(struct domain *d, struct vcpu *v,
>                                         unsigned int virq)
>   {
> 

Cheers,
-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 51/57] ARM: new VGIC: Add preliminary stub implementation

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> The ARM arch code requires an interrupt controller emulation to implement
> vgic_clear_pending_irqs(), although it is suspected that it is actually
> not necessary. Go with a stub for now to make the linker happy.

The implementation of that function is fundamentally wrong on the 
current vGIC for a few reasons:
	- lr_mask is reset but the LRs are not. This means when we context 
switch back, the LR might still be written and injecting unexpected 
interrupt (whoops).
	- both lists (inflight and pending) are cleared which means that a 
physical interrupt pending on that vCPU is lost forever (stay active in 
the physical so never going to fire again).

Furthermore, I don't think that Xen business to reset the GIC on cpu_on. 
If anything should be done, then is it on CPU_off to migrate the current 
interrupts to another vCPU. But IIRC the OS is responsible for that.

So I would kill that function. Any opinions?

> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - split off from former patch, otherwise unchanged
> 
>   xen/arch/arm/vgic/vgic.c | 8 ++++++++
>   1 file changed, 8 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index 5e767927c0..5d84a4d81a 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -790,6 +790,14 @@ void gic_dump_vgic_info(struct vcpu *v)
>       spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
>   }
>   
> +void vgic_clear_pending_irqs(struct vcpu *v)
> +{
> +    /*
> +     * TODO: It is unclear whether we really need this, so we might instead
> +     * remove it on the caller site.
> +     */
> +}
> +
>   /**
>    * arch_move_irqs() - migrate the physical affinity of hardware mapped vIRQs
>    * @v:  the vCPU, already assigned to the new pCPU
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 53/57] ARM: new VGIC: vgic-init: implement vgic_init

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> This patch allocates and initializes the data structures used to model
> the vgic distributor and virtual cpu interfaces. At that stage the
> number of IRQs and number of virtual CPUs is frozen.
> Implement the various functions that the Xen arch code is expecting to
> call during domain and VCPU setup to initialize the VGIC.
> Their prototypes are already in existing header files.
> 
> This is based on Linux commit ad275b8bb1e6, written by Eric Auger.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - adapt to former changes
> - add missing comment line
> - extend commit message
> 
>   xen/arch/arm/vgic/vgic-init.c | 196 ++++++++++++++++++++++++++++++++++++++++++
>   1 file changed, 196 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic-init.c b/xen/arch/arm/vgic/vgic-init.c
> index d091c92ed0..8bc83f677b 100644
> --- a/xen/arch/arm/vgic/vgic-init.c
> +++ b/xen/arch/arm/vgic/vgic-init.c
> @@ -20,6 +20,77 @@
>   
>   #include "vgic.h"
>   
> +/*
> + * Initialization rules: there are multiple stages to the vgic
> + * initialization, both for the distributor and the CPU interfaces.  The basic
> + * idea is that even though the VGIC is not functional or not requested from
> + * user space, the critical path of the run loop can still call VGIC functions
> + * that just won't do anything, without them having to check additional
> + * initialization flags to ensure they don't look at uninitialized data
> + * structures.
> + *
> + * Distributor:
> + *
> + * - vgic_early_init(): initialization of static data that doesn't
> + *   depend on any sizing information or emulation type. No allocation
> + *   is allowed there.
> + *
> + * - vgic_init(): allocation and initialization of the generic data
> + *   structures that depend on sizing information (number of CPUs,
> + *   number of interrupts). Also initializes the vcpu specific data
> + *   structures. Can be executed lazily for GICv2.
> + *
> + * CPU Interface:
> + *
> + * - kvm_vgic_vcpu_early_init(): initialization of static data that
> + *   doesn't depend on any sizing information or emulation type. No
> + *   allocation is allowed there.
> + */
> +
> +/**
> + * vgic_vcpu_early_init() - Initialize static VGIC VCPU data structures
> + * @vcpu: The VCPU whose VGIC data structures whould be initialized
> + *
> + * Only do initialization, but do not actually enable the VGIC CPU interface
> + * yet.
> + */
> +static void vgic_vcpu_early_init(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
> +    int i;

unsigned please.

> +
> +    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
> +    spin_lock_init(&vgic_cpu->ap_list_lock);

Do we need something similar to 23b40df6f098e3bcb2f105a4909860240976e40f 
"xen/arm: vgic: Make sure the number of SPIs is a multiple of 32"?

> +
> +    /*
> +     * Enable and configure all SGIs to be edge-triggered and
> +     * configure all PPIs as level-triggered.
> +     */
> +    for ( i = 0; i < VGIC_NR_PRIVATE_IRQS; i++ )
> +    {
> +        struct vgic_irq *irq = &vgic_cpu->private_irqs[i];
> +
> +        INIT_LIST_HEAD(&irq->ap_list);
> +        spin_lock_init(&irq->irq_lock);
> +        irq->intid = i;
> +        irq->vcpu = NULL;
> +        irq->target_vcpu = vcpu;
> +        irq->targets = 1U << vcpu->vcpu_id;
> +        atomic_set(&irq->refcount, 0);
> +        if ( vgic_irq_is_sgi(i) )
> +        {
> +            /* SGIs */
> +            irq->enabled = 1;
> +            irq->config = VGIC_CONFIG_EDGE;
> +        }
> +        else
> +        {
> +            /* PPIs */
> +            irq->config = VGIC_CONFIG_LEVEL;
> +        }
> +    }
> +}
> +
>   /* CREATION */
>   
>   /**
> @@ -50,6 +121,131 @@ int domain_vgic_register(struct domain *d, int *mmio_count)
>       return 0;
>   }
>   
> +/* INIT/DESTROY */
> +
> +/**
> + * domain_vgic_init: initialize the dist data structures
> + * @d: domain pointer
> + * @nr_spis: number of SPIs
> + */
> +int domain_vgic_init(struct domain *d, unsigned int nr_spis)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +    int i, ret;

Ditto for i.

> +
> +    /* Limit the number of virtual SPIs supported to (1020 - 32) = 988  */
> +    if ( nr_spis > (1020 - NR_LOCAL_IRQS) )
> +        return -EINVAL;
> +
> +    dist->nr_spis = nr_spis;
> +    dist->spis = xzalloc_array(struct vgic_irq, nr_spis);
> +    if ( !dist->spis )
> +        return  -ENOMEM;
> +
> +    /*
> +     * In the following code we do not take the irq struct lock since
> +     * no other action on irq structs can happen while the VGIC is
> +     * not initialized yet:
> +     * If someone wants to inject an interrupt or does a MMIO access, we
> +     * require prior initialization in case of a virtual GICv3 or trigger
> +     * initialization when using a virtual GICv2.
> +     */
> +    for ( i = 0; i < nr_spis; i++ )
> +    {
> +        struct vgic_irq *irq = &dist->spis[i];
> +
> +        irq->intid = i + VGIC_NR_PRIVATE_IRQS;
> +        INIT_LIST_HEAD(&irq->ap_list);
> +        spin_lock_init(&irq->irq_lock);
> +        irq->vcpu = NULL;
> +        irq->target_vcpu = NULL;
> +        atomic_set(&irq->refcount, 0);
> +        if ( dist->version == GIC_V2 )
> +            irq->targets = 0;
> +        else
> +            irq->mpidr = 0;
> +    }
> +
> +    INIT_LIST_HEAD(&dist->lpi_list_head);
> +    spin_lock_init(&dist->lpi_list_lock);
> +
> +    if ( dist->version == GIC_V2 )
> +        ret = vgic_v2_map_resources(d);
> +    else
> +        ret = -ENXIO;
> +
> +    if ( ret )
> +        return ret;
> +
> +    /* allocated_irqs() is used by Xen to find available vIRQs */
> +    d->arch.vgic.allocated_irqs =
> +        xzalloc_array(unsigned long, BITS_TO_LONGS(vgic_num_irqs(d)));
> +    if ( !d->arch.vgic.allocated_irqs )
> +        return -ENOMEM;
> +
> +    /* vIRQ0-15 (SGIs) are reserved */
> +    for ( i = 0; i < NR_GIC_SGI; i++ )
> +        set_bit(i, d->arch.vgic.allocated_irqs);
> +
> +    return 0;
> +}
> +
> +/**
> + * vcpu_vgic_init() - Register VCPU-specific KVM iodevs
> + * was: kvm_vgic_vcpu_init()
> + * Xen: adding vgic_vx_enable() call
> + * @vcpu: pointer to the VCPU being created and initialized
> + */
> +int vcpu_vgic_init(struct vcpu *vcpu)
> +{
> +    int ret = 0;
> +
> +    vgic_vcpu_early_init(vcpu);
> +
> +    if ( gic_hw_version() == GIC_V2 )
> +        vgic_v2_enable(vcpu);
> +    else
> +        ret = -ENXIO;
> +
> +    return ret;
> +}
> +
> +void domain_vgic_free(struct domain *d)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +        int i, ret;
> +
> +    for ( i = 0; i < dist->nr_spis; i++ )
> +    {
> +        struct vgic_irq *irq = vgic_get_irq(d, NULL, 32 + i);
> +
> +        if ( !irq->hw )
> +            continue;
> +
> +        ret = release_guest_irq(d, irq->hwintid);
> +        if ( ret )
> +            dprintk(XENLOG_G_WARNING,
> +                "d%u: Failed to release virq %u ret = %d\n",
> +                d->domain_id, 32 + i, ret);

Indentation looks wrong.

> +    }
> +
> +    dist->ready = false;
> +    dist->initialized = false;
> +
> +    xfree(dist->spis);
> +    xfree(dist->allocated_irqs);
> +    dist->nr_spis = 0;
> +}
> +
> +int vcpu_vgic_free(struct vcpu *vcpu)
> +{
> +    struct vgic_cpu *vgic_cpu = &vcpu->arch.vgic;
> +
> +    INIT_LIST_HEAD(&vgic_cpu->ap_list_head);
> +
> +    return 0;
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 54/57] ARM: new VGIC: vgic-init: implement map_resources

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> map_resources is the last initialization step needed before the first
> VCPU is run. At that stage the code stores the MMIO base addresses used.
> Also it registers the respective register frames with the MMIO framework.
> 
> This is based on Linux commit cbae53e663ea, written by Eric Auger.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - adapting to previous changes
> 
>   xen/arch/arm/vgic/vgic-v2.c | 66 +++++++++++++++++++++++++++++++++++++++++++++
>   xen/arch/arm/vgic/vgic.h    |  1 +
>   2 files changed, 67 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
> index 4e74ebf7f5..da64b4758c 100644
> --- a/xen/arch/arm/vgic/vgic-v2.c
> +++ b/xen/arch/arm/vgic/vgic-v2.c
> @@ -221,6 +221,72 @@ void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)
>       gic_hw_ops->write_lr(lr, &lr_val);
>   }
>   
> +int vgic_v2_map_resources(struct domain *d)
> +{
> +    struct vgic_dist *dist = &d->arch.vgic;
> +    paddr_t cbase, csize;
> +    paddr_t vbase;
> +    int ret;
> +
> +    /*
> +     * The hardware domain gets the hardware address.
> +     * Guests get the virtual platform layout.
> +     */
> +    if ( is_hardware_domain(d) )
> +    {
> +        d->arch.vgic.vgic_dist_base = gic_v2_hw_data.dbase;
> +        /*
> +         * For the hardware domain, we always map the whole HW CPU
> +         * interface region in order to match the device tree (the "reg"
> +         * properties is copied as it is).
> +         * Note that we assume the size of the CPU interface is always
> +         * aligned to PAGE_SIZE.
> +         */
> +        cbase = gic_v2_hw_data.cbase;   /* was: dist->vgic_cpu_base */
> +        csize = gic_v2_hw_data.csize;
> +        vbase = gic_v2_hw_data.vbase; /* was: kvm_vgic_global_state.vcpu_base */

NIT: Please keep the comment indented the same way.

> +    }
> +    else
> +    {
> +        d->arch.vgic.vgic_dist_base = GUEST_GICD_BASE;
> +        /*
> +         * The CPU interface exposed to the guest is always 8kB. We may
> +         * need to add an offset to the virtual CPU interface base
> +         * address when in the GIC is aliased to get a 8kB contiguous
> +         * region.
> +         */
> +        BUILD_BUG_ON(GUEST_GICC_SIZE != SZ_8K);
> +        cbase = GUEST_GICC_BASE;
> +        csize = GUEST_GICC_SIZE;
> +        vbase = gic_v2_hw_data.vbase + gic_v2_hw_data.aliased_offset;
> +    }
> +
> +
> +    ret = vgic_register_dist_iodev(d, gaddr_to_gfn(dist->vgic_dist_base),
> +                                   VGIC_V2);
> +    if ( ret )
> +    {
> +        gdprintk(XENLOG_ERR, "Unable to register VGIC MMIO regions\n");
> +        return ret;
> +    }
> +
> +    /*
> +     * Map the gic virtual cpu interface in the gic cpu interface
> +     * region of the guest.
> +     */
> +    ret = map_mmio_regions(d, gaddr_to_gfn(cbase), csize / PAGE_SIZE,
> +                   maddr_to_mfn(vbase));

Indentation.

> +    if ( ret )
> +    {
> +        gdprintk(XENLOG_ERR, "Unable to remap VGIC CPU to VCPU\n");
> +        return ret;
> +    }
> +
> +    dist->ready = true;
> +
> +        return 0;
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index f19dc9502f..6fab994b9c 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -61,6 +61,7 @@ void vgic_sync_hardware_irq(struct domain *d,
>   void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
>   void vgic_v2_set_underflow(struct vcpu *vcpu);
> +int vgic_v2_map_resources(struct domain *d);
>   int vgic_register_dist_iodev(struct domain *d, gfn_t dist_base_fn,
>                                enum vgic_type);
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 55/57] ARM: new VGIC: Add vgic_v2_enable

[Julien Grall]

Hi Andre,

On 05/03/18 16:04, Andre Przywara wrote:
> Enable the VGIC operation by properly initialising the registers
> in the hypervisor GIC interface.
> 
> This is based on Linux commit f7b6985cc3d0, written by Eric Auger.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>

Would you mind to move that patch before #53? This feels more logical as 
the first user in there.

> ---
> Changelog RFC ... v1:
> - drop unneeded vgic_vmcr initialization
> - use update_hcr_status wrapper
> 
>   xen/arch/arm/vgic/vgic-v2.c | 6 ++++++
>   xen/arch/arm/vgic/vgic.h    | 1 +
>   2 files changed, 7 insertions(+)
> 
> diff --git a/xen/arch/arm/vgic/vgic-v2.c b/xen/arch/arm/vgic/vgic-v2.c
> index da64b4758c..b7d6493e5a 100644
> --- a/xen/arch/arm/vgic/vgic-v2.c
> +++ b/xen/arch/arm/vgic/vgic-v2.c
> @@ -221,6 +221,12 @@ void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr)
>       gic_hw_ops->write_lr(lr, &lr_val);
>   }
>   
> +void vgic_v2_enable(struct vcpu *vcpu)
> +{
> +    /* Get the show on the road... */
> +    gic_hw_ops->update_hcr_status(GICH_HCR_EN, 1);
> +}
> +
>   int vgic_v2_map_resources(struct domain *d)
>   {
>       struct vgic_dist *dist = &d->arch.vgic;
> diff --git a/xen/arch/arm/vgic/vgic.h b/xen/arch/arm/vgic/vgic.h
> index 6fab994b9c..bd0c3fe5ab 100644
> --- a/xen/arch/arm/vgic/vgic.h
> +++ b/xen/arch/arm/vgic/vgic.h
> @@ -61,6 +61,7 @@ void vgic_sync_hardware_irq(struct domain *d,
>   void vgic_v2_fold_lr_state(struct vcpu *vcpu);
>   void vgic_v2_populate_lr(struct vcpu *vcpu, struct vgic_irq *irq, int lr);
>   void vgic_v2_set_underflow(struct vcpu *vcpu);
> +void vgic_v2_enable(struct vcpu *vcpu);
>   int vgic_v2_map_resources(struct domain *d);
>   int vgic_register_dist_iodev(struct domain *d, gfn_t dist_base_fn,
>                                enum vgic_type);
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 56/57] ARM: allocate two pages for struct vcpu

[Julien Grall]

Hi,

On 05/03/18 16:04, Andre Przywara wrote:
> At the moment we allocate exactly one page for struct vcpu on ARM, also
> have a check in place to prevent it growing beyond 4KB.
> As the struct includes the state of all 32 private (per-VCPU) interrupts,
> we are at 3840 bytes on arm64 at the moment already. Growing the per-IRQ
> VGIC structure even slightly makes the VCPU quickly exceed the 4K limit.
> The new VGIC will need more space per virtual IRQ. I spent a few hours
> trying to trim this down, but couldn't get it below 4KB, even with the
> nasty hacks piling up to save some bytes here and there.
> It turns out that beyond efficiency, maybe, there is no real technical
> reason this struct has to fit in one page, so lifting the limit to two
> pages seems like the most pragmatic solution.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - no changes
> 
>   xen/arch/arm/domain.c | 9 ++++++---
>   1 file changed, 6 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/domain.c b/xen/arch/arm/domain.c
> index 11a46aa27f..0bec6aad17 100644
> --- a/xen/arch/arm/domain.c
> +++ b/xen/arch/arm/domain.c
> @@ -502,10 +502,13 @@ void dump_pageframe_info(struct domain *d)
>   struct vcpu *alloc_vcpu_struct(void)
>   {
>       struct vcpu *v;
> -    BUILD_BUG_ON(sizeof(*v) > PAGE_SIZE);
> -    v = alloc_xenheap_pages(0, 0);
> -    if ( v != NULL )
> +
> +    BUILD_BUG_ON(sizeof(*v) > 2 * PAGE_SIZE);
> +    v = alloc_xenheap_pages(1, 0);

While I am still not a big fan of increasing struct vcpu. We should at 
least not blindly increase it for anyone. There are indeed some setup 
where the 8KB solution is not necessary. This is the case of Arm32, and 
Arm64 (with the old vGIC).

I would also like to hear Stefano's opinion on that one.

> +    if ( v != NULL ) {
>           clear_page(v);
> +        clear_page((void *)v + PAGE_SIZE);
> +    }
>       return v;
>   }
>   
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 57/57] ARM: VGIC: wire new VGIC(-v2) files into Xen build system

[Julien Grall]

Hi,

On 05/03/18 16:04, Andre Przywara wrote:
> Now that we have both the old VGIC prepared to cope with a sibling and
> the code for the new VGIC in place, lets add a Kconfig option to enable
> the new code and wire it into the Xen build system.
> This will add a compile time option to use either the "old" or the "new"
> VGIC.
> In the moment this is restricted to a vGIC-v2. To make the build system
> happy, we provide a temporary dummy implementation of
> vgic_v3_setup_hw() to allow building for now.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
> Changelog RFC ... v1:
> - no changes
> 
>   xen/arch/arm/Kconfig     |  6 +++++-
>   xen/arch/arm/Makefile    | 10 +++++++++-
>   xen/arch/arm/vgic/vgic.c |  8 ++++++++
>   xen/common/Makefile      |  1 +
>   4 files changed, 23 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 2782ee6589..aad19927ce 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -48,7 +48,11 @@ config HAS_GICV3
>   config HAS_ITS
>           bool
>           prompt "GICv3 ITS MSI controller support" if EXPERT = "y"
> -        depends on HAS_GICV3
> +        depends on HAS_GICV3 && !NEW_VGIC
> +
> +config NEW_VGIC
> +        bool
> +        prompt "Use new VGIC implementation"

You potentially want to add a description encouraging people to test and 
making aware of this is new (not security supported). Or maybe we need 
to do that in SUPPORT.MD?

>   
>   config SBSA_VUART_CONSOLE
>   	bool "Emulated SBSA UART console support"
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 41d7366527..2a3ec94a18 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -16,7 +16,6 @@ obj-y += domain_build.o
>   obj-y += domctl.o
>   obj-$(EARLY_PRINTK) += early_printk.o
>   obj-y += gic.o
> -obj-y += gic-vgic.o
>   obj-y += gic-v2.o
>   obj-$(CONFIG_HAS_GICV3) += gic-v3.o
>   obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
> @@ -47,10 +46,19 @@ obj-y += sysctl.o
>   obj-y += time.o
>   obj-y += traps.o
>   obj-y += vcpreg.o
> +ifeq ($(CONFIG_NEW_VGIC),y)
> +obj-y += vgic/vgic.o
> +obj-y += vgic/vgic-v2.o
> +obj-y += vgic/vgic-mmio.o
> +obj-y += vgic/vgic-mmio-v2.o
> +obj-y += vgic/vgic-init.o

Hmmm, I would much prefer to see a Makefile introduced in vgic.

> +else
> +obj-y += gic-vgic.o
>   obj-y += vgic.o
>   obj-y += vgic-v2.o
>   obj-$(CONFIG_HAS_GICV3) += vgic-v3.o
>   obj-$(CONFIG_HAS_ITS) += vgic-v3-its.o
> +endif
>   obj-y += vm_event.o
>   obj-y += vtimer.o
>   obj-$(CONFIG_SBSA_VUART_CONSOLE) += vpl011.o
> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
> index f42092fec3..f7b4779a71 100644
> --- a/xen/arch/arm/vgic/vgic.c
> +++ b/xen/arch/arm/vgic/vgic.c
> @@ -978,6 +978,14 @@ unsigned int domain_max_vcpus(const struct domain *d)
>       return min_t(unsigned int, MAX_VIRT_CPUS, vgic_vcpu_limit);
>   }
>   
> +void vgic_v3_setup_hw(paddr_t dbase,
> +                      unsigned int nr_rdist_regions,
> +                      const struct rdist_region *regions,
> +                      unsigned int intid_bits)
> +{
> +    /* Dummy implementation to allow building without actual vGICv3 support. */
> +}
> +
>   /*
>    * Local variables:
>    * mode: C
> diff --git a/xen/common/Makefile b/xen/common/Makefile
> index 3a349f478b..92a1d1fa58 100644
> --- a/xen/common/Makefile
> +++ b/xen/common/Makefile

This would require to CC the "REST".

> @@ -19,6 +19,7 @@ obj-y += keyhandler.o
>   obj-$(CONFIG_KEXEC) += kexec.o
>   obj-$(CONFIG_KEXEC) += kimage.o
>   obj-y += lib.o
> +obj-$(CONFIG_NEW_VGIC) += list_sort.

This does not look right to me. I would much prefer a new config or 
building by default. I would like to hear the opinion of others here.

>   obj-$(CONFIG_LIVEPATCH) += livepatch.o livepatch_elf.o
>   obj-y += lzo.o
>   obj-$(CONFIG_HAS_MEM_ACCESS) += mem_access.o
> 

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 51/57] ARM: new VGIC: Add preliminary stub implementation

[Andre Przywara]

Hi,

On 09/03/18 18:18, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:04, Andre Przywara wrote:
>> The ARM arch code requires an interrupt controller emulation to implement
>> vgic_clear_pending_irqs(), although it is suspected that it is actually
>> not necessary. Go with a stub for now to make the linker happy.
> 
> The implementation of that function is fundamentally wrong on the
> current vGIC for a few reasons:
>     - lr_mask is reset but the LRs are not. This means when we context
> switch back, the LR might still be written and injecting unexpected
> interrupt (whoops).
>     - both lists (inflight and pending) are cleared which means that a
> physical interrupt pending on that vCPU is lost forever (stay active in
> the physical so never going to fire again).
> 
> Furthermore, I don't think that Xen business to reset the GIC on cpu_on.
> If anything should be done, then is it on CPU_off to migrate the current
> interrupts to another vCPU. But IIRC the OS is responsible for that.
> 
> So I would kill that function. Any opinions?

So I guess given that the patch is pretty small, we are good with
keeping that for now, and solve this together with the old VGIC later.

Cheers,
Andre.

>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - split off from former patch, otherwise unchanged
>>
>>   xen/arch/arm/vgic/vgic.c | 8 ++++++++
>>   1 file changed, 8 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> index 5e767927c0..5d84a4d81a 100644
>> --- a/xen/arch/arm/vgic/vgic.c
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -790,6 +790,14 @@ void gic_dump_vgic_info(struct vcpu *v)
>>       spin_unlock_irqrestore(&v->arch.vgic.ap_list_lock, flags);
>>   }
>>   +void vgic_clear_pending_irqs(struct vcpu *v)
>> +{
>> +    /*
>> +     * TODO: It is unclear whether we really need this, so we might
>> instead
>> +     * remove it on the caller site.
>> +     */
>> +}
>> +
>>   /**
>>    * arch_move_irqs() - migrate the physical affinity of hardware
>> mapped vIRQs
>>    * @v:  the vCPU, already assigned to the new pCPU
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Andre Przywara]

Hi,

On 08/03/18 15:39, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> The active register handlers are shared between the v2 and v3 emulation,
>> so their implementation goes into vgic-mmio.c, to be easily referenced
>> from the v3 emulation as well later.
>> Since activation/deactivation of an interrupt may happen entirely in the
>> guest without it ever exiting, we need some extra logic to properly track
>> the active state.
>> For clearing the active state, we would basically have to halt the guest
>> to make sure this is properly propagated into the respective VCPUs.
>> This is not yet implemented in Xen.
>> Fortunately this feature is mostly used to reset a just in initialised
>> GIC, so chances are we are tasked to clear bits that are already zero.
>> Add some simple check to avoid a pointless warning in this case.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - remove premature "proper ACTIVE" handler stub
>> - avoid unnecessary warnings on NO-OP register writes
>> - extend comments
>>
>>   xen/arch/arm/vgic/vgic-mmio-v2.c |   4 +-
>>   xen/arch/arm/vgic/vgic-mmio.c    | 103
>> +++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic-mmio.h    |  11 +++++
>>   3 files changed, 116 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> index efdd73301d..c93455fbb2 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio-v2.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio-v2.c
>> @@ -92,10 +92,10 @@ static const struct vgic_register_region
>> vgic_v2_dist_registers[] = {
>>           vgic_mmio_read_pending, vgic_mmio_write_cpending, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ISACTIVER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
>> +        vgic_mmio_read_active, vgic_mmio_write_sactive, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_ICACTIVER,
>> -        vgic_mmio_read_raz, vgic_mmio_write_wi, 1,
>> +        vgic_mmio_read_active, vgic_mmio_write_cactive, 1,
>>           VGIC_ACCESS_32bit),
>>       REGISTER_DESC_WITH_BITS_PER_IRQ(GICD_IPRIORITYR,
>>           vgic_mmio_read_raz, vgic_mmio_write_wi, 8,
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.c
>> b/xen/arch/arm/vgic/vgic-mmio.c
>> index 2e939d5e39..c44d67082f 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.c
>> +++ b/xen/arch/arm/vgic/vgic-mmio.c
>> @@ -281,6 +281,109 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
>>       }
>>   }
>>   +/*
>> + * The actual active bit for a virtual IRQ is held in the LR. Our shadow
>> + * copy in struct vgic_irq is only synced when needed and may not be
>> + * up-to-date all of the time.
>> + * Returning the actual active state is quite costly (stopping all
>> + * VCPUs processing any affected vIRQs), so we use a simple
>> implementation
>> + * to get the best possible answer.
>> + */
>> +unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
>> +                                    paddr_t addr, unsigned int len)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    uint32_t value = 0;
>> +    unsigned int i;
>> +
>> +    /* Loop over all IRQs affected by this read */
>> +    for ( i = 0; i < len * 8; i++ )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        if ( irq->active )
>> +            value |= (1U << i);
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +    }
>> +
>> +    return value;
>> +}
>> +
>> +/*
>> + * We don't actually support clearing the active state of an IRQ (yet).
>> + * However there is a chance that most guests use this for
>> initialization.
>> + * We check whether this MMIO access would actually affect any active
>> IRQ,
>> + * and only print our warning in this case. So clearing already
>> non-active
>> + * IRQs would not be moaned about in the logs.
>> + */
>> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val)
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    unsigned int i;
>> +    bool bail_out = false;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        /*
>> +         * If we know that the IRQ is active or we can't be sure about
>> +         * it (because it is currently in a CPU), log the not properly
>> +         * emulated MMIO access.
>> +         */
>> +        if ( irq->active || irq->vcpu )
>> +        {
>> +            gdprintk(XENLOG_ERR,
>> +                     "%pv: vGICD: IRQ%d: clearing active state not
>> supported\n",
> 
> s/%d/%u/
> 
>> +                     vcpu, irq->intid);
> 
> gdprintk will always print the vCPU. Thought it is the current which
> might be different from vcpu (mostly in the re-dist case).

Ah, thanks. I always get confused about what which version of *printk does.

> So I would use dprintk(XENLOG_G_ERR, "%pv: ..."). I would even be tempt
> to use printk(....) so we can spot potential issue on non-debug build.

Well, in the true spirit of Xen paranoia ;-) I wanted to avoid a guest
spamming the console. And in the end there is nothing a administrator
could really do about it. In my experience those messages tend to really
scare users ("I could boot Dom0 but I see those error messages ....").

>> +            bail_out = true;
> 
> I admit the bailout is a bit weird here. You would only print the
> warning for the first activated IRQ and give the impression it is fine
> for the rest. So maybe you want to drop IRQ%d?

For the above reasons I wanted to keep them concise, so that we see that
the issue has happened, but avoid getting tons of error messages about
the same problem (as this may affect up to 32 IRQs).
But for debugging it might be good to know which IRQ was affected. I see
two use cases for a guest:
- (De-)activating a single IRQ: we get one message and know which IRQ it
was, so an admin can chase this down to a certain device (driver).
- (De-)activating *every* IRQ in this range (~0): we still get one
message per 32 IRQs, but can see whether it covers SPIs only (IRQ>=32)
and which ones.

So what about a compromise: I use dprintk(XENLOG_G_ERR, "%pv ...), print
the (first) IRQ and the *value* to be written. So a knowledgeable admin
can tell whether it's a single IRQ or a "clear/set-all" case. That
should also give enough info for debugging, but keeps it short.

Does that sound OK?

Cheers,
Andre.

>> +        }
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +        if ( bail_out )
>> +            return;
>> +    }
>> +}
>> +
>> +/*
>> + * We don't actually support setting the active state of an IRQ (yet).
>> + * We check whether this MMIO access would actually affect any
>> non-active IRQ,
>> + * and only print our warning in this case.
>> + */
>> +void vgic_mmio_write_sactive(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val)
> 
> See my comments on cactive.
> 
>> +{
>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>> +    unsigned int i;
>> +    bool bail_out = false;
>> +
>> +    for_each_set_bit( i, &val, len * 8 )
>> +    {
>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>> + i);
>> +
>> +        /*
>> +         * If we know that the IRQ is not active or we can't be sure
>> about
>> +         * it (because it is currently in a CPU), log the not properly
>> +         * emulated MMIO access.
>> +         */
>> +        if ( !irq->active || irq->vcpu )
>> +        {
>> +            gdprintk(XENLOG_ERR,
>> +                     "%pv: vGICD: IRQ%d: setting active state not
>> supported\n",
>> +                     vcpu, irq->intid);
>> +            bail_out = true;
>> +        }
>> +
>> +        vgic_put_irq(vcpu->domain, irq);
>> +        if ( bail_out )
>> +            return;
>> +    }
>> +}
>> +
>>   static int match_region(const void *key, const void *elt)
>>   {
>>       const unsigned int offset = (unsigned long)key;
>> diff --git a/xen/arch/arm/vgic/vgic-mmio.h
>> b/xen/arch/arm/vgic/vgic-mmio.h
>> index 4465f3b7e5..8604720628 100644
>> --- a/xen/arch/arm/vgic/vgic-mmio.h
>> +++ b/xen/arch/arm/vgic/vgic-mmio.h
>> @@ -118,6 +118,17 @@ void vgic_mmio_write_cpending(struct vcpu *vcpu,
>>                                 paddr_t addr, unsigned int len,
>>                                 unsigned long val);
>>   +unsigned long vgic_mmio_read_active(struct vcpu *vcpu,
>> +                                    paddr_t addr, unsigned int len);
>> +
>> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val);
>> +
>> +void vgic_mmio_write_sactive(struct vcpu *vcpu,
>> +                             paddr_t addr, unsigned int len,
>> +                             unsigned long val);
>> +
>>   unsigned int vgic_v2_init_dist_iodev(struct vgic_io_device *dev);
>>     #endif
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Julien Grall]

Hi Andre,

On 13/03/18 17:02, Andre Przywara wrote:
> On 08/03/18 15:39, Julien Grall wrote:
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> +/*
>>> + * We don't actually support clearing the active state of an IRQ (yet).
>>> + * However there is a chance that most guests use this for
>>> initialization.
>>> + * We check whether this MMIO access would actually affect any active
>>> IRQ,
>>> + * and only print our warning in this case. So clearing already
>>> non-active
>>> + * IRQs would not be moaned about in the logs.
>>> + */
>>> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
>>> +                             paddr_t addr, unsigned int len,
>>> +                             unsigned long val)
>>> +{
>>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>>> +    unsigned int i;
>>> +    bool bail_out = false;
>>> +
>>> +    for_each_set_bit( i, &val, len * 8 )
>>> +    {
>>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>>> + i);
>>> +
>>> +        /*
>>> +         * If we know that the IRQ is active or we can't be sure about
>>> +         * it (because it is currently in a CPU), log the not properly
>>> +         * emulated MMIO access.
>>> +         */
>>> +        if ( irq->active || irq->vcpu )
>>> +        {
>>> +            gdprintk(XENLOG_ERR,
>>> +                     "%pv: vGICD: IRQ%d: clearing active state not
>>> supported\n",
>>
>> s/%d/%u/
>>
>>> +                     vcpu, irq->intid);
>>
>> gdprintk will always print the vCPU. Thought it is the current which
>> might be different from vcpu (mostly in the re-dist case).
> 
> Ah, thanks. I always get confused about what which version of *printk does.
> 
>> So I would use dprintk(XENLOG_G_ERR, "%pv: ..."). I would even be tempt
>> to use printk(....) so we can spot potential issue on non-debug build.
> 
> Well, in the true spirit of Xen paranoia ;-) I wanted to avoid a guest
> spamming the console.

The guests messages are rate limited.

> And in the end there is nothing a administrator
> could really do about it. In my experience those messages tend to really
> scare users ("I could boot Dom0 but I see those error messages ....").

Xen message are not only here for the administrator, they are also here 
to help for the developer to get log to dissect.

I think that particular message should be printed in non-debug build 
because if the interrupt was active and can't clear it. Then something
will go wrong later on.

> 
>>> +            bail_out = true;
>>
>> I admit the bailout is a bit weird here. You would only print the
>> warning for the first activated IRQ and give the impression it is fine
>> for the rest. So maybe you want to drop IRQ%d?
> 
> For the above reasons I wanted to keep them concise, so that we see that
> the issue has happened, but avoid getting tons of error messages about
> the same problem (as this may affect up to 32 IRQs).
> But for debugging it might be good to know which IRQ was affected. I see
> two use cases for a guest:
> - (De-)activating a single IRQ: we get one message and know which IRQ it
> was, so an admin can chase this down to a certain device (driver).
> - (De-)activating *every* IRQ in this range (~0): we still get one
> message per 32 IRQs, but can see whether it covers SPIs only (IRQ>=32)
> and which ones.
> 
> So what about a compromise: I use dprintk(XENLOG_G_ERR, "%pv ...), print
> the (first) IRQ and the *value* to be written. So a knowledgeable admin
> can tell whether it's a single IRQ or a "clear/set-all" case. That
> should also give enough info for debugging, but keeps it short.

I can't see how a knowledgeable admin will be able to know that IRQ 2 is 
active with just the register value.

> 
> Does that sound OK?

I would still prefer the one per IRQ and using printk(XENLOG_G_*). I 
don't much care about the spam, see why above.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Julien Grall]

On 13/03/18 17:14, Julien Grall wrote:
> On 13/03/18 17:02, Andre Przywara wrote:
>> On 08/03/18 15:39, Julien Grall wrote:
>>> On 05/03/18 16:03, Andre Przywara wrote:
>>> I admit the bailout is a bit weird here. You would only print the
>>> warning for the first activated IRQ and give the impression it is fine
>>> for the rest. So maybe you want to drop IRQ%d?
>>
>> For the above reasons I wanted to keep them concise, so that we see that
>> the issue has happened, but avoid getting tons of error messages about
>> the same problem (as this may affect up to 32 IRQs).
>> But for debugging it might be good to know which IRQ was affected. I see
>> two use cases for a guest:
>> - (De-)activating a single IRQ: we get one message and know which IRQ it
>> was, so an admin can chase this down to a certain device (driver).
>> - (De-)activating *every* IRQ in this range (~0): we still get one
>> message per 32 IRQs, but can see whether it covers SPIs only (IRQ>=32)
>> and which ones.
>>
>> So what about a compromise: I use dprintk(XENLOG_G_ERR, "%pv ...), print
>> the (first) IRQ and the *value* to be written. So a knowledgeable admin
>> can tell whether it's a single IRQ or a "clear/set-all" case. That
>> should also give enough info for debugging, but keeps it short.
> 
> I can't see how a knowledgeable admin will be able to know that IRQ 2 is 
> active with just the register value.
> 
>>
>> Does that sound OK?
> 
> I would still prefer the one per IRQ and using printk(XENLOG_G_*). I  > don't much care about the spam, see why above.

XENLOG_G_DEBUG is a good candidate actually.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Andre Przywara]

Hi,

On 13/03/18 17:14, Julien Grall wrote:
> Hi Andre,
> 
> On 13/03/18 17:02, Andre Przywara wrote:
>> On 08/03/18 15:39, Julien Grall wrote:
>>> On 05/03/18 16:03, Andre Przywara wrote:
>>>> +/*
>>>> + * We don't actually support clearing the active state of an IRQ
>>>> (yet).
>>>> + * However there is a chance that most guests use this for
>>>> initialization.
>>>> + * We check whether this MMIO access would actually affect any active
>>>> IRQ,
>>>> + * and only print our warning in this case. So clearing already
>>>> non-active
>>>> + * IRQs would not be moaned about in the logs.
>>>> + */
>>>> +void vgic_mmio_write_cactive(struct vcpu *vcpu,
>>>> +                             paddr_t addr, unsigned int len,
>>>> +                             unsigned long val)
>>>> +{
>>>> +    uint32_t intid = VGIC_ADDR_TO_INTID(addr, 1);
>>>> +    unsigned int i;
>>>> +    bool bail_out = false;
>>>> +
>>>> +    for_each_set_bit( i, &val, len * 8 )
>>>> +    {
>>>> +        struct vgic_irq *irq = vgic_get_irq(vcpu->domain, vcpu, intid
>>>> + i);
>>>> +
>>>> +        /*
>>>> +         * If we know that the IRQ is active or we can't be sure about
>>>> +         * it (because it is currently in a CPU), log the not properly
>>>> +         * emulated MMIO access.
>>>> +         */
>>>> +        if ( irq->active || irq->vcpu )
>>>> +        {
>>>> +            gdprintk(XENLOG_ERR,
>>>> +                     "%pv: vGICD: IRQ%d: clearing active state not
>>>> supported\n",
>>>
>>> s/%d/%u/
>>>
>>>> +                     vcpu, irq->intid);
>>>
>>> gdprintk will always print the vCPU. Thought it is the current which
>>> might be different from vcpu (mostly in the re-dist case).
>>
>> Ah, thanks. I always get confused about what which version of *printk
>> does.
>>
>>> So I would use dprintk(XENLOG_G_ERR, "%pv: ..."). I would even be tempt
>>> to use printk(....) so we can spot potential issue on non-debug build.
>>
>> Well, in the true spirit of Xen paranoia ;-) I wanted to avoid a guest
>> spamming the console.
> 
> The guests messages are rate limited.

Ah ...

>> And in the end there is nothing a administrator
>> could really do about it. In my experience those messages tend to really
>> scare users ("I could boot Dom0 but I see those error messages ....").
> 
> Xen message are not only here for the administrator, they are also here
> to help for the developer to get log to dissect.

Sure, see below ...

> I think that particular message should be printed in non-debug build
> because if the interrupt was active and can't clear it. Then something
> will go wrong later on.

Fair enough, if it's rate limited ...

>>>> +            bail_out = true;
>>>
>>> I admit the bailout is a bit weird here. You would only print the
>>> warning for the first activated IRQ and give the impression it is fine
>>> for the rest. So maybe you want to drop IRQ%d?
>>
>> For the above reasons I wanted to keep them concise, so that we see that
>> the issue has happened, but avoid getting tons of error messages about
>> the same problem (as this may affect up to 32 IRQs).
>> But for debugging it might be good to know which IRQ was affected. I see
>> two use cases for a guest:
>> - (De-)activating a single IRQ: we get one message and know which IRQ it
>> was, so an admin can chase this down to a certain device (driver).
>> - (De-)activating *every* IRQ in this range (~0): we still get one
>> message per 32 IRQs, but can see whether it covers SPIs only (IRQ>=32)
>> and which ones.
>>
>> So what about a compromise: I use dprintk(XENLOG_G_ERR, "%pv ...), print
>> the (first) IRQ and the *value* to be written. So a knowledgeable admin
>> can tell whether it's a single IRQ or a "clear/set-all" case. That
>> should also give enough info for debugging, but keeps it short.
> 
> I can't see how a knowledgeable admin will be able to know that IRQ 2 is
> active with just the register value.

Well, I was assuming that a really knowledgeable admin would somehow
forward the error message either to the ML or at least to $search_engine.
...

>> Does that sound OK?
> 
> I would still prefer the one per IRQ and using printk(XENLOG_G_*).

I really don't think one per IRQ is too useful. A developer however can
easily deal with "IRQ45, value: 0x00802000" from a log. And can deduce
from there that it's about IRQ45 and IRQ55. Following the example above
you would either see one "IRQ32, value: 0xffffffff" or "IRQ 45, value:
0x00002000".
That looks like a good compromise between readability (having the IRQ
number for admins) and brevity.

I changed it now to output:
%pv: vGICD: clearing active state not supported (IRQ%u, value: 0x%08lx)

> I don't much care about the spam, see why above.

Having them on the console between Dom0 messages is really scary, but
not helpful if it's *more* than one. Since it's a known limitation of
the VGIC emulation, not a real "error" in that sense.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Julien Grall]

Hi,

On 13/03/18 17:34, Andre Przywara wrote:
> On 13/03/18 17:14, Julien Grall wrote:
>> On 13/03/18 17:02, Andre Przywara wrote:
>>> On 08/03/18 15:39, Julien Grall wrote:
>>>> On 05/03/18 16:03, Andre Przywara wrote:
>> I can't see how a knowledgeable admin will be able to know that IRQ 2 is
>> active with just the register value.
> 
> Well, I was assuming that a really knowledgeable admin would somehow
> forward the error message either to the ML or at least to $search_engine.
> ...

Surely, but it does not mean the message should be clueless for the 
developer. I would rather no spent 10 min to try to find out what's 
going on where reading logs...

> 
>>> Does that sound OK?
>>
>> I would still prefer the one per IRQ and using printk(XENLOG_G_*).
> 
> I really don't think one per IRQ is too useful. A developer however can
> easily deal with "IRQ45, value: 0x00802000" from a log. And can deduce
> from there that it's about IRQ45 and IRQ55. Following the example above
> you would either see one "IRQ32, value: 0xffffffff" or "IRQ 45, value:
> 0x00002000".

I still can't see how the developer would know the IRQ55 is active or 
not. That's the whole purpose of the per IRQ.

> That looks like a good compromise between readability (having the IRQ
> number for admins) and brevity.

You may save 10 characters on the logs, you likely going to waste 10 min 
of the developer to understand what that messages really mean.

> 
> I changed it now to output:
> %pv: vGICD: clearing active state not supported (IRQ%u, value: 0x%08lx)
> 
>> I don't much care about the spam, see why above.
> 
> Having them on the console between Dom0 messages is really scary, but
> not helpful if it's *more* than one. Since it's a known limitation of
> the VGIC emulation, not a real "error" in that sense.

It is the same as having any Xen messages interleaved with Dom0 
messages. If the user is not happy with that, then it can divert Dom0 
console to another UART.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 51/57] ARM: new VGIC: Add preliminary stub implementation

[Julien Grall]

Hi,

On 03/13/2018 03:55 PM, Andre Przywara wrote:
> Hi,
> 
> On 09/03/18 18:18, Julien Grall wrote:
>> Hi Andre,
>>
>> On 05/03/18 16:04, Andre Przywara wrote:
>>> The ARM arch code requires an interrupt controller emulation to implement
>>> vgic_clear_pending_irqs(), although it is suspected that it is actually
>>> not necessary. Go with a stub for now to make the linker happy.
>>
>> The implementation of that function is fundamentally wrong on the
>> current vGIC for a few reasons:
>>      - lr_mask is reset but the LRs are not. This means when we context
>> switch back, the LR might still be written and injecting unexpected
>> interrupt (whoops).
>>      - both lists (inflight and pending) are cleared which means that a
>> physical interrupt pending on that vCPU is lost forever (stay active in
>> the physical so never going to fire again).
>>
>> Furthermore, I don't think that Xen business to reset the GIC on cpu_on.
>> If anything should be done, then is it on CPU_off to migrate the current
>> interrupts to another vCPU. But IIRC the OS is responsible for that.
>>
>> So I would kill that function. Any opinions?
> 
> So I guess given that the patch is pretty small, we are good with
> keeping that for now, and solve this together with the old VGIC later.

I am ok with that.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Andre Przywara]

Hi,

On 13/03/18 17:42, Julien Grall wrote:
> Hi,
> 
> On 13/03/18 17:34, Andre Przywara wrote:
>> On 13/03/18 17:14, Julien Grall wrote:
>>> On 13/03/18 17:02, Andre Przywara wrote:
>>>> On 08/03/18 15:39, Julien Grall wrote:
>>>>> On 05/03/18 16:03, Andre Przywara wrote:
>>> I can't see how a knowledgeable admin will be able to know that IRQ 2 is
>>> active with just the register value.
>>
>> Well, I was assuming that a really knowledgeable admin would somehow
>> forward the error message either to the ML or at least to $search_engine.
>> ...
> 
> Surely, but it does not mean the message should be clueless for the
> developer. I would rather no spent 10 min to try to find out what's
> going on where reading logs...
> 
>>
>>>> Does that sound OK?
>>>
>>> I would still prefer the one per IRQ and using printk(XENLOG_G_*).
>>
>> I really don't think one per IRQ is too useful. A developer however can
>> easily deal with "IRQ45, value: 0x00802000" from a log. And can deduce
>> from there that it's about IRQ45 and IRQ55. Following the example above
>> you would either see one "IRQ32, value: 0xffffffff" or "IRQ 45, value:
>> 0x00002000".
> 
> I still can't see how the developer would know the IRQ55 is active or
> not. That's the whole purpose of the per IRQ.
> 
>> That looks like a good compromise between readability (having the IRQ
>> number for admins) and brevity.
> 
> You may save 10 characters on the logs, you likely going to waste 10 min
> of the developer to understand what that messages really mean.

It's not about 10 characters, it's about 31 *lines*.
At the moment a write to I[CS]ACTIVER triggers *one* line in the log:
%pv: vGICD: unhandled word write %#"PRIregister" to ISACTIVER%d

Now a write to this register would potentially trigger *32* lines:
%pv: vGICD: IRQ%u: setting active state not supported

By dumping the line as I proposed, I basically mimic the current line,
plus give some information about one IRQ affected:
%pv: vGICD: setting active state not supported (IRQ%u, value: 0x%08lx)

So this is not a regression, but an improvement.

Cheers,
Andre.

>> I changed it now to output:
>> %pv: vGICD: clearing active state not supported (IRQ%u, value: 0x%08lx)
>>
>>> I don't much care about the spam, see why above.
>>
>> Having them on the console between Dom0 messages is really scary, but
>> not helpful if it's *more* than one. Since it's a known limitation of
>> the VGIC emulation, not a real "error" in that sense.
> 
> It is the same as having any Xen messages interleaved with Dom0
> messages. If the user is not happy with that, then it can divert Dom0
> console to another UART.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 39/57] ARM: new VGIC: Add ACTIVE registers handlers

[Julien Grall]

Hi,

On 03/14/2018 02:30 PM, Andre Przywara wrote:
> Hi,
> 
> On 13/03/18 17:42, Julien Grall wrote:
>> Hi,
>>
>> On 13/03/18 17:34, Andre Przywara wrote:
>>> On 13/03/18 17:14, Julien Grall wrote:
>>>> On 13/03/18 17:02, Andre Przywara wrote:
>>>>> On 08/03/18 15:39, Julien Grall wrote:
>>>>>> On 05/03/18 16:03, Andre Przywara wrote:
>>>> I can't see how a knowledgeable admin will be able to know that IRQ 2 is
>>>> active with just the register value.
>>>
>>> Well, I was assuming that a really knowledgeable admin would somehow
>>> forward the error message either to the ML or at least to $search_engine.
>>> ...
>>
>> Surely, but it does not mean the message should be clueless for the
>> developer. I would rather no spent 10 min to try to find out what's
>> going on where reading logs...
>>
>>>
>>>>> Does that sound OK?
>>>>
>>>> I would still prefer the one per IRQ and using printk(XENLOG_G_*).
>>>
>>> I really don't think one per IRQ is too useful. A developer however can
>>> easily deal with "IRQ45, value: 0x00802000" from a log. And can deduce
>>> from there that it's about IRQ45 and IRQ55. Following the example above
>>> you would either see one "IRQ32, value: 0xffffffff" or "IRQ 45, value:
>>> 0x00002000".
>>
>> I still can't see how the developer would know the IRQ55 is active or
>> not. That's the whole purpose of the per IRQ.
>>
>>> That looks like a good compromise between readability (having the IRQ
>>> number for admins) and brevity.
>>
>> You may save 10 characters on the logs, you likely going to waste 10 min
>> of the developer to understand what that messages really mean.
> 
> It's not about 10 characters, it's about 31 *lines*.
> At the moment a write to I[CS]ACTIVER triggers *one* line in the log:
> %pv: vGICD: unhandled word write %#"PRIregister" to ISACTIVER%d
> 
> Now a write to this register would potentially trigger *32* lines:
> %pv: vGICD: IRQ%u: setting active state not supported

Very likely there will 0 lines printed because clearing an active 
interrupt should never happen in Xen guest today. So if there is 32 
lines printed, then having 32 lines in the log is your last of your concern.

> 
> By dumping the line as I proposed, I basically mimic the current line,
> plus give some information about one IRQ affected:
> %pv: vGICD: setting active state not supported (IRQ%u, value: 0x%08lx)
> 
> So this is not a regression, but an improvement.

I never said it was a regression. I said your new message and bail out 
is counter-intuitive because the developer can't guess if there are 
other IRQs active with just "value".

If you want to make an improvement, do it properly. In that case, I am 
only asking to drop the counter-intuitive bail_out.

Cheers.

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 28/57] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Andre Przywara]

Hi,

On 06/03/18 18:13, Julien Grall wrote:
> Hi Andre,
> 
> On 05/03/18 16:03, Andre Przywara wrote:
>> The new VGIC implementation centers around a struct vgic_irq instance
>> per virtual IRQ.
>> Provide a function to retrieve the right instance for a given IRQ
>> number and (in case of private interrupts) the right VCPU.
>> This also includes the corresponding put function, which does nothing
>> for private interrupts and SPIs, but handles the ref-counting for LPIs.
>>
>> This is based on Linux commit 64a959d66e47, written by Christoffer Dall.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>> ---
>> Changelog RFC ... v1:
>> - add kernel-doc comments to exported functions
>> - adapt to previous changes (new_vgic.h, arch_vcpu member name)
>> - use ASSERT_UNREACHABLE
>>
>>   xen/arch/arm/vgic/vgic.c | 124
>> +++++++++++++++++++++++++++++++++++++++++++++++
>>   xen/arch/arm/vgic/vgic.h |  41 ++++++++++++++++
>>   2 files changed, 165 insertions(+)
>>   create mode 100644 xen/arch/arm/vgic/vgic.c
>>   create mode 100644 xen/arch/arm/vgic/vgic.h
>>
>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>> new file mode 100644
>> index 0000000000..ace30f78d0
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic/vgic.c
>> @@ -0,0 +1,124 @@
>> +/*
>> + * Copyright (C) 2015, 2016 ARM Ltd.
>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <asm/bug.h>
>> +#include <asm/new_vgic.h>
>> +#include <xen/sched.h>
>> +
>> +#include "vgic.h"
>> +
>> +/*
>> + * Iterate over the VM's list of mapped LPIs to find the one with a
>> + * matching interrupt ID and return a reference to the IRQ structure.
>> + */
>> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
>> +{
>> +    struct vgic_dist *dist = &d->arch.vgic;
>> +    struct vgic_irq *irq = NULL;
>> +
>> +    spin_lock(&dist->lpi_list_lock);
>> +
>> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
> 
> I am still not a big fan of the list solution. Strictly speaking nobody
> is populating that list and likely going to be too slow in Xen case (I
> am thinking for the hardware domain). So I think I would prefer to see
> the LPI related code disappear for this cut. This could easily be added
> back as they are standalone.

I was thinking about that, but dismissed the idea:
Considering LPIs as first class citizens is a crucial property of the
new VGIC and actually the main driver for its creation: the refcounting
is solely in for that purpose, and the per-IRQ data structure and its
lock are mainly driven by it. So removing the LPI support completely
will make the refcounting look very awkward (since useless and unused).
Consequently one would need to remove that as well. In the worst case we
run into issues with unused functions, like vgic_get_irq_kref().

I already removed a lot of code from the KVM VGIC, which I feel we will
need back one day. So I really like to keep this one in, and be it just
for documentation how the refcounting is supposed to be used.
I will add a comment to this respect.

And I believe replacing the list_for_each_entry() with something more
sophisticated is the least of our problems later on.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 28/57] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Julien Grall]

On 03/19/2018 05:32 PM, Andre Przywara wrote:
> Hi,

Hi,

> On 06/03/18 18:13, Julien Grall wrote:
>> Hi Andre,
>>
>> On 05/03/18 16:03, Andre Przywara wrote:
>>> The new VGIC implementation centers around a struct vgic_irq instance
>>> per virtual IRQ.
>>> Provide a function to retrieve the right instance for a given IRQ
>>> number and (in case of private interrupts) the right VCPU.
>>> This also includes the corresponding put function, which does nothing
>>> for private interrupts and SPIs, but handles the ref-counting for LPIs.
>>>
>>> This is based on Linux commit 64a959d66e47, written by Christoffer Dall.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>> ---
>>> Changelog RFC ... v1:
>>> - add kernel-doc comments to exported functions
>>> - adapt to previous changes (new_vgic.h, arch_vcpu member name)
>>> - use ASSERT_UNREACHABLE
>>>
>>>    xen/arch/arm/vgic/vgic.c | 124
>>> +++++++++++++++++++++++++++++++++++++++++++++++
>>>    xen/arch/arm/vgic/vgic.h |  41 ++++++++++++++++
>>>    2 files changed, 165 insertions(+)
>>>    create mode 100644 xen/arch/arm/vgic/vgic.c
>>>    create mode 100644 xen/arch/arm/vgic/vgic.h
>>>
>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>> new file mode 100644
>>> index 0000000000..ace30f78d0
>>> --- /dev/null
>>> +++ b/xen/arch/arm/vgic/vgic.c
>>> @@ -0,0 +1,124 @@
>>> +/*
>>> + * Copyright (C) 2015, 2016 ARM Ltd.
>>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License version 2 as
>>> + * published by the Free Software Foundation.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program.  If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <asm/bug.h>
>>> +#include <asm/new_vgic.h>
>>> +#include <xen/sched.h>
>>> +
>>> +#include "vgic.h"
>>> +
>>> +/*
>>> + * Iterate over the VM's list of mapped LPIs to find the one with a
>>> + * matching interrupt ID and return a reference to the IRQ structure.
>>> + */
>>> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
>>> +{
>>> +    struct vgic_dist *dist = &d->arch.vgic;
>>> +    struct vgic_irq *irq = NULL;
>>> +
>>> +    spin_lock(&dist->lpi_list_lock);
>>> +
>>> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
>>
>> I am still not a big fan of the list solution. Strictly speaking nobody
>> is populating that list and likely going to be too slow in Xen case (I
>> am thinking for the hardware domain). So I think I would prefer to see
>> the LPI related code disappear for this cut. This could easily be added
>> back as they are standalone.
> 
> I was thinking about that, but dismissed the idea:
> Considering LPIs as first class citizens is a crucial property of the
> new VGIC and actually the main driver for its creation: the refcounting
> is solely in for that purpose, and the per-IRQ data structure and its
> lock are mainly driven by it. So removing the LPI support completely
> will make the refcounting look very awkward (since useless and unused).
> Consequently one would need to remove that as well. In the worst case we
> run into issues with unused functions, like vgic_get_irq_kref().
> 
> I already removed a lot of code from the KVM VGIC, which I feel we will
> need back one day. So I really like to keep this one in, and be it just
> for documentation how the refcounting is supposed to be used.
> I will add a comment to this respect.
> And I believe replacing the list_for_each_entry() with something more
> sophisticated is the least of our problems later on.

Using a list for LPIs means the search is O(n). That search will be done 
each time you want to get the LPI information.

While in guest you may have limited LPIs, this will not be the case for 
the hardware domain. Indeed that domain has access to all devices and 
likely going to use a lot of MSIs.

So I still can't understand why you dismiss that problem.

To be honest, I don't mind to keep the refcounting around. What I don't 
want to see in Xen is using a data structure that we know will not fit 
Xen (even for well-behave guests).

You may remove a lot of code, and re-add the code later but that's the 
price to convert KVM to Xen code. It makes much more sense to get all 
the LPIs infrastructure together.

And to be clear, what I am asking for is implementing vgic_get_lpi with 
return NULL + a comment.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 28/57] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Andre Przywara]

Hi,

On 19/03/18 21:53, Julien Grall wrote:
> 
> 
> On 03/19/2018 05:32 PM, Andre Przywara wrote:
>> Hi,
> 
> Hi,
> 
>> On 06/03/18 18:13, Julien Grall wrote:
>>> Hi Andre,
>>>
>>> On 05/03/18 16:03, Andre Przywara wrote:
>>>> The new VGIC implementation centers around a struct vgic_irq instance
>>>> per virtual IRQ.
>>>> Provide a function to retrieve the right instance for a given IRQ
>>>> number and (in case of private interrupts) the right VCPU.
>>>> This also includes the corresponding put function, which does nothing
>>>> for private interrupts and SPIs, but handles the ref-counting for LPIs.
>>>>
>>>> This is based on Linux commit 64a959d66e47, written by Christoffer
>>>> Dall.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
>>>> ---
>>>> Changelog RFC ... v1:
>>>> - add kernel-doc comments to exported functions
>>>> - adapt to previous changes (new_vgic.h, arch_vcpu member name)
>>>> - use ASSERT_UNREACHABLE
>>>>
>>>>    xen/arch/arm/vgic/vgic.c | 124
>>>> +++++++++++++++++++++++++++++++++++++++++++++++
>>>>    xen/arch/arm/vgic/vgic.h |  41 ++++++++++++++++
>>>>    2 files changed, 165 insertions(+)
>>>>    create mode 100644 xen/arch/arm/vgic/vgic.c
>>>>    create mode 100644 xen/arch/arm/vgic/vgic.h
>>>>
>>>> diff --git a/xen/arch/arm/vgic/vgic.c b/xen/arch/arm/vgic/vgic.c
>>>> new file mode 100644
>>>> index 0000000000..ace30f78d0
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/vgic/vgic.c
>>>> @@ -0,0 +1,124 @@
>>>> +/*
>>>> + * Copyright (C) 2015, 2016 ARM Ltd.
>>>> + * Imported from Linux ("new" KVM VGIC) and heavily adapted to Xen.
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or
>>>> modify
>>>> + * it under the terms of the GNU General Public License version 2 as
>>>> + * published by the Free Software Foundation.
>>>> + *
>>>> + * This program is distributed in the hope that it will be useful,
>>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>>> + * GNU General Public License for more details.
>>>> + *
>>>> + * You should have received a copy of the GNU General Public License
>>>> + * along with this program.  If not, see
>>>> <http://www.gnu.org/licenses/>.
>>>> + */
>>>> +
>>>> +#include <asm/bug.h>
>>>> +#include <asm/new_vgic.h>
>>>> +#include <xen/sched.h>
>>>> +
>>>> +#include "vgic.h"
>>>> +
>>>> +/*
>>>> + * Iterate over the VM's list of mapped LPIs to find the one with a
>>>> + * matching interrupt ID and return a reference to the IRQ structure.
>>>> + */
>>>> +static struct vgic_irq *vgic_get_lpi(struct domain *d, u32 intid)
>>>> +{
>>>> +    struct vgic_dist *dist = &d->arch.vgic;
>>>> +    struct vgic_irq *irq = NULL;
>>>> +
>>>> +    spin_lock(&dist->lpi_list_lock);
>>>> +
>>>> +    list_for_each_entry( irq, &dist->lpi_list_head, lpi_list )
>>>
>>> I am still not a big fan of the list solution. Strictly speaking nobody
>>> is populating that list and likely going to be too slow in Xen case (I
>>> am thinking for the hardware domain). So I think I would prefer to see
>>> the LPI related code disappear for this cut. This could easily be added
>>> back as they are standalone.
>>
>> I was thinking about that, but dismissed the idea:
>> Considering LPIs as first class citizens is a crucial property of the
>> new VGIC and actually the main driver for its creation: the refcounting
>> is solely in for that purpose, and the per-IRQ data structure and its
>> lock are mainly driven by it. So removing the LPI support completely
>> will make the refcounting look very awkward (since useless and unused).
>> Consequently one would need to remove that as well. In the worst case we
>> run into issues with unused functions, like vgic_get_irq_kref().
>>
>> I already removed a lot of code from the KVM VGIC, which I feel we will
>> need back one day. So I really like to keep this one in, and be it just
>> for documentation how the refcounting is supposed to be used.
>> I will add a comment to this respect.
>> And I believe replacing the list_for_each_entry() with something more
>> sophisticated is the least of our problems later on.
> 
> Using a list for LPIs means the search is O(n). That search will be done
> each time you want to get the LPI information.
> 
> While in guest you may have limited LPIs, this will not be the case for
> the hardware domain. Indeed that domain has access to all devices and
> likely going to use a lot of MSIs.
> 
> So I still can't understand why you dismiss that problem.

I think you got me wrong. I am happy with eventually using another data
structure, but want to keep the current list stub in as documentation,
as it motivates a crucial parts of the VGIC, namely refcounting and the
possible dynamic allocation of struct vgic_irq's. If that would be left
out, anyone reading the code would seriously scratch their head why we
have all this code, possibly removing it.
Actually the list we are talking about consists of a root pointer and
this single "list_for_each_entry()" call in vgic_get_lpi(). That's it.
There is no code to populate the list at the moment. So I consider this
more *documentation* on how it's supposed to be done, especially the
refcounting, allocation, and where to initialise things. And as I
mentioned before, a list is a good example, because it's easy to
understand: You immediately understand what list_for_each_entry() means,
and the code shows where to take and drop the locks and how to refcount.

Whoever implements LPI support, can just grep for lpi_list_head and
replace the respective code with something more appropriate.
I added comments to that regard.

> To be honest, I don't mind to keep the refcounting around. What I don't
> want to see in Xen is using a data structure that we know will not fit
> Xen (even for well-behave guests).

As said, no one is really using this data structure ;-)
But there is good code to keep in: the root pointer in the struct,
where the data structure should be initialized and freed, how the
refcounting is supposed to be done and why we have vgic_put_irq() and
what it should do.

> You may remove a lot of code, and re-add the code later but that's the
> price to convert KVM to Xen code. It makes much more sense to get all
> the LPIs infrastructure together.

Re-adding code is not trivial, due to the changes required to make it
work in Xen. For instance I ported the refcounting to Xen. So I prefer
to not throw away the work I already did.

> And to be clear, what I am asking for is implementing vgic_get_lpi with
> return NULL + a comment.

It does that already: lpi_list_head is always empty, so it will skip the
loop and eventually return NULL. This also tells the reader how to
handle non-allocated LPIs.

I am happy to add a comment saying that a list is not the right data
structure.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

Re: [PATCH 28/57] ARM: new VGIC: Add acccessor to new struct vgic_irq instance

[Julien Grall]

[-- Attachment #1.1: Type: text/plain, Size: 496 bytes --]

Hi,

Sorry for the formatting.

On 20 Mar 2018 19:01, "Andre Przywara" <andre.przywara@linaro.org> wrote:


I am happy to add a comment saying that a list is not the right data
structure.


Fair enough. I won't argue more, let's keep the list and add a comment. I
do want to see that series merged in Xen 4.11.

Cheers,


Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel

[-- Attachment #1.2: Type: text/html, Size: 1518 bytes --]

[-- Attachment #2: Type: text/plain, Size: 157 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel