All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v2 00/27] arm64: Dom0 ITS emulation
@ 2017-03-16 11:20 Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT Andre Przywara
                   ` (26 more replies)
  0 siblings, 27 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

a revised version of the series to emulate an ARM GICv3 ITS interrupt
controller, for Dom0 only at the moment.
The ITS is an interrupt controller widget providing a sophisticated way
of dealing with MSIs in a scalable manner. For more explanations, see
below.
Some remarks on this version:
- I kept the command line parameters and Kconfig options in for now.
  I believe we have to impose _some_ kind of limit on the number of
  supported LPIs and devices, Linux does certainly do as well. Until
  we know some one-fits-all numbers and have confirmation from several
  platforms, I'd like to keep this easily configurable.
  I am happy to remove this from the final drop should people still have
  concerns.
- Two-level tables for devices are missing from this version. This
  shouldn't be hard to implement, though, I just didn't find the time
  yet. I suspect it will be part of the next drop.
- To allow mapping devices with non-trivial device ID mappings (platform
  devices, for instance), we deviate from our premature allocation
  scheme and move to mapping devices on demand upon Dom0's MAPD calls.
  I am not a fan of this approach, but this should solve the immediate
  issues we see on actual hardware. Advantage is that we know the device
  ID and the number of MSIs very precisely.
  I hope that this approach can go away once we have a proper way of
  dealing with that (DT/IORT parsing in Xen), but it definitely should
  stay restricted to Dom0.
  So I removed the code which created device mapping on the PCI physops
  calls, however have the patch still around here in case we need it.
- The teardown path needs some more love. AFAICT Dom0 does not take part
  in the usual domain destroy procedure, so we can't really test this.
  For now I kept one function in to show what needs to be done, but this
  is not plugged anywhere. So to avoid untested and possibly buggy code
  to sneak in, we might think about removing it.
- I tried to exterminate any signed ints and also get rid of any unattended
  (as in not-named) constants, but I am sure I missed some. So let your
  OCD kick in and find all of them! Also I am sure the massive rebasing
  and patch squashing created some new funny bugs and artifacts.

For other changes, see the changelog below.
----------------------------------
This series adds support for emulation of an ARM GICv3 ITS interrupt
controller. For hardware which relies on the ITS to provide interrupts for
its peripherals this code is needed to get a machine booted into Dom0 at
all. ITS emulation for DomUs is only really useful with PCI passthrough,
which is not yet available for ARM. It is expected that this feature
will be co-developed with the ITS DomU code. However this code drop here
considered DomU emulation already, to keep later architectural changes
to a minimum.

Some generic design principles:

* The current GIC code statically allocates structures for each supported
IRQ (both for the host and the guest), which due to the potentially
millions of LPI interrupts is not feasible to copy for the ITS.
So we refrain from introducing the ITS as a first class Xen interrupt
controller, also we don't hold struct irq_desc's or struct pending_irq's
for each possible LPI.
Fortunately LPIs are only interesting to guests, so we get away with
storing only the virtual IRQ number and the guest VCPU for each allocated
host LPI, which can be stashed into one uint64_t. This data is stored in
a two-level table, which is both memory efficient and quick to access.
We hook into the existing IRQ handling and VGIC code to avoid accessing
the normal structures, providing alternative methods for getting the
needed information (priority, is enabled?) for LPIs.
For interrupts which are queued to or are actually in a guest we
allocate struct pending_irq's on demand. As it is expected that only a
very small number of interrupts is ever on a VCPU at the same time, this
seems like the best approach. For now allocated structs are re-used and
held in a linked list. Should it emerge that traversing a linked list
is a performance issue, this can be changed to use a hash table.

* On the guest side we (later will) have to deal with malicious guests
trying to hog Xen with mapping requests for a lot of LPIs, for instance.
As the ITS actually uses system memory for storing status information,
we use this memory (which the guest has to provide) to naturally limit
a guest. For those tables which are page sized (devices, collections (CPUs),
LPI properties) we map those pages into Xen, so we can easily access
them from the virtual GIC code.
Unfortunately the actual interrupt mapping tables are not necessarily
page aligned, also can be much smaller than a page, so mapping all of
them permanently is fiddly. As ITS commands in need to iterate those
tables are pretty rare after all, we for now map them on demand upon
emulating a virtual ITS command. This is acceptable because "mapping"
them is actually very cheap on arm64. Also as we can't properly protect
those areas due to their sub-page-size property, we validate the data
in there before actually using it. The vITS code basically just stores
the data in there which the guest has actually transferred via the
virtual ITS command queue before, so there is no secret revealed nor
does it create an attack vector for a malicious guest.

* An obvious approach to handling some guest ITS commands would be to
propagate them to the host, for instance to map devices and LPIs and
to enable or disable LPIs.
However this (later with DomU support) will create an attack vector, as
a malicious guest could try to fill the host command queue with
propagated commands.
So we try to avoid this situation: Dom0 sending a device mapping (MAPD)
command is the only time we allow queuing commands to the host ITS command
queue, as this seems to be the only reliable way of getting the
required information at the moment. However at the same time we map all
events to LPIs already, also enable them. This avoids sending commands
later at runtime, as we can deal with mappings and LPI enabling/disabling
internally.

As it is expected that the ITS support will become a tech preview in the
first release, there is a Kconfig option to enable it. Also it is
supported on arm64 only, which will most likely not change in the future.
This leads to some hideous constructs like an #ifdef'ed header file with
empty function stubs, I have some hope we can still clean this up.
Also some parameters are config options which can be overridden on the
Xen commandline. This is to support experimentation and adaption to
various platforms, ideally we find either one-size-fits-all values or
find another way of getting rid of this.

This code boots Dom0 on an ARM Fast Model with ITS support. I tried to
address the issues seen by people running the previous version on real
hardware, though couldn't verify this here for myself.
So any testing, bug reports (and possibly even fixes) are very welcome.

The code can also be found on the its/v2 branch here:
git://linux-arm.org/xen-ap.git
http://www.linux-arm.org/git?p=xen-ap.git;a=shortlog;h=refs/heads/its/v2

Cheers,
Andre

Changelog v1 .. v2:
- clean up header file inclusion
- rework host ITS table allocation: observe attributes, many fixes
- remove patch 1 to export __flush_dcache_area, use existing function instead
- use number of LPIs internally instead of number of bits
- keep host_its_list as private as possible
- keep struct its_devices private
- rework gicv3_its_map_guest_devices
- fix rbtree issues
- more error handling and propagation
- cope with GICv4 implementations (but no virtual LPI features!)
- abstract host and guest ITSes by using doorbell addresses
- join per-redistributor variables into one per-CPU structure
- fix data types (unsigned int)
- many minor bug fixes

(Rough) changelog RFC-v2 .. v1:
- split host ITS driver into gic-v3-lpi.c and gic-v3-its.c part
- rename virtual ITS driver file to vgic-v3-its.c
- use macros and named constants for all magic numbers
- use atomic accessors for accessing the host LPI data
- remove leftovers from connecting virtual and host ITSes
- bail out if host ITS is disabled in the DT
- rework map/unmap_guest_pages():
    - split off p2m part as get/put_guest_pages (to be done on allocation)
    - get rid of vmap, using map_domain_page() instead
- delay allocation of virtual tables until actual LPI/ITS enablement
- properly size both virtual and physical tables upon allocation
- fix put_domain() locking issues in physdev_op and LPI handling code
- add and extend comments in various areas
- fix lotsa coding style and white space issues, including comment style
- add locking to data structures not yet covered
- fix various locking issues
- use an rbtree to deal with ITS devices (instead of a list)
- properly handle memory attributes for ITS tables
- handle cacheable/non-cacheable ITS table mappings
- sanitize guest provided ITS/LPI table attributes
- fix breakage on non-GICv2 compatible host GICv3 controllers
- add command line parameters on top of Kconfig options
- properly wait for an ITS to become quiescient before enabling it
- handle host ITS command queue errors
- actually wait for host ITS command completion (READR==WRITER)
- fix ARM32 compilation
- various patch splits and reorderings

*** BLURB HERE ***

Andre Przywara (27):
  ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
  ARM: GICv3: allocate LPI pending and property table
  ARM: GICv3 ITS: allocate device and collection table
  ARM: GICv3 ITS: map ITS command buffer
  ARM: GICv3 ITS: introduce ITS command handling
  ARM: GICv3 ITS: introduce device mapping
  ARM: arm64: activate atomic 64-bit accessors
  ARM: GICv3 ITS: introduce host LPI array
  ARM: GICv3: introduce separate pending_irq structs for LPIs
  ARM: GICv3: forward pending LPIs to guests
  ARM: GICv3: enable ITS and LPIs on the host
  ARM: vGICv3: handle virtual LPI pending and property tables
  ARM: vGICv3: Handle disabled LPIs
  ARM: vGICv3: introduce basic ITS emulation bits
  ARM: vITS: introduce translation table walks
  ARM: vITS: handle CLEAR command
  ARM: vITS: handle INT command
  ARM: vITS: handle MAPC command
  ARM: vITS: handle MAPD command
  ARM: vITS: handle MAPTI command
  ARM: vITS: handle MOVI command
  ARM: vITS: handle DISCARD command
  ARM: vITS: handle INV command
  ARM: vITS: handle INVALL command
  ARM: vITS: create and initialize virtual ITSes for Dom0
  ARM: vITS: create ITS subnodes for Dom0 DT
  ARM: vGIC: advertise LPI support

 docs/misc/xen-command-line.markdown |  17 +
 xen/arch/arm/Kconfig                |  33 ++
 xen/arch/arm/Makefile               |   3 +
 xen/arch/arm/gic-v3-its.c           | 925 ++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3-lpi.c           | 491 +++++++++++++++++++
 xen/arch/arm/gic-v3.c               |  70 ++-
 xen/arch/arm/gic.c                  |   9 +-
 xen/arch/arm/vgic-v3-its.c          | 953 ++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic-v3.c              | 347 +++++++++++--
 xen/arch/arm/vgic.c                 |  68 ++-
 xen/include/asm-arm/atomic.h        |   6 +-
 xen/include/asm-arm/bitops.h        |   1 +
 xen/include/asm-arm/domain.h        |  14 +-
 xen/include/asm-arm/gic.h           |   7 +
 xen/include/asm-arm/gic_v3_defs.h   |  73 ++-
 xen/include/asm-arm/gic_v3_its.h    | 250 ++++++++++
 xen/include/asm-arm/irq.h           |   8 +
 xen/include/asm-arm/vgic.h          |  34 ++
 18 files changed, 3263 insertions(+), 46 deletions(-)
 create mode 100644 xen/arch/arm/gic-v3-its.c
 create mode 100644 xen/arch/arm/gic-v3-lpi.c
 create mode 100644 xen/arch/arm/vgic-v3-its.c
 create mode 100644 xen/include/asm-arm/gic_v3_its.h

-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-21 20:17   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table Andre Przywara
                   ` (25 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Parse the DT GIC subnodes to find every ITS MSI controller the hardware
offers. Store that information in a list to both propagate all of them
later to Dom0, but also to be able to iterate over all ITSes.
This introduces an ITS Kconfig option.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/Kconfig             |  4 +++
 xen/arch/arm/Makefile            |  1 +
 xen/arch/arm/gic-v3-its.c        | 73 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3.c            | 10 +++---
 xen/include/asm-arm/gic_v3_its.h | 67 ++++++++++++++++++++++++++++++++++++
 5 files changed, 151 insertions(+), 4 deletions(-)
 create mode 100644 xen/arch/arm/gic-v3-its.c
 create mode 100644 xen/include/asm-arm/gic_v3_its.h

diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 2e023d1..bf64c61 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -45,6 +45,10 @@ config ACPI
 config HAS_GICV3
 	bool
 
+config HAS_ITS
+        bool "GICv3 ITS MSI controller support"
+        depends on HAS_GICV3
+
 endmenu
 
 menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 7afb8a3..54860e0 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -18,6 +18,7 @@ obj-$(EARLY_PRINTK) += early_printk.o
 obj-y += gic.o
 obj-y += gic-v2.o
 obj-$(CONFIG_HAS_GICV3) += gic-v3.o
+obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
 obj-y += guestcopy.o
 obj-y += hvm.o
 obj-y += io.o
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
new file mode 100644
index 0000000..4056e5b
--- /dev/null
+++ b/xen/arch/arm/gic-v3-its.c
@@ -0,0 +1,73 @@
+/*
+ * xen/arch/arm/gic-v3-its.c
+ *
+ * ARM GICv3 Interrupt Translation Service (ITS) support
+ *
+ * Copyright (C) 2016,2017 - ARM Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; under version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/lib.h>
+#include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
+
+LIST_HEAD(host_its_list);
+
+bool gicv3_its_host_has_its(void)
+{
+    return !list_empty(&host_its_list);
+}
+
+/* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
+void gicv3_its_dt_init(const struct dt_device_node *node)
+{
+    const struct dt_device_node *its = NULL;
+    struct host_its *its_data;
+
+    /*
+     * Check for ITS MSI subnodes. If any, add the ITS register
+     * frames to the ITS list.
+     */
+    dt_for_each_child_node(node, its)
+    {
+        uint64_t addr, size;
+
+        if ( !dt_device_is_compatible(its, "arm,gic-v3-its") )
+            continue;
+
+        if ( dt_device_get_address(its, 0, &addr, &size) )
+            panic("GICv3: Cannot find a valid ITS frame address");
+
+        its_data = xzalloc(struct host_its);
+        if ( !its_data )
+            panic("GICv3: Cannot allocate memory for ITS frame");
+
+        its_data->addr = addr;
+        its_data->size = size;
+        its_data->dt_node = its;
+
+        printk("GICv3: Found ITS @0x%lx\n", addr);
+
+        list_add_tail(&its_data->entry, &host_its_list);
+    }
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 955591b..1512521 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -43,6 +43,7 @@
 #include <asm/device.h>
 #include <asm/gic.h>
 #include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
 #include <asm/cpufeature.h>
 #include <asm/acpi.h>
 
@@ -1228,11 +1229,12 @@ static void __init gicv3_dt_init(void)
      */
     res = dt_device_get_address(node, 1 + gicv3.rdist_count,
                                 &cbase, &csize);
-    if ( res )
-        return;
+    if ( !res )
+        dt_device_get_address(node, 1 + gicv3.rdist_count + 2,
+                              &vbase, &vsize);
 
-    dt_device_get_address(node, 1 + gicv3.rdist_count + 2,
-                          &vbase, &vsize);
+    /* Check for ITS child nodes and build the host ITS list accordingly. */
+    gicv3_its_dt_init(node);
 }
 
 static int gicv3_iomem_deny_access(const struct domain *d)
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
new file mode 100644
index 0000000..765a655
--- /dev/null
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -0,0 +1,67 @@
+/*
+ * ARM GICv3 ITS support
+ *
+ * Andre Przywara <andre.przywara@arm.com>
+ * Copyright (c) 2016,2017 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; under version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#ifndef __ASM_ARM_ITS_H__
+#define __ASM_ARM_ITS_H__
+
+#include <xen/device_tree.h>
+
+/* data structure for each hardware ITS */
+struct host_its {
+    struct list_head entry;
+    const struct dt_device_node *dt_node;
+    paddr_t addr;
+    paddr_t size;
+};
+
+
+#ifdef CONFIG_HAS_ITS
+
+extern struct list_head host_its_list;
+
+/* Parse the host DT and pick up all host ITSes. */
+void gicv3_its_dt_init(const struct dt_device_node *node);
+
+bool gicv3_its_host_has_its(void);
+
+#else
+
+static LIST_HEAD(host_its_list);
+
+static inline void gicv3_its_dt_init(const struct dt_device_node *node)
+{
+}
+
+static inline bool gicv3_its_host_has_its(void)
+{
+    return false;
+}
+
+#endif /* CONFIG_HAS_ITS */
+
+#endif
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-21 21:23   ` Julien Grall
  2017-03-21 22:57   ` Stefano Stabellini
  2017-03-16 11:20 ` [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table Andre Przywara
                   ` (24 subsequent siblings)
  26 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The ARM GICv3 provides a new kind of interrupt called LPIs.
The pending bits and the configuration data (priority, enable bits) for
those LPIs are stored in tables in normal memory, which software has to
provide to the hardware.
Allocate the required memory, initialize it and hand it over to each
redistributor. The maximum number of LPIs to be used can be adjusted with
the command line option "max_lpi_bits", which defaults to a compile time
constant exposed in Kconfig.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 docs/misc/xen-command-line.markdown |   9 ++
 xen/arch/arm/Kconfig                |  15 +++
 xen/arch/arm/Makefile               |   1 +
 xen/arch/arm/gic-v3-lpi.c           | 201 ++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3.c               |  17 +++
 xen/include/asm-arm/bitops.h        |   1 +
 xen/include/asm-arm/gic.h           |   2 +
 xen/include/asm-arm/gic_v3_defs.h   |  52 +++++++++-
 xen/include/asm-arm/gic_v3_its.h    |  14 +++
 9 files changed, 311 insertions(+), 1 deletion(-)
 create mode 100644 xen/arch/arm/gic-v3-lpi.c

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index a11fdf9..619016d 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1158,6 +1158,15 @@ based interrupts. Any higher IRQs will be available for use via PCI MSI.
 ### maxcpus
 > `= <integer>`
 
+### max\_lpi\_bits
+> `= <integer>`
+
+Specifies the number of ARM GICv3 LPI interrupts to allocate on the host,
+presented as the number of bits needed to encode it. This must be at least
+14 and not exceed 32, and each LPI requires one byte (configuration) and
+one pending bit to be allocated.
+Defaults to 20 bits (to cover at most 1048576 interrupts).
+
 ### mce
 > `= <integer>`
 
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index bf64c61..86f7b53 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -49,6 +49,21 @@ config HAS_ITS
         bool "GICv3 ITS MSI controller support"
         depends on HAS_GICV3
 
+config MAX_PHYS_LPI_BITS
+        depends on HAS_ITS
+        int "Maximum bits for GICv3 host LPIs (14-32)"
+        range 14 32
+        default "20"
+        help
+          Specifies the maximum number of LPIs (in bits) Xen should take
+          care of. The host ITS may provide support for a very large number
+          of supported LPIs, for all of which we may not want to allocate
+          memory, so this number here allows to limit this.
+          Xen itself does not know how many LPIs domains will ever need
+          beforehand.
+          This can be overridden on the command line with the max_lpi_bits
+          parameter.
+
 endmenu
 
 menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 54860e0..02a8737 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -19,6 +19,7 @@ obj-y += gic.o
 obj-y += gic-v2.o
 obj-$(CONFIG_HAS_GICV3) += gic-v3.o
 obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
+obj-$(CONFIG_HAS_ITS) += gic-v3-lpi.o
 obj-y += guestcopy.o
 obj-y += hvm.o
 obj-y += io.o
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
new file mode 100644
index 0000000..4f8414b
--- /dev/null
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -0,0 +1,201 @@
+/*
+ * xen/arch/arm/gic-v3-lpi.c
+ *
+ * ARM GICv3 Locality-specific Peripheral Interrupts (LPI) support
+ *
+ * Copyright (C) 2016,2017 - ARM Ltd
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; under version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/lib.h>
+#include <xen/mm.h>
+#include <asm/gic.h>
+#include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
+#include <asm/io.h>
+#include <asm/page.h>
+
+#define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
+/* Global state */
+static struct {
+    /* The global LPI property table, shared by all redistributors. */
+    uint8_t *lpi_property;
+    /*
+     * Number of physical LPIs the host supports. This is a property of
+     * the GIC hardware. We depart from the habit of naming these things
+     * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
+     * in a different context to differentiate them from "virtual LPIs".
+     */
+    unsigned long int nr_host_lpis;
+    unsigned int flags;
+} lpi_data;
+
+struct lpi_redist_data {
+    void                *pending_table;
+};
+
+static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
+
+#define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
+
+static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
+{
+    uint64_t val;
+    unsigned int order;
+    void *pendtable;
+
+    if ( this_cpu(lpi_redist).pending_table )
+        return -EBUSY;
+
+    val  = GIC_BASER_CACHE_RaWaWb << GICR_PENDBASER_INNER_CACHEABILITY_SHIFT;
+    val |= GIC_BASER_CACHE_SameAsInner << GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT;
+    val |= GIC_BASER_InnerShareable << GICR_PENDBASER_SHAREABILITY_SHIFT;
+
+    /*
+     * The pending table holds one bit per LPI and even covers bits for
+     * interrupt IDs below 8192, so we allocate the full range.
+     * The GICv3 imposes a 64KB alignment requirement, also requires
+     * physically contigious memory.
+     */
+    order = max(get_order_from_bytes(lpi_data.nr_host_lpis / 8), 4U);
+    pendtable = alloc_xenheap_pages(order, 0);
+    if ( !pendtable )
+        return -ENOMEM;
+
+    /* Make sure the physical address can be encoded in the register. */
+    if ( (virt_to_maddr(pendtable) & ~GENMASK(51, 16)) )
+    {
+        free_xenheap_pages(pendtable, 0);
+        return -ERANGE;
+    }
+    memset(pendtable, 0, lpi_data.nr_host_lpis / 8);
+    clean_and_invalidate_dcache_va_range(pendtable,
+                                         lpi_data.nr_host_lpis / 8);
+
+    this_cpu(lpi_redist).pending_table = pendtable;
+
+    val |= GICR_PENDBASER_PTZ;
+
+    val |= virt_to_maddr(pendtable);
+
+    *reg = val;
+
+    return 0;
+}
+
+/*
+ * Tell a redistributor about the (shared) property table, allocating one
+ * if not already done.
+ */
+static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
+{
+    uint64_t reg;
+
+    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
+    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
+    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
+
+    /*
+     * The property table is shared across all redistributors, so allocate
+     * this only once, but return the same value on subsequent calls.
+     */
+    if ( !lpi_data.lpi_property )
+    {
+        /* The property table holds one byte per LPI. */
+        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
+        void *table = alloc_xenheap_pages(order, 0);
+
+        if ( !table )
+            return -ENOMEM;
+
+        /* Make sure the physical address can be encoded in the register. */
+        if ( (virt_to_maddr(table) & ~GENMASK(51, 12)) )
+        {
+            free_xenheap_pages(table, 0);
+            return -ERANGE;
+        }
+        memset(table, GIC_PRI_IRQ | LPI_PROP_RES1, MAX_PHYS_LPIS);
+        clean_and_invalidate_dcache_va_range(table, MAX_PHYS_LPIS);
+        lpi_data.lpi_property = table;
+    }
+
+    /* Encode the number of bits needed, minus one */
+    reg |= ((fls(lpi_data.nr_host_lpis - 1) - 1) << 0);
+
+    reg |= virt_to_maddr(lpi_data.lpi_property);
+
+    writeq_relaxed(reg, rdist_base + GICR_PROPBASER);
+    reg = readq_relaxed(rdist_base + GICR_PROPBASER);
+
+    /* If we can't do shareable, we have to drop cacheability as well. */
+    if ( !(reg & GICR_PROPBASER_SHAREABILITY_MASK) )
+    {
+        reg &= ~GICR_PROPBASER_INNER_CACHEABILITY_MASK;
+        reg |= GIC_BASER_CACHE_nC << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
+    }
+
+    /* Remember that we have to flush the property table if non-cacheable. */
+    if ( (reg & GICR_PROPBASER_INNER_CACHEABILITY_MASK) <= GIC_BASER_CACHE_nC )
+    {
+        lpi_data.flags |= LPI_PROPTABLE_NEEDS_FLUSHING;
+        /* Update the redistributors knowledge about the attributes. */
+        writeq_relaxed(reg, rdist_base + GICR_PROPBASER);
+    }
+
+    return 0;
+}
+
+int gicv3_lpi_init_rdist(void __iomem * rdist_base)
+{
+    uint32_t reg;
+    uint64_t table_reg;
+    int ret;
+
+    /* We don't support LPIs without an ITS. */
+    if ( !gicv3_its_host_has_its() )
+        return -ENODEV;
+
+    /* Make sure LPIs are disabled before setting up the tables. */
+    reg = readl_relaxed(rdist_base + GICR_CTLR);
+    if ( reg & GICR_CTLR_ENABLE_LPIS )
+        return -EBUSY;
+
+    ret = gicv3_lpi_allocate_pendtable(&table_reg);
+    if (ret)
+        return ret;
+    writeq_relaxed(table_reg, rdist_base + GICR_PENDBASER);
+
+    return gicv3_lpi_set_proptable(rdist_base);
+}
+
+static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
+integer_param("max_lpi_bits", max_lpi_bits);
+
+int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
+{
+    lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
+
+    printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);
+
+    return 0;
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 1512521..ed78363 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -548,6 +548,9 @@ static void __init gicv3_dist_init(void)
     type = readl_relaxed(GICD + GICD_TYPER);
     nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
 
+    if ( type & GICD_TYPE_LPIS )
+        gicv3_lpi_init_host_lpis(((type >> GICD_TYPE_ID_BITS_SHIFT) & 0x1f) + 1);
+
     printk("GICv3: %d lines, (IID %8.8x).\n",
            nr_lines, readl_relaxed(GICD + GICD_IIDR));
 
@@ -660,6 +663,20 @@ static int __init gicv3_populate_rdist(void)
             if ( (typer >> 32) == aff )
             {
                 this_cpu(rbase) = ptr;
+
+                if ( typer & GICR_TYPER_PLPIS )
+                {
+                    int ret;
+
+                    ret = gicv3_lpi_init_rdist(ptr);
+                    if ( ret && ret != -ENODEV )
+                    {
+                        printk("GICv3: CPU%d: Cannot initialize LPIs: %d\n",
+                               smp_processor_id(), ret);
+                        break;
+                    }
+                }
+
                 printk("GICv3: CPU%d: Found redistributor in region %d @%p\n",
                         smp_processor_id(), i, ptr);
                 return 0;
diff --git a/xen/include/asm-arm/bitops.h b/xen/include/asm-arm/bitops.h
index bda8898..1cbfb9e 100644
--- a/xen/include/asm-arm/bitops.h
+++ b/xen/include/asm-arm/bitops.h
@@ -24,6 +24,7 @@
 #define BIT(nr)                 (1UL << (nr))
 #define BIT_MASK(nr)            (1UL << ((nr) % BITS_PER_WORD))
 #define BIT_WORD(nr)            ((nr) / BITS_PER_WORD)
+#define BIT_ULL(nr)             (1ULL << (nr))
 #define BITS_PER_BYTE           8
 
 #define ADDR (*(volatile int *) addr)
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 836a103..12bd155 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -220,6 +220,8 @@ enum gic_version {
     GIC_V3,
 };
 
+#define LPI_OFFSET      8192
+
 extern enum gic_version gic_hw_version(void);
 
 /* Program the IRQ type into the GIC */
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
index 6bd25a5..b307322 100644
--- a/xen/include/asm-arm/gic_v3_defs.h
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -44,7 +44,8 @@
 #define GICC_SRE_EL2_ENEL1           (1UL << 3)
 
 /* Additional bits in GICD_TYPER defined by GICv3 */
-#define GICD_TYPE_ID_BITS_SHIFT 19
+#define GICD_TYPE_ID_BITS_SHIFT      19
+#define GICD_TYPE_LPIS               (1U << 17)
 
 #define GICD_CTLR_RWP                (1UL << 31)
 #define GICD_CTLR_ARE_NS             (1U << 4)
@@ -95,12 +96,61 @@
 #define GICR_IGRPMODR0               (0x0D00)
 #define GICR_NSACR                   (0x0E00)
 
+#define GICR_CTLR_ENABLE_LPIS        (1U << 0)
+
 #define GICR_TYPER_PLPIS             (1U << 0)
 #define GICR_TYPER_VLPIS             (1U << 1)
 #define GICR_TYPER_LAST              (1U << 4)
 
+/* For specifying the inner cacheability type only */
+#define GIC_BASER_CACHE_nCnB         0ULL
+/* For specifying the outer cacheability type only */
+#define GIC_BASER_CACHE_SameAsInner  0ULL
+#define GIC_BASER_CACHE_nC           1ULL
+#define GIC_BASER_CACHE_RaWt         2ULL
+#define GIC_BASER_CACHE_RaWb         3ULL
+#define GIC_BASER_CACHE_WaWt         4ULL
+#define GIC_BASER_CACHE_WaWb         5ULL
+#define GIC_BASER_CACHE_RaWaWt       6ULL
+#define GIC_BASER_CACHE_RaWaWb       7ULL
+#define GIC_BASER_CACHE_MASK         7ULL
+
+#define GIC_BASER_NonShareable       0ULL
+#define GIC_BASER_InnerShareable     1ULL
+#define GIC_BASER_OuterShareable     2ULL
+
+#define GICR_PROPBASER_SHAREABILITY_SHIFT               10
+#define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT         7
+#define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT         56
+#define GICR_PROPBASER_SHAREABILITY_MASK                     \
+        (3UL << GICR_PROPBASER_SHAREABILITY_SHIFT)
+#define GICR_PROPBASER_INNER_CACHEABILITY_MASK               \
+        (7UL << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT)
+#define GICR_PROPBASER_OUTER_CACHEABILITY_MASK               \
+        (7UL << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT)
+#define GICR_PROPBASER_RES0_MASK                             \
+        (GENMASK(63, 59) | GENMASK(55, 52) | GENMASK(6, 5))
+
+#define GICR_PENDBASER_SHAREABILITY_SHIFT               10
+#define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT         7
+#define GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT         56
+#define GICR_PENDBASER_SHAREABILITY_MASK                     \
+	(3UL << GICR_PENDBASER_SHAREABILITY_SHIFT)
+#define GICR_PENDBASER_INNER_CACHEABILITY_MASK               \
+	(7UL << GICR_PENDBASER_INNER_CACHEABILITY_SHIFT)
+#define GICR_PENDBASER_OUTER_CACHEABILITY_MASK               \
+        (7UL << GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT)
+#define GICR_PENDBASER_PTZ                              BIT(62)
+#define GICR_PENDBASER_RES0_MASK                             \
+        (BIT(63) | GENMASK(61, 59) | GENMASK(55, 52) |       \
+         GENMASK(15, 12) | GENMASK(6, 0))
+
 #define DEFAULT_PMR_VALUE            0xff
 
+#define LPI_PROP_PRIO_MASK           0xfc
+#define LPI_PROP_RES1                (1 << 1)
+#define LPI_PROP_ENABLED             (1 << 0)
+
 #define GICH_VMCR_EOI                (1 << 9)
 #define GICH_VMCR_VENG1              (1 << 1)
 
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 765a655..219d109 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -40,6 +40,11 @@ void gicv3_its_dt_init(const struct dt_device_node *node);
 
 bool gicv3_its_host_has_its(void);
 
+int gicv3_lpi_init_rdist(void __iomem * rdist_base);
+
+/* Initialize the host structures for LPIs. */
+int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
+
 #else
 
 static LIST_HEAD(host_its_list);
@@ -53,6 +58,15 @@ static inline bool gicv3_its_host_has_its(void)
     return false;
 }
 
+static inline int gicv3_lpi_init_rdist(void __iomem * rdist_base)
+{
+    return -ENODEV;
+}
+
+static inline int gicv3_lpi_init_host_lpis(unsigned int nr_lpis)
+{
+    return 0;
+}
 #endif /* CONFIG_HAS_ITS */
 
 #endif
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-21 23:29   ` Stefano Stabellini
  2017-03-22 13:52   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer Andre Przywara
                   ` (23 subsequent siblings)
  26 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Each ITS maps a pair of a DeviceID (for instance derived from a PCI
b/d/f triplet) and an EventID (the MSI payload or interrupt ID) to a
pair of LPI number and collection ID, which points to the target CPU.
This mapping is stored in the device and collection tables, which software
has to provide for the ITS to use.
Allocate the required memory and hand it to the ITS.
The maximum number of devices is limited to a compile-time constant
exposed in Kconfig.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 docs/misc/xen-command-line.markdown |   8 ++
 xen/arch/arm/Kconfig                |  14 ++++
 xen/arch/arm/gic-v3-its.c           | 163 ++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3.c               |   3 +
 xen/include/asm-arm/gic_v3_its.h    |  63 +++++++++++++-
 5 files changed, 250 insertions(+), 1 deletion(-)

diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
index 619016d..068d116 100644
--- a/docs/misc/xen-command-line.markdown
+++ b/docs/misc/xen-command-line.markdown
@@ -1158,6 +1158,14 @@ based interrupts. Any higher IRQs will be available for use via PCI MSI.
 ### maxcpus
 > `= <integer>`
 
+### max\_its\_device\_bits
+> `= <integer>`
+
+Specifies the maximum number of devices using MSIs on the ARM GICv3 ITS
+controller to allocate table entries for. Each table entry uses a hardware
+specific size, typically 8 or 16 bytes.
+Defaults to 10 bits (to cover at most 1024 devices).
+
 ### max\_lpi\_bits
 > `= <integer>`
 
diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
index 86f7b53..0d50156 100644
--- a/xen/arch/arm/Kconfig
+++ b/xen/arch/arm/Kconfig
@@ -64,6 +64,20 @@ config MAX_PHYS_LPI_BITS
           This can be overridden on the command line with the max_lpi_bits
           parameter.
 
+config MAX_PHYS_ITS_DEVICE_BITS
+        depends on HAS_ITS
+        int "Number of device bits the ITS supports"
+        range 1 32
+        default "10"
+        help
+          Specifies the maximum number of devices which want to use the ITS.
+          Xen needs to allocates memory for the whole range very early.
+          The allocation scheme may be sparse, so a much larger number must
+          be supported to cover devices with a high bus number or those on
+          separate bus segments.
+          This can be overridden on the command line with the
+          max_its_device_bits parameter.
+
 endmenu
 
 menu "ARM errata workaround via the alternative framework"
diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 4056e5b..9982fe9 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -19,8 +19,10 @@
  */
 
 #include <xen/lib.h>
+#include <xen/mm.h>
 #include <asm/gic_v3_defs.h>
 #include <asm/gic_v3_its.h>
+#include <asm/io.h>
 
 LIST_HEAD(host_its_list);
 
@@ -29,6 +31,167 @@ bool gicv3_its_host_has_its(void)
     return !list_empty(&host_its_list);
 }
 
+#define BASER_ATTR_MASK                                           \
+        ((0x3UL << GITS_BASER_SHAREABILITY_SHIFT)               | \
+         (0x7UL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)         | \
+         (0x7UL << GITS_BASER_INNER_CACHEABILITY_SHIFT))
+#define BASER_RO_MASK   (GENMASK(58, 56) | GENMASK(52, 48))
+
+/* Check that the physical address can be encoded in the PROPBASER register. */
+static bool check_propbaser_phys_addr(void *vaddr, unsigned int page_bits)
+{
+    paddr_t paddr = virt_to_maddr(vaddr);
+
+    return (!(paddr & ~GENMASK(page_bits < 16 ? 47 : 51, page_bits)));
+}
+
+static uint64_t encode_propbaser_phys_addr(paddr_t addr, unsigned int page_bits)
+{
+    uint64_t ret = addr & GENMASK(47, page_bits);
+
+    if ( page_bits < 16 )
+        return ret;
+
+    /* For 64K pages address bits 51-48 are encoded in bits 15-12. */
+    return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
+}
+
+/* The ITS BASE registers work with page sizes of 4K, 16K or 64K. */
+#define BASER_PAGE_BITS(sz) ((sz) * 2 + 12)
+
+static int its_map_baser(void __iomem *basereg, uint64_t regc,
+                         unsigned int nr_items)
+{
+    uint64_t attr, reg;
+    unsigned int entry_size = GITS_BASER_ENTRY_SIZE(regc);
+    unsigned int pagesz = 2, order, table_size;
+    void *buffer;
+
+    attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
+    attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
+    attr |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
+
+    /*
+     * Setup the BASE register with the attributes that we like. Then read
+     * it back and see what sticks (page size, cacheability and shareability
+     * attributes), retrying if necessary.
+     */
+retry:
+    table_size = ROUNDUP(nr_items * entry_size, BIT(BASER_PAGE_BITS(pagesz)));
+    /* The BASE registers support at most 256 pages. */
+    table_size = min(table_size, 256U << BASER_PAGE_BITS(pagesz));
+    /* The memory block must be aligned to the requested page size. */
+    order = max(get_order_from_bytes(table_size), pagesz * 2);
+
+    buffer = alloc_xenheap_pages(order, 0);
+    if ( !buffer )
+        return -ENOMEM;
+
+    if ( !check_propbaser_phys_addr(buffer, BASER_PAGE_BITS(pagesz)) )
+    {
+        free_xenheap_pages(buffer, 0);
+        return -ERANGE;
+    }
+    memset(buffer, 0, table_size);
+
+    reg  = attr;
+    reg |= (pagesz << GITS_BASER_PAGE_SIZE_SHIFT);
+    reg |= (table_size >> BASER_PAGE_BITS(pagesz)) - 1;
+    reg |= regc & BASER_RO_MASK;
+    reg |= GITS_VALID_BIT;
+    reg |= encode_propbaser_phys_addr(virt_to_maddr(buffer),
+                                      BASER_PAGE_BITS(pagesz));
+
+    writeq_relaxed(reg, basereg);
+    regc = readq_relaxed(basereg);
+
+    /* The host didn't like our attributes, just use what it returned. */
+    if ( (regc & BASER_ATTR_MASK) != attr )
+    {
+        /* If we can't map it shareable, drop cacheability as well. */
+        if ( (regc & GITS_BASER_SHAREABILITY_MASK) == GIC_BASER_NonShareable )
+        {
+            regc &= ~GITS_BASER_INNER_CACHEABILITY_MASK;
+            writeq_relaxed(regc, basereg);
+        }
+        attr = regc & BASER_ATTR_MASK;
+    }
+    if ( (regc & GITS_BASER_INNER_CACHEABILITY_MASK) <= GIC_BASER_CACHE_nC )
+        clean_and_invalidate_dcache_va_range(buffer, table_size);
+
+    /* If the host accepted our page size, we are done. */
+    if ( ((regc >> GITS_BASER_PAGE_SIZE_SHIFT) & 0x3UL) == pagesz )
+        return 0;
+
+    free_xenheap_pages(buffer, order);
+
+    if ( pagesz-- > 0 )
+        goto retry;
+
+    /* None of the page sizes was accepted, give up */
+    return -EINVAL;
+}
+
+static unsigned int max_its_device_bits = CONFIG_MAX_PHYS_ITS_DEVICE_BITS;
+integer_param("max_its_device_bits", max_its_device_bits);
+
+static int gicv3_its_init_single_its(struct host_its *hw_its)
+{
+    uint64_t reg;
+    int i;
+    unsigned int devid_bits;
+
+    hw_its->its_base = ioremap_nocache(hw_its->addr, hw_its->size);
+    if ( !hw_its->its_base )
+        return -ENOMEM;
+
+    reg = readq_relaxed(hw_its->its_base + GITS_TYPER);
+    devid_bits = GITS_TYPER_DEVICE_ID_BITS(reg);
+    devid_bits = min(devid_bits, max_its_device_bits);
+
+    for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
+    {
+        void __iomem *basereg = hw_its->its_base + GITS_BASER0 + i * 8;
+        unsigned int type;
+
+        reg = readq_relaxed(basereg);
+        type = (reg & GITS_BASER_TYPE_MASK) >> GITS_BASER_TYPE_SHIFT;
+        switch ( type )
+        {
+        case GITS_BASER_TYPE_NONE:
+            continue;
+        case GITS_BASER_TYPE_DEVICE:
+            its_map_baser(basereg, reg, BIT(devid_bits));
+            break;
+        case GITS_BASER_TYPE_COLLECTION:
+            its_map_baser(basereg, reg, NR_CPUS);
+            break;
+        /* In case this is a GICv4, provide a (dummy) vPE table as well. */
+        case GITS_BASER_TYPE_VCPU:
+            its_map_baser(basereg, reg, 1);
+            break;
+        default:
+            continue;
+        }
+    }
+
+    return 0;
+}
+
+int gicv3_its_init(void)
+{
+    struct host_its *hw_its;
+    int ret;
+
+    list_for_each_entry(hw_its, &host_its_list, entry) {
+        ret = gicv3_its_init_single_its(hw_its);
+        if ( ret )
+            return ret;
+    }
+
+    return 0;
+}
+
 /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
 void gicv3_its_dt_init(const struct dt_device_node *node)
 {
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index ed78363..cc1e219 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1590,6 +1590,9 @@ static int __init gicv3_init(void)
     spin_lock(&gicv3.lock);
 
     gicv3_dist_init();
+    res = gicv3_its_init();
+    if ( res )
+        printk(XENLOG_WARNING "GICv3: ITS: initialization failed: %d\n", res);
     res = gicv3_cpu_init();
     gicv3_hyp_init();
 
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 219d109..a6c0acc 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -20,6 +20,60 @@
 #ifndef __ASM_ARM_ITS_H__
 #define __ASM_ARM_ITS_H__
 
+#define GITS_CTLR                       0x000
+#define GITS_IIDR                       0x004
+#define GITS_TYPER                      0x008
+#define GITS_CBASER                     0x080
+#define GITS_CWRITER                    0x088
+#define GITS_CREADR                     0x090
+#define GITS_BASER_NR_REGS              8
+#define GITS_BASER0                     0x100
+#define GITS_BASER1                     0x108
+#define GITS_BASER2                     0x110
+#define GITS_BASER3                     0x118
+#define GITS_BASER4                     0x120
+#define GITS_BASER5                     0x128
+#define GITS_BASER6                     0x130
+#define GITS_BASER7                     0x138
+
+/* Register bits */
+#define GITS_VALID_BIT                  BIT_ULL(63)
+
+#define GITS_CTLR_QUIESCENT             BIT(31)
+#define GITS_CTLR_ENABLE                BIT(0)
+
+#define GITS_TYPER_DEVIDS_SHIFT         13
+#define GITS_TYPER_DEVIDS_MASK          (0x1fUL << GITS_TYPER_DEVIDS_SHIFT)
+#define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
+                                               GITS_TYPER_DEVIDS_SHIFT) + 1)
+
+#define GITS_IIDR_VALUE                 0x34c
+
+#define GITS_BASER_INDIRECT             BIT_ULL(62)
+#define GITS_BASER_INNER_CACHEABILITY_SHIFT        59
+#define GITS_BASER_TYPE_SHIFT           56
+#define GITS_BASER_TYPE_MASK            (7ULL << GITS_BASER_TYPE_SHIFT)
+#define GITS_BASER_OUTER_CACHEABILITY_SHIFT        53
+#define GITS_BASER_TYPE_NONE            0UL
+#define GITS_BASER_TYPE_DEVICE          1UL
+#define GITS_BASER_TYPE_VCPU            2UL
+#define GITS_BASER_TYPE_CPU             3UL
+#define GITS_BASER_TYPE_COLLECTION      4UL
+#define GITS_BASER_TYPE_RESERVED5       5UL
+#define GITS_BASER_TYPE_RESERVED6       6UL
+#define GITS_BASER_TYPE_RESERVED7       7UL
+#define GITS_BASER_ENTRY_SIZE_SHIFT     48
+#define GITS_BASER_ENTRY_SIZE(reg)                                       \
+                        (((reg >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
+#define GITS_BASER_SHAREABILITY_SHIFT   10
+#define GITS_BASER_PAGE_SIZE_SHIFT      8
+#define GITS_BASER_RO_MASK              (GITS_BASER_TYPE_MASK | \
+                                        (31UL << GITS_BASER_ENTRY_SIZE_SHIFT) |\
+                                        GITS_BASER_INDIRECT)
+#define GITS_BASER_SHAREABILITY_MASK   (0x3ULL << GITS_BASER_SHAREABILITY_SHIFT)
+#define GITS_BASER_OUTER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)
+#define GITS_BASER_INNER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_INNER_CACHEABILITY_SHIFT)
+
 #include <xen/device_tree.h>
 
 /* data structure for each hardware ITS */
@@ -28,6 +82,7 @@ struct host_its {
     const struct dt_device_node *dt_node;
     paddr_t addr;
     paddr_t size;
+    void __iomem *its_base;
 };
 
 
@@ -42,8 +97,9 @@ bool gicv3_its_host_has_its(void);
 
 int gicv3_lpi_init_rdist(void __iomem * rdist_base);
 
-/* Initialize the host structures for LPIs. */
+/* Initialize the host structures for LPIs and the host ITSes. */
 int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
+int gicv3_its_init(void);
 
 #else
 
@@ -67,6 +123,11 @@ static inline int gicv3_lpi_init_host_lpis(unsigned int nr_lpis)
 {
     return 0;
 }
+
+static inline int gicv3_its_init(void)
+{
+    return 0;
+}
 #endif /* CONFIG_HAS_ITS */
 
 #endif
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (2 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-21 23:48   ` Stefano Stabellini
  2017-03-22 15:23   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling Andre Przywara
                   ` (22 subsequent siblings)
  26 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Instead of directly manipulating the tables in memory, an ITS driver
sends commands via a ring buffer in normal system memory to the ITS h/w
to create or alter the LPI mappings.
Allocate memory for that buffer and tell the ITS about it to be able
to send ITS commands.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c        | 57 ++++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/gic_v3_its.h |  6 +++++
 2 files changed, 63 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 9982fe9..e5601ed 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -20,10 +20,13 @@
 
 #include <xen/lib.h>
 #include <xen/mm.h>
+#include <xen/sizes.h>
 #include <asm/gic_v3_defs.h>
 #include <asm/gic_v3_its.h>
 #include <asm/io.h>
 
+#define ITS_CMD_QUEUE_SZ                SZ_64K
+
 LIST_HEAD(host_its_list);
 
 bool gicv3_its_host_has_its(void)
@@ -56,6 +59,55 @@ static uint64_t encode_propbaser_phys_addr(paddr_t addr, unsigned int page_bits)
     return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
 }
 
+static void *its_map_cbaser(struct host_its *its)
+{
+    void __iomem *cbasereg = its->its_base + GITS_CBASER;
+    uint64_t reg;
+    void *buffer;
+    unsigned int order;
+
+    reg  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
+    reg |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
+    reg |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
+
+    /* The ITS command buffer needs to be 64K aligned. */
+    order = max(get_order_from_bytes(ITS_CMD_QUEUE_SZ), 16U - PAGE_SHIFT);
+    buffer = alloc_xenheap_pages(order, 0);
+    if ( !buffer )
+        return NULL;
+
+    if ( virt_to_maddr(buffer) & ~GENMASK(51, 12) )
+    {
+        free_xenheap_pages(buffer, 0);
+        return NULL;
+    }
+    memset(buffer, 0, ITS_CMD_QUEUE_SZ);
+
+    reg |= GITS_VALID_BIT | virt_to_maddr(buffer);
+    reg |= ((ITS_CMD_QUEUE_SZ / SZ_4K) - 1) & GITS_CBASER_SIZE_MASK;
+    writeq_relaxed(reg, cbasereg);
+    reg = readq_relaxed(cbasereg);
+
+    /* If the ITS dropped shareability, drop cacheability as well. */
+    if ( (reg & GITS_BASER_SHAREABILITY_MASK) == 0 )
+    {
+        reg &= ~GITS_BASER_INNER_CACHEABILITY_MASK;
+        writeq_relaxed(reg, cbasereg);
+    }
+
+    /*
+     * If the command queue memory is mapped as uncached, we need to flush
+     * it on every access.
+     */
+    if ( !(reg & GITS_BASER_INNER_CACHEABILITY_MASK) )
+    {
+        its->flags |= HOST_ITS_FLUSH_CMD_QUEUE;
+        dprintk(XENLOG_WARNING, "using non-cacheable ITS command queue\n");
+    }
+
+    return buffer;
+}
+
 /* The ITS BASE registers work with page sizes of 4K, 16K or 64K. */
 #define BASER_PAGE_BITS(sz) ((sz) * 2 + 12)
 
@@ -175,6 +227,11 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
         }
     }
 
+    hw_its->cmd_buf = its_map_cbaser(hw_its);
+    if ( !hw_its->cmd_buf )
+        return -ENOMEM;
+    writeq_relaxed(0, hw_its->its_base + GITS_CWRITER);
+
     return 0;
 }
 
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index a6c0acc..d5facf0 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -74,8 +74,12 @@
 #define GITS_BASER_OUTER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)
 #define GITS_BASER_INNER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_INNER_CACHEABILITY_SHIFT)
 
+#define GITS_CBASER_SIZE_MASK           0xff
+
 #include <xen/device_tree.h>
 
+#define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
+
 /* data structure for each hardware ITS */
 struct host_its {
     struct list_head entry;
@@ -83,6 +87,8 @@ struct host_its {
     paddr_t addr;
     paddr_t size;
     void __iomem *its_base;
+    void *cmd_buf;
+    unsigned int flags;
 };
 
 
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (3 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 15:05   ` Shanker Donthineni
                     ` (2 more replies)
  2017-03-16 11:20 ` [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping Andre Przywara
                   ` (21 subsequent siblings)
  26 siblings, 3 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

To be able to easily send commands to the ITS, create the respective
wrapper functions, which take care of the ring buffer.
The first two commands we implement provide methods to map a collection
to a redistributor (aka host core) and to flush the command queue (SYNC).
Start using these commands for mapping one collection to each host CPU.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c         | 181 +++++++++++++++++++++++++++++++++++++-
 xen/arch/arm/gic-v3-lpi.c         |  22 +++++
 xen/arch/arm/gic-v3.c             |  19 +++-
 xen/include/asm-arm/gic_v3_defs.h |   2 +
 xen/include/asm-arm/gic_v3_its.h  |  38 ++++++++
 5 files changed, 260 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index e5601ed..5c11b0d 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -19,11 +19,14 @@
  */
 
 #include <xen/lib.h>
+#include <xen/delay.h>
 #include <xen/mm.h>
 #include <xen/sizes.h>
+#include <asm/gic.h>
 #include <asm/gic_v3_defs.h>
 #include <asm/gic_v3_its.h>
 #include <asm/io.h>
+#include <asm/page.h>
 
 #define ITS_CMD_QUEUE_SZ                SZ_64K
 
@@ -34,6 +37,145 @@ bool gicv3_its_host_has_its(void)
     return !list_empty(&host_its_list);
 }
 
+#define BUFPTR_MASK                     GENMASK(19, 5)
+static int its_send_command(struct host_its *hw_its, const void *its_cmd)
+{
+    s_time_t deadline = NOW() + MILLISECS(1);
+    uint64_t readp, writep;
+    int ret = -EBUSY;
+
+    /* No ITS commands from an interrupt handler (at the moment). */
+    ASSERT(!in_irq());
+
+    spin_lock(&hw_its->cmd_lock);
+
+    do {
+        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
+        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
+
+        if ( ((writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ) != readp )
+        {
+            ret = 0;
+            break;
+        }
+
+        /*
+         * If the command queue is full, wait for a bit in the hope it drains
+         * before giving up.
+         */
+        spin_unlock(&hw_its->cmd_lock);
+        cpu_relax();
+        udelay(1);
+        spin_lock(&hw_its->cmd_lock);
+    } while ( NOW() <= deadline );
+
+    if ( ret )
+    {
+        spin_unlock(&hw_its->cmd_lock);
+        printk(XENLOG_WARNING "ITS: command queue full.\n");
+        return ret;
+    }
+
+    memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE);
+    if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
+        clean_and_invalidate_dcache_va_range(hw_its->cmd_buf + writep,
+                                             ITS_CMD_SIZE);
+    else
+        dsb(ishst);
+
+    writep = (writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ;
+    writeq_relaxed(writep & BUFPTR_MASK, hw_its->its_base + GITS_CWRITER);
+
+    spin_unlock(&hw_its->cmd_lock);
+
+    return 0;
+}
+
+/* Wait for an ITS to finish processing all commands. */
+static int gicv3_its_wait_commands(struct host_its *hw_its)
+{
+    s_time_t deadline = NOW() + MILLISECS(100);
+    uint64_t readp, writep;
+
+    do {
+        spin_lock(&hw_its->cmd_lock);
+        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
+        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
+        spin_unlock(&hw_its->cmd_lock);
+
+        if ( readp == writep )
+            return 0;
+
+        cpu_relax();
+        udelay(1);
+    } while ( NOW() <= deadline );
+
+    return -ETIMEDOUT;
+}
+
+static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
+                              uint64_t reg)
+{
+    reg &= ~GENMASK(51, 16);
+
+    reg |= gicv3_get_redist_address(cpu, hw_its->flags & HOST_ITS_USES_PTA);
+
+    return reg;
+}
+
+static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
+{
+    uint64_t cmd[4];
+
+    cmd[0] = GITS_CMD_SYNC;
+    cmd[1] = 0x00;
+    cmd[2] = encode_rdbase(its, cpu, 0x0);
+    cmd[3] = 0x00;
+
+    return its_send_command(its, cmd);
+}
+
+static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
+                             unsigned int cpu)
+{
+    uint64_t cmd[4];
+
+    cmd[0] = GITS_CMD_MAPC;
+    cmd[1] = 0x00;
+    cmd[2] = encode_rdbase(its, cpu, (collection_id & GENMASK(15, 0)));
+    cmd[2] |= GITS_VALID_BIT;
+    cmd[3] = 0x00;
+
+    return its_send_command(its, cmd);
+}
+
+/* Set up the (1:1) collection mapping for the given host CPU. */
+int gicv3_its_setup_collection(unsigned int cpu)
+{
+    struct host_its *its;
+    int ret;
+
+    list_for_each_entry(its, &host_its_list, entry)
+    {
+        if ( !its->cmd_buf )
+            continue;
+
+        ret = its_send_cmd_mapc(its, cpu, cpu);
+        if ( ret )
+            return ret;
+
+        ret = its_send_cmd_sync(its, cpu);
+        if ( ret )
+            return ret;
+
+        ret = gicv3_its_wait_commands(its);
+        if ( ret )
+            return ret;
+    }
+
+    return 0;
+}
+
 #define BASER_ATTR_MASK                                           \
         ((0x3UL << GITS_BASER_SHAREABILITY_SHIFT)               | \
          (0x7UL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)         | \
@@ -184,22 +326,59 @@ retry:
     return -EINVAL;
 }
 
+/*
+ * Before an ITS gets initialized, it should be in a quiescent state, where
+ * all outstanding commands and transactions have finished.
+ * So if the ITS is already enabled, turn it off and wait for all outstanding
+ * operations to get processed by polling the QUIESCENT bit.
+ */
+static int gicv3_disable_its(struct host_its *hw_its)
+{
+    uint32_t reg;
+    s_time_t deadline = NOW() + MILLISECS(100);
+
+    reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
+    if ( (reg & GITS_CTLR_QUIESCENT) && !(reg & GITS_CTLR_ENABLE) )
+        return 0;
+
+    writel_relaxed(reg & ~GITS_CTLR_ENABLE, hw_its->its_base + GITS_CTLR);
+
+    do {
+        reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
+        if ( reg & GITS_CTLR_QUIESCENT )
+            return 0;
+
+        cpu_relax();
+        udelay(1);
+    } while ( NOW() <= deadline );
+
+    dprintk(XENLOG_ERR, "ITS not quiescent.\n");
+
+    return -ETIMEDOUT;
+}
+
 static unsigned int max_its_device_bits = CONFIG_MAX_PHYS_ITS_DEVICE_BITS;
 integer_param("max_its_device_bits", max_its_device_bits);
 
 static int gicv3_its_init_single_its(struct host_its *hw_its)
 {
     uint64_t reg;
-    int i;
+    int i ,ret;
     unsigned int devid_bits;
 
     hw_its->its_base = ioremap_nocache(hw_its->addr, hw_its->size);
     if ( !hw_its->its_base )
         return -ENOMEM;
 
+    ret = gicv3_disable_its(hw_its);
+    if ( ret )
+        return ret;
+
     reg = readq_relaxed(hw_its->its_base + GITS_TYPER);
     devid_bits = GITS_TYPER_DEVICE_ID_BITS(reg);
     devid_bits = min(devid_bits, max_its_device_bits);
+    if ( reg & GITS_TYPER_PTA )
+        hw_its->flags |= HOST_ITS_USES_PTA;
 
     for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
     {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 4f8414b..51d7425 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -42,6 +42,8 @@ static struct {
 } lpi_data;
 
 struct lpi_redist_data {
+    paddr_t             redist_addr;
+    unsigned int        redist_id;
     void                *pending_table;
 };
 
@@ -49,6 +51,26 @@ static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
 
 #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
 
+/* Stores this redistributor's physical address and ID in a per-CPU variable */
+void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
+{
+    this_cpu(lpi_redist).redist_addr = address;
+    this_cpu(lpi_redist).redist_id = redist_id;
+}
+
+/*
+ * Returns a redistributor's ID (either as an address or as an ID).
+ * This must be (and is) called only after it has been setup by the above
+ * function.
+ */
+uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta)
+{
+    if ( use_pta )
+        return per_cpu(lpi_redist, cpu).redist_addr & GENMASK(51, 16);
+    else
+        return per_cpu(lpi_redist, cpu).redist_id << 16;
+}
+
 static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
 {
     uint64_t val;
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index cc1e219..38dafe7 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -666,7 +666,21 @@ static int __init gicv3_populate_rdist(void)
 
                 if ( typer & GICR_TYPER_PLPIS )
                 {
-                    int ret;
+                    paddr_t rdist_addr;
+                    int procnum, ret;
+
+                    /*
+                     * The ITS refers to redistributors either by their physical
+                     * address or by their ID. Determine those two values and
+                     * let the ITS code store them in per host CPU variables to
+                     * later be able to address those redistributors.
+                     */
+                    rdist_addr = gicv3.rdist_regions[i].base;
+                    rdist_addr += ptr - gicv3.rdist_regions[i].map_base;
+                    procnum = (typer & GICR_TYPER_PROC_NUM_MASK);
+                    procnum >>= GICR_TYPER_PROC_NUM_SHIFT;
+
+                    gicv3_set_redist_address(rdist_addr, procnum);
 
                     ret = gicv3_lpi_init_rdist(ptr);
                     if ( ret && ret != -ENODEV )
@@ -715,6 +729,9 @@ static int gicv3_cpu_init(void)
     if ( gicv3_enable_redist() )
         return -ENODEV;
 
+    if ( gicv3_its_host_has_its() )
+        gicv3_its_setup_collection(smp_processor_id());
+
     /* Set priority on PPI and SGI interrupts */
     priority = (GIC_PRI_IPI << 24 | GIC_PRI_IPI << 16 | GIC_PRI_IPI << 8 |
                 GIC_PRI_IPI);
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
index b307322..878bae2 100644
--- a/xen/include/asm-arm/gic_v3_defs.h
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -101,6 +101,8 @@
 #define GICR_TYPER_PLPIS             (1U << 0)
 #define GICR_TYPER_VLPIS             (1U << 1)
 #define GICR_TYPER_LAST              (1U << 4)
+#define GICR_TYPER_PROC_NUM_SHIFT    8
+#define GICR_TYPER_PROC_NUM_MASK     (0xffff << GICR_TYPER_PROC_NUM_SHIFT)
 
 /* For specifying the inner cacheability type only */
 #define GIC_BASER_CACHE_nCnB         0ULL
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index d5facf0..8b493fb 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -42,10 +42,12 @@
 #define GITS_CTLR_QUIESCENT             BIT(31)
 #define GITS_CTLR_ENABLE                BIT(0)
 
+#define GITS_TYPER_PTA                  BIT_ULL(19)
 #define GITS_TYPER_DEVIDS_SHIFT         13
 #define GITS_TYPER_DEVIDS_MASK          (0x1fUL << GITS_TYPER_DEVIDS_SHIFT)
 #define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
                                                GITS_TYPER_DEVIDS_SHIFT) + 1)
+#define GITS_TYPER_IDBITS_SHIFT         8
 
 #define GITS_IIDR_VALUE                 0x34c
 
@@ -76,9 +78,26 @@
 
 #define GITS_CBASER_SIZE_MASK           0xff
 
+/* ITS command definitions */
+#define ITS_CMD_SIZE                    32
+
+#define GITS_CMD_MOVI                   0x01
+#define GITS_CMD_INT                    0x03
+#define GITS_CMD_CLEAR                  0x04
+#define GITS_CMD_SYNC                   0x05
+#define GITS_CMD_MAPD                   0x08
+#define GITS_CMD_MAPC                   0x09
+#define GITS_CMD_MAPTI                  0x0a
+#define GITS_CMD_MAPI                   0x0b
+#define GITS_CMD_INV                    0x0c
+#define GITS_CMD_INVALL                 0x0d
+#define GITS_CMD_MOVALL                 0x0e
+#define GITS_CMD_DISCARD                0x0f
+
 #include <xen/device_tree.h>
 
 #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
+#define HOST_ITS_USES_PTA               (1U << 1)
 
 /* data structure for each hardware ITS */
 struct host_its {
@@ -87,6 +106,7 @@ struct host_its {
     paddr_t addr;
     paddr_t size;
     void __iomem *its_base;
+    spinlock_t cmd_lock;
     void *cmd_buf;
     unsigned int flags;
 };
@@ -107,6 +127,13 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base);
 int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
 int gicv3_its_init(void);
 
+/* Store the physical address and ID for each redistributor as read from DT. */
+void gicv3_set_redist_address(paddr_t address, unsigned int redist_id);
+uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
+
+/* Map a collection for this host CPU to each host ITS. */
+int gicv3_its_setup_collection(unsigned int cpu);
+
 #else
 
 static LIST_HEAD(host_its_list);
@@ -134,6 +161,17 @@ static inline int gicv3_its_init(void)
 {
     return 0;
 }
+
+static inline void gicv3_set_redist_address(paddr_t address,
+                                            unsigned int redist_id)
+{
+}
+
+static inline int gicv3_its_setup_collection(unsigned int cpu)
+{
+    return 0;
+}
+
 #endif /* CONFIG_HAS_ITS */
 
 #endif
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (4 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-22 17:29   ` Julien Grall
                     ` (2 more replies)
  2017-03-16 11:20 ` [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors Andre Przywara
                   ` (20 subsequent siblings)
  26 siblings, 3 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The ITS uses device IDs to map LPIs to a device. Dom0 will later use
those IDs, which we directly pass on to the host.
For this we have to map each device that Dom0 may request to a host
ITS device with the same identifier.
Allocate the respective memory and enter each device into an rbtree to
later be able to iterate over it or to easily teardown guests.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c        | 207 +++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic-v3.c           |   3 +
 xen/include/asm-arm/domain.h     |   3 +
 xen/include/asm-arm/gic_v3_its.h |  18 ++++
 4 files changed, 231 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 5c11b0d..60b15b5 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -21,6 +21,8 @@
 #include <xen/lib.h>
 #include <xen/delay.h>
 #include <xen/mm.h>
+#include <xen/rbtree.h>
+#include <xen/sched.h>
 #include <xen/sizes.h>
 #include <asm/gic.h>
 #include <asm/gic_v3_defs.h>
@@ -32,6 +34,17 @@
 
 LIST_HEAD(host_its_list);
 
+struct its_devices {
+    struct rb_node rbnode;
+    struct host_its *hw_its;
+    void *itt_addr;
+    paddr_t guest_doorbell;
+    uint32_t host_devid;
+    uint32_t guest_devid;
+    uint32_t eventids;
+    uint32_t *host_lpis;
+};
+
 bool gicv3_its_host_has_its(void)
 {
     return !list_empty(&host_its_list);
@@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
     return its_send_command(its, cmd);
 }
 
+static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
+                             uint8_t size_bits, paddr_t itt_addr, bool valid)
+{
+    uint64_t cmd[4];
+
+    if ( valid )
+    {
+        ASSERT(size_bits < 32);
+        ASSERT(!(itt_addr & ~GENMASK(51, 8)));
+    }
+    cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
+    cmd[1] = valid ? size_bits : 0x00;
+    cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
+    cmd[3] = 0x00;
+
+    return its_send_command(its, cmd);
+}
+
 /* Set up the (1:1) collection mapping for the given host CPU. */
 int gicv3_its_setup_collection(unsigned int cpu)
 {
@@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
     devid_bits = min(devid_bits, max_its_device_bits);
     if ( reg & GITS_TYPER_PTA )
         hw_its->flags |= HOST_ITS_USES_PTA;
+    hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
 
     for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
     {
@@ -428,6 +460,180 @@ int gicv3_its_init(void)
     return 0;
 }
 
+static int remove_mapped_guest_device(struct its_devices *dev)
+{
+    int ret;
+
+    if ( dev->hw_its )
+    {
+        int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);
+        if ( ret )
+            return ret;
+    }
+
+    ret = gicv3_its_wait_commands(dev->hw_its);
+    if ( ret )
+        return ret;
+
+    xfree(dev->itt_addr);
+    xfree(dev);
+
+    return 0;
+}
+
+static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
+{
+    struct host_its *hw_its;
+
+    list_for_each_entry(hw_its, &host_its_list, entry)
+    {
+        if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
+            return hw_its;
+    }
+
+    return NULL;
+}
+
+static int compare_its_guest_devices(struct its_devices *dev,
+                                     paddr_t doorbell, uint32_t devid)
+{
+    if ( dev->guest_doorbell < doorbell )
+        return -1;
+
+    if ( dev->guest_doorbell > doorbell )
+        return 1;
+
+    if ( dev->guest_devid < devid )
+        return -1;
+
+    if ( dev->guest_devid > devid )
+        return 1;
+
+    return 0;
+}
+
+/*
+ * Map a hardware device, identified by a certain host ITS and its device ID
+ * to domain d, a guest ITS (identified by its doorbell address) and device ID.
+ * Also provide the number of events (MSIs) needed for that device.
+ * This does not check if this particular hardware device is already mapped
+ * at another domain, it is expected that this would be done by the caller.
+ */
+int gicv3_its_map_guest_device(struct domain *d,
+                               paddr_t host_doorbell, uint32_t host_devid,
+                               paddr_t guest_doorbell, uint32_t guest_devid,
+                               uint32_t nr_events, bool valid)
+{
+    void *itt_addr = NULL;
+    struct host_its *hw_its;
+    struct its_devices *dev = NULL, *temp;
+    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
+    int ret = -ENOENT;
+
+    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
+    if ( !hw_its )
+        return ret;
+
+    /* check for already existing mappings */
+    spin_lock(&d->arch.vgic.its_devices_lock);
+    while ( *new )
+    {
+        temp = rb_entry(*new, struct its_devices, rbnode);
+
+        parent = *new;
+        if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
+        {
+            if ( !valid )
+                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
+
+            spin_unlock(&d->arch.vgic.its_devices_lock);
+
+            if ( valid )
+                return -EBUSY;
+
+            return remove_mapped_guest_device(temp);
+        }
+
+        if ( compare_its_guest_devices(temp, guest_doorbell, guest_devid) > 0 )
+            new = &((*new)->rb_left);
+        else
+            new = &((*new)->rb_right);
+    }
+
+    if ( !valid )
+        goto out_unlock;
+
+    ret = -ENOMEM;
+
+    /* An Interrupt Translation Table needs to be 256-byte aligned. */
+    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
+    if ( !itt_addr )
+        goto out_unlock;
+
+    dev = xzalloc(struct its_devices);
+    if ( !dev )
+        goto out_unlock;
+
+    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
+                            virt_to_maddr(itt_addr), true);
+    if ( ret )
+        goto out_unlock;
+
+    dev->itt_addr = itt_addr;
+    dev->hw_its = hw_its;
+    dev->guest_doorbell = guest_doorbell;
+    dev->guest_devid = guest_devid;
+    dev->host_devid = host_devid;
+    dev->eventids = nr_events;
+
+    rb_link_node(&dev->rbnode, parent, new);
+    rb_insert_color(&dev->rbnode, &d->arch.vgic.its_devices);
+
+    spin_unlock(&d->arch.vgic.its_devices_lock);
+
+    return 0;
+
+out_unlock:
+    spin_unlock(&d->arch.vgic.its_devices_lock);
+    if ( dev )
+        xfree(dev->host_lpis);
+    xfree(itt_addr);
+    xfree(dev);
+    return ret;
+}
+
+/* Removing any connections a domain had to any ITS in the system. */
+void gicv3_its_unmap_all_devices(struct domain *d)
+{
+    struct rb_node *victim;
+    struct its_devices *dev;
+
+    /*
+     * This is an easily readable, yet inefficient implementation.
+     * It uses the provided iteration wrapper and erases each node, which
+     * possibly triggers rebalancing.
+     * This seems overkill since we are going to abolish the whole tree, but
+     * avoids an open-coded re-implementation of the traversal functions with
+     * some recursive function calls.
+     * Performance does not matter here, since we are destroying a domain.
+     */
+restart:
+    spin_lock(&d->arch.vgic.its_devices_lock);
+    if ( (victim = rb_first(&d->arch.vgic.its_devices)) )
+    {
+        dev = rb_entry(victim, struct its_devices, rbnode);
+        rb_erase(victim, &d->arch.vgic.its_devices);
+
+        spin_unlock(&d->arch.vgic.its_devices_lock);
+
+        remove_mapped_guest_device(dev);
+
+        goto restart;
+    }
+
+    spin_unlock(&d->arch.vgic.its_devices_lock);
+}
+
 /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
 void gicv3_its_dt_init(const struct dt_device_node *node)
 {
@@ -455,6 +661,7 @@ void gicv3_its_dt_init(const struct dt_device_node *node)
         its_data->addr = addr;
         its_data->size = size;
         its_data->dt_node = its;
+        spin_lock_init(&its_data->cmd_lock);
 
         printk("GICv3: Found ITS @0x%lx\n", addr);
 
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index d61479d..1fadb00 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
     d->arch.vgic.nr_regions = rdist_count;
     d->arch.vgic.rdist_regions = rdist_regions;
 
+    spin_lock_init(&d->arch.vgic.its_devices_lock);
+    d->arch.vgic.its_devices = RB_ROOT;
+
     /*
      * Domain 0 gets the hardware address.
      * Guests get the virtual platform layout.
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 2d6fbb1..00b9c1a 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -11,6 +11,7 @@
 #include <asm/gic.h>
 #include <public/hvm/params.h>
 #include <xen/serial.h>
+#include <xen/rbtree.h>
 
 struct hvm_domain
 {
@@ -109,6 +110,8 @@ struct arch_domain
         } *rdist_regions;
         int nr_regions;                     /* Number of rdist regions */
         uint32_t rdist_stride;              /* Re-Distributor stride */
+        struct rb_root its_devices;         /* devices mapped to an ITS */
+        spinlock_t its_devices_lock;        /* protects the its_devices tree */
 #endif
     } vgic;
 
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 8b493fb..3421ea0 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -48,6 +48,10 @@
 #define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
                                                GITS_TYPER_DEVIDS_SHIFT) + 1)
 #define GITS_TYPER_IDBITS_SHIFT         8
+#define GITS_TYPER_ITT_SIZE_SHIFT       4
+#define GITS_TYPER_ITT_SIZE_MASK        (0xfUL << GITS_TYPER_ITT_SIZE_SHIFT)
+#define GITS_TYPER_ITT_SIZE(r)          ((((r) & GITS_TYPER_ITT_SIZE_MASK) >> \
+                                                GITS_TYPER_ITT_SIZE_SHIFT) + 1)
 
 #define GITS_IIDR_VALUE                 0x34c
 
@@ -94,7 +98,10 @@
 #define GITS_CMD_MOVALL                 0x0e
 #define GITS_CMD_DISCARD                0x0f
 
+#define ITS_DOORBELL_OFFSET             0x10040
+
 #include <xen/device_tree.h>
+#include <xen/rbtree.h>
 
 #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
 #define HOST_ITS_USES_PTA               (1U << 1)
@@ -108,6 +115,7 @@ struct host_its {
     void __iomem *its_base;
     spinlock_t cmd_lock;
     void *cmd_buf;
+    unsigned int itte_size;
     unsigned int flags;
 };
 
@@ -134,6 +142,16 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
 /* Map a collection for this host CPU to each host ITS. */
 int gicv3_its_setup_collection(unsigned int cpu);
 
+/*
+ * Map a device on the host by allocating an ITT on the host (ITS).
+ * "nr_event" specifies how many events (interrupts) this device will need.
+ * Setting "valid" to false deallocates the device.
+ */
+int gicv3_its_map_guest_device(struct domain *d,
+                               paddr_t host_doorbell, uint32_t host_devid,
+                               paddr_t guest_doorbell, uint32_t guest_devid,
+                               uint32_t nr_events, bool valid);
+
 #else
 
 static LIST_HEAD(host_its_list);
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (5 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-22 17:30   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array Andre Przywara
                   ` (19 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

For some reason (probably because there was no user before) the 64-bit
atomic access wrappers were commented out so far.
As we will need them in the next patch, active (and fix) them now.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/include/asm-arm/atomic.h | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/xen/include/asm-arm/atomic.h b/xen/include/asm-arm/atomic.h
index 22a5036..df9de6a 100644
--- a/xen/include/asm-arm/atomic.h
+++ b/xen/include/asm-arm/atomic.h
@@ -53,9 +53,9 @@ build_atomic_write(write_u16_atomic, "h", WORD, uint16_t, "r")
 build_atomic_write(write_u32_atomic, "",  WORD, uint32_t, "r")
 build_atomic_write(write_int_atomic, "",  WORD, int, "r")
 
-#if 0 /* defined (CONFIG_ARM_64) */
-build_atomic_read(read_u64_atomic, "x", uint64_t, "=r")
-build_atomic_write(write_u64_atomic, "x", uint64_t, "r")
+#if defined (CONFIG_ARM_64)
+build_atomic_read(read_u64_atomic, "", "", uint64_t, "=r")
+build_atomic_write(write_u64_atomic, "", "", uint64_t, "r")
 #endif
 
 build_add_sized(add_u8_sized, "b", BYTE, uint8_t, "ri")
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (6 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-22 23:38   ` Stefano Stabellini
  2017-03-23 19:08   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs Andre Przywara
                   ` (18 subsequent siblings)
  26 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The number of LPIs on a host can be potentially huge (millions),
although in practise will be mostly reasonable. So prematurely allocating
an array of struct irq_desc's for each LPI is not an option.
However Xen itself does not care about LPIs, as every LPI will be injected
into a guest (Dom0 for now).
Create a dense data structure (8 Bytes) for each LPI which holds just
enough information to determine the virtual IRQ number and the VCPU into
which the LPI needs to be injected.
Also to not artificially limit the number of LPIs, we create a 2-level
table for holding those structures.
This patch introduces functions to initialize these tables and to
create, lookup and destroy entries for a given LPI.
By using the naturally atomic access guarantee the native uint64_t data
type gives us, we allocate and access LPI information in a way that does
not require a lock.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c        |  90 ++++++++++++++++++-
 xen/arch/arm/gic-v3-lpi.c        | 188 +++++++++++++++++++++++++++++++++++++++
 xen/include/asm-arm/gic.h        |   5 ++
 xen/include/asm-arm/gic_v3_its.h |  11 +++
 4 files changed, 292 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 60b15b5..ed14d95 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -148,6 +148,20 @@ static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
     return its_send_command(its, cmd);
 }
 
+static int its_send_cmd_mapti(struct host_its *its,
+                              uint32_t deviceid, uint32_t eventid,
+                              uint32_t pintid, uint16_t icid)
+{
+    uint64_t cmd[4];
+
+    cmd[0] = GITS_CMD_MAPTI | ((uint64_t)deviceid << 32);
+    cmd[1] = eventid | ((uint64_t)pintid << 32);
+    cmd[2] = icid;
+    cmd[3] = 0x00;
+
+    return its_send_command(its, cmd);
+}
+
 static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
                              unsigned int cpu)
 {
@@ -180,6 +194,19 @@ static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
     return its_send_command(its, cmd);
 }
 
+static int its_send_cmd_inv(struct host_its *its,
+                            uint32_t deviceid, uint32_t eventid)
+{
+    uint64_t cmd[4];
+
+    cmd[0] = GITS_CMD_INV | ((uint64_t)deviceid << 32);
+    cmd[1] = eventid;
+    cmd[2] = 0x00;
+    cmd[3] = 0x00;
+
+    return its_send_command(its, cmd);
+}
+
 /* Set up the (1:1) collection mapping for the given host CPU. */
 int gicv3_its_setup_collection(unsigned int cpu)
 {
@@ -462,7 +489,7 @@ int gicv3_its_init(void)
 
 static int remove_mapped_guest_device(struct its_devices *dev)
 {
-    int ret;
+    int ret, i;
 
     if ( dev->hw_its )
     {
@@ -471,11 +498,19 @@ static int remove_mapped_guest_device(struct its_devices *dev)
             return ret;
     }
 
+    /*
+     * The only error the function below would return is -ENOENT, in which
+     * case there is nothing to free here. So we just ignore it.
+     */
+    for ( i = 0; i < DIV_ROUND_UP(dev->eventids, LPI_BLOCK); i++ )
+        gicv3_free_host_lpi_block(dev->hw_its, dev->host_lpis[i]);
+
     ret = gicv3_its_wait_commands(dev->hw_its);
     if ( ret )
         return ret;
 
     xfree(dev->itt_addr);
+    xfree(dev->host_lpis);
     xfree(dev);
 
     return 0;
@@ -513,6 +548,36 @@ static int compare_its_guest_devices(struct its_devices *dev,
 }
 
 /*
+ * On the host ITS @its, map @nr_events consecutive LPIs.
+ * The mapping connects a device @devid and event @eventid pair to LPI @lpi,
+ * increasing both @eventid and @lpi to cover the number of requested LPIs.
+ */
+int gicv3_its_map_host_events(struct host_its *its,
+                              uint32_t devid, uint32_t eventid, uint32_t lpi,
+                              uint32_t nr_events)
+{
+    uint32_t i;
+    int ret;
+
+    for ( i = 0; i < nr_events; i++ )
+    {
+        /* For now we map every host LPI to host CPU 0 */
+        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
+        if ( ret )
+            return ret;
+        ret = its_send_cmd_inv(its, devid, eventid + i);
+        if ( ret )
+            return ret;
+    }
+
+    ret = its_send_cmd_sync(its, 0);
+    if ( ret )
+        return ret;
+
+    return gicv3_its_wait_commands(its);
+}
+
+/*
  * Map a hardware device, identified by a certain host ITS and its device ID
  * to domain d, a guest ITS (identified by its doorbell address) and device ID.
  * Also provide the number of events (MSIs) needed for that device.
@@ -528,7 +593,7 @@ int gicv3_its_map_guest_device(struct domain *d,
     struct host_its *hw_its;
     struct its_devices *dev = NULL, *temp;
     struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
-    int ret = -ENOENT;
+    int ret = -ENOENT, i;
 
     hw_its = gicv3_its_find_by_doorbell(host_doorbell);
     if ( !hw_its )
@@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
     if ( !dev )
         goto out_unlock;
 
+    dev->host_lpis = xzalloc_array(uint32_t,
+                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
+    if ( !dev->host_lpis )
+        goto out_unlock;
+
     ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
                             virt_to_maddr(itt_addr), true);
     if ( ret )
@@ -591,10 +661,26 @@ int gicv3_its_map_guest_device(struct domain *d,
 
     spin_unlock(&d->arch.vgic.its_devices_lock);
 
+    /*
+     * Map all host LPIs within this device already. We can't afford to queue
+     * any host ITS commands later on during the guest's runtime.
+     */
+    for ( i = 0; i < DIV_ROUND_UP(nr_events, LPI_BLOCK); i++ )
+    {
+        ret = gicv3_allocate_host_lpi_block(hw_its, d, host_devid,
+                                            i * LPI_BLOCK);
+        if ( ret < 0 )
+            goto out;
+
+        dev->host_lpis[i] = ret;
+    }
+
     return 0;
 
 out_unlock:
     spin_unlock(&d->arch.vgic.its_devices_lock);
+
+out:
     if ( dev )
         xfree(dev->host_lpis);
     xfree(itt_addr);
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 51d7425..59d3ba4 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -20,24 +20,44 @@
 
 #include <xen/lib.h>
 #include <xen/mm.h>
+#include <xen/sched.h>
+#include <asm/atomic.h>
+#include <asm/domain.h>
 #include <asm/gic.h>
 #include <asm/gic_v3_defs.h>
 #include <asm/gic_v3_its.h>
 #include <asm/io.h>
 #include <asm/page.h>
 
+/* LPIs on the host always go to a guest, so no struct irq_desc for them. */
+union host_lpi {
+    uint64_t data;
+    struct {
+        uint32_t virt_lpi;
+        uint16_t dom_id;
+        uint16_t vcpu_id;
+    };
+};
+
 #define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
 /* Global state */
 static struct {
     /* The global LPI property table, shared by all redistributors. */
     uint8_t *lpi_property;
     /*
+     * A two-level table to lookup LPIs firing on the host and look up the
+     * domain and virtual LPI number to inject.
+     */
+    union host_lpi **host_lpis;
+    /*
      * Number of physical LPIs the host supports. This is a property of
      * the GIC hardware. We depart from the habit of naming these things
      * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
      * in a different context to differentiate them from "virtual LPIs".
      */
     unsigned long int nr_host_lpis;
+    /* Protects allocation and deallocation of host LPIs, but not the access */
+    spinlock_t host_lpis_lock;
     unsigned int flags;
 } lpi_data;
 
@@ -50,6 +70,19 @@ struct lpi_redist_data {
 static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
 
 #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
+#define HOST_LPIS_PER_PAGE      (PAGE_SIZE / sizeof(union host_lpi))
+
+static union host_lpi *gic_get_host_lpi(uint32_t plpi)
+{
+    if ( !is_lpi(plpi) || plpi >= MAX_PHYS_LPIS + LPI_OFFSET )
+        return NULL;
+
+    plpi -= LPI_OFFSET;
+    if ( !lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE] )
+        return NULL;
+
+    return &lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE][plpi % HOST_LPIS_PER_PAGE];
+}
 
 /* Stores this redistributor's physical address and ID in a per-CPU variable */
 void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
@@ -204,15 +237,170 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base)
 static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
 integer_param("max_lpi_bits", max_lpi_bits);
 
+/*
+ * Allocate the 2nd level array for host LPIs. This one holds pointers
+ * to the page with the actual "union host_lpi" entries. Our LPI limit
+ * avoids excessive memory usage.
+ */
 int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
 {
+    int nr_lpi_ptrs;
+
+    /* We rely on the data structure being atomically accessible. */
+    BUILD_BUG_ON(sizeof(union host_lpi) > sizeof(unsigned long));
+
     lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
 
+    spin_lock_init(&lpi_data.host_lpis_lock);
+
+    nr_lpi_ptrs = MAX_PHYS_LPIS / (PAGE_SIZE / sizeof(union host_lpi));
+    lpi_data.host_lpis = xzalloc_array(union host_lpi *, nr_lpi_ptrs);
+    if ( !lpi_data.host_lpis )
+        return -ENOMEM;
+
     printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);
 
     return 0;
 }
 
+/* Must be called with host_lpis_lock held. */
+static int find_unused_host_lpi(int start, uint32_t *index)
+{
+    int chunk;
+    uint32_t i = *index;
+
+    for ( chunk = start; chunk < MAX_PHYS_LPIS / HOST_LPIS_PER_PAGE; chunk++ )
+    {
+        /* If we hit an unallocated chunk, use entry 0 in that one. */
+        if ( !lpi_data.host_lpis[chunk] )
+        {
+            *index = 0;
+            return chunk;
+        }
+
+        /* Find an unallocated entry in this chunk. */
+        for ( ; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
+        {
+            if ( lpi_data.host_lpis[chunk][i].dom_id == INVALID_DOMID )
+            {
+                *index = i;
+                return chunk;
+            }
+        }
+        i = 0;
+    }
+
+    return -1;
+}
+
+/*
+ * Allocate a block of 32 LPIs on the given host ITS for device "devid",
+ * starting with "eventid". Put them into the respective ITT by issuing a
+ * MAPTI command for each of them.
+ */
+int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
+                                  uint32_t host_devid, uint32_t eventid)
+{
+    static uint32_t next_lpi = 0;
+    uint32_t lpi, lpi_idx = next_lpi % HOST_LPIS_PER_PAGE;
+    int chunk;
+    int i;
+
+    spin_lock(&lpi_data.host_lpis_lock);
+    chunk = find_unused_host_lpi(next_lpi / HOST_LPIS_PER_PAGE, &lpi_idx);
+
+    if ( chunk == - 1 )          /* rescan for a hole from the beginning */
+    {
+        lpi_idx = 0;
+        chunk = find_unused_host_lpi(0, &lpi_idx);
+        if ( chunk == -1 )
+        {
+            spin_unlock(&lpi_data.host_lpis_lock);
+            return -ENOSPC;
+        }
+    }
+
+    /* If we hit an unallocated chunk, we initialize it and use entry 0. */
+    if ( !lpi_data.host_lpis[chunk] )
+    {
+        union host_lpi *new_chunk;
+
+        new_chunk = alloc_xenheap_pages(0, 0);
+        if ( !new_chunk )
+        {
+            spin_unlock(&lpi_data.host_lpis_lock);
+            return -ENOMEM;
+        }
+
+        for ( i = 0; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
+            new_chunk[i].dom_id = INVALID_DOMID;
+
+        lpi_data.host_lpis[chunk] = new_chunk;
+        lpi_idx = 0;
+    }
+
+    lpi = chunk * HOST_LPIS_PER_PAGE + lpi_idx;
+
+    for ( i = 0; i < LPI_BLOCK; i++ )
+    {
+        union host_lpi hlpi;
+
+        /*
+         * Mark this host LPI as belonging to the domain, but don't assign
+         * any virtual LPI or a VCPU yet.
+         */
+        hlpi.virt_lpi = INVALID_LPI;
+        hlpi.dom_id = d->domain_id;
+        hlpi.vcpu_id = INVALID_DOMID;
+        write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
+                         hlpi.data);
+
+        /*
+         * Enable this host LPI, so we don't have to do this during the
+         * guest's runtime.
+         */
+        lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
+    }
+
+    /*
+     * We have allocated and initialized the host LPI entries, so it's safe
+     * to drop the lock now. Access to the structures can be done concurrently
+     * as it involves only an atomic uint64_t access.
+     */
+    spin_unlock(&lpi_data.host_lpis_lock);
+
+    if ( lpi_data.flags & LPI_PROPTABLE_NEEDS_FLUSHING )
+        clean_and_invalidate_dcache_va_range(&lpi_data.lpi_property[lpi],
+                                             LPI_BLOCK);
+
+    gicv3_its_map_host_events(its, host_devid, eventid, lpi + LPI_OFFSET,
+                              LPI_BLOCK);
+
+    next_lpi = lpi + LPI_BLOCK;
+    return lpi + LPI_OFFSET;
+}
+
+int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi)
+{
+    union host_lpi *hlpi, empty_lpi = { .dom_id = INVALID_DOMID };
+    int i;
+
+    hlpi = gic_get_host_lpi(lpi);
+    if ( !hlpi )
+        return -ENOENT;
+
+    spin_lock(&lpi_data.host_lpis_lock);
+
+    for ( i = 0; i < LPI_BLOCK; i++ )
+        write_u64_atomic(&hlpi[i].data, empty_lpi.data);
+
+    /* TODO: Call a function in gic-v3-its.c to send DISCARDs */
+
+    spin_unlock(&lpi_data.host_lpis_lock);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
index 12bd155..7825575 100644
--- a/xen/include/asm-arm/gic.h
+++ b/xen/include/asm-arm/gic.h
@@ -220,7 +220,12 @@ enum gic_version {
     GIC_V3,
 };
 
+#define INVALID_LPI     0
 #define LPI_OFFSET      8192
+static inline bool is_lpi(unsigned int irq)
+{
+    return irq >= LPI_OFFSET;
+}
 
 extern enum gic_version gic_hw_version(void);
 
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 3421ea0..2387972 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -106,6 +106,11 @@
 #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
 #define HOST_ITS_USES_PTA               (1U << 1)
 
+#define INVALID_DOMID ((uint16_t)~0)
+
+/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
+#define LPI_BLOCK                       32
+
 /* data structure for each hardware ITS */
 struct host_its {
     struct list_head entry;
@@ -151,6 +156,12 @@ int gicv3_its_map_guest_device(struct domain *d,
                                paddr_t host_doorbell, uint32_t host_devid,
                                paddr_t guest_doorbell, uint32_t guest_devid,
                                uint32_t nr_events, bool valid);
+int gicv3_its_map_host_events(struct host_its *its,
+                              uint32_t host_devid, uint32_t eventid,
+                              uint32_t lpi, uint32_t nrevents);
+int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
+                                  uint32_t host_devid, uint32_t eventid);
+int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
 
 #else
 
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (7 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-22 23:44   ` Stefano Stabellini
  2017-03-24 11:40   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests Andre Przywara
                   ` (17 subsequent siblings)
  26 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

For the same reason that allocating a struct irq_desc for each
possible LPI is not an option, having a struct pending_irq for each LPI
is also not feasible. However we actually only need those when an
interrupt is on a vCPU (or is about to be injected).
Maintain a list of those structs that we can use for the lifecycle of
a guest LPI. We allocate new entries if necessary, however reuse
pre-owned entries whenever possible.
I added some locking around this list here, however my gut feeling is
that we don't need one because this a per-VCPU structure anyway.
If someone could confirm this, I'd be grateful.
Teach the existing VGIC functions to find the right pointer when being
given a virtual LPI number.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic.c           |  3 +++
 xen/arch/arm/vgic-v3.c       |  3 +++
 xen/arch/arm/vgic.c          | 64 +++++++++++++++++++++++++++++++++++++++++---
 xen/include/asm-arm/domain.h |  2 ++
 xen/include/asm-arm/vgic.h   | 14 ++++++++++
 5 files changed, 83 insertions(+), 3 deletions(-)

diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index a5348f2..bd3c032 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -509,6 +509,9 @@ static void gic_update_one_lr(struct vcpu *v, int i)
                 struct vcpu *v_target = vgic_get_target_vcpu(v, irq);
                 irq_set_affinity(p->desc, cpumask_of(v_target->processor));
             }
+            /* If this was an LPI, mark this struct as available again. */
+            if ( is_lpi(p->irq) )
+                p->irq = 0;
         }
     }
 }
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 1fadb00..b0653c2 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -1426,6 +1426,9 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
     if ( v->vcpu_id == last_cpu || (v->vcpu_id == (d->max_vcpus - 1)) )
         v->arch.vgic.flags |= VGIC_V3_RDIST_LAST;
 
+    spin_lock_init(&v->arch.vgic.pending_lpi_list_lock);
+    INIT_LIST_HEAD(&v->arch.vgic.pending_lpi_list);
+
     return 0;
 }
 
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index 364d5f0..e5cfa54 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -30,6 +30,8 @@
 
 #include <asm/mmio.h>
 #include <asm/gic.h>
+#include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
 #include <asm/vgic.h>
 
 static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v, int rank)
@@ -61,7 +63,7 @@ struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq)
     return vgic_get_rank(v, rank);
 }
 
-static void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
+void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
 {
     INIT_LIST_HEAD(&p->inflight);
     INIT_LIST_HEAD(&p->lr_queue);
@@ -244,10 +246,14 @@ struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq)
 
 static int vgic_get_virq_priority(struct vcpu *v, unsigned int virq)
 {
-    struct vgic_irq_rank *rank = vgic_rank_irq(v, virq);
+    struct vgic_irq_rank *rank;
     unsigned long flags;
     int priority;
 
+    if ( is_lpi(virq) )
+        return vgic_lpi_get_priority(v->domain, virq);
+
+    rank = vgic_rank_irq(v, virq);
     vgic_lock_rank(v, rank, flags);
     priority = rank->priority[virq & INTERRUPT_RANK_MASK];
     vgic_unlock_rank(v, rank, flags);
@@ -446,13 +452,63 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
     return true;
 }
 
+/*
+ * Holding struct pending_irq's for each possible virtual LPI in each domain
+ * requires too much Xen memory, also a malicious guest could potentially
+ * spam Xen with LPI map requests. We cannot cover those with (guest allocated)
+ * ITS memory, so we use a dynamic scheme of allocating struct pending_irq's
+ * on demand.
+ */
+struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
+                                   bool allocate)
+{
+    struct lpi_pending_irq *lpi_irq, *empty = NULL;
+
+    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
+    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
+    {
+        if ( lpi_irq->pirq.irq == lpi )
+        {
+            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
+            return &lpi_irq->pirq;
+        }
+
+        if ( lpi_irq->pirq.irq == 0 && !empty )
+            empty = lpi_irq;
+    }
+
+    if ( !allocate )
+    {
+        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
+        return NULL;
+    }
+
+    if ( !empty )
+    {
+        empty = xzalloc(struct lpi_pending_irq);
+        vgic_init_pending_irq(&empty->pirq, lpi);
+        list_add_tail(&empty->entry, &v->arch.vgic.pending_lpi_list);
+    } else
+    {
+        empty->pirq.status = 0;
+        empty->pirq.irq = lpi;
+    }
+
+    spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
+
+    return &empty->pirq;
+}
+
 struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
 {
     struct pending_irq *n;
+
     /* Pending irqs allocation strategy: the first vgic.nr_spis irqs
      * are used for SPIs; the rests are used for per cpu irqs */
     if ( irq < 32 )
         n = &v->arch.vgic.pending_irqs[irq];
+    else if ( is_lpi(irq) )
+        n = lpi_to_pending(v, irq, true);
     else
         n = &v->domain->arch.vgic.pending_irqs[irq - 32];
     return n;
@@ -480,7 +536,7 @@ void vgic_clear_pending_irqs(struct vcpu *v)
 void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
 {
     uint8_t priority;
-    struct pending_irq *iter, *n = irq_to_pending(v, virq);
+    struct pending_irq *iter, *n;
     unsigned long flags;
     bool running;
 
@@ -488,6 +544,8 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
 
     spin_lock_irqsave(&v->arch.vgic.lock, flags);
 
+    n = irq_to_pending(v, virq);
+
     /* vcpu offline */
     if ( test_bit(_VPF_down, &v->pause_flags) )
     {
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 00b9c1a..f44a84b 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -257,6 +257,8 @@ struct arch_vcpu
         paddr_t rdist_base;
 #define VGIC_V3_RDIST_LAST  (1 << 0)        /* last vCPU of the rdist */
         uint8_t flags;
+        struct list_head pending_lpi_list;
+        spinlock_t pending_lpi_list_lock;   /* protects the pending_lpi_list */
     } vgic;
 
     /* Timer registers  */
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 467333c..8f1099c 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -83,6 +83,12 @@ struct pending_irq
     struct list_head lr_queue;
 };
 
+struct lpi_pending_irq
+{
+    struct list_head entry;
+    struct pending_irq pirq;
+};
+
 #define NR_INTERRUPT_PER_RANK   32
 #define INTERRUPT_RANK_MASK (NR_INTERRUPT_PER_RANK - 1)
 
@@ -296,13 +302,21 @@ extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
 extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
 extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
 extern void vgic_clear_pending_irqs(struct vcpu *v);
+extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
 extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
 extern struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq);
+extern struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int irq,
+                                          bool allocate);
 extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v, int b, int n, int s);
 extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
 extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
 extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
 extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
+/* placeholder function until the property table gets introduced */
+static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
+{
+    return 0xa;
+}
 extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
 int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (8 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 12:03   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 11/27] ARM: GICv3: enable ITS and LPIs on the host Andre Przywara
                   ` (16 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Upon receiving an LPI, we need to find the right VCPU and virtual IRQ
number to get this IRQ injected.
Iterate our two-level LPI table to find this information quickly when
the host takes an LPI. Call the existing injection function to let the
GIC emulation deal with this interrupt.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-lpi.c | 41 +++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic.c        |  6 ++++--
 xen/include/asm-arm/irq.h |  8 ++++++++
 3 files changed, 53 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 59d3ba4..0579976 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -104,6 +104,47 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta)
         return per_cpu(lpi_redist, cpu).redist_id << 16;
 }
 
+/*
+ * Handle incoming LPIs, which are a bit special, because they are potentially
+ * numerous and also only get injected into guests. Treat them specially here,
+ * by just looking up their target vCPU and virtual LPI number and hand it
+ * over to the injection function.
+ */
+void do_LPI(unsigned int lpi)
+{
+    struct domain *d;
+    union host_lpi *hlpip, hlpi;
+    struct vcpu *vcpu;
+
+    WRITE_SYSREG32(lpi, ICC_EOIR1_EL1);
+
+    hlpip = gic_get_host_lpi(lpi);
+    if ( !hlpip )
+        return;
+
+    hlpi.data = read_u64_atomic(&hlpip->data);
+
+    /* We may have mapped more host LPIs than the guest actually asked for. */
+    if ( !hlpi.virt_lpi )
+        return;
+
+    d = get_domain_by_id(hlpi.dom_id);
+    if ( !d )
+        return;
+
+    if ( hlpi.vcpu_id >= d->max_vcpus )
+    {
+        put_domain(d);
+        return;
+    }
+
+    vcpu = d->vcpu[hlpi.vcpu_id];
+
+    put_domain(d);
+
+    vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
+}
+
 static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
 {
     uint64_t val;
diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
index bd3c032..7286e5d 100644
--- a/xen/arch/arm/gic.c
+++ b/xen/arch/arm/gic.c
@@ -700,8 +700,10 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
             local_irq_enable();
             do_IRQ(regs, irq, is_fiq);
             local_irq_disable();
-        }
-        else if (unlikely(irq < 16))
+        } else if ( is_lpi(irq) )
+        {
+            do_LPI(irq);
+        } else if ( unlikely(irq < 16) )
         {
             do_sgi(regs, irq);
         }
diff --git a/xen/include/asm-arm/irq.h b/xen/include/asm-arm/irq.h
index 8f7a167..ee47de8 100644
--- a/xen/include/asm-arm/irq.h
+++ b/xen/include/asm-arm/irq.h
@@ -34,6 +34,14 @@ struct irq_desc *__irq_to_desc(int irq);
 
 void do_IRQ(struct cpu_user_regs *regs, unsigned int irq, int is_fiq);
 
+#ifdef CONFIG_HAS_ITS
+void do_LPI(unsigned int irq);
+#else
+static inline void do_LPI(unsigned int irq)
+{
+}
+#endif
+
 #define domain_pirq_to_irq(d, pirq) (pirq)
 
 bool_t is_assignable_irq(unsigned int irq);
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 11/27] ARM: GICv3: enable ITS and LPIs on the host
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (9 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 12/27] ARM: vGICv3: handle virtual LPI pending and property tables Andre Przywara
                   ` (15 subsequent siblings)
  26 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Now that the host part of the ITS code is in place, we can enable the
ITS and also LPIs on each redistributor to get the show rolling.
At this point there would be no LPIs mapped, as guests don't know about
the ITS yet.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c |  4 ++++
 xen/arch/arm/gic-v3.c     | 19 +++++++++++++++++++
 2 files changed, 23 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index ed14d95..5a2dbec 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -470,6 +470,10 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
         return -ENOMEM;
     writeq_relaxed(0, hw_its->its_base + GITS_CWRITER);
 
+    /* Now enable interrupt translation and command processing on that ITS. */
+    reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
+    writel_relaxed(reg | GITS_CTLR_ENABLE, hw_its->its_base + GITS_CTLR);
+
     return 0;
 }
 
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 38dafe7..83a839a 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -621,6 +621,21 @@ static int gicv3_enable_redist(void)
     return 0;
 }
 
+/* Enable LPIs on this redistributor (only useful when the host has an ITS). */
+static bool gicv3_enable_lpis(void)
+{
+    uint32_t val;
+
+    val = readl_relaxed(GICD_RDIST_BASE + GICR_TYPER);
+    if ( !(val & GICR_TYPER_PLPIS) )
+        return false;
+
+    val = readl_relaxed(GICD_RDIST_BASE + GICR_CTLR);
+    writel_relaxed(val | GICR_CTLR_ENABLE_LPIS, GICD_RDIST_BASE + GICR_CTLR);
+
+    return true;
+}
+
 static int __init gicv3_populate_rdist(void)
 {
     int i;
@@ -729,8 +744,12 @@ static int gicv3_cpu_init(void)
     if ( gicv3_enable_redist() )
         return -ENODEV;
 
+    /* If the host has any ITSes, enable LPIs now. */
     if ( gicv3_its_host_has_its() )
+    {
         gicv3_its_setup_collection(smp_processor_id());
+        gicv3_enable_lpis();
+    }
 
     /* Set priority on PPI and SGI interrupts */
     priority = (GIC_PRI_IPI << 24 | GIC_PRI_IPI << 16 | GIC_PRI_IPI << 8 |
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 12/27] ARM: vGICv3: handle virtual LPI pending and property tables
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (10 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 11/27] ARM: GICv3: enable ITS and LPIs on the host Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 12:09   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 13/27] ARM: vGICv3: Handle disabled LPIs Andre Przywara
                   ` (14 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Allow a guest to provide the address and size for the memory regions
it has reserved for the GICv3 pending and property tables.
We sanitise the various fields of the respective redistributor
registers and map those pages into Xen's address space to have easy
access.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3.c       | 220 +++++++++++++++++++++++++++++++++++++++----
 xen/arch/arm/vgic.c          |   4 +
 xen/include/asm-arm/domain.h |   8 +-
 xen/include/asm-arm/vgic.h   |  24 ++++-
 4 files changed, 233 insertions(+), 23 deletions(-)

diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index b0653c2..c6db2d7 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -20,12 +20,14 @@
 
 #include <xen/bitops.h>
 #include <xen/config.h>
+#include <xen/domain_page.h>
 #include <xen/lib.h>
 #include <xen/init.h>
 #include <xen/softirq.h>
 #include <xen/irq.h>
 #include <xen/sched.h>
 #include <xen/sizes.h>
+#include <xen/vmap.h>
 #include <asm/current.h>
 #include <asm/mmio.h>
 #include <asm/gic_v3_defs.h>
@@ -229,12 +231,15 @@ static int __vgic_v3_rdistr_rd_mmio_read(struct vcpu *v, mmio_info_t *info,
         goto read_reserved;
 
     case VREG64(GICR_PROPBASER):
-        /* LPI's not implemented */
-        goto read_as_zero_64;
+        if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
+        *r = vgic_reg64_extract(v->domain->arch.vgic.rdist_propbase, info);
+        return 1;
 
     case VREG64(GICR_PENDBASER):
-        /* LPI's not implemented */
-        goto read_as_zero_64;
+        if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
+        *r = vgic_reg64_extract(v->arch.vgic.rdist_pendbase, info);
+        *r &= ~GICR_PENDBASER_PTZ;       /* WO, reads as 0 */
+        return 1;
 
     case 0x0080:
         goto read_reserved;
@@ -302,11 +307,6 @@ bad_width:
     domain_crash_synchronous();
     return 0;
 
-read_as_zero_64:
-    if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
-    *r = 0;
-    return 1;
-
 read_as_zero_32:
     if ( dabt.size != DABT_WORD ) goto bad_width;
     *r = 0;
@@ -331,11 +331,179 @@ read_unknown:
     return 1;
 }
 
+static uint64_t vgic_sanitise_field(uint64_t reg, uint64_t field_mask,
+                                    int field_shift,
+                                    uint64_t (*sanitise_fn)(uint64_t))
+{
+    uint64_t field = (reg & field_mask) >> field_shift;
+
+    field = sanitise_fn(field) << field_shift;
+
+    return (reg & ~field_mask) | field;
+}
+
+/* We want to avoid outer shareable. */
+static uint64_t vgic_sanitise_shareability(uint64_t field)
+{
+    switch (field) {
+    case GIC_BASER_OuterShareable:
+        return GIC_BASER_InnerShareable;
+    default:
+        return field;
+    }
+}
+
+/* Avoid any inner non-cacheable mapping. */
+static uint64_t vgic_sanitise_inner_cacheability(uint64_t field)
+{
+    switch (field) {
+    case GIC_BASER_CACHE_nCnB:
+    case GIC_BASER_CACHE_nC:
+        return GIC_BASER_CACHE_RaWb;
+    default:
+        return field;
+    }
+}
+
+/* Non-cacheable or same-as-inner are OK. */
+static uint64_t vgic_sanitise_outer_cacheability(uint64_t field)
+{
+    switch (field) {
+    case GIC_BASER_CACHE_SameAsInner:
+    case GIC_BASER_CACHE_nC:
+        return field;
+    default:
+        return GIC_BASER_CACHE_nC;
+    }
+}
+
+static uint64_t sanitize_propbaser(uint64_t reg)
+{
+    reg = vgic_sanitise_field(reg, GICR_PROPBASER_SHAREABILITY_MASK,
+                              GICR_PROPBASER_SHAREABILITY_SHIFT,
+                              vgic_sanitise_shareability);
+    reg = vgic_sanitise_field(reg, GICR_PROPBASER_INNER_CACHEABILITY_MASK,
+                              GICR_PROPBASER_INNER_CACHEABILITY_SHIFT,
+                              vgic_sanitise_inner_cacheability);
+    reg = vgic_sanitise_field(reg, GICR_PROPBASER_OUTER_CACHEABILITY_MASK,
+                              GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT,
+                              vgic_sanitise_outer_cacheability);
+
+    reg &= ~GICR_PROPBASER_RES0_MASK;
+    return reg;
+}
+
+static uint64_t sanitize_pendbaser(uint64_t reg)
+{
+    reg = vgic_sanitise_field(reg, GICR_PENDBASER_SHAREABILITY_MASK,
+                              GICR_PENDBASER_SHAREABILITY_SHIFT,
+                              vgic_sanitise_shareability);
+    reg = vgic_sanitise_field(reg, GICR_PENDBASER_INNER_CACHEABILITY_MASK,
+                              GICR_PENDBASER_INNER_CACHEABILITY_SHIFT,
+                              vgic_sanitise_inner_cacheability);
+    reg = vgic_sanitise_field(reg, GICR_PENDBASER_OUTER_CACHEABILITY_MASK,
+                              GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT,
+                              vgic_sanitise_outer_cacheability);
+
+    reg &= ~GICR_PENDBASER_RES0_MASK;
+    return reg;
+}
+
+/*
+ * Mark a given number of guest pages as used (by increasing their refcount),
+ * starting with the given guest address. This needs to be called once before
+ * calling (possibly repeatedly) map_guest_pages().
+ * Before the domain gets destroyed, call put_guest_pages() to drop the
+ * reference.
+ */
+int get_guest_pages(struct domain *d, paddr_t gpa, int nr_pages)
+{
+    int i;
+    struct page_info *page;
+
+    for ( i = 0; i < nr_pages; i++ )
+    {
+        page = get_page_from_gfn(d, (gpa >> PAGE_SHIFT) + i, NULL, P2M_ALLOC);
+        if ( ! page )
+            return -EINVAL;
+    }
+
+    return 0;
+}
+
+void put_guest_pages(struct domain *d, paddr_t gpa, int nr_pages)
+{
+    mfn_t mfn;
+    int i;
+
+    p2m_read_lock(&d->arch.p2m);
+    for ( i = 0; i < nr_pages; i++ )
+    {
+        mfn = p2m_get_entry(&d->arch.p2m, _gfn((gpa >> PAGE_SHIFT) + i),
+                            NULL, NULL, NULL);
+        if ( mfn_eq(mfn, INVALID_MFN) )
+            continue;
+        put_page(mfn_to_page(mfn_x(mfn)));
+    }
+    p2m_read_unlock(&d->arch.p2m);
+}
+
+/*
+ * Provides easy access to guest memory by "mapping" some parts of it into
+ * Xen's VA space. In fact it relies on the memory being already mapped
+ * and just provides a pointer to it.
+ * This allows the ITS configuration data to be held in guest memory and
+ * avoids using Xen's memory for that.
+ */
+void *map_guest_pages(struct domain *d, paddr_t guest_addr, int nr_pages)
+{
+    int i;
+    void *ptr, *follow;
+
+    ptr = map_domain_page(_mfn(guest_addr >> PAGE_SHIFT));
+
+    /* Make sure subsequent pages are mapped in a virtually contigious way. */
+    for ( i = 1; i < nr_pages; i++ )
+    {
+        follow = map_domain_page(_mfn((guest_addr >> PAGE_SHIFT) + i));
+        if ( follow != ptr + ((long)i << PAGE_SHIFT) )
+            return NULL;
+    }
+
+    return ptr + (guest_addr & ~PAGE_MASK);
+}
+
+/* "Unmap" previously mapped guest pages. Should be optimized away on arm64. */
+void unmap_guest_pages(void *va, int nr_pages)
+{
+    long i;
+
+    for ( i = nr_pages - 1; i >= 0; i-- )
+        unmap_domain_page(((uintptr_t)va & PAGE_MASK) + (i << PAGE_SHIFT));
+}
+
+int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
+{
+    if ( vlpi >= d->arch.vgic.nr_lpis )
+        return GIC_PRI_IRQ;
+
+    return d->arch.vgic.proptable[vlpi - LPI_OFFSET] & LPI_PROP_PRIO_MASK;
+}
+
+bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi)
+{
+    if ( vlpi >= d->arch.vgic.nr_lpis )
+        return false;
+
+    return d->arch.vgic.proptable[vlpi - LPI_OFFSET] & LPI_PROP_ENABLED;
+}
+
 static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
                                           uint32_t gicr_reg,
                                           register_t r)
 {
     struct hsr_dabt dabt = info->dabt;
+    uint64_t reg;
 
     switch ( gicr_reg )
     {
@@ -366,36 +534,54 @@ static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
         goto write_impl_defined;
 
     case VREG64(GICR_SETLPIR):
-        /* LPI is not implemented */
+        /* LPIs without an ITS are not implemented */
         goto write_ignore_64;
 
     case VREG64(GICR_CLRLPIR):
-        /* LPI is not implemented */
+        /* LPIs without an ITS are not implemented */
         goto write_ignore_64;
 
     case 0x0050:
         goto write_reserved;
 
     case VREG64(GICR_PROPBASER):
-        /* LPI is not implemented */
-        goto write_ignore_64;
+        if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
+
+        /* Writing PROPBASER with LPIs enabled is UNPREDICTABLE. */
+        if ( v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED )
+            return 1;
+
+        reg = v->domain->arch.vgic.rdist_propbase;
+        vgic_reg64_update(&reg, r, info);
+        reg = sanitize_propbaser(reg);
+        v->domain->arch.vgic.rdist_propbase = reg;
+        return 1;
 
     case VREG64(GICR_PENDBASER):
-        /* LPI is not implemented */
-        goto write_ignore_64;
+        if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
+
+        /* Writing PENDBASER with LPIs enabled is UNPREDICTABLE. */
+        if ( v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED )
+            return 1;
+
+	reg = v->arch.vgic.rdist_pendbase;
+	vgic_reg64_update(&reg, r, info);
+	reg = sanitize_pendbaser(reg);
+	v->arch.vgic.rdist_pendbase = reg;
+	return 1;
 
     case 0x0080:
         goto write_reserved;
 
     case VREG64(GICR_INVLPIR):
-        /* LPI is not implemented */
+        /* LPIs without an ITS are not implemented */
         goto write_ignore_64;
 
     case 0x00A8:
         goto write_reserved;
 
     case VREG64(GICR_INVALLR):
-        /* LPI is not implemented */
+        /* LPIs without an ITS are not implemented */
         goto write_ignore_64;
 
     case 0x00B8:
diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
index e5cfa54..47ba8c0 100644
--- a/xen/arch/arm/vgic.c
+++ b/xen/arch/arm/vgic.c
@@ -494,6 +494,10 @@ struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
         empty->pirq.irq = lpi;
     }
 
+    /* Update the enabled status */
+    if ( vgic_lpi_is_enabled(v->domain, lpi) )
+        set_bit(GIC_IRQ_GUEST_ENABLED, &empty->pirq.status);
+
     spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
 
     return &empty->pirq;
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index f44a84b..33c1851 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -110,6 +110,9 @@ struct arch_domain
         } *rdist_regions;
         int nr_regions;                     /* Number of rdist regions */
         uint32_t rdist_stride;              /* Re-Distributor stride */
+        int nr_lpis;
+        uint64_t rdist_propbase;
+        uint8_t *proptable;
         struct rb_root its_devices;         /* devices mapped to an ITS */
         spinlock_t its_devices_lock;        /* protects the its_devices tree */
 #endif
@@ -255,7 +258,10 @@ struct arch_vcpu
 
         /* GICv3: redistributor base and flags for this vCPU */
         paddr_t rdist_base;
-#define VGIC_V3_RDIST_LAST  (1 << 0)        /* last vCPU of the rdist */
+#define VGIC_V3_RDIST_LAST      (1 << 0)        /* last vCPU of the rdist */
+#define VGIC_V3_LPIS_ENABLED    (1 << 1)
+        uint64_t rdist_pendbase;
+        unsigned long *pendtable;
         uint8_t flags;
         struct list_head pending_lpi_list;
         spinlock_t pending_lpi_list_lock;   /* protects the pending_lpi_list */
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index 8f1099c..f8bccfa 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -285,6 +285,11 @@ VGIC_REG_HELPERS(32, 0x3);
 
 #undef VGIC_REG_HELPERS
 
+int get_guest_pages(struct domain *d, paddr_t gpa, int nr_pages);
+void put_guest_pages(struct domain *d, paddr_t gpa, int nr_pages);
+void *map_guest_pages(struct domain *d, paddr_t guest_addr, int nr_pages);
+void unmap_guest_pages(void *va, int nr_pages);
+
 enum gic_sgi_mode;
 
 /*
@@ -312,14 +317,23 @@ extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
 extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
 extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
 extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
-/* placeholder function until the property table gets introduced */
-static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
-{
-    return 0xa;
-}
 extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
 int vgic_v2_init(struct domain *d, int *mmio_count);
 int vgic_v3_init(struct domain *d, int *mmio_count);
+#ifdef CONFIG_HAS_GICV3
+extern int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi);
+extern bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi);
+#else
+static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
+{
+    return 0xa0;
+}
+
+static inline bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi)
+{
+    return false;
+}
+#endif
 
 extern int domain_vgic_register(struct domain *d, int *mmio_count);
 extern int vcpu_vgic_free(struct vcpu *v);
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 13/27] ARM: vGICv3: Handle disabled LPIs
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (11 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 12/27] ARM: vGICv3: handle virtual LPI pending and property tables Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 12:20   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits Andre Przywara
                   ` (13 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

If a guest disables an LPI, we do not forward this to the associated
host LPI to avoid queueing commands to the host ITS command queue.
So it may happen that an LPI fires nevertheless on the host. In this
case we can bail out early, but have to save the pending state on the
virtual side.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-lpi.c  |  8 ++++++++
 xen/arch/arm/vgic-v3.c     | 12 ++++++++++++
 xen/include/asm-arm/vgic.h |  6 ++++++
 3 files changed, 26 insertions(+)

diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 0579976..994698e 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -142,6 +142,14 @@ void do_LPI(unsigned int lpi)
 
     put_domain(d);
 
+    /*
+     * We keep all host LPIs enabled, so check if it's disabled on the guest
+     * side and just record this LPI in the virtual pending table in this case.
+     * The guest picks it up once it gets enabled again.
+     */
+    if ( !vgic_can_inject_lpi(vcpu, hlpi.virt_lpi) )
+        return;
+
     vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
 }
 
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index c6db2d7..de625bf 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -498,6 +498,18 @@ bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi)
     return d->arch.vgic.proptable[vlpi - LPI_OFFSET] & LPI_PROP_ENABLED;
 }
 
+bool vgic_can_inject_lpi(struct vcpu *vcpu, uint32_t vlpi)
+{
+    if ( vlpi >= vcpu->domain->arch.vgic.nr_lpis )
+        return false;
+
+    if ( vgic_lpi_is_enabled(vcpu->domain, vlpi) )
+        return true;
+
+    set_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable);
+    return false;
+}
+
 static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
                                           uint32_t gicr_reg,
                                           register_t r)
diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
index f8bccfa..302702e 100644
--- a/xen/include/asm-arm/vgic.h
+++ b/xen/include/asm-arm/vgic.h
@@ -323,6 +323,7 @@ int vgic_v3_init(struct domain *d, int *mmio_count);
 #ifdef CONFIG_HAS_GICV3
 extern int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi);
 extern bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi);
+extern bool vgic_can_inject_lpi(struct vcpu *v, uint32_t vlpi);
 #else
 static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
 {
@@ -333,6 +334,11 @@ static inline bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi)
 {
     return false;
 }
+
+static inline bool vgic_can_inject_lpi(struct vcpu *v, uint32_t vlpi)
+{
+    return false;
+}
 #endif
 
 extern int domain_vgic_register(struct domain *d, int *mmio_count);
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (12 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 13/27] ARM: vGICv3: Handle disabled LPIs Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 16:25   ` Shanker Donthineni
  2017-03-24 12:41   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 15/27] ARM: vITS: introduce translation table walks Andre Przywara
                   ` (12 subsequent siblings)
  26 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Create a new file to hold the emulation code for the ITS widget.
For now we emulate the memory mapped ITS registers and provide a stub
to introduce the ITS command handling framework (but without actually
emulating any commands at this time).

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/Makefile             |   1 +
 xen/arch/arm/vgic-v3-its.c        | 487 ++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/vgic-v3.c            |   9 -
 xen/include/asm-arm/gic_v3_defs.h |  19 ++
 4 files changed, 507 insertions(+), 9 deletions(-)
 create mode 100644 xen/arch/arm/vgic-v3-its.c

diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
index 02a8737..e7ce2c83 100644
--- a/xen/arch/arm/Makefile
+++ b/xen/arch/arm/Makefile
@@ -47,6 +47,7 @@ obj-y += traps.o
 obj-y += vgic.o
 obj-y += vgic-v2.o
 obj-$(CONFIG_HAS_GICV3) += vgic-v3.o
+obj-$(CONFIG_HAS_ITS) += vgic-v3-its.o
 obj-y += vm_event.o
 obj-y += vtimer.o
 obj-y += vpsci.o
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
new file mode 100644
index 0000000..5337638
--- /dev/null
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -0,0 +1,487 @@
+/*
+ * xen/arch/arm/vgic-v3-its.c
+ *
+ * ARM Interrupt Translation Service (ITS) emulation
+ *
+ * Andre Przywara <andre.przywara@arm.com>
+ * Copyright (c) 2016,2017 ARM Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License as published by
+ * the Free Software Foundation; under version 2 of the License.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <xen/bitops.h>
+#include <xen/config.h>
+#include <xen/domain_page.h>
+#include <xen/lib.h>
+#include <xen/init.h>
+#include <xen/softirq.h>
+#include <xen/irq.h>
+#include <xen/sched.h>
+#include <xen/sizes.h>
+#include <asm/current.h>
+#include <asm/mmio.h>
+#include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
+#include <asm/vgic.h>
+#include <asm/vgic-emul.h>
+
+/* Data structure to describe a virtual ITS */
+struct virt_its {
+    struct domain *d;
+    spinlock_t vcmd_lock;       /* protects the virtual command buffer */
+    uint64_t cbaser;
+    uint64_t *cmdbuf;
+    int cwriter;
+    int creadr;
+    spinlock_t its_lock;        /* protects the collection and device tables */
+    uint64_t baser0, baser1;
+    uint16_t *coll_table;
+    int max_collections;
+    uint64_t *dev_table;
+    int max_devices;
+    bool enabled;
+};
+
+/*
+ * An Interrupt Translation Table Entry: this is indexed by a
+ * DeviceID/EventID pair and is located in guest memory.
+ */
+struct vits_itte
+{
+    uint32_t vlpi;
+    uint16_t collection;
+};
+
+/**************************************
+ * Functions that handle ITS commands *
+ **************************************/
+
+static uint64_t its_cmd_mask_field(uint64_t *its_cmd,
+                                   int word, int shift, int size)
+{
+    return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT(size) - 1);
+}
+
+#define its_cmd_get_command(cmd)        its_cmd_mask_field(cmd, 0,  0,  8)
+#define its_cmd_get_deviceid(cmd)       its_cmd_mask_field(cmd, 0, 32, 32)
+#define its_cmd_get_size(cmd)           its_cmd_mask_field(cmd, 1,  0,  5)
+#define its_cmd_get_id(cmd)             its_cmd_mask_field(cmd, 1,  0, 32)
+#define its_cmd_get_physical_id(cmd)    its_cmd_mask_field(cmd, 1, 32, 32)
+#define its_cmd_get_collection(cmd)     its_cmd_mask_field(cmd, 2,  0, 16)
+#define its_cmd_get_target_addr(cmd)    its_cmd_mask_field(cmd, 2, 16, 32)
+#define its_cmd_get_validbit(cmd)       its_cmd_mask_field(cmd, 2, 63,  1)
+
+#define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
+
+static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
+                                uint32_t writer)
+{
+    uint64_t *cmdptr;
+
+    if ( !its->cmdbuf )
+        return -1;
+
+    if ( writer >= ITS_CMD_BUFFER_SIZE(its->cbaser) )
+        return -1;
+
+    spin_lock(&its->vcmd_lock);
+
+    while ( its->creadr != writer )
+    {
+        cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
+        switch (its_cmd_get_command(cmdptr))
+        {
+        case GITS_CMD_SYNC:
+            /* We handle ITS commands synchronously, so we ignore SYNC. */
+	    break;
+        default:
+            gdprintk(XENLOG_G_WARNING, "ITS: unhandled ITS command %ld\n",
+                   its_cmd_get_command(cmdptr));
+            break;
+        }
+
+        its->creadr += ITS_CMD_SIZE;
+        if ( its->creadr == ITS_CMD_BUFFER_SIZE(its->cbaser) )
+            its->creadr = 0;
+    }
+    its->cwriter = writer;
+
+    spin_unlock(&its->vcmd_lock);
+
+    return 0;
+}
+
+/*****************************
+ * ITS registers read access *
+ *****************************/
+
+/*
+ * The physical address is encoded slightly differently depending on
+ * the used page size: the highest four bits are stored in the lowest
+ * four bits of the field for 64K pages.
+ */
+static paddr_t get_baser_phys_addr(uint64_t reg)
+{
+    if ( reg & BIT(9) )
+        return (reg & GENMASK(47, 16)) | ((reg & GENMASK(15, 12)) << 36);
+    else
+        return reg & GENMASK(47, 12);
+}
+
+static int vgic_v3_its_mmio_read(struct vcpu *v, mmio_info_t *info,
+                                 register_t *r, void *priv)
+{
+    struct virt_its *its = priv;
+
+    switch ( info->gpa & 0xffff )
+    {
+    case VREG32(GITS_CTLR):
+        if ( info->dabt.size != DABT_WORD ) goto bad_width;
+        *r = vgic_reg32_extract(its->enabled | BIT(31), info);
+	break;
+    case VREG32(GITS_IIDR):
+        if ( info->dabt.size != DABT_WORD ) goto bad_width;
+        *r = vgic_reg32_extract(GITS_IIDR_VALUE, info);
+        break;
+    case VREG64(GITS_TYPER):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(0x1eff1, info);
+        break;
+    case VREG64(GITS_CBASER):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(its->cbaser, info);
+        break;
+    case VREG64(GITS_CWRITER):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(its->cwriter, info);
+        break;
+    case VREG64(GITS_CREADR):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(its->creadr, info);
+        break;
+    case VREG64(GITS_BASER0):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(its->baser0, info);
+        break;
+    case VREG64(GITS_BASER1):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(its->baser1, info);
+        break;
+    case VRANGE64(GITS_BASER2, GITS_BASER7):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        *r = vgic_reg64_extract(0, info);
+        break;
+    case VREG32(GICD_PIDR2):
+        if ( info->dabt.size != DABT_WORD ) goto bad_width;
+        *r = vgic_reg32_extract(GICV3_GICD_PIDR2, info);
+        break;
+    }
+
+    return 1;
+
+bad_width:
+    domain_crash_synchronous();
+
+    return 0;
+}
+
+/******************************
+ * ITS registers write access *
+ ******************************/
+
+static int its_baser_table_size(uint64_t baser)
+{
+    int page_size = 0;
+
+    switch ( (baser >> 8) & 3 )
+    {
+    case 0: page_size = SZ_4K; break;
+    case 1: page_size = SZ_16K; break;
+    case 2:
+    case 3: page_size = SZ_64K; break;
+    }
+
+    return page_size * ((baser & GENMASK(7, 0)) + 1);
+}
+
+static int its_baser_nr_entries(uint64_t baser)
+{
+    int entry_size = ((baser & GENMASK(52, 48)) >> 48) + 1;
+
+    return its_baser_table_size(baser) / entry_size;
+}
+
+static void vgic_its_map_cmdbuf(struct virt_its *its)
+{
+    if ( its->cmdbuf || !(its->cbaser & GITS_VALID_BIT) )
+        return;
+
+    get_guest_pages(its->d, its->cbaser & GENMASK(51, 12),
+                    (its->cbaser & 0xff) + 1);
+    its->cmdbuf = map_guest_pages(its->d, its->cbaser & GENMASK(51, 12),
+                                  (its->cbaser & 0xff) + 1);
+}
+
+static void vgic_its_unmap_cmdbuf(struct virt_its *its)
+{
+    int nr_pages = (its->cbaser & 0xff) + 1;
+
+    if ( !its->cmdbuf )
+        return;
+
+    unmap_guest_pages(its->cmdbuf, nr_pages);
+    put_guest_pages(its->d, its->cbaser & GENMASK(51, 12), nr_pages);
+
+    its->cmdbuf = NULL;
+}
+
+static void* vgic_its_map_its_table(struct virt_its *its, uint64_t reg)
+{
+    void *ret;
+    int table_size = its_baser_table_size(reg);
+
+    if ( !(reg & GITS_VALID_BIT) )
+        return NULL;
+
+    get_guest_pages(its->d, get_baser_phys_addr(reg), table_size >> PAGE_SHIFT);
+    ret = map_guest_pages(its->d, get_baser_phys_addr(reg),
+                          table_size >> PAGE_SHIFT);
+    memset(ret, 0, table_size);
+
+    return ret;
+}
+
+static void vgic_its_unmap_its_table(struct domain *d, void *table,
+                                     uint64_t reg)
+{
+    if ( !table )
+        return;
+
+    unmap_guest_pages(table, its_baser_table_size(reg) >> PAGE_SHIFT);
+    put_guest_pages(d, get_baser_phys_addr(reg),
+                    its_baser_table_size(reg) >> PAGE_SHIFT);
+}
+
+static void vgic_v3_its_change_its_status(struct virt_its *its, bool status)
+{
+    if ( !status )
+    {
+        its->enabled = false;
+        return;
+    }
+
+    vgic_its_map_cmdbuf(its);
+
+    if ( !its->dev_table )
+        its->dev_table = vgic_its_map_its_table(its, its->baser0);
+
+    if ( !its->coll_table )
+        its->coll_table = vgic_its_map_its_table(its, its->baser1);
+
+    its->enabled = true;
+}
+
+static void sanitize_its_base_reg(uint64_t *reg)
+{
+    uint64_t r = *reg;
+
+    /* Avoid outer shareable. */
+    switch ( (r >> GITS_BASER_SHAREABILITY_SHIFT) & 0x03 )
+    {
+    case GIC_BASER_OuterShareable:
+        r = r & ~GITS_BASER_SHAREABILITY_MASK;
+        r |= GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
+        break;
+    default:
+        break;
+    }
+
+    /* Avoid any inner non-cacheable mapping. */
+    switch ( (r >> GITS_BASER_INNER_CACHEABILITY_SHIFT) & 0x07 )
+    {
+    case GIC_BASER_CACHE_nCnB:
+    case GIC_BASER_CACHE_nC:
+        r = r & ~GITS_BASER_INNER_CACHEABILITY_MASK;
+        r |= GIC_BASER_CACHE_RaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
+        break;
+    default:
+        break;
+    }
+
+    /* Only allow non-cacheable or same-as-inner. */
+    switch ( (r >> GITS_BASER_OUTER_CACHEABILITY_SHIFT) & 0x07 )
+    {
+    case GIC_BASER_CACHE_SameAsInner:
+    case GIC_BASER_CACHE_nC:
+        break;
+    default:
+        r = r & ~GITS_BASER_OUTER_CACHEABILITY_MASK;
+        r |= GIC_BASER_CACHE_nC << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
+        break;
+    }
+
+    *reg = r;
+}
+
+static int vgic_v3_its_mmio_write(struct vcpu *v, mmio_info_t *info,
+                                  register_t r, void *priv)
+{
+    struct domain *d = v->domain;
+    struct virt_its *its = priv;
+    uint64_t reg;
+    uint32_t reg32, ctlr;
+
+    switch ( info->gpa & 0xffff )
+    {
+    case VREG32(GITS_CTLR):
+        if ( info->dabt.size != DABT_WORD ) goto bad_width;
+
+        ctlr = its->enabled ? GITS_CTLR_ENABLE : 0;
+        reg32 = ctlr;
+        vgic_reg32_update(&reg32, r, info);
+        its->enabled = reg32 & GITS_CTLR_ENABLE;
+
+        if ( ctlr ^ reg32 )
+            vgic_v3_its_change_its_status(its, its->enabled);
+        return 1;
+
+    case VREG32(GITS_IIDR):
+        goto write_ignore_32;
+    case VREG32(GITS_TYPER):
+        goto write_ignore_32;
+    case VREG64(GITS_CBASER):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+
+        /* Changing base registers with the ITS enabled is UNPREDICTABLE. */
+        if ( its->enabled )
+            return 1;
+
+        reg = its->cbaser;
+        vgic_reg64_update(&reg, r, info);
+        sanitize_its_base_reg(&reg);
+
+        vgic_its_unmap_cmdbuf(its);
+        its->cbaser = reg;
+
+	return 1;
+
+    case VREG64(GITS_CWRITER):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+        reg = its->cwriter & 0xfffe0;
+        vgic_reg64_update(&reg, r, info);
+        its->cwriter = reg & 0xfffe0;
+
+        if ( its->enabled )
+            vgic_its_handle_cmds(d, its, reg);
+
+        return 1;
+
+    case VREG64(GITS_CREADR):
+        goto write_ignore_64;
+    case VREG64(GITS_BASER0):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+
+        /* Changing base registers with the ITS enabled is UNPREDICTABLE. */
+        if ( its->enabled )
+            return 1;
+
+        reg = its->baser0;
+        vgic_reg64_update(&reg, r, info);
+
+        reg &= ~GITS_BASER_RO_MASK;
+        reg |= (sizeof(uint64_t) - 1) << GITS_BASER_ENTRY_SIZE_SHIFT;
+        reg |= GITS_BASER_TYPE_DEVICE << GITS_BASER_TYPE_SHIFT;
+        sanitize_its_base_reg(&reg);
+
+        /* Has the table address been changed or invalidated? */
+        if ( !(reg & GITS_VALID_BIT) ||
+             get_baser_phys_addr(reg) != get_baser_phys_addr(its->baser0) )
+        {
+            vgic_its_unmap_its_table(its->d, its->dev_table, reg);
+            its->dev_table = NULL;
+        }
+
+        if ( reg & GITS_VALID_BIT )
+            its->max_devices = its_baser_nr_entries(reg);
+        else
+            its->max_devices = 0;
+
+        its->baser0 = reg;
+        return 1;
+    case VREG64(GITS_BASER1):
+        if ( info->dabt.size < DABT_WORD ) goto bad_width;
+
+        /* Changing base registers with the ITS enabled is UNPREDICTABLE. */
+        if ( its->enabled )
+            return 1;
+
+        reg = its->baser1;
+        vgic_reg64_update(&reg, r, info);
+        reg &= ~GITS_BASER_RO_MASK;
+        reg |= (sizeof(uint16_t) - 1) << GITS_BASER_ENTRY_SIZE_SHIFT;
+        reg |= GITS_BASER_TYPE_COLLECTION << GITS_BASER_TYPE_SHIFT;
+        sanitize_its_base_reg(&reg);
+
+        if ( !(reg & GITS_VALID_BIT) ||
+             get_baser_phys_addr(reg) != get_baser_phys_addr(its->baser1) )
+        {
+            vgic_its_unmap_its_table(its->d, its->coll_table, reg);
+            its->coll_table = NULL;
+        }
+
+        if ( reg & GITS_VALID_BIT )
+            its->max_collections = its_baser_nr_entries(reg);
+        else
+            its->max_collections = 0;
+        its->baser1 = reg;
+        return 1;
+    case VRANGE64(GITS_BASER2, GITS_BASER7):
+        goto write_ignore_64;
+    default:
+        gdprintk(XENLOG_G_WARNING, "ITS: unhandled ITS register 0x%lx\n",
+                 info->gpa & 0xffff);
+        return 0;
+    }
+
+    return 1;
+
+write_ignore_64:
+    if ( ! vgic_reg64_check_access(info->dabt) ) goto bad_width;
+    return 1;
+
+write_ignore_32:
+    if ( info->dabt.size != DABT_WORD ) goto bad_width;
+    return 1;
+
+bad_width:
+    printk(XENLOG_G_ERR "%pv vGICR: bad read width %d r%d offset %#08lx\n",
+           v, info->dabt.size, info->dabt.reg, info->gpa & 0xffff);
+
+    domain_crash_synchronous();
+
+    return 0;
+}
+
+static const struct mmio_handler_ops vgic_its_mmio_handler = {
+    .read  = vgic_v3_its_mmio_read,
+    .write = vgic_v3_its_mmio_write,
+};
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index de625bf..d1382be 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -159,15 +159,6 @@ static void vgic_store_irouter(struct domain *d, struct vgic_irq_rank *rank,
     rank->vcpu[offset] = new_vcpu->vcpu_id;
 }
 
-static inline bool vgic_reg64_check_access(struct hsr_dabt dabt)
-{
-    /*
-     * 64 bits registers can be accessible using 32-bit and 64-bit unless
-     * stated otherwise (See 8.1.3 ARM IHI 0069A).
-     */
-    return ( dabt.size == DABT_DOUBLE_WORD || dabt.size == DABT_WORD );
-}
-
 static int __vgic_v3_rdistr_rd_mmio_read(struct vcpu *v, mmio_info_t *info,
                                          uint32_t gicr_reg,
                                          register_t *r)
diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
index 878bae2..1e88d6b 100644
--- a/xen/include/asm-arm/gic_v3_defs.h
+++ b/xen/include/asm-arm/gic_v3_defs.h
@@ -153,6 +153,16 @@
 #define LPI_PROP_RES1                (1 << 1)
 #define LPI_PROP_ENABLED             (1 << 0)
 
+/*
+ * PIDR2: Only bits[7:4] are not implementation defined. We are
+ * emulating a GICv3 ([7:4] = 0x3).
+ *
+ * We don't emulate a specific registers scheme so implement the others
+ * bits as RES0 as recommended by the spec (see 8.1.13 in ARM IHI 0069A).
+ */
+#define GICV3_GICD_PIDR2  0x30
+#define GICV3_GICR_PIDR2  GICV3_GICD_PIDR2
+
 #define GICH_VMCR_EOI                (1 << 9)
 #define GICH_VMCR_VENG1              (1 << 1)
 
@@ -196,6 +206,15 @@ struct rdist_region {
     bool single_rdist;
 };
 
+/*
+ * 64 bits registers can be accessible using 32-bit and 64-bit unless
+ * stated otherwise (See 8.1.3 ARM IHI 0069A).
+ */
+static inline bool vgic_reg64_check_access(struct hsr_dabt dabt)
+{
+    return ( dabt.size == DABT_DOUBLE_WORD || dabt.size == DABT_WORD );
+}
+
 #endif /* __ASM_ARM_GIC_V3_DEFS_H__ */
 
 /*
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 15/27] ARM: vITS: introduce translation table walks
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (13 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 13:00   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 16/27] ARM: vITS: handle CLEAR command Andre Przywara
                   ` (11 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The ITS stores the target (v)CPU and the (virtual) LPI number in tables.
Introduce functions to walk those tables and translate an device ID -
event ID pair into a pair of virtual LPI and vCPU.
Since the final interrupt translation tables can be smaller than a page,
we map them on demand (which is cheap on arm64). Also we take care of
the locking on the way, since we can't easily protect those ITTs from
being altered by the guest.

To allow compiling without warnings, we declare two functions as
non-static for the moment.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 135 +++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 135 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 5337638..267a573 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -62,6 +62,141 @@ struct vits_itte
     uint16_t collection;
 };
 
+#define UNMAPPED_COLLECTION      ((uint16_t)~0)
+
+/* Must be called with the ITS lock held. */
+static struct vcpu *get_vcpu_from_collection(struct virt_its *its, int collid)
+{
+    uint16_t vcpu_id;
+
+    if ( collid >= its->max_collections )
+        return NULL;
+
+    vcpu_id = its->coll_table[collid];
+    if ( vcpu_id == UNMAPPED_COLLECTION || vcpu_id >= its->d->max_vcpus )
+        return NULL;
+
+    return its->d->vcpu[vcpu_id];
+}
+
+#define DEV_TABLE_ITT_ADDR(x) ((x) & GENMASK(51, 8))
+#define DEV_TABLE_ITT_SIZE(x) (BIT(((x) & GENMASK(7, 0)) + 1))
+#define DEV_TABLE_ENTRY(addr, bits)                     \
+        (((addr) & GENMASK(51, 8)) | (((bits) - 1) & GENMASK(7, 0)))
+
+static paddr_t get_itte_address(struct virt_its *its,
+                                uint32_t devid, uint32_t evid)
+{
+    paddr_t addr;
+
+    if ( devid >= its->max_devices )
+        return ~0;
+
+    if ( evid >= DEV_TABLE_ITT_SIZE(its->dev_table[devid]) )
+        return ~0;
+
+    addr = DEV_TABLE_ITT_ADDR(its->dev_table[devid]);
+
+    return addr + evid * sizeof(struct vits_itte);
+}
+
+/*
+ * Looks up a given deviceID/eventID pair on an ITS and returns a pointer to
+ * the corresponding ITTE. This maps the respective guest page into Xen.
+ * Once finished with handling the ITTE, call put_devid_evid() to unmap
+ * the page again.
+ * Must be called with the ITS lock held.
+ */
+static struct vits_itte *get_devid_evid(struct virt_its *its,
+                                        uint32_t devid, uint32_t evid)
+{
+    paddr_t addr = get_itte_address(its, devid, evid);
+
+    if ( addr == ~0 )
+        return NULL;
+
+    return map_guest_pages(its->d, addr, 1);
+}
+
+/* Must be called with the ITS lock held. */
+static void put_devid_evid(struct virt_its *its, struct vits_itte *itte)
+{
+    unmap_guest_pages(itte, 1);
+}
+
+/*
+ * Queries the collection and device tables to get the vCPU and virtual
+ * LPI number for a given guest event. This takes care of mapping the
+ * respective tables and validating the values, since we can't efficiently
+ * protect the ITTs with their less-than-page-size granularity.
+ * Takes and drops the its_lock.
+ */
+bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
+               struct vcpu **vcpu, uint32_t *vlpi)
+{
+    struct vits_itte *itte;
+    int collid;
+    uint32_t _vlpi;
+    struct vcpu *_vcpu;
+
+    spin_lock(&its->its_lock);
+    itte = get_devid_evid(its, devid, evid);
+    if ( !itte )
+    {
+        spin_unlock(&its->its_lock);
+        return false;
+    }
+    collid = itte->collection;
+    _vlpi = itte->vlpi;
+    put_devid_evid(its, itte);
+
+    _vcpu = get_vcpu_from_collection(its, collid);
+    spin_unlock(&its->its_lock);
+
+    if ( !_vcpu )
+        return false;
+
+    if ( collid >= its->max_collections )
+        return false;
+
+    *vcpu = _vcpu;
+    *vlpi = _vlpi;
+
+    return true;
+}
+
+#define SKIP_LPI_UPDATE 1
+bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
+                uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
+{
+    struct vits_itte *itte;
+
+    if ( collid >= its->max_collections )
+        return false;
+
+    /* TODO: validate vlpi */
+
+    spin_lock(&its->its_lock);
+    itte = get_devid_evid(its, devid, evid);
+    if ( !itte )
+    {
+        spin_unlock(&its->its_lock);
+        return false;
+    }
+
+    itte->collection = collid;
+    if ( vlpi != SKIP_LPI_UPDATE )
+        itte->vlpi = vlpi;
+
+    if ( vcpu )
+        *vcpu = get_vcpu_from_collection(its, collid);
+
+    put_devid_evid(its, itte);
+    spin_unlock(&its->its_lock);
+
+    return true;
+}
+
 /**************************************
  * Functions that handle ITS commands *
  **************************************/
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 16/27] ARM: vITS: handle CLEAR command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (14 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 15/27] ARM: vITS: introduce translation table walks Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 14:27   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 17/27] ARM: vITS: handle INT command Andre Przywara
                   ` (10 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

This introduces the ITS command handler for the CLEAR command, which
clears the pending state of an LPI.
This removes a not-yet injected, but already queued IRQ from a VCPU.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 35 +++++++++++++++++++++++++++++++++--
 1 file changed, 33 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 267a573..e808f43 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -131,8 +131,8 @@ static void put_devid_evid(struct virt_its *its, struct vits_itte *itte)
  * protect the ITTs with their less-than-page-size granularity.
  * Takes and drops the its_lock.
  */
-bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
-               struct vcpu **vcpu, uint32_t *vlpi)
+static bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
+                      struct vcpu **vcpu, uint32_t *vlpi)
 {
     struct vits_itte *itte;
     int collid;
@@ -216,6 +216,34 @@ static uint64_t its_cmd_mask_field(uint64_t *its_cmd,
 #define its_cmd_get_target_addr(cmd)    its_cmd_mask_field(cmd, 2, 16, 32)
 #define its_cmd_get_validbit(cmd)       its_cmd_mask_field(cmd, 2, 63,  1)
 
+static int its_handle_clear(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    uint32_t eventid = its_cmd_get_id(cmdptr);
+    struct pending_irq *pirq;
+    struct vcpu *vcpu;
+    uint32_t vlpi;
+
+    if ( !read_itte(its, devid, eventid, &vcpu, &vlpi) )
+        return -1;
+
+    /* Remove a pending, but not yet injected guest IRQ. */
+    pirq = lpi_to_pending(vcpu, vlpi, false);
+    if ( pirq )
+    {
+        clear_bit(GIC_IRQ_GUEST_QUEUED, &pirq->status);
+        gic_remove_from_queues(vcpu, vlpi);
+
+        /* Mark this pending IRQ struct as availabe again. */
+        if ( !test_bit(GIC_IRQ_GUEST_VISIBLE, &pirq->status) )
+            pirq->irq = 0;
+    }
+
+    clear_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable);
+
+    return 0;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -236,6 +264,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
         switch (its_cmd_get_command(cmdptr))
         {
+        case GITS_CMD_CLEAR:
+            its_handle_clear(its, cmdptr);
+            break;
         case GITS_CMD_SYNC:
             /* We handle ITS commands synchronously, so we ignore SYNC. */
 	    break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 17/27] ARM: vITS: handle INT command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (15 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 16/27] ARM: vITS: handle CLEAR command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 14:38   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 18/27] ARM: vITS: handle MAPC command Andre Przywara
                   ` (9 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The INT command sets a given LPI identified by a DeviceID/EventID pair
as pending and thus triggers it to be injected.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index e808f43..23476f9 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -244,6 +244,26 @@ static int its_handle_clear(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+static int its_handle_int(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    uint32_t eventid = its_cmd_get_id(cmdptr);
+    struct vcpu *vcpu;
+    uint32_t vlpi;
+    uint8_t prop;
+
+    if ( !read_itte(its, devid, eventid, &vcpu, &vlpi) )
+        return -1;
+
+    prop = vcpu->domain->arch.vgic.proptable[vlpi - LPI_OFFSET];
+    if ( prop & LPI_PROP_ENABLED )
+        vgic_vcpu_inject_irq(vcpu, vlpi);
+    else
+        set_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable);
+
+    return 0;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -267,6 +287,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_CLEAR:
             its_handle_clear(its, cmdptr);
             break;
+        case GITS_CMD_INT:
+            its_handle_int(its, cmdptr);
+            break;
         case GITS_CMD_SYNC:
             /* We handle ITS commands synchronously, so we ignore SYNC. */
 	    break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 18/27] ARM: vITS: handle MAPC command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (16 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 17/27] ARM: vITS: handle INT command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 19/27] ARM: vITS: handle MAPD command Andre Przywara
                   ` (8 subsequent siblings)
  26 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The MAPC command associates a given collection ID with a given
redistributor, thus mapping collections to VCPUs.
We just store the vcpu_id in the collection table for that.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 30 ++++++++++++++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 23476f9..175a213 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -264,6 +264,33 @@ static int its_handle_int(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+static int its_handle_mapc(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t collid = its_cmd_get_collection(cmdptr);
+    uint64_t rdbase = its_cmd_mask_field(cmdptr, 2, 16, 44);
+    int ret = -1;
+
+    if ( collid >= its->max_collections )
+        return ret;
+
+    if ( rdbase >= its->d->max_vcpus )
+        return ret;
+
+    spin_lock(&its->its_lock);
+    if ( its->coll_table )
+    {
+        if ( its_cmd_get_validbit(cmdptr) )
+            its->coll_table[collid] = rdbase;
+        else
+            its->coll_table[collid] = UNMAPPED_COLLECTION;
+
+        ret = 0;
+    }
+    spin_unlock(&its->its_lock);
+
+    return ret;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -290,6 +317,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_INT:
             its_handle_int(its, cmdptr);
             break;
+        case GITS_CMD_MAPC:
+            its_handle_mapc(its, cmdptr);
+            break;
         case GITS_CMD_SYNC:
             /* We handle ITS commands synchronously, so we ignore SYNC. */
 	    break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 19/27] ARM: vITS: handle MAPD command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (17 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 18/27] ARM: vITS: handle MAPC command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 14:41   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 20/27] ARM: vITS: handle MAPTI command Andre Przywara
                   ` (7 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The MAPD command maps a device by associating a memory region for
storing ITEs with a certain device ID.
We store the given guest physical address in the device table, and, if
this command comes from Dom0, tell the host ITS driver about this new
mapping, so it can issue the corresponing host MAPD command and create
the required tables.
We don't map the device tables permanently, as their alignment
requirement is only 256 Bytes, thus making mapping of several tables
complicated. Instead we map the device tables on demand when we need
them later.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 44 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 175a213..c26d5d4 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -38,6 +38,7 @@
 /* Data structure to describe a virtual ITS */
 struct virt_its {
     struct domain *d;
+    paddr_t doorbell_address;
     spinlock_t vcmd_lock;       /* protects the virtual command buffer */
     uint64_t cbaser;
     uint64_t *cmdbuf;
@@ -291,6 +292,46 @@ static int its_handle_mapc(struct virt_its *its, uint64_t *cmdptr)
     return ret;
 }
 
+static int its_handle_mapd(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    int size = its_cmd_get_size(cmdptr);
+    bool valid = its_cmd_get_validbit(cmdptr);
+    paddr_t itt_addr = its_cmd_mask_field(cmdptr, 2, 0, 52) & GENMASK(51, 8);
+    int ret;
+
+    if ( !its->dev_table )
+        return -1;
+
+    /*
+     * There is no easy and clean way for Xen to know the ITS device ID of a
+     * particular (PCI) device, so we have to rely on the guest telling
+     * us about it. For *now* we are just using the device ID *Dom0* uses,
+     * because the driver there has the actual knowledge.
+     * Eventually this will be replaced with a dedicated hypercall to
+     * announce pass-through of devices.
+     */
+    if ( is_hardware_domain(its->d) )
+    {
+        /* Dom0's ITSes are mapped 1:1, so both address are the same. */
+        ret = gicv3_its_map_guest_device(its->d, its->doorbell_address, devid,
+                                         its->doorbell_address, devid,
+                                         BIT(size + 1), valid);
+        if ( ret )
+            return ret;
+    }
+
+    spin_lock(&its->its_lock);
+    if ( valid )
+        its->dev_table[devid] = DEV_TABLE_ENTRY(itt_addr, size + 1);
+    else
+        its->dev_table[devid] = 0;
+
+    spin_unlock(&its->its_lock);
+
+    return 0;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -320,6 +361,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_MAPC:
             its_handle_mapc(its, cmdptr);
             break;
+        case GITS_CMD_MAPD:
+            its_handle_mapd(its, cmdptr);
+	    break;
         case GITS_CMD_SYNC:
             /* We handle ITS commands synchronously, so we ignore SYNC. */
 	    break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 20/27] ARM: vITS: handle MAPTI command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (18 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 19/27] ARM: vITS: handle MAPD command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 14:54   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 21/27] ARM: vITS: handle MOVI command Andre Przywara
                   ` (6 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The MAPTI commands associates a DeviceID/EventID pair with a LPI/CPU
pair and actually instantiates LPI interrupts.
We connect the already allocated host LPI to this virtual LPI, so that
any triggering IRQ on the host can be quickly forwarded to a guest.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c        | 63 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3-lpi.c        | 18 ++++++++++++
 xen/arch/arm/vgic-v3-its.c       | 27 +++++++++++++++--
 xen/include/asm-arm/gic_v3_its.h |  6 ++++
 4 files changed, 112 insertions(+), 2 deletions(-)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index 5a2dbec..e2fcf50 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -724,6 +724,69 @@ restart:
     spin_unlock(&d->arch.vgic.its_devices_lock);
 }
 
+/*
+ * Translates an event for a given guest device ID into the associated host
+ * LPI number. This can be used to look up the mapped guest LPI.
+ */
+static uint32_t translate_event(struct domain *d, paddr_t doorbell,
+                                uint32_t devid, uint32_t eventid)
+{
+    struct rb_node *node;
+    struct its_devices *dev;
+    uint32_t host_lpi = 0;
+    int cmp;
+
+    spin_lock(&d->arch.vgic.its_devices_lock);
+    node = d->arch.vgic.its_devices.rb_node;
+    while (node)
+    {
+        dev = rb_entry(node, struct its_devices, rbnode);
+        cmp = compare_its_guest_devices(dev, doorbell, devid);
+
+        if ( !cmp )
+        {
+            if ( eventid >= dev->eventids )
+                goto out;
+
+            host_lpi = dev->host_lpis[eventid / LPI_BLOCK] +
+                                (eventid % LPI_BLOCK);
+            if ( !is_lpi(host_lpi) )
+                host_lpi = 0;
+            goto out;
+        }
+
+        if ( cmp > 0 )
+            node = node->rb_left;
+        else
+            node = node->rb_right;
+    }
+
+out:
+    spin_unlock(&d->arch.vgic.its_devices_lock);
+
+    return host_lpi;
+}
+
+/*
+ * Connects the event ID for an already assigned device to the given VCPU/vLPI
+ * pair. The corresponding physical LPI is already mapped on the host side
+ * (when assigning the physical device to the guest), so we just connect the
+ * target VCPU/vLPI pair to that interrupt to inject it properly if it fires.
+ */
+int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell_address,
+                             uint32_t devid, uint32_t eventid,
+                             struct vcpu *v, uint32_t virt_lpi)
+{
+    uint32_t host_lpi = translate_event(d, doorbell_address, devid, eventid);
+
+    if ( !host_lpi )
+        return -ENOENT;
+
+    gicv3_lpi_update_host_entry(host_lpi, d->domain_id, v->vcpu_id, virt_lpi);
+
+    return 0;
+}
+
 /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
 void gicv3_its_dt_init(const struct dt_device_node *node)
 {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index 994698e..c110ec9 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -153,6 +153,24 @@ void do_LPI(unsigned int lpi)
     vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
 }
 
+int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
+                                unsigned int vcpu_id, uint32_t virt_lpi)
+{
+    union host_lpi *hlpip, hlpi;
+
+    host_lpi -= LPI_OFFSET;
+
+    hlpip = &lpi_data.host_lpis[host_lpi / HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
+
+    hlpi.virt_lpi = virt_lpi;
+    hlpi.dom_id = domain_id;
+    hlpi.vcpu_id = vcpu_id;
+
+    write_u64_atomic(&hlpip->data, hlpi.data);
+
+    return 0;
+}
+
 static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
 {
     uint64_t val;
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index c26d5d4..600ff69 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -167,8 +167,8 @@ static bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
 }
 
 #define SKIP_LPI_UPDATE 1
-bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
-                uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
+static bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
+                       uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
 {
     struct vits_itte *itte;
 
@@ -332,6 +332,25 @@ static int its_handle_mapd(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    uint32_t eventid = its_cmd_get_id(cmdptr);
+    uint32_t intid = its_cmd_get_physical_id(cmdptr);
+    int collid = its_cmd_get_collection(cmdptr);
+    struct vcpu *vcpu;
+
+    if ( its_cmd_get_command(cmdptr) == GITS_CMD_MAPI )
+        intid = eventid;
+
+    if ( !write_itte(its, devid, eventid, collid, intid, &vcpu) )
+        return -1;
+
+    gicv3_assign_guest_event(its->d, its->doorbell_address, devid, eventid, vcpu, intid);
+
+    return 0;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -364,6 +383,10 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_MAPD:
             its_handle_mapd(its, cmdptr);
 	    break;
+        case GITS_CMD_MAPI:
+        case GITS_CMD_MAPTI:
+            its_handle_mapti(its, cmdptr);
+            break;
         case GITS_CMD_SYNC:
             /* We handle ITS commands synchronously, so we ignore SYNC. */
 	    break;
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 2387972..1ea00e9 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -163,6 +163,12 @@ int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
                                   uint32_t host_devid, uint32_t eventid);
 int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
 
+int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell,
+                             uint32_t devid, uint32_t eventid,
+                             struct vcpu *v, uint32_t virt_lpi);
+int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
+                                unsigned int vcpu_id, uint32_t virt_lpi);
+
 #else
 
 static LIST_HEAD(host_its_list);
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 21/27] ARM: vITS: handle MOVI command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (19 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 20/27] ARM: vITS: handle MAPTI command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 15:00   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 22/27] ARM: vITS: handle DISCARD command Andre Przywara
                   ` (5 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The MOVI command moves the interrupt affinity from one redistributor
(read: VCPU) to another.
For now migration of "live" LPIs is not yet implemented, but we store
the changed affinity in the host LPI structure and in our virtual ITTE.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c        | 15 +++++++++++++++
 xen/arch/arm/gic-v3-lpi.c        | 13 +++++++++++++
 xen/arch/arm/vgic-v3-its.c       | 24 ++++++++++++++++++++++++
 xen/include/asm-arm/gic_v3_its.h |  4 ++++
 4 files changed, 56 insertions(+)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index e2fcf50..aa9b1b2 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -787,6 +787,21 @@ int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell_address,
     return 0;
 }
 
+/* Changes the target VCPU for a given host LPI assigned to a domain. */
+int gicv3_lpi_change_vcpu(struct domain *d, paddr_t doorbell,
+                          uint32_t devid, uint32_t eventid,
+                          unsigned int vcpu_id)
+{
+    uint32_t host_lpi = translate_event(d, doorbell, devid, eventid);
+
+    if ( !host_lpi )
+        return -ENOENT;
+
+    gicv3_lpi_update_host_vcpuid(host_lpi, vcpu_id);
+
+    return 0;
+}
+
 /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
 void gicv3_its_dt_init(const struct dt_device_node *node)
 {
diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
index c110ec9..8c7cea1 100644
--- a/xen/arch/arm/gic-v3-lpi.c
+++ b/xen/arch/arm/gic-v3-lpi.c
@@ -171,6 +171,19 @@ int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
     return 0;
 }
 
+int gicv3_lpi_update_host_vcpuid(uint32_t host_lpi, unsigned int vcpu_id)
+{
+    union host_lpi *hlpip;
+
+    host_lpi -= LPI_OFFSET;
+
+    hlpip = &lpi_data.host_lpis[host_lpi / HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
+
+    write_u16_atomic(&hlpip->vcpu_id, vcpu_id);
+
+    return 0;
+}
+
 static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
 {
     uint64_t val;
diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 600ff69..bb573de 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -351,6 +351,24 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+static int its_handle_movi(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    uint32_t eventid = its_cmd_get_id(cmdptr);
+    int collid = its_cmd_get_collection(cmdptr);
+    struct vcpu *vcpu;
+
+    if ( !write_itte(its, devid, eventid, collid, SKIP_LPI_UPDATE, &vcpu) )
+        return -1;
+
+    /* TODO: lookup currently-in-guest virtual IRQs and migrate them */
+
+    gicv3_lpi_change_vcpu(its->d,
+                          its->doorbell_address, devid, eventid, vcpu->vcpu_id);
+
+    return 0;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -387,6 +405,12 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_MAPTI:
             its_handle_mapti(its, cmdptr);
             break;
+        case GITS_CMD_MOVALL:
+            gdprintk(XENLOG_G_INFO, "ITS: ignoring MOVALL command\n");
+            break;
+        case GITS_CMD_MOVI:
+            its_handle_movi(its, cmdptr);
+            break;
         case GITS_CMD_SYNC:
             /* We handle ITS commands synchronously, so we ignore SYNC. */
 	    break;
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 1ea00e9..25b6b3c 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -166,8 +166,12 @@ int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
 int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell,
                              uint32_t devid, uint32_t eventid,
                              struct vcpu *v, uint32_t virt_lpi);
+int gicv3_lpi_change_vcpu(struct domain *d, paddr_t doorbell,
+                          uint32_t devid, uint32_t eventid,
+                          unsigned int vcpu_id);
 int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
                                 unsigned int vcpu_id, uint32_t virt_lpi);
+int gicv3_lpi_update_host_vcpuid(uint32_t host_lpi, unsigned int vcpu_id);
 
 #else
 
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 22/27] ARM: vITS: handle DISCARD command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (20 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 21/27] ARM: vITS: handle MOVI command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 23/27] ARM: vITS: handle INV command Andre Przywara
                   ` (4 subsequent siblings)
  26 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The DISCARD command drops the connection between a DeviceID/EventID
and an LPI/collection pair.
We mark the respective structure entries as not allocated and make
sure that any queued IRQs are removed.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index bb573de..21a663e 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -369,6 +369,36 @@ static int its_handle_movi(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+static int its_handle_discard(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    uint32_t eventid = its_cmd_get_id(cmdptr);
+    struct pending_irq *pirq;
+    struct vcpu *vcpu;
+    uint32_t vlpi;
+
+    if ( !read_itte(its, devid, eventid, &vcpu, &vlpi) )
+        return -1;
+
+    pirq = lpi_to_pending(vcpu, vlpi, false);
+    if ( pirq )
+    {
+        clear_bit(GIC_IRQ_GUEST_QUEUED, &pirq->status);
+        gic_remove_from_queues(vcpu, vlpi);
+
+        /* Mark this pending IRQ struct as availabe again. */
+        if ( !test_bit(GIC_IRQ_GUEST_VISIBLE, &pirq->status) )
+            pirq->irq = 0;
+    }
+
+    if ( !write_itte(its, devid, eventid, ~0, INVALID_LPI, NULL) )
+        return -1;
+
+    gicv3_assign_guest_event(its->d, its->doorbell_address, devid, eventid, NULL, 0);
+
+    return 0;
+}
+
 #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
 
 static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
@@ -392,6 +422,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_CLEAR:
             its_handle_clear(its, cmdptr);
             break;
+        case GITS_CMD_DISCARD:
+            its_handle_discard(its, cmdptr);
+            break;
         case GITS_CMD_INT:
             its_handle_int(its, cmdptr);
             break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 23/27] ARM: vITS: handle INV command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (21 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 22/27] ARM: vITS: handle DISCARD command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 24/27] ARM: vITS: handle INVALL command Andre Przywara
                   ` (3 subsequent siblings)
  26 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The INV command instructs the ITS to update the configuration data for
a given LPI by re-reading its entry from the property table.
We don't need to care so much about the priority value, but enabling
or disabling an LPI has some effect: We remove or push virtual LPIs
to their VCPUs, also check the virtual pending bit if an LPI gets enabled.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 57 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 21a663e..8af06ac 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -265,6 +265,60 @@ static int its_handle_int(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+/*
+ * For a given virtual LPI read the enabled bit from the virtual property
+ * table and update the virtual IRQ's state.
+ * This takes care of removing or pushing of virtual LPIs to their VCPUs.
+ */
+static void update_lpi_enabled_status(struct virt_its* its,
+                                      struct vcpu *vcpu, uint32_t vlpi)
+{
+    struct pending_irq *pirq = lpi_to_pending(vcpu, vlpi, false);
+    uint8_t property = its->d->arch.vgic.proptable[vlpi - LPI_OFFSET];
+
+    if ( property & LPI_PROP_ENABLED )
+    {
+        if ( pirq )
+        {
+            unsigned long flags;
+
+            set_bit(GIC_IRQ_GUEST_ENABLED, &pirq->status);
+            spin_lock_irqsave(&vcpu->arch.vgic.lock, flags);
+            if ( !list_empty(&pirq->inflight) &&
+                 !test_bit(GIC_IRQ_GUEST_VISIBLE, &pirq->status) )
+                gic_raise_guest_irq(vcpu, vlpi, property & LPI_PROP_PRIO_MASK);
+            spin_unlock_irqrestore(&vcpu->arch.vgic.lock, flags);
+        }
+
+        /* Check whether the LPI has fired while the guest had it disabled. */
+        if ( test_and_clear_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable) )
+            vgic_vcpu_inject_irq(vcpu, vlpi);
+    }
+    else
+    {
+        if ( pirq )
+        {
+            clear_bit(GIC_IRQ_GUEST_ENABLED, &pirq->status);
+            gic_remove_from_queues(vcpu, vlpi);
+        }
+    }
+}
+
+static int its_handle_inv(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t devid = its_cmd_get_deviceid(cmdptr);
+    uint32_t eventid = its_cmd_get_id(cmdptr);
+    struct vcpu *vcpu;
+    uint32_t vlpi;
+
+    if ( !read_itte(its, devid, eventid, &vcpu, &vlpi) )
+        return -1;
+
+    update_lpi_enabled_status(its, vcpu, vlpi);
+
+    return 0;
+}
+
 static int its_handle_mapc(struct virt_its *its, uint64_t *cmdptr)
 {
     uint32_t collid = its_cmd_get_collection(cmdptr);
@@ -428,6 +482,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_INT:
             its_handle_int(its, cmdptr);
             break;
+        case GITS_CMD_INV:
+            its_handle_inv(its, cmdptr);
+	    break;
         case GITS_CMD_MAPC:
             its_handle_mapc(its, cmdptr);
             break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 24/27] ARM: vITS: handle INVALL command
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (22 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 23/27] ARM: vITS: handle INV command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 15:12   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 25/27] ARM: vITS: create and initialize virtual ITSes for Dom0 Andre Przywara
                   ` (2 subsequent siblings)
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

The INVALL command instructs an ITS to invalidate the configuration
data for all LPIs associated with a given redistributor (read: VCPU).
This is nasty to emulate exactly with our architecture, so we just scan
the pending table and inject _every_ LPI found there that got enabled.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index 8af06ac..cc12c1c 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -319,6 +319,40 @@ static int its_handle_inv(struct virt_its *its, uint64_t *cmdptr)
     return 0;
 }
 
+/*
+ * INVALL updates the per-LPI configuration status for every LPI mapped to
+ * a particular redistributor. Since our property table is referenced when
+ * needed, we don't need to sync anything, really. But we have to take care
+ * of LPIs getting enabled if there is an interrupt pending.
+ * To catch every LPI without iterating through the device table we just
+ * look for set bits in our virtual pending table and check the status of
+ * the enabled bit in the respective property table entry.
+ * This actually covers every (pending) LPI from every redistributor,
+ * but update_lpi_enabled_status() is a NOP for LPIs not being mapped
+ * to the redistributor/VCPU we are interested in.
+ */
+static int its_handle_invall(struct virt_its *its, uint64_t *cmdptr)
+{
+    uint32_t collid = its_cmd_get_collection(cmdptr);
+    struct vcpu *vcpu;
+    int vlpi = 0;
+
+    spin_lock(&its->its_lock);
+    vcpu = get_vcpu_from_collection(its, collid);
+    spin_unlock(&its->its_lock);
+
+    do {
+        vlpi = find_next_bit(vcpu->arch.vgic.pendtable,
+                             its->d->arch.vgic.nr_lpis, vlpi);
+        if ( vlpi >= its->d->arch.vgic.nr_lpis )
+            break;
+
+        update_lpi_enabled_status(its, vcpu, vlpi);
+    } while (1);
+
+    return 0;
+}
+
 static int its_handle_mapc(struct virt_its *its, uint64_t *cmdptr)
 {
     uint32_t collid = its_cmd_get_collection(cmdptr);
@@ -485,6 +519,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
         case GITS_CMD_INV:
             its_handle_inv(its, cmdptr);
 	    break;
+        case GITS_CMD_INVALL:
+            its_handle_invall(its, cmdptr);
+	    break;
         case GITS_CMD_MAPC:
             its_handle_mapc(its, cmdptr);
             break;
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 25/27] ARM: vITS: create and initialize virtual ITSes for Dom0
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (23 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 24/27] ARM: vITS: handle INVALL command Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 15:18   ` Julien Grall
  2017-03-16 11:20 ` [PATCH v2 26/27] ARM: vITS: create ITS subnodes for Dom0 DT Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 27/27] ARM: vGIC: advertise LPI support Andre Przywara
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

For each hardware ITS create and initialize a virtual ITS for Dom0.
We use the same memory mapped address to keep the doorbell working.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3-its.c       | 29 +++++++++++++++++++++++++++++
 xen/arch/arm/vgic-v3.c           | 12 ++++++++++++
 xen/include/asm-arm/domain.h     |  1 +
 xen/include/asm-arm/gic_v3_its.h | 12 ++++++++++++
 4 files changed, 54 insertions(+)

diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
index cc12c1c..caa7354 100644
--- a/xen/arch/arm/vgic-v3-its.c
+++ b/xen/arch/arm/vgic-v3-its.c
@@ -914,6 +914,35 @@ static const struct mmio_handler_ops vgic_its_mmio_handler = {
     .write = vgic_v3_its_mmio_write,
 };
 
+int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr)
+{
+    struct virt_its *its;
+    uint64_t base_attr;
+
+    its = xzalloc(struct virt_its);
+    if ( ! its )
+        return -ENOMEM;
+
+    base_attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
+    base_attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
+    base_attr |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
+
+    its->cbaser  = base_attr;
+    base_attr |= 0ULL << GITS_BASER_PAGE_SIZE_SHIFT;
+    its->baser0  = GITS_BASER_TYPE_DEVICE << GITS_BASER_TYPE_SHIFT;
+    its->baser0 |= (7ULL << GITS_BASER_ENTRY_SIZE_SHIFT) | base_attr;
+    its->baser1  = GITS_BASER_TYPE_COLLECTION << GITS_BASER_TYPE_SHIFT;
+    its->baser1 |= (1ULL << GITS_BASER_ENTRY_SIZE_SHIFT) | base_attr;
+    its->d = d;
+    its->doorbell_address = guest_addr + ITS_DOORBELL_OFFSET;
+    spin_lock_init(&its->vcmd_lock);
+    spin_lock_init(&its->its_lock);
+
+    register_mmio_handler(d, &vgic_its_mmio_handler, guest_addr, SZ_64K, its);
+
+    return 0;
+}
+
 /*
  * Local variables:
  * mode: C
diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index d1382be..8faec95 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -31,6 +31,7 @@
 #include <asm/current.h>
 #include <asm/mmio.h>
 #include <asm/gic_v3_defs.h>
+#include <asm/gic_v3_its.h>
 #include <asm/vgic.h>
 #include <asm/vgic-emul.h>
 #include <asm/vreg.h>
@@ -1651,6 +1652,7 @@ static int vgic_v3_domain_init(struct domain *d)
      */
     if ( is_hardware_domain(d) )
     {
+        struct host_its *hw_its;
         unsigned int first_cpu = 0;
 
         d->arch.vgic.dbase = vgic_v3_hw.dbase;
@@ -1676,6 +1678,16 @@ static int vgic_v3_domain_init(struct domain *d)
 
             first_cpu += size / d->arch.vgic.rdist_stride;
         }
+        d->arch.vgic.nr_regions = vgic_v3_hw.nr_rdist_regions;
+
+        list_for_each_entry(hw_its, &host_its_list, entry)
+        {
+            /* Emulate the control registers frame (lower 64K). */
+            vgic_v3_its_init_virtual(d, hw_its->addr);
+
+            d->arch.vgic.has_its = true;
+        }
+
     }
     else
     {
diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
index 33c1851..27cc310 100644
--- a/xen/include/asm-arm/domain.h
+++ b/xen/include/asm-arm/domain.h
@@ -115,6 +115,7 @@ struct arch_domain
         uint8_t *proptable;
         struct rb_root its_devices;         /* devices mapped to an ITS */
         spinlock_t its_devices_lock;        /* protects the its_devices tree */
+        bool has_its;
 #endif
     } vgic;
 
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 25b6b3c..55ef143 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -148,6 +148,13 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
 int gicv3_its_setup_collection(unsigned int cpu);
 
 /*
+ * Create and register a virtual ITS at the given guest address.
+ * If a host ITS is specified, a hardware domain can reach out to that host
+ * ITS to deal with devices and LPI mappings and can enable/disable LPIs.
+ */
+int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr);
+
+/*
  * Map a device on the host by allocating an ITT on the host (ITS).
  * "nr_event" specifies how many events (interrupts) this device will need.
  * Setting "valid" to false deallocates the device.
@@ -211,6 +218,11 @@ static inline int gicv3_its_setup_collection(unsigned int cpu)
     return 0;
 }
 
+static inline int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr)
+{
+    return 0;
+}
+
 #endif /* CONFIG_HAS_ITS */
 
 #endif
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 26/27] ARM: vITS: create ITS subnodes for Dom0 DT
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (24 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 25/27] ARM: vITS: create and initialize virtual ITSes for Dom0 Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-16 11:20 ` [PATCH v2 27/27] ARM: vGIC: advertise LPI support Andre Przywara
  26 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Dom0 expects all ITSes in the system to be propagated to be able to
use MSIs.
Create Dom0 DT nodes for each hardware ITS, keeping the register frame
address the same, as the doorbell address that the Dom0 drivers program
into the BARs has to match the hardware.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/gic-v3-its.c        | 78 ++++++++++++++++++++++++++++++++++++++++
 xen/arch/arm/gic-v3.c            |  4 ++-
 xen/include/asm-arm/gic_v3_its.h | 13 +++++++
 3 files changed, 94 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
index aa9b1b2..b8fe5a3 100644
--- a/xen/arch/arm/gic-v3-its.c
+++ b/xen/arch/arm/gic-v3-its.c
@@ -20,6 +20,7 @@
 
 #include <xen/lib.h>
 #include <xen/delay.h>
+#include <xen/libfdt/libfdt.h>
 #include <xen/mm.h>
 #include <xen/rbtree.h>
 #include <xen/sched.h>
@@ -802,6 +803,83 @@ int gicv3_lpi_change_vcpu(struct domain *d, paddr_t doorbell,
     return 0;
 }
 
+/*
+ * Create the respective guest DT nodes for a list of host ITSes.
+ * This copies the reg property, so the guest sees the ITS at the same address
+ * as the host.
+ * Giving NULL for the its_list will make it use the list of host ITSes.
+ */
+int gicv3_its_make_dt_nodes(struct list_head *its_list,
+                            const struct domain *d,
+                            const struct dt_device_node *gic,
+                            void *fdt)
+{
+    uint32_t len;
+    int res;
+    const void *prop = NULL;
+    const struct dt_device_node *its = NULL;
+    const struct host_its *its_data;
+
+    if ( !its_list )
+        its_list = &host_its_list;
+
+    if ( list_empty(its_list) )
+        return 0;
+
+    /* The sub-nodes require the ranges property */
+    prop = dt_get_property(gic, "ranges", &len);
+    if ( !prop )
+    {
+        printk(XENLOG_ERR "Can't find ranges property for the gic node\n");
+        return -FDT_ERR_XEN(ENOENT);
+    }
+
+    res = fdt_property(fdt, "ranges", prop, len);
+    if ( res )
+        return res;
+
+    list_for_each_entry(its_data, its_list, entry)
+    {
+        its = its_data->dt_node;
+
+        res = fdt_begin_node(fdt, its->name);
+        if ( res )
+            return res;
+
+        res = fdt_property_string(fdt, "compatible", "arm,gic-v3-its");
+        if ( res )
+            return res;
+
+        res = fdt_property(fdt, "msi-controller", NULL, 0);
+        if ( res )
+            return res;
+
+        if ( its->phandle )
+        {
+            res = fdt_property_cell(fdt, "phandle", its->phandle);
+            if ( res )
+                return res;
+        }
+
+        /* Use the same reg regions as the ITS node in host DTB. */
+        prop = dt_get_property(its, "reg", &len);
+        if ( !prop )
+        {
+            printk(XENLOG_ERR "GICv3: Can't find ITS reg property.\n");
+            res = -FDT_ERR_XEN(ENOENT);
+            return res;
+        }
+
+        res = fdt_property(fdt, "reg", prop, len);
+        if ( res )
+            return res;
+
+        fdt_end_node(fdt);
+    }
+
+    return res;
+}
+
 /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
 void gicv3_its_dt_init(const struct dt_device_node *node)
 {
diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
index 83a839a..f9193a6 100644
--- a/xen/arch/arm/gic-v3.c
+++ b/xen/arch/arm/gic-v3.c
@@ -1167,8 +1167,10 @@ static int gicv3_make_hwdom_dt_node(const struct domain *d,
 
     res = fdt_property(fdt, "reg", new_cells, len);
     xfree(new_cells);
+    if ( res )
+        return res;
 
-    return res;
+    return gicv3_its_make_dt_nodes(NULL, d, gic, fdt);
 }
 
 static const hw_irq_controller gicv3_host_irq_type = {
diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
index 55ef143..67cd48d 100644
--- a/xen/include/asm-arm/gic_v3_its.h
+++ b/xen/include/asm-arm/gic_v3_its.h
@@ -154,6 +154,12 @@ int gicv3_its_setup_collection(unsigned int cpu);
  */
 int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr);
 
+/* Given a list of ITSes, create the appropriate DT nodes for a domain. */
+int gicv3_its_make_dt_nodes(struct list_head *its_list,
+                            const struct domain *d,
+                            const struct dt_device_node *gic,
+                            void *fdt);
+
 /*
  * Map a device on the host by allocating an ITT on the host (ITS).
  * "nr_event" specifies how many events (interrupts) this device will need.
@@ -222,6 +228,13 @@ static inline int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr)
 {
     return 0;
 }
+static inline int gicv3_its_make_dt_nodes(struct list_head *its_list,
+                                       const struct domain *d,
+                                       const struct dt_device_node *gic,
+                                       void *fdt)
+{
+    return 0;
+}
 
 #endif /* CONFIG_HAS_ITS */
 
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* [PATCH v2 27/27] ARM: vGIC: advertise LPI support
  2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
                   ` (25 preceding siblings ...)
  2017-03-16 11:20 ` [PATCH v2 26/27] ARM: vITS: create ITS subnodes for Dom0 DT Andre Przywara
@ 2017-03-16 11:20 ` Andre Przywara
  2017-03-24 15:25   ` Julien Grall
  26 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 11:20 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

To let a guest know about the availability of virtual LPIs, set the
respective bits in the virtual GIC registers and let a guest control
the LPI enable bit.
Only report the LPI capability if the host has initialized at least
one ITS.

Signed-off-by: Andre Przywara <andre.przywara@arm.com>
---
 xen/arch/arm/vgic-v3.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 83 insertions(+), 5 deletions(-)

diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
index 8faec95..6d5b7f4 100644
--- a/xen/arch/arm/vgic-v3.c
+++ b/xen/arch/arm/vgic-v3.c
@@ -169,8 +169,10 @@ static int __vgic_v3_rdistr_rd_mmio_read(struct vcpu *v, mmio_info_t *info,
     switch ( gicr_reg )
     {
     case VREG32(GICR_CTLR):
-        /* We have not implemented LPI's, read zero */
-        goto read_as_zero_32;
+        if ( dabt.size != DABT_WORD ) goto bad_width;
+        *r = vgic_reg32_extract(!!(v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED),
+                                info);
+        return 1;
 
     case VREG32(GICR_IIDR):
         if ( dabt.size != DABT_WORD ) goto bad_width;
@@ -182,16 +184,19 @@ static int __vgic_v3_rdistr_rd_mmio_read(struct vcpu *v, mmio_info_t *info,
         uint64_t typer, aff;
 
         if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
-        /* TBD: Update processor id in [23:8] when ITS support is added */
         aff = (MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 3) << 56 |
                MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 2) << 48 |
                MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 1) << 40 |
                MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 0) << 32);
         typer = aff;
+        typer |= (v->vcpu_id & 0xffff) << 8;
 
         if ( v->arch.vgic.flags & VGIC_V3_RDIST_LAST )
             typer |= GICR_TYPER_LAST;
 
+        if ( v->domain->arch.vgic.has_its )
+            typer |= GICR_TYPER_PLPIS;
+
         *r = vgic_reg64_extract(typer, info);
 
         return 1;
@@ -502,6 +507,40 @@ bool vgic_can_inject_lpi(struct vcpu *vcpu, uint32_t vlpi)
     return false;
 }
 
+static void vgic_vcpu_enable_lpis(struct vcpu *v)
+{
+    uint64_t reg = v->domain->arch.vgic.rdist_propbase;
+    unsigned int nr_lpis = BIT((reg & 0x1f) + 1) - LPI_OFFSET;
+    int nr_pages;
+
+    /* The first VCPU to enable LPIs maps the property table. */
+    if ( !v->domain->arch.vgic.proptable )
+    {
+        v->domain->arch.vgic.nr_lpis = nr_lpis;
+        nr_pages = DIV_ROUND_UP(nr_lpis, PAGE_SIZE);
+
+        get_guest_pages(v->domain, reg & GENMASK(51, 12), nr_pages);
+        v->domain->arch.vgic.proptable = map_guest_pages(v->domain,
+                                                         reg & GENMASK(51, 12),
+                                                         nr_pages);
+        printk("VGIC-v3: VCPU%d mapped %d pages for property table\n",
+               v->vcpu_id, nr_pages);
+    }
+    nr_pages = DIV_ROUND_UP(((nr_lpis + LPI_OFFSET) / 8), PAGE_SIZE);
+    reg = v->arch.vgic.rdist_pendbase;
+
+    get_guest_pages(v->domain, reg & GENMASK(51, 12), nr_pages);
+    v->arch.vgic.pendtable = map_guest_pages(v->domain,
+                                             reg & GENMASK(51, 12), nr_pages);
+
+    printk("VGIC-v3: VCPU%d mapped %d pages for pending table\n",
+           v->vcpu_id, nr_pages);
+
+    v->arch.vgic.flags |= VGIC_V3_LPIS_ENABLED;
+
+    printk("VGICv3: enabled %d LPIs for VCPU%d\n", nr_lpis, v->vcpu_id);
+}
+
 static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
                                           uint32_t gicr_reg,
                                           register_t r)
@@ -512,8 +551,18 @@ static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
     switch ( gicr_reg )
     {
     case VREG32(GICR_CTLR):
-        /* LPI's not implemented */
-        goto write_ignore_32;
+        if ( dabt.size != DABT_WORD ) goto bad_width;
+        if ( !v->domain->arch.vgic.has_its )
+            return 1;
+
+        /* LPIs can only be enabled once, but never disabled again. */
+        if ( !(r & GICR_CTLR_ENABLE_LPIS) ||
+             (v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED) )
+            return 1;
+
+        vgic_vcpu_enable_lpis(v);
+
+        return 1;
 
     case VREG32(GICR_IIDR):
         /* RO */
@@ -1113,6 +1162,11 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
         typer = ((ncpus - 1) << GICD_TYPE_CPUS_SHIFT |
                  DIV_ROUND_UP(v->domain->arch.vgic.nr_spis, 32));
 
+        if ( v->domain->arch.vgic.has_its )
+        {
+            typer |= GICD_TYPE_LPIS;
+            irq_bits = 16;
+        }
         typer |= (irq_bits - 1) << GICD_TYPE_ID_BITS_SHIFT;
 
         *r = vgic_reg32_extract(typer, info);
@@ -1729,6 +1783,30 @@ static int vgic_v3_domain_init(struct domain *d)
 
 static void vgic_v3_domain_free(struct domain *d)
 {
+    int nr_pages;
+    struct vcpu *v;
+
+    if ( d->arch.vgic.proptable )
+    {
+        nr_pages = DIV_ROUND_UP(d->arch.vgic.nr_lpis, PAGE_SIZE);
+
+        unmap_guest_pages(d->arch.vgic.proptable, nr_pages);
+        put_guest_pages(d, d->arch.vgic.rdist_propbase & GENMASK(51, 12),
+                        nr_pages);
+        printk("VGICv3: freeing PROPBASE for Domain%d\n", d->domain_id);
+    }
+
+    nr_pages = DIV_ROUND_UP((d->arch.vgic.nr_lpis + LPI_OFFSET) / 8, PAGE_SIZE);
+    for_each_vcpu(d, v)
+    {
+        if ( !v->arch.vgic.pendtable )
+            continue;
+
+        unmap_guest_pages(v->arch.vgic.pendtable, nr_pages);
+        put_guest_pages(d, v->arch.vgic.rdist_pendbase & GENMASK(51, 12),
+                        nr_pages);
+        printk("VGICv3: freeing PENDBASE for VCPU%d\n", v->vcpu_id);
+    }
     xfree(d->arch.vgic.rdist_regions);
 }
 
-- 
2.9.0


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply related	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-03-16 11:20 ` [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling Andre Przywara
@ 2017-03-16 15:05   ` Shanker Donthineni
  2017-03-16 15:18     ` Andre Przywara
  2017-03-22  0:02   ` Stefano Stabellini
  2017-03-22 15:59   ` Julien Grall
  2 siblings, 1 reply; 119+ messages in thread
From: Shanker Donthineni @ 2017-03-16 15:05 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini, Julien Grall; +Cc: xen-devel, Vijay Kilari

Hi Andre,


On 03/16/2017 06:20 AM, Andre Przywara wrote:
> To be able to easily send commands to the ITS, create the respective
> wrapper functions, which take care of the ring buffer.
> The first two commands we implement provide methods to map a collection
> to a redistributor (aka host core) and to flush the command queue (SYNC).
> Start using these commands for mapping one collection to each host CPU.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c         | 181 +++++++++++++++++++++++++++++++++++++-
>  xen/arch/arm/gic-v3-lpi.c         |  22 +++++
>  xen/arch/arm/gic-v3.c             |  19 +++-
>  xen/include/asm-arm/gic_v3_defs.h |   2 +
>  xen/include/asm-arm/gic_v3_its.h  |  38 ++++++++
>  5 files changed, 260 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index e5601ed..5c11b0d 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -19,11 +19,14 @@
>   */
>  
>  #include <xen/lib.h>
> +#include <xen/delay.h>
>  #include <xen/mm.h>
>  #include <xen/sizes.h>
> +#include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
> +#include <asm/page.h>
>  
>  #define ITS_CMD_QUEUE_SZ                SZ_64K
>  
> @@ -34,6 +37,145 @@ bool gicv3_its_host_has_its(void)
>      return !list_empty(&host_its_list);
>  }
>  
> +#define BUFPTR_MASK                     GENMASK(19, 5)
> +static int its_send_command(struct host_its *hw_its, const void *its_cmd)
> +{
> +    s_time_t deadline = NOW() + MILLISECS(1);
> +    uint64_t readp, writep;
> +    int ret = -EBUSY;
> +
> +    /* No ITS commands from an interrupt handler (at the moment). */
> +    ASSERT(!in_irq());
> +
> +    spin_lock(&hw_its->cmd_lock);
> +
> +    do {
> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
> +
> +        if ( ((writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ) != readp )
> +        {
> +            ret = 0;
> +            break;
> +        }
> +
> +        /*
> +         * If the command queue is full, wait for a bit in the hope it drains
> +         * before giving up.
> +         */
> +        spin_unlock(&hw_its->cmd_lock);
> +        cpu_relax();
> +        udelay(1);
> +        spin_lock(&hw_its->cmd_lock);
> +    } while ( NOW() <= deadline );
> +
> +    if ( ret )
> +    {
> +        spin_unlock(&hw_its->cmd_lock);
> +        printk(XENLOG_WARNING "ITS: command queue full.\n");
> +        return ret;
> +    }
> +
> +    memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE);
> +    if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
> +        clean_and_invalidate_dcache_va_range(hw_its->cmd_buf + writep,
> +                                             ITS_CMD_SIZE);
> +    else
> +        dsb(ishst);
> +
> +    writep = (writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ;
> +    writeq_relaxed(writep & BUFPTR_MASK, hw_its->its_base + GITS_CWRITER);
> +
> +    spin_unlock(&hw_its->cmd_lock);
> +
> +    return 0;
> +}
> +
> +/* Wait for an ITS to finish processing all commands. */
> +static int gicv3_its_wait_commands(struct host_its *hw_its)
> +{
> +    s_time_t deadline = NOW() + MILLISECS(100);
> +    uint64_t readp, writep;
> +
> +    do {
> +        spin_lock(&hw_its->cmd_lock);
> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
> +        spin_unlock(&hw_its->cmd_lock);
> +
> +        if ( readp == writep )
> +            return 0;
> +
> +        cpu_relax();
> +        udelay(1);
> +    } while ( NOW() <= deadline );
> +
> +    return -ETIMEDOUT;
> +}
> +
> +static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
> +                              uint64_t reg)
> +{
> +    reg &= ~GENMASK(51, 16);
> +
> +    reg |= gicv3_get_redist_address(cpu, hw_its->flags & HOST_ITS_USES_PTA);
> +
> +    return reg;
> +}
> +
> +static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_SYNC;
> +    cmd[1] = 0x00;
> +    cmd[2] = encode_rdbase(its, cpu, 0x0);
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
> +static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
> +                             unsigned int cpu)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_MAPC;
> +    cmd[1] = 0x00;
> +    cmd[2] = encode_rdbase(its, cpu, (collection_id & GENMASK(15, 0)));
> +    cmd[2] |= GITS_VALID_BIT;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
> +/* Set up the (1:1) collection mapping for the given host CPU. */
> +int gicv3_its_setup_collection(unsigned int cpu)
> +{
> +    struct host_its *its;
> +    int ret;
> +
> +    list_for_each_entry(its, &host_its_list, entry)
> +    {
> +        if ( !its->cmd_buf )
> +            continue;
> +
> +        ret = its_send_cmd_mapc(its, cpu, cpu);
> +        if ( ret )
> +            return ret;
> +
> +        ret = its_send_cmd_sync(its, cpu);
> +        if ( ret )
> +            return ret;
> +
> +        ret = gicv3_its_wait_commands(its);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    return 0;
> +}
> +
>  #define BASER_ATTR_MASK                                           \
>          ((0x3UL << GITS_BASER_SHAREABILITY_SHIFT)               | \
>           (0x7UL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)         | \
> @@ -184,22 +326,59 @@ retry:
>      return -EINVAL;
>  }
>  
> +/*
> + * Before an ITS gets initialized, it should be in a quiescent state, where
> + * all outstanding commands and transactions have finished.
> + * So if the ITS is already enabled, turn it off and wait for all outstanding
> + * operations to get processed by polling the QUIESCENT bit.
> + */
> +static int gicv3_disable_its(struct host_its *hw_its)
> +{
> +    uint32_t reg;
> +    s_time_t deadline = NOW() + MILLISECS(100);
> +
> +    reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
> +    if ( (reg & GITS_CTLR_QUIESCENT) && !(reg & GITS_CTLR_ENABLE) )

nit: I prefer changing to 'if ( !(reg & GITS_CTLR_ENABLE) && (reg & GITS_CTLR_QUIESCENT) ) ' because bit GITS_CTLR_QUIESCENT is not valid if ITS hardware is in enabled state.

>  #endif

-- 
Shanker Donthineni
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-03-16 15:05   ` Shanker Donthineni
@ 2017-03-16 15:18     ` Andre Przywara
  0 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-16 15:18 UTC (permalink / raw)
  To: shankerd, Stefano Stabellini, Julien Grall; +Cc: xen-devel, Vijay Kilari

Hi Shanker,

On 16/03/17 15:05, Shanker Donthineni wrote:
> Hi Andre,
> 
> 
> On 03/16/2017 06:20 AM, Andre Przywara wrote:
>> To be able to easily send commands to the ITS, create the respective
>> wrapper functions, which take care of the ring buffer.
>> The first two commands we implement provide methods to map a collection
>> to a redistributor (aka host core) and to flush the command queue (SYNC).
>> Start using these commands for mapping one collection to each host CPU.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---

...

>> @@ -184,22 +326,59 @@ retry:
>>      return -EINVAL;
>>  }
>>  
>> +/*
>> + * Before an ITS gets initialized, it should be in a quiescent state, where
>> + * all outstanding commands and transactions have finished.
>> + * So if the ITS is already enabled, turn it off and wait for all outstanding
>> + * operations to get processed by polling the QUIESCENT bit.
>> + */
>> +static int gicv3_disable_its(struct host_its *hw_its)
>> +{
>> +    uint32_t reg;
>> +    s_time_t deadline = NOW() + MILLISECS(100);
>> +
>> +    reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
>> +    if ( (reg & GITS_CTLR_QUIESCENT) && !(reg & GITS_CTLR_ENABLE) )
> 
> nit: I prefer changing to 'if ( !(reg & GITS_CTLR_ENABLE) && (reg & GITS_CTLR_QUIESCENT) ) ' because bit GITS_CTLR_QUIESCENT is not valid if ITS hardware is in enabled state.

Sure, makes sense. I will change this.

Thanks for having a look!

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits
  2017-03-16 11:20 ` [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits Andre Przywara
@ 2017-03-16 16:25   ` Shanker Donthineni
  2017-03-20 12:17     ` Vijay Kilari
  2017-03-24 12:41   ` Julien Grall
  1 sibling, 1 reply; 119+ messages in thread
From: Shanker Donthineni @ 2017-03-16 16:25 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini, Julien Grall; +Cc: xen-devel, Vijay Kilari

Hi Andre,


On 03/16/2017 06:20 AM, Andre Przywara wrote:
> Create a new file to hold the emulation code for the ITS widget.
> For now we emulate the memory mapped ITS registers and provide a stub
> to introduce the ITS command handling framework (but without actually
> emulating any commands at this time).
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/Makefile             |   1 +
>  xen/arch/arm/vgic-v3-its.c        | 487 ++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vgic-v3.c            |   9 -
>  xen/include/asm-arm/gic_v3_defs.h |  19 ++
>  4 files changed, 507 insertions(+), 9 deletions(-)
>  create mode 100644 xen/arch/arm/vgic-v3-its.c
>
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 02a8737..e7ce2c83 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -47,6 +47,7 @@ obj-y += traps.o
>  obj-y += vgic.o
>  obj-y += vgic-v2.o
>  obj-$(CONFIG_HAS_GICV3) += vgic-v3.o
> +obj-$(CONFIG_HAS_ITS) += vgic-v3-its.o
>  obj-y += vm_event.o
>  obj-y += vtimer.o
>  obj-y += vpsci.o
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> new file mode 100644
> index 0000000..5337638
> --- /dev/null
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -0,0 +1,487 @@
> +/*
> + * xen/arch/arm/vgic-v3-its.c
> + *
> + * ARM Interrupt Translation Service (ITS) emulation
> + *
> + * Andre Przywara <andre.przywara@arm.com>
> + * Copyright (c) 2016,2017 ARM Ltd.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; under version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/bitops.h>
> +#include <xen/config.h>
> +#include <xen/domain_page.h>
> +#include <xen/lib.h>
> +#include <xen/init.h>
> +#include <xen/softirq.h>
> +#include <xen/irq.h>
> +#include <xen/sched.h>
> +#include <xen/sizes.h>
> +#include <asm/current.h>
> +#include <asm/mmio.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
> +#include <asm/vgic.h>
> +#include <asm/vgic-emul.h>
> +
> +/* Data structure to describe a virtual ITS */
> +struct virt_its {
> +    struct domain *d;
> +    spinlock_t vcmd_lock;       /* protects the virtual command buffer */
> +    uint64_t cbaser;
> +    uint64_t *cmdbuf;
> +    int cwriter;
> +    int creadr;
> +    spinlock_t its_lock;        /* protects the collection and device tables */
> +    uint64_t baser0, baser1;
> +    uint16_t *coll_table;
> +    int max_collections;
> +    uint64_t *dev_table;
> +    int max_devices;
> +    bool enabled;
> +};
> +
> +/*
> + * An Interrupt Translation Table Entry: this is indexed by a
> + * DeviceID/EventID pair and is located in guest memory.
> + */
> +struct vits_itte
> +{
> +    uint32_t vlpi;
> +    uint16_t collection;
> +};
> +
> +/**************************************
> + * Functions that handle ITS commands *
> + **************************************/
> +
> +static uint64_t its_cmd_mask_field(uint64_t *its_cmd,
> +                                   int word, int shift, int size)
> +{
> +    return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT(size) - 1);
> +}
> +
> +#define its_cmd_get_command(cmd)        its_cmd_mask_field(cmd, 0,  0,  8)
> +#define its_cmd_get_deviceid(cmd)       its_cmd_mask_field(cmd, 0, 32, 32)
> +#define its_cmd_get_size(cmd)           its_cmd_mask_field(cmd, 1,  0,  5)
> +#define its_cmd_get_id(cmd)             its_cmd_mask_field(cmd, 1,  0, 32)
> +#define its_cmd_get_physical_id(cmd)    its_cmd_mask_field(cmd, 1, 32, 32)
> +#define its_cmd_get_collection(cmd)     its_cmd_mask_field(cmd, 2,  0, 16)
> +#define its_cmd_get_target_addr(cmd)    its_cmd_mask_field(cmd, 2, 16, 32)
> +#define its_cmd_get_validbit(cmd)       its_cmd_mask_field(cmd, 2, 63,  1)
> +
> +#define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
> +
> +static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
> +                                uint32_t writer)
> +{
> +    uint64_t *cmdptr;
> +
> +    if ( !its->cmdbuf )
> +        return -1;
> +
> +    if ( writer >= ITS_CMD_BUFFER_SIZE(its->cbaser) )
> +        return -1;
> +
> +    spin_lock(&its->vcmd_lock);
> +
> +    while ( its->creadr != writer )
> +    {
> +        cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
> +        switch (its_cmd_get_command(cmdptr))
> +        {
> +        case GITS_CMD_SYNC:
> +            /* We handle ITS commands synchronously, so we ignore SYNC. */
> +	    break;
> +        default:
> +            gdprintk(XENLOG_G_WARNING, "ITS: unhandled ITS command %ld\n",
> +                   its_cmd_get_command(cmdptr));
> +            break;
> +        }
> +
> +        its->creadr += ITS_CMD_SIZE;
> +        if ( its->creadr == ITS_CMD_BUFFER_SIZE(its->cbaser) )
> +            its->creadr = 0;
> +    }
> +    its->cwriter = writer;
> +
> +    spin_unlock(&its->vcmd_lock);
> +
> +    return 0;
> +}
> +
> +/*****************************
> + * ITS registers read access *
> + *****************************/
> +
> +/*
> + * The physical address is encoded slightly differently depending on
> + * the used page size: the highest four bits are stored in the lowest
> + * four bits of the field for 64K pages.
> + */
> +static paddr_t get_baser_phys_addr(uint64_t reg)
> +{
> +    if ( reg & BIT(9) )
> +        return (reg & GENMASK(47, 16)) | ((reg & GENMASK(15, 12)) << 36);
> +    else
> +        return reg & GENMASK(47, 12);
> +}
> +
> +static int vgic_v3_its_mmio_read(struct vcpu *v, mmio_info_t *info,
> +                                 register_t *r, void *priv)
> +{
> +    struct virt_its *its = priv;
> +
> +    switch ( info->gpa & 0xffff )
> +    {
> +    case VREG32(GITS_CTLR):
> +        if ( info->dabt.size != DABT_WORD ) goto bad_width;
> +        *r = vgic_reg32_extract(its->enabled | BIT(31), info);
> +	break;
> +    case VREG32(GITS_IIDR):
> +        if ( info->dabt.size != DABT_WORD ) goto bad_width;
> +        *r = vgic_reg32_extract(GITS_IIDR_VALUE, info);
> +        break;
> +    case VREG64(GITS_TYPER):
> +        if ( info->dabt.size < DABT_WORD ) goto bad_width;
> +        *r = vgic_reg64_extract(0x1eff1, info);

Please don't hard-code Devbits (bits 17-13) value to 16 here. For dom0, at least match to Kconfig option, or use ITS hardware reported value. On Qualcomm server chips, GITS_TYPER.Devbit is set to 32bits.

-- 
Shanker Donthineni
Qualcomm Datacenter Technologies, Inc. as an affiliate of Qualcomm Technologies, Inc.
Qualcomm Technologies, Inc. is a member of the Code Aurora Forum, a Linux Foundation Collaborative Project.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits
  2017-03-16 16:25   ` Shanker Donthineni
@ 2017-03-20 12:17     ` Vijay Kilari
  0 siblings, 0 replies; 119+ messages in thread
From: Vijay Kilari @ 2017-03-20 12:17 UTC (permalink / raw)
  To: Shanker Donthineni
  Cc: Andre Przywara, Julien Grall, Stefano Stabellini, xen-devel

On Thu, Mar 16, 2017 at 9:55 PM, Shanker Donthineni
<shankerd@codeaurora.org> wrote:
> Hi Andre,
>
>
> On 03/16/2017 06:20 AM, Andre Przywara wrote:
>> Create a new file to hold the emulation code for the ITS widget.
>> For now we emulate the memory mapped ITS registers and provide a stub
>> to introduce the ITS command handling framework (but without actually
>> emulating any commands at this time).
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/Makefile             |   1 +
>>  xen/arch/arm/vgic-v3-its.c        | 487 ++++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/vgic-v3.c            |   9 -
>>  xen/include/asm-arm/gic_v3_defs.h |  19 ++
>>  4 files changed, 507 insertions(+), 9 deletions(-)
>>  create mode 100644 xen/arch/arm/vgic-v3-its.c
>>
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 02a8737..e7ce2c83 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -47,6 +47,7 @@ obj-y += traps.o
>>  obj-y += vgic.o
>>  obj-y += vgic-v2.o
>>  obj-$(CONFIG_HAS_GICV3) += vgic-v3.o
>> +obj-$(CONFIG_HAS_ITS) += vgic-v3-its.o
>>  obj-y += vm_event.o
>>  obj-y += vtimer.o
>>  obj-y += vpsci.o
>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
>> new file mode 100644
>> index 0000000..5337638
>> --- /dev/null
>> +++ b/xen/arch/arm/vgic-v3-its.c
>> @@ -0,0 +1,487 @@
>> +/*
>> + * xen/arch/arm/vgic-v3-its.c
>> + *
>> + * ARM Interrupt Translation Service (ITS) emulation
>> + *
>> + * Andre Przywara <andre.przywara@arm.com>
>> + * Copyright (c) 2016,2017 ARM Ltd.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; under version 2 of the License.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/bitops.h>
>> +#include <xen/config.h>
>> +#include <xen/domain_page.h>
>> +#include <xen/lib.h>
>> +#include <xen/init.h>
>> +#include <xen/softirq.h>
>> +#include <xen/irq.h>
>> +#include <xen/sched.h>
>> +#include <xen/sizes.h>
>> +#include <asm/current.h>
>> +#include <asm/mmio.h>
>> +#include <asm/gic_v3_defs.h>
>> +#include <asm/gic_v3_its.h>
>> +#include <asm/vgic.h>
>> +#include <asm/vgic-emul.h>
>> +
>> +/* Data structure to describe a virtual ITS */
>> +struct virt_its {
>> +    struct domain *d;
>> +    spinlock_t vcmd_lock;       /* protects the virtual command buffer */
>> +    uint64_t cbaser;
>> +    uint64_t *cmdbuf;
>> +    int cwriter;
>> +    int creadr;
>> +    spinlock_t its_lock;        /* protects the collection and device tables */
>> +    uint64_t baser0, baser1;
>> +    uint16_t *coll_table;
>> +    int max_collections;
>> +    uint64_t *dev_table;
>> +    int max_devices;
>> +    bool enabled;
>> +};
>> +
>> +/*
>> + * An Interrupt Translation Table Entry: this is indexed by a
>> + * DeviceID/EventID pair and is located in guest memory.
>> + */
>> +struct vits_itte
>> +{
>> +    uint32_t vlpi;
>> +    uint16_t collection;
>> +};
>> +
>> +/**************************************
>> + * Functions that handle ITS commands *
>> + **************************************/
>> +
>> +static uint64_t its_cmd_mask_field(uint64_t *its_cmd,
>> +                                   int word, int shift, int size)
>> +{
>> +    return (le64_to_cpu(its_cmd[word]) >> shift) & (BIT(size) - 1);
>> +}
>> +
>> +#define its_cmd_get_command(cmd)        its_cmd_mask_field(cmd, 0,  0,  8)
>> +#define its_cmd_get_deviceid(cmd)       its_cmd_mask_field(cmd, 0, 32, 32)
>> +#define its_cmd_get_size(cmd)           its_cmd_mask_field(cmd, 1,  0,  5)
>> +#define its_cmd_get_id(cmd)             its_cmd_mask_field(cmd, 1,  0, 32)
>> +#define its_cmd_get_physical_id(cmd)    its_cmd_mask_field(cmd, 1, 32, 32)
>> +#define its_cmd_get_collection(cmd)     its_cmd_mask_field(cmd, 2,  0, 16)
>> +#define its_cmd_get_target_addr(cmd)    its_cmd_mask_field(cmd, 2, 16, 32)
>> +#define its_cmd_get_validbit(cmd)       its_cmd_mask_field(cmd, 2, 63,  1)
>> +
>> +#define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
>> +
>> +static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
>> +                                uint32_t writer)
>> +{
>> +    uint64_t *cmdptr;
>> +
>> +    if ( !its->cmdbuf )
>> +        return -1;
>> +
>> +    if ( writer >= ITS_CMD_BUFFER_SIZE(its->cbaser) )
>> +        return -1;
>> +
>> +    spin_lock(&its->vcmd_lock);
>> +
>> +    while ( its->creadr != writer )
>> +    {
>> +        cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
>> +        switch (its_cmd_get_command(cmdptr))
>> +        {
>> +        case GITS_CMD_SYNC:
>> +            /* We handle ITS commands synchronously, so we ignore SYNC. */
>> +         break;
>> +        default:
>> +            gdprintk(XENLOG_G_WARNING, "ITS: unhandled ITS command %ld\n",
>> +                   its_cmd_get_command(cmdptr));
>> +            break;
>> +        }
>> +
>> +        its->creadr += ITS_CMD_SIZE;
>> +        if ( its->creadr == ITS_CMD_BUFFER_SIZE(its->cbaser) )
>> +            its->creadr = 0;
>> +    }
>> +    its->cwriter = writer;
>> +
>> +    spin_unlock(&its->vcmd_lock);
>> +
>> +    return 0;
>> +}
>> +
>> +/*****************************
>> + * ITS registers read access *
>> + *****************************/
>> +
>> +/*
>> + * The physical address is encoded slightly differently depending on
>> + * the used page size: the highest four bits are stored in the lowest
>> + * four bits of the field for 64K pages.
>> + */
>> +static paddr_t get_baser_phys_addr(uint64_t reg)
>> +{
>> +    if ( reg & BIT(9) )
>> +        return (reg & GENMASK(47, 16)) | ((reg & GENMASK(15, 12)) << 36);
>> +    else
>> +        return reg & GENMASK(47, 12);
>> +}
>> +
>> +static int vgic_v3_its_mmio_read(struct vcpu *v, mmio_info_t *info,
>> +                                 register_t *r, void *priv)
>> +{
>> +    struct virt_its *its = priv;
>> +
>> +    switch ( info->gpa & 0xffff )
>> +    {
>> +    case VREG32(GITS_CTLR):
>> +        if ( info->dabt.size != DABT_WORD ) goto bad_width;
>> +        *r = vgic_reg32_extract(its->enabled | BIT(31), info);
>> +     break;
>> +    case VREG32(GITS_IIDR):
>> +        if ( info->dabt.size != DABT_WORD ) goto bad_width;
>> +        *r = vgic_reg32_extract(GITS_IIDR_VALUE, info);
>> +        break;
>> +    case VREG64(GITS_TYPER):
>> +        if ( info->dabt.size < DABT_WORD ) goto bad_width;
>> +        *r = vgic_reg64_extract(0x1eff1, info);
>
> Please don't hard-code Devbits (bits 17-13) value to 16 here. For dom0, at least match to Kconfig option, or use ITS hardware reported value. On Qualcomm server chips, GITS_TYPER.Devbit is set to 32bits.

On Cavium also, the number Devbits is more than 17. IRC it is 22

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
  2017-03-16 11:20 ` [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT Andre Przywara
@ 2017-03-21 20:17   ` Julien Grall
  2017-03-23 10:57     ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-21 20:17 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 2e023d1..bf64c61 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -45,6 +45,10 @@ config ACPI
>  config HAS_GICV3
>  	bool
>
> +config HAS_ITS
> +        bool "GICv3 ITS MSI controller support"
> +        depends on HAS_GICV3

IIRC, we discussed that GICv3 ITS will be a technical preview for Xen 
4.9. I think case I think it should depends on EXPERT mode to avoid been 
shipped by default.

> +
>  endmenu
>
>  menu "ARM errata workaround via the alternative framework"

[...]

> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> new file mode 100644
> index 0000000..4056e5b
> --- /dev/null
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -0,0 +1,73 @@
> +/*
> + * xen/arch/arm/gic-v3-its.c
> + *
> + * ARM GICv3 Interrupt Translation Service (ITS) support
> + *
> + * Copyright (C) 2016,2017 - ARM Ltd
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; under version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/lib.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
> +
> +LIST_HEAD(host_its_list);

On the previous version we agreed that host_its_list should be confined 
to gic-v3-its.c. So why is it still exported?

[...]

> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> new file mode 100644
> index 0000000..765a655
> --- /dev/null
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -0,0 +1,67 @@


[...]

> +#else
> +
> +static LIST_HEAD(host_its_list);

This is quite ugly and could really be avoided if the host_its_list was 
confined as we agreed in the previous version.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-16 11:20 ` [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table Andre Przywara
@ 2017-03-21 21:23   ` Julien Grall
  2017-03-23 14:40     ` Andre Przywara
  2017-03-21 22:57   ` Stefano Stabellini
  1 sibling, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-21 21:23 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index bf64c61..86f7b53 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -49,6 +49,21 @@ config HAS_ITS
>          bool "GICv3 ITS MSI controller support"
>          depends on HAS_GICV3
>
> +config MAX_PHYS_LPI_BITS
> +        depends on HAS_ITS
> +        int "Maximum bits for GICv3 host LPIs (14-32)"
> +        range 14 32
> +        default "20"
> +        help
> +          Specifies the maximum number of LPIs (in bits) Xen should take
> +          care of. The host ITS may provide support for a very large number
> +          of supported LPIs, for all of which we may not want to allocate
> +          memory, so this number here allows to limit this.
> +          Xen itself does not know how many LPIs domains will ever need
> +          beforehand.
> +          This can be overridden on the command line with the max_lpi_bits
> +          parameter.

I continue to disagree on this Kconfig option. There is no sensible 
default value and I don't see how a distribution will be able to pick-up 
one value here.

If the number of LPIs have to be restricted, this should be done via the 
command line or a per-platform option.

I have already made my point on previous e-mails so I am not going to 
argue more.

> +
>  endmenu
>
>  menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 54860e0..02a8737 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -19,6 +19,7 @@ obj-y += gic.o
>  obj-y += gic-v2.o
>  obj-$(CONFIG_HAS_GICV3) += gic-v3.o
>  obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
> +obj-$(CONFIG_HAS_ITS) += gic-v3-lpi.o
>  obj-y += guestcopy.o
>  obj-y += hvm.o
>  obj-y += io.o
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> new file mode 100644
> index 0000000..4f8414b
> --- /dev/null
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -0,0 +1,201 @@
> +/*
> + * xen/arch/arm/gic-v3-lpi.c
> + *
> + * ARM GICv3 Locality-specific Peripheral Interrupts (LPI) support
> + *
> + * Copyright (C) 2016,2017 - ARM Ltd
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; under version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/lib.h>
> +#include <xen/mm.h>
> +#include <asm/gic.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
> +#include <asm/io.h>
> +#include <asm/page.h>
> +
> +#define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)

NIT: Newline here please.

> +/* Global state */
> +static struct {
> +    /* The global LPI property table, shared by all redistributors. */
> +    uint8_t *lpi_property;
> +    /*
> +     * Number of physical LPIs the host supports. This is a property of
> +     * the GIC hardware. We depart from the habit of naming these things
> +     * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
> +     * in a different context to differentiate them from "virtual LPIs".
> +     */
> +    unsigned long int nr_host_lpis;

Why unsigned long?

> +    unsigned int flags;
> +} lpi_data;
> +
> +struct lpi_redist_data {
> +    void                *pending_table;
> +};
> +
> +static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
> +
> +#define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)

This is fairly confusing. When I read "nr_host_lpis" I understand that 
you store the number of LPIs. But in fact you store the maximum LPI ID.

Also it is very unclear the difference between MAX_PHYS_LPIS and 
nr_host_lpis and will likely cause trouble in the future.

So it looks like to me that nr_host_lpis should be named 
max_host_lpis_ids and MAX_PHYS_LPIS should be MAX_NR_PHYS_LPIS.

Also, please stay consistent with the naming. If you use host then use 
host everywhere and not a mix between phys and host for the same purpose.

> +
> +static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
> +{
> +    uint64_t val;
> +    unsigned int order;
> +    void *pendtable;
> +
> +    if ( this_cpu(lpi_redist).pending_table )
> +        return -EBUSY;
> +
> +    val  = GIC_BASER_CACHE_RaWaWb << GICR_PENDBASER_INNER_CACHEABILITY_SHIFT;
> +    val |= GIC_BASER_CACHE_SameAsInner << GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT;
> +    val |= GIC_BASER_InnerShareable << GICR_PENDBASER_SHAREABILITY_SHIFT;
> +
> +    /*
> +     * The pending table holds one bit per LPI and even covers bits for
> +     * interrupt IDs below 8192, so we allocate the full range.
> +     * The GICv3 imposes a 64KB alignment requirement, also requires
> +     * physically contigious memory.

The bit after the comma seems quite obvious. Both xalloc and 
alloc_xenheap_pages will ensure that the region will be contiguous in 
the physical address space.

[...]

> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
> +{
> +    uint64_t reg;
> +
> +    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
> +    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
> +    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
> +
> +    /*
> +     * The property table is shared across all redistributors, so allocate
> +     * this only once, but return the same value on subsequent calls.
> +     */
> +    if ( !lpi_data.lpi_property )
> +    {
> +        /* The property table holds one byte per LPI. */
> +        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
> +        void *table = alloc_xenheap_pages(order, 0);
> +
> +        if ( !table )
> +            return -ENOMEM;
> +
> +        /* Make sure the physical address can be encoded in the register. */
> +        if ( (virt_to_maddr(table) & ~GENMASK(51, 12)) )
> +        {
> +            free_xenheap_pages(table, 0);
> +            return -ERANGE;
> +        }
> +        memset(table, GIC_PRI_IRQ | LPI_PROP_RES1, MAX_PHYS_LPIS);
> +        clean_and_invalidate_dcache_va_range(table, MAX_PHYS_LPIS);
> +        lpi_data.lpi_property = table;
> +    }
> +
> +    /* Encode the number of bits needed, minus one */
> +    reg |= ((fls(lpi_data.nr_host_lpis - 1) - 1) << 0);

As said on the previous version, please avoid hardcoded shift.

[...]

> +int gicv3_lpi_init_rdist(void __iomem * rdist_base)
> +{
> +    uint32_t reg;
> +    uint64_t table_reg;
> +    int ret;
> +
> +    /* We don't support LPIs without an ITS. */
> +    if ( !gicv3_its_host_has_its() )
> +        return -ENODEV;
> +
> +    /* Make sure LPIs are disabled before setting up the tables. */
> +    reg = readl_relaxed(rdist_base + GICR_CTLR);
> +    if ( reg & GICR_CTLR_ENABLE_LPIS )
> +        return -EBUSY;
> +
> +    ret = gicv3_lpi_allocate_pendtable(&table_reg);
> +    if (ret)
> +        return ret;
> +    writeq_relaxed(table_reg, rdist_base + GICR_PENDBASER);

Same question as on the previous version. The cacheability and 
shareability may not stick. This was a bug fix in Linux (see commit 
241a386) and I am struggling to understand why Xen would not be affected?

> +
> +    return gicv3_lpi_set_proptable(rdist_base);
> +}
> +
> +static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
> +integer_param("max_lpi_bits", max_lpi_bits);
> +
> +int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
> +{


Please add a comment to explain why you don't sanitize the value from 
the command line.

> +    lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));

nr_host_lpis is "unsigned long" so why are you using BIT_ULL?

> +
> +    printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);

s/%ld/%lu/

[...]

> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 1512521..ed78363 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -548,6 +548,9 @@ static void __init gicv3_dist_init(void)
>      type = readl_relaxed(GICD + GICD_TYPER);
>      nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
>
> +    if ( type & GICD_TYPE_LPIS )
> +        gicv3_lpi_init_host_lpis(((type >> GICD_TYPE_ID_BITS_SHIFT) & 0x1f) + 1);

Again, a macro has been suggested on in the last 2 versions to avoid 
hardcoded value. You said you will fix it and it is not done. Please do it.

[...]

> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 836a103..12bd155 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -220,6 +220,8 @@ enum gic_version {
>      GIC_V3,
>  };
>
> +#define LPI_OFFSET      8192

My comments on the previous version about LPI_OFFSET are not addressed. 
And one question was left unanswered.

> +
>  extern enum gic_version gic_hw_version(void);
>
>  /* Program the IRQ type into the GIC */

[...]

> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 765a655..219d109 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -40,6 +40,11 @@ void gicv3_its_dt_init(const struct dt_device_node *node);
>
>  bool gicv3_its_host_has_its(void);
>
> +int gicv3_lpi_init_rdist(void __iomem * rdist_base);
> +
> +/* Initialize the host structures for LPIs. */
> +int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);

In the implementation, the parameter is called "hw_lpi_bits". Please 
stay consistent.

> +
>  #else
>
>  static LIST_HEAD(host_its_list);
> @@ -53,6 +58,15 @@ static inline bool gicv3_its_host_has_its(void)
>      return false;
>  }
>
> +static inline int gicv3_lpi_init_rdist(void __iomem * rdist_base)
> +{
> +    return -ENODEV;
> +}
> +
> +static inline int gicv3_lpi_init_host_lpis(unsigned int nr_lpis)

Ditto.

> +{
> +    return 0;
> +}
>  #endif /* CONFIG_HAS_ITS */
>
>  #endif
>

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-16 11:20 ` [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table Andre Przywara
  2017-03-21 21:23   ` Julien Grall
@ 2017-03-21 22:57   ` Stefano Stabellini
  2017-03-21 23:08     ` André Przywara
  1 sibling, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-21 22:57 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> The ARM GICv3 provides a new kind of interrupt called LPIs.
> The pending bits and the configuration data (priority, enable bits) for
> those LPIs are stored in tables in normal memory, which software has to
> provide to the hardware.
> Allocate the required memory, initialize it and hand it over to each
> redistributor. The maximum number of LPIs to be used can be adjusted with
> the command line option "max_lpi_bits", which defaults to a compile time
> constant exposed in Kconfig.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  docs/misc/xen-command-line.markdown |   9 ++
>  xen/arch/arm/Kconfig                |  15 +++
>  xen/arch/arm/Makefile               |   1 +
>  xen/arch/arm/gic-v3-lpi.c           | 201 ++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/gic-v3.c               |  17 +++
>  xen/include/asm-arm/bitops.h        |   1 +
>  xen/include/asm-arm/gic.h           |   2 +
>  xen/include/asm-arm/gic_v3_defs.h   |  52 +++++++++-
>  xen/include/asm-arm/gic_v3_its.h    |  14 +++
>  9 files changed, 311 insertions(+), 1 deletion(-)
>  create mode 100644 xen/arch/arm/gic-v3-lpi.c
> 
> diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
> index a11fdf9..619016d 100644
> --- a/docs/misc/xen-command-line.markdown
> +++ b/docs/misc/xen-command-line.markdown
> @@ -1158,6 +1158,15 @@ based interrupts. Any higher IRQs will be available for use via PCI MSI.
>  ### maxcpus
>  > `= <integer>`
>  
> +### max\_lpi\_bits
> +> `= <integer>`
> +
> +Specifies the number of ARM GICv3 LPI interrupts to allocate on the host,
> +presented as the number of bits needed to encode it. This must be at least
> +14 and not exceed 32, and each LPI requires one byte (configuration) and
> +one pending bit to be allocated.
> +Defaults to 20 bits (to cover at most 1048576 interrupts).
> +
>  ### mce
>  > `= <integer>`
>  
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index bf64c61..86f7b53 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -49,6 +49,21 @@ config HAS_ITS
>          bool "GICv3 ITS MSI controller support"
>          depends on HAS_GICV3
>  
> +config MAX_PHYS_LPI_BITS
> +        depends on HAS_ITS
> +        int "Maximum bits for GICv3 host LPIs (14-32)"
> +        range 14 32
> +        default "20"
> +        help
> +          Specifies the maximum number of LPIs (in bits) Xen should take
> +          care of. The host ITS may provide support for a very large number
> +          of supported LPIs, for all of which we may not want to allocate
> +          memory, so this number here allows to limit this.
> +          Xen itself does not know how many LPIs domains will ever need
> +          beforehand.
> +          This can be overridden on the command line with the max_lpi_bits
> +          parameter.
> +
>  endmenu
>  
>  menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> index 54860e0..02a8737 100644
> --- a/xen/arch/arm/Makefile
> +++ b/xen/arch/arm/Makefile
> @@ -19,6 +19,7 @@ obj-y += gic.o
>  obj-y += gic-v2.o
>  obj-$(CONFIG_HAS_GICV3) += gic-v3.o
>  obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
> +obj-$(CONFIG_HAS_ITS) += gic-v3-lpi.o
>  obj-y += guestcopy.o
>  obj-y += hvm.o
>  obj-y += io.o
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> new file mode 100644
> index 0000000..4f8414b
> --- /dev/null
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -0,0 +1,201 @@
> +/*
> + * xen/arch/arm/gic-v3-lpi.c
> + *
> + * ARM GICv3 Locality-specific Peripheral Interrupts (LPI) support
> + *
> + * Copyright (C) 2016,2017 - ARM Ltd
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License as published by
> + * the Free Software Foundation; under version 2 of the License.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
> + */
> +
> +#include <xen/lib.h>
> +#include <xen/mm.h>
> +#include <asm/gic.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
> +#include <asm/io.h>
> +#include <asm/page.h>
> +
> +#define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
> +/* Global state */
> +static struct {
> +    /* The global LPI property table, shared by all redistributors. */
> +    uint8_t *lpi_property;
> +    /*
> +     * Number of physical LPIs the host supports. This is a property of
> +     * the GIC hardware. We depart from the habit of naming these things
> +     * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
> +     * in a different context to differentiate them from "virtual LPIs".
> +     */
> +    unsigned long int nr_host_lpis;
> +    unsigned int flags;
> +} lpi_data;
> +
> +struct lpi_redist_data {
> +    void                *pending_table;
> +};
> +
> +static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
> +
> +#define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
> +
> +static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
> +{
> +    uint64_t val;
> +    unsigned int order;
> +    void *pendtable;
> +
> +    if ( this_cpu(lpi_redist).pending_table )
> +        return -EBUSY;
> +
> +    val  = GIC_BASER_CACHE_RaWaWb << GICR_PENDBASER_INNER_CACHEABILITY_SHIFT;
> +    val |= GIC_BASER_CACHE_SameAsInner << GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT;
> +    val |= GIC_BASER_InnerShareable << GICR_PENDBASER_SHAREABILITY_SHIFT;
> +
> +    /*
> +     * The pending table holds one bit per LPI and even covers bits for
> +     * interrupt IDs below 8192, so we allocate the full range.
> +     * The GICv3 imposes a 64KB alignment requirement, also requires
> +     * physically contigious memory.
> +     */
> +    order = max(get_order_from_bytes(lpi_data.nr_host_lpis / 8), 4U);
> +    pendtable = alloc_xenheap_pages(order, 0);
> +    if ( !pendtable )
> +        return -ENOMEM;
> +
> +    /* Make sure the physical address can be encoded in the register. */
> +    if ( (virt_to_maddr(pendtable) & ~GENMASK(51, 16)) )
> +    {
> +        free_xenheap_pages(pendtable, 0);
> +        return -ERANGE;
> +    }
> +    memset(pendtable, 0, lpi_data.nr_host_lpis / 8);
> +    clean_and_invalidate_dcache_va_range(pendtable,
> +                                         lpi_data.nr_host_lpis / 8);
> +
> +    this_cpu(lpi_redist).pending_table = pendtable;
> +
> +    val |= GICR_PENDBASER_PTZ;
> +
> +    val |= virt_to_maddr(pendtable);
> +
> +    *reg = val;
> +
> +    return 0;
> +}
> +
> +/*
> + * Tell a redistributor about the (shared) property table, allocating one
> + * if not already done.
> + */
> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
> +{
> +    uint64_t reg;
> +
> +    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
> +    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
> +    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
> +
> +    /*
> +     * The property table is shared across all redistributors, so allocate
> +     * this only once, but return the same value on subsequent calls.
> +     */
> +    if ( !lpi_data.lpi_property )
> +    {
> +        /* The property table holds one byte per LPI. */
> +        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
> +        void *table = alloc_xenheap_pages(order, 0);

you investigated the possibility of using _xmalloc to 64K align it?


> +        if ( !table )
> +            return -ENOMEM;
> +
> +        /* Make sure the physical address can be encoded in the register. */
> +        if ( (virt_to_maddr(table) & ~GENMASK(51, 12)) )
> +        {
> +            free_xenheap_pages(table, 0);
> +            return -ERANGE;
> +        }
> +        memset(table, GIC_PRI_IRQ | LPI_PROP_RES1, MAX_PHYS_LPIS);
> +        clean_and_invalidate_dcache_va_range(table, MAX_PHYS_LPIS);
> +        lpi_data.lpi_property = table;
> +    }
> +
> +    /* Encode the number of bits needed, minus one */
> +    reg |= ((fls(lpi_data.nr_host_lpis - 1) - 1) << 0);
> +
> +    reg |= virt_to_maddr(lpi_data.lpi_property);
> +
> +    writeq_relaxed(reg, rdist_base + GICR_PROPBASER);
> +    reg = readq_relaxed(rdist_base + GICR_PROPBASER);
> +
> +    /* If we can't do shareable, we have to drop cacheability as well. */
> +    if ( !(reg & GICR_PROPBASER_SHAREABILITY_MASK) )
> +    {
> +        reg &= ~GICR_PROPBASER_INNER_CACHEABILITY_MASK;
> +        reg |= GIC_BASER_CACHE_nC << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
> +    }
> +
> +    /* Remember that we have to flush the property table if non-cacheable. */
> +    if ( (reg & GICR_PROPBASER_INNER_CACHEABILITY_MASK) <= GIC_BASER_CACHE_nC )
> +    {
> +        lpi_data.flags |= LPI_PROPTABLE_NEEDS_FLUSHING;
> +        /* Update the redistributors knowledge about the attributes. */
> +        writeq_relaxed(reg, rdist_base + GICR_PROPBASER);
> +    }
> +
> +    return 0;
> +}
> +
> +int gicv3_lpi_init_rdist(void __iomem * rdist_base)
> +{
> +    uint32_t reg;
> +    uint64_t table_reg;
> +    int ret;
> +
> +    /* We don't support LPIs without an ITS. */
> +    if ( !gicv3_its_host_has_its() )
> +        return -ENODEV;
> +
> +    /* Make sure LPIs are disabled before setting up the tables. */
> +    reg = readl_relaxed(rdist_base + GICR_CTLR);
> +    if ( reg & GICR_CTLR_ENABLE_LPIS )
> +        return -EBUSY;
> +
> +    ret = gicv3_lpi_allocate_pendtable(&table_reg);
> +    if (ret)
> +        return ret;
> +    writeq_relaxed(table_reg, rdist_base + GICR_PENDBASER);
> +
> +    return gicv3_lpi_set_proptable(rdist_base);
> +}
> +
> +static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
> +integer_param("max_lpi_bits", max_lpi_bits);
> +
> +int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
> +{
> +    lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
> +
> +    printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);
> +
> +    return 0;
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-file-style: "BSD"
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index 1512521..ed78363 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -548,6 +548,9 @@ static void __init gicv3_dist_init(void)
>      type = readl_relaxed(GICD + GICD_TYPER);
>      nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
>  
> +    if ( type & GICD_TYPE_LPIS )
> +        gicv3_lpi_init_host_lpis(((type >> GICD_TYPE_ID_BITS_SHIFT) & 0x1f) + 1);
> +
>      printk("GICv3: %d lines, (IID %8.8x).\n",
>             nr_lines, readl_relaxed(GICD + GICD_IIDR));
>  
> @@ -660,6 +663,20 @@ static int __init gicv3_populate_rdist(void)
>              if ( (typer >> 32) == aff )
>              {
>                  this_cpu(rbase) = ptr;
> +
> +                if ( typer & GICR_TYPER_PLPIS )
> +                {
> +                    int ret;
> +
> +                    ret = gicv3_lpi_init_rdist(ptr);
> +                    if ( ret && ret != -ENODEV )
> +                    {
> +                        printk("GICv3: CPU%d: Cannot initialize LPIs: %d\n",
> +                               smp_processor_id(), ret);
> +                        break;
> +                    }
> +                }
> +
>                  printk("GICv3: CPU%d: Found redistributor in region %d @%p\n",
>                          smp_processor_id(), i, ptr);
>                  return 0;
> diff --git a/xen/include/asm-arm/bitops.h b/xen/include/asm-arm/bitops.h
> index bda8898..1cbfb9e 100644
> --- a/xen/include/asm-arm/bitops.h
> +++ b/xen/include/asm-arm/bitops.h
> @@ -24,6 +24,7 @@
>  #define BIT(nr)                 (1UL << (nr))
>  #define BIT_MASK(nr)            (1UL << ((nr) % BITS_PER_WORD))
>  #define BIT_WORD(nr)            ((nr) / BITS_PER_WORD)
> +#define BIT_ULL(nr)             (1ULL << (nr))
>  #define BITS_PER_BYTE           8
>  
>  #define ADDR (*(volatile int *) addr)
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 836a103..12bd155 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -220,6 +220,8 @@ enum gic_version {
>      GIC_V3,
>  };
>  
> +#define LPI_OFFSET      8192
> +
>  extern enum gic_version gic_hw_version(void);
>  
>  /* Program the IRQ type into the GIC */
> diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
> index 6bd25a5..b307322 100644
> --- a/xen/include/asm-arm/gic_v3_defs.h
> +++ b/xen/include/asm-arm/gic_v3_defs.h
> @@ -44,7 +44,8 @@
>  #define GICC_SRE_EL2_ENEL1           (1UL << 3)
>  
>  /* Additional bits in GICD_TYPER defined by GICv3 */
> -#define GICD_TYPE_ID_BITS_SHIFT 19
> +#define GICD_TYPE_ID_BITS_SHIFT      19
> +#define GICD_TYPE_LPIS               (1U << 17)
>  
>  #define GICD_CTLR_RWP                (1UL << 31)
>  #define GICD_CTLR_ARE_NS             (1U << 4)
> @@ -95,12 +96,61 @@
>  #define GICR_IGRPMODR0               (0x0D00)
>  #define GICR_NSACR                   (0x0E00)
>  
> +#define GICR_CTLR_ENABLE_LPIS        (1U << 0)
> +
>  #define GICR_TYPER_PLPIS             (1U << 0)
>  #define GICR_TYPER_VLPIS             (1U << 1)
>  #define GICR_TYPER_LAST              (1U << 4)
>  
> +/* For specifying the inner cacheability type only */
> +#define GIC_BASER_CACHE_nCnB         0ULL
> +/* For specifying the outer cacheability type only */
> +#define GIC_BASER_CACHE_SameAsInner  0ULL
> +#define GIC_BASER_CACHE_nC           1ULL
> +#define GIC_BASER_CACHE_RaWt         2ULL
> +#define GIC_BASER_CACHE_RaWb         3ULL
> +#define GIC_BASER_CACHE_WaWt         4ULL
> +#define GIC_BASER_CACHE_WaWb         5ULL
> +#define GIC_BASER_CACHE_RaWaWt       6ULL
> +#define GIC_BASER_CACHE_RaWaWb       7ULL
> +#define GIC_BASER_CACHE_MASK         7ULL
> +
> +#define GIC_BASER_NonShareable       0ULL
> +#define GIC_BASER_InnerShareable     1ULL
> +#define GIC_BASER_OuterShareable     2ULL
> +
> +#define GICR_PROPBASER_SHAREABILITY_SHIFT               10
> +#define GICR_PROPBASER_INNER_CACHEABILITY_SHIFT         7
> +#define GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT         56
> +#define GICR_PROPBASER_SHAREABILITY_MASK                     \
> +        (3UL << GICR_PROPBASER_SHAREABILITY_SHIFT)
> +#define GICR_PROPBASER_INNER_CACHEABILITY_MASK               \
> +        (7UL << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT)
> +#define GICR_PROPBASER_OUTER_CACHEABILITY_MASK               \
> +        (7UL << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT)
> +#define GICR_PROPBASER_RES0_MASK                             \
> +        (GENMASK(63, 59) | GENMASK(55, 52) | GENMASK(6, 5))
> +
> +#define GICR_PENDBASER_SHAREABILITY_SHIFT               10
> +#define GICR_PENDBASER_INNER_CACHEABILITY_SHIFT         7
> +#define GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT         56
> +#define GICR_PENDBASER_SHAREABILITY_MASK                     \
> +	(3UL << GICR_PENDBASER_SHAREABILITY_SHIFT)
> +#define GICR_PENDBASER_INNER_CACHEABILITY_MASK               \
> +	(7UL << GICR_PENDBASER_INNER_CACHEABILITY_SHIFT)
> +#define GICR_PENDBASER_OUTER_CACHEABILITY_MASK               \
> +        (7UL << GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT)
> +#define GICR_PENDBASER_PTZ                              BIT(62)
> +#define GICR_PENDBASER_RES0_MASK                             \
> +        (BIT(63) | GENMASK(61, 59) | GENMASK(55, 52) |       \
> +         GENMASK(15, 12) | GENMASK(6, 0))
> +
>  #define DEFAULT_PMR_VALUE            0xff
>  
> +#define LPI_PROP_PRIO_MASK           0xfc
> +#define LPI_PROP_RES1                (1 << 1)
> +#define LPI_PROP_ENABLED             (1 << 0)
> +
>  #define GICH_VMCR_EOI                (1 << 9)
>  #define GICH_VMCR_VENG1              (1 << 1)
>  
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 765a655..219d109 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -40,6 +40,11 @@ void gicv3_its_dt_init(const struct dt_device_node *node);
>  
>  bool gicv3_its_host_has_its(void);
>  
> +int gicv3_lpi_init_rdist(void __iomem * rdist_base);
> +
> +/* Initialize the host structures for LPIs. */
> +int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
> +
>  #else
>  
>  static LIST_HEAD(host_its_list);
> @@ -53,6 +58,15 @@ static inline bool gicv3_its_host_has_its(void)
>      return false;
>  }
>  
> +static inline int gicv3_lpi_init_rdist(void __iomem * rdist_base)
> +{
> +    return -ENODEV;
> +}
> +
> +static inline int gicv3_lpi_init_host_lpis(unsigned int nr_lpis)
> +{
> +    return 0;
> +}
>  #endif /* CONFIG_HAS_ITS */
>  
>  #endif
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-21 22:57   ` Stefano Stabellini
@ 2017-03-21 23:08     ` André Przywara
  2017-03-21 23:27       ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: André Przywara @ 2017-03-21 23:08 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

On 21/03/17 22:57, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Andre Przywara wrote:
>> The ARM GICv3 provides a new kind of interrupt called LPIs.
>> The pending bits and the configuration data (priority, enable bits) for
>> those LPIs are stored in tables in normal memory, which software has to
>> provide to the hardware.
>> Allocate the required memory, initialize it and hand it over to each
>> redistributor. The maximum number of LPIs to be used can be adjusted with
>> the command line option "max_lpi_bits", which defaults to a compile time
>> constant exposed in Kconfig.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

....

>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>> new file mode 100644
>> index 0000000..4f8414b
>> --- /dev/null
>> +++ b/xen/arch/arm/gic-v3-lpi.c

....

>> +/*
>> + * Tell a redistributor about the (shared) property table, allocating one
>> + * if not already done.
>> + */
>> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
>> +{
>> +    uint64_t reg;
>> +
>> +    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
>> +    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
>> +    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
>> +
>> +    /*
>> +     * The property table is shared across all redistributors, so allocate
>> +     * this only once, but return the same value on subsequent calls.
>> +     */
>> +    if ( !lpi_data.lpi_property )
>> +    {
>> +        /* The property table holds one byte per LPI. */
>> +        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
>> +        void *table = alloc_xenheap_pages(order, 0);
> 
> you investigated the possibility of using _xmalloc to 64K align it?

Yes, _xmalloc gives me the alignment, but AFAICT not the physical
contiguous memory guarantee the table needs. If there are other ways to
satisfy those requirements, I am all ears. Otherwise I'd just add a
comment to motivate this.

Cheers,
Andre.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-21 23:08     ` André Przywara
@ 2017-03-21 23:27       ` Stefano Stabellini
  2017-03-23 10:50         ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-21 23:27 UTC (permalink / raw)
  To: André Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

[-- Attachment #1: Type: TEXT/PLAIN, Size: 2383 bytes --]

On Tue, 21 Mar 2017, André Przywara wrote:
> On 21/03/17 22:57, Stefano Stabellini wrote:
> > On Thu, 16 Mar 2017, Andre Przywara wrote:
> >> The ARM GICv3 provides a new kind of interrupt called LPIs.
> >> The pending bits and the configuration data (priority, enable bits) for
> >> those LPIs are stored in tables in normal memory, which software has to
> >> provide to the hardware.
> >> Allocate the required memory, initialize it and hand it over to each
> >> redistributor. The maximum number of LPIs to be used can be adjusted with
> >> the command line option "max_lpi_bits", which defaults to a compile time
> >> constant exposed in Kconfig.
> >>
> >> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> 
> ....
> 
> >> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> >> new file mode 100644
> >> index 0000000..4f8414b
> >> --- /dev/null
> >> +++ b/xen/arch/arm/gic-v3-lpi.c
> 
> ....
> 
> >> +/*
> >> + * Tell a redistributor about the (shared) property table, allocating one
> >> + * if not already done.
> >> + */
> >> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
> >> +{
> >> +    uint64_t reg;
> >> +
> >> +    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
> >> +    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
> >> +    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
> >> +
> >> +    /*
> >> +     * The property table is shared across all redistributors, so allocate
> >> +     * this only once, but return the same value on subsequent calls.
> >> +     */
> >> +    if ( !lpi_data.lpi_property )
> >> +    {
> >> +        /* The property table holds one byte per LPI. */
> >> +        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
> >> +        void *table = alloc_xenheap_pages(order, 0);
> > 
> > you investigated the possibility of using _xmalloc to 64K align it?
> 
> Yes, _xmalloc gives me the alignment, but AFAICT not the physical
> contiguous memory guarantee the table needs. If there are other ways to
> satisfy those requirements, I am all ears. Otherwise I'd just add a
> comment to motivate this.

OK, but the problem is that I am not sure whether alloc_xenheap_pages
makes any guarantees on the alignment of the allocated pages. Is it
possible to allocate 1<<4 pages that are not 64K aligned with it?

[-- Attachment #2: Type: text/plain, Size: 127 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table
  2017-03-16 11:20 ` [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table Andre Przywara
@ 2017-03-21 23:29   ` Stefano Stabellini
  2017-03-22 13:52   ` Julien Grall
  1 sibling, 0 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-21 23:29 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> Each ITS maps a pair of a DeviceID (for instance derived from a PCI
> b/d/f triplet) and an EventID (the MSI payload or interrupt ID) to a
> pair of LPI number and collection ID, which points to the target CPU.
> This mapping is stored in the device and collection tables, which software
> has to provide for the ITS to use.
> Allocate the required memory and hand it to the ITS.
> The maximum number of devices is limited to a compile-time constant
> exposed in Kconfig.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  docs/misc/xen-command-line.markdown |   8 ++
>  xen/arch/arm/Kconfig                |  14 ++++
>  xen/arch/arm/gic-v3-its.c           | 163 ++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/gic-v3.c               |   3 +
>  xen/include/asm-arm/gic_v3_its.h    |  63 +++++++++++++-
>  5 files changed, 250 insertions(+), 1 deletion(-)
> 
> diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
> index 619016d..068d116 100644
> --- a/docs/misc/xen-command-line.markdown
> +++ b/docs/misc/xen-command-line.markdown
> @@ -1158,6 +1158,14 @@ based interrupts. Any higher IRQs will be available for use via PCI MSI.
>  ### maxcpus
>  > `= <integer>`
>  
> +### max\_its\_device\_bits
> +> `= <integer>`
> +
> +Specifies the maximum number of devices using MSIs on the ARM GICv3 ITS
> +controller to allocate table entries for. Each table entry uses a hardware
> +specific size, typically 8 or 16 bytes.
> +Defaults to 10 bits (to cover at most 1024 devices).
> +
>  ### max\_lpi\_bits
>  > `= <integer>`
>  
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 86f7b53..0d50156 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -64,6 +64,20 @@ config MAX_PHYS_LPI_BITS
>            This can be overridden on the command line with the max_lpi_bits
>            parameter.
>  
> +config MAX_PHYS_ITS_DEVICE_BITS
> +        depends on HAS_ITS
> +        int "Number of device bits the ITS supports"
> +        range 1 32
> +        default "10"
> +        help
> +          Specifies the maximum number of devices which want to use the ITS.
> +          Xen needs to allocates memory for the whole range very early.
> +          The allocation scheme may be sparse, so a much larger number must
> +          be supported to cover devices with a high bus number or those on
> +          separate bus segments.
> +          This can be overridden on the command line with the
> +          max_its_device_bits parameter.
> +
>  endmenu
>  
>  menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 4056e5b..9982fe9 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -19,8 +19,10 @@
>   */
>  
>  #include <xen/lib.h>
> +#include <xen/mm.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
> +#include <asm/io.h>
>  
>  LIST_HEAD(host_its_list);
>  
> @@ -29,6 +31,167 @@ bool gicv3_its_host_has_its(void)
>      return !list_empty(&host_its_list);
>  }
>  
> +#define BASER_ATTR_MASK                                           \
> +        ((0x3UL << GITS_BASER_SHAREABILITY_SHIFT)               | \
> +         (0x7UL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)         | \
> +         (0x7UL << GITS_BASER_INNER_CACHEABILITY_SHIFT))
> +#define BASER_RO_MASK   (GENMASK(58, 56) | GENMASK(52, 48))
> +
> +/* Check that the physical address can be encoded in the PROPBASER register. */
> +static bool check_propbaser_phys_addr(void *vaddr, unsigned int page_bits)
> +{
> +    paddr_t paddr = virt_to_maddr(vaddr);
> +
> +    return (!(paddr & ~GENMASK(page_bits < 16 ? 47 : 51, page_bits)));
> +}
> +
> +static uint64_t encode_propbaser_phys_addr(paddr_t addr, unsigned int page_bits)
> +{
> +    uint64_t ret = addr & GENMASK(47, page_bits);
> +
> +    if ( page_bits < 16 )
> +        return ret;
> +
> +    /* For 64K pages address bits 51-48 are encoded in bits 15-12. */
> +    return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
> +}
> +
> +/* The ITS BASE registers work with page sizes of 4K, 16K or 64K. */
> +#define BASER_PAGE_BITS(sz) ((sz) * 2 + 12)
> +
> +static int its_map_baser(void __iomem *basereg, uint64_t regc,
> +                         unsigned int nr_items)
> +{
> +    uint64_t attr, reg;
> +    unsigned int entry_size = GITS_BASER_ENTRY_SIZE(regc);
> +    unsigned int pagesz = 2, order, table_size;
> +    void *buffer;
> +
> +    attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
> +    attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
> +    attr |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
> +
> +    /*
> +     * Setup the BASE register with the attributes that we like. Then read
> +     * it back and see what sticks (page size, cacheability and shareability
> +     * attributes), retrying if necessary.
> +     */
> +retry:
> +    table_size = ROUNDUP(nr_items * entry_size, BIT(BASER_PAGE_BITS(pagesz)));
> +    /* The BASE registers support at most 256 pages. */
> +    table_size = min(table_size, 256U << BASER_PAGE_BITS(pagesz));
> +    /* The memory block must be aligned to the requested page size. */
> +    order = max(get_order_from_bytes(table_size), pagesz * 2);
> +
> +    buffer = alloc_xenheap_pages(order, 0);
> +    if ( !buffer )
> +        return -ENOMEM;
> +
> +    if ( !check_propbaser_phys_addr(buffer, BASER_PAGE_BITS(pagesz)) )
> +    {
> +        free_xenheap_pages(buffer, 0);
> +        return -ERANGE;
> +    }
> +    memset(buffer, 0, table_size);
> +
> +    reg  = attr;
> +    reg |= (pagesz << GITS_BASER_PAGE_SIZE_SHIFT);
> +    reg |= (table_size >> BASER_PAGE_BITS(pagesz)) - 1;
> +    reg |= regc & BASER_RO_MASK;
> +    reg |= GITS_VALID_BIT;
> +    reg |= encode_propbaser_phys_addr(virt_to_maddr(buffer),
> +                                      BASER_PAGE_BITS(pagesz));
> +
> +    writeq_relaxed(reg, basereg);
> +    regc = readq_relaxed(basereg);
> +
> +    /* The host didn't like our attributes, just use what it returned. */
> +    if ( (regc & BASER_ATTR_MASK) != attr )
> +    {
> +        /* If we can't map it shareable, drop cacheability as well. */
> +        if ( (regc & GITS_BASER_SHAREABILITY_MASK) == GIC_BASER_NonShareable )
> +        {
> +            regc &= ~GITS_BASER_INNER_CACHEABILITY_MASK;
> +            writeq_relaxed(regc, basereg);
> +        }
> +        attr = regc & BASER_ATTR_MASK;
> +    }
> +    if ( (regc & GITS_BASER_INNER_CACHEABILITY_MASK) <= GIC_BASER_CACHE_nC )
> +        clean_and_invalidate_dcache_va_range(buffer, table_size);
> +
> +    /* If the host accepted our page size, we are done. */
> +    if ( ((regc >> GITS_BASER_PAGE_SIZE_SHIFT) & 0x3UL) == pagesz )
> +        return 0;
> +
> +    free_xenheap_pages(buffer, order);
> +
> +    if ( pagesz-- > 0 )
> +        goto retry;
> +
> +    /* None of the page sizes was accepted, give up */
> +    return -EINVAL;
> +}
> +
> +static unsigned int max_its_device_bits = CONFIG_MAX_PHYS_ITS_DEVICE_BITS;
> +integer_param("max_its_device_bits", max_its_device_bits);
> +
> +static int gicv3_its_init_single_its(struct host_its *hw_its)
> +{
> +    uint64_t reg;
> +    int i;
> +    unsigned int devid_bits;
> +
> +    hw_its->its_base = ioremap_nocache(hw_its->addr, hw_its->size);
> +    if ( !hw_its->its_base )
> +        return -ENOMEM;
> +
> +    reg = readq_relaxed(hw_its->its_base + GITS_TYPER);
> +    devid_bits = GITS_TYPER_DEVICE_ID_BITS(reg);
> +    devid_bits = min(devid_bits, max_its_device_bits);
> +
> +    for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
> +    {
> +        void __iomem *basereg = hw_its->its_base + GITS_BASER0 + i * 8;
> +        unsigned int type;
> +
> +        reg = readq_relaxed(basereg);
> +        type = (reg & GITS_BASER_TYPE_MASK) >> GITS_BASER_TYPE_SHIFT;
> +        switch ( type )
> +        {
> +        case GITS_BASER_TYPE_NONE:
> +            continue;
> +        case GITS_BASER_TYPE_DEVICE:
> +            its_map_baser(basereg, reg, BIT(devid_bits));

Check return value of its_map_baser?


> +            break;

> +        case GITS_BASER_TYPE_COLLECTION:
> +            its_map_baser(basereg, reg, NR_CPUS);

We should use num_possible_cpus() instead of NR_CPUS here


> +            break;
> +        /* In case this is a GICv4, provide a (dummy) vPE table as well. */
> +        case GITS_BASER_TYPE_VCPU:
> +            its_map_baser(basereg, reg, 1);
> +            break;
> +        default:
> +            continue;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
> +int gicv3_its_init(void)
> +{
> +    struct host_its *hw_its;
> +    int ret;
> +
> +    list_for_each_entry(hw_its, &host_its_list, entry) {
> +        ret = gicv3_its_init_single_its(hw_its);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    return 0;
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index ed78363..cc1e219 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -1590,6 +1590,9 @@ static int __init gicv3_init(void)
>      spin_lock(&gicv3.lock);
>  
>      gicv3_dist_init();
> +    res = gicv3_its_init();
> +    if ( res )
> +        printk(XENLOG_WARNING "GICv3: ITS: initialization failed: %d\n", res);
>      res = gicv3_cpu_init();
>      gicv3_hyp_init();
>  
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 219d109..a6c0acc 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -20,6 +20,60 @@
>  #ifndef __ASM_ARM_ITS_H__
>  #define __ASM_ARM_ITS_H__
>  
> +#define GITS_CTLR                       0x000
> +#define GITS_IIDR                       0x004
> +#define GITS_TYPER                      0x008
> +#define GITS_CBASER                     0x080
> +#define GITS_CWRITER                    0x088
> +#define GITS_CREADR                     0x090
> +#define GITS_BASER_NR_REGS              8
> +#define GITS_BASER0                     0x100
> +#define GITS_BASER1                     0x108
> +#define GITS_BASER2                     0x110
> +#define GITS_BASER3                     0x118
> +#define GITS_BASER4                     0x120
> +#define GITS_BASER5                     0x128
> +#define GITS_BASER6                     0x130
> +#define GITS_BASER7                     0x138
> +
> +/* Register bits */
> +#define GITS_VALID_BIT                  BIT_ULL(63)
> +
> +#define GITS_CTLR_QUIESCENT             BIT(31)
> +#define GITS_CTLR_ENABLE                BIT(0)
> +
> +#define GITS_TYPER_DEVIDS_SHIFT         13
> +#define GITS_TYPER_DEVIDS_MASK          (0x1fUL << GITS_TYPER_DEVIDS_SHIFT)
> +#define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
> +                                               GITS_TYPER_DEVIDS_SHIFT) + 1)
> +
> +#define GITS_IIDR_VALUE                 0x34c
> +
> +#define GITS_BASER_INDIRECT             BIT_ULL(62)
> +#define GITS_BASER_INNER_CACHEABILITY_SHIFT        59
> +#define GITS_BASER_TYPE_SHIFT           56
> +#define GITS_BASER_TYPE_MASK            (7ULL << GITS_BASER_TYPE_SHIFT)
> +#define GITS_BASER_OUTER_CACHEABILITY_SHIFT        53
> +#define GITS_BASER_TYPE_NONE            0UL
> +#define GITS_BASER_TYPE_DEVICE          1UL
> +#define GITS_BASER_TYPE_VCPU            2UL
> +#define GITS_BASER_TYPE_CPU             3UL
> +#define GITS_BASER_TYPE_COLLECTION      4UL
> +#define GITS_BASER_TYPE_RESERVED5       5UL
> +#define GITS_BASER_TYPE_RESERVED6       6UL
> +#define GITS_BASER_TYPE_RESERVED7       7UL
> +#define GITS_BASER_ENTRY_SIZE_SHIFT     48
> +#define GITS_BASER_ENTRY_SIZE(reg)                                       \
> +                        (((reg >> GITS_BASER_ENTRY_SIZE_SHIFT) & 0x1f) + 1)
> +#define GITS_BASER_SHAREABILITY_SHIFT   10
> +#define GITS_BASER_PAGE_SIZE_SHIFT      8
> +#define GITS_BASER_RO_MASK              (GITS_BASER_TYPE_MASK | \
> +                                        (31UL << GITS_BASER_ENTRY_SIZE_SHIFT) |\
> +                                        GITS_BASER_INDIRECT)
> +#define GITS_BASER_SHAREABILITY_MASK   (0x3ULL << GITS_BASER_SHAREABILITY_SHIFT)
> +#define GITS_BASER_OUTER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)
> +#define GITS_BASER_INNER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_INNER_CACHEABILITY_SHIFT)
> +
>  #include <xen/device_tree.h>
>  
>  /* data structure for each hardware ITS */
> @@ -28,6 +82,7 @@ struct host_its {
>      const struct dt_device_node *dt_node;
>      paddr_t addr;
>      paddr_t size;
> +    void __iomem *its_base;
>  };
>  
>  
> @@ -42,8 +97,9 @@ bool gicv3_its_host_has_its(void);
>  
>  int gicv3_lpi_init_rdist(void __iomem * rdist_base);
>  
> -/* Initialize the host structures for LPIs. */
> +/* Initialize the host structures for LPIs and the host ITSes. */
>  int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
> +int gicv3_its_init(void);
>  
>  #else
>  
> @@ -67,6 +123,11 @@ static inline int gicv3_lpi_init_host_lpis(unsigned int nr_lpis)
>  {
>      return 0;
>  }
> +
> +static inline int gicv3_its_init(void)
> +{
> +    return 0;
> +}
>  #endif /* CONFIG_HAS_ITS */
>  
>  #endif
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer
  2017-03-16 11:20 ` [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer Andre Przywara
@ 2017-03-21 23:48   ` Stefano Stabellini
  2017-03-22 15:23   ` Julien Grall
  1 sibling, 0 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-21 23:48 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> Instead of directly manipulating the tables in memory, an ITS driver
> sends commands via a ring buffer in normal system memory to the ITS h/w
> to create or alter the LPI mappings.
> Allocate memory for that buffer and tell the ITS about it to be able
> to send ITS commands.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>


> ---
>  xen/arch/arm/gic-v3-its.c        | 57 ++++++++++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/gic_v3_its.h |  6 +++++
>  2 files changed, 63 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 9982fe9..e5601ed 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -20,10 +20,13 @@
>  
>  #include <xen/lib.h>
>  #include <xen/mm.h>
> +#include <xen/sizes.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
>  
> +#define ITS_CMD_QUEUE_SZ                SZ_64K
> +
>  LIST_HEAD(host_its_list);
>  
>  bool gicv3_its_host_has_its(void)
> @@ -56,6 +59,55 @@ static uint64_t encode_propbaser_phys_addr(paddr_t addr, unsigned int page_bits)
>      return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
>  }
>  
> +static void *its_map_cbaser(struct host_its *its)
> +{
> +    void __iomem *cbasereg = its->its_base + GITS_CBASER;
> +    uint64_t reg;
> +    void *buffer;
> +    unsigned int order;
> +
> +    reg  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
> +    reg |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
> +    reg |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
> +
> +    /* The ITS command buffer needs to be 64K aligned. */
> +    order = max(get_order_from_bytes(ITS_CMD_QUEUE_SZ), 16U - PAGE_SHIFT);
> +    buffer = alloc_xenheap_pages(order, 0);
> +    if ( !buffer )
> +        return NULL;
> +
> +    if ( virt_to_maddr(buffer) & ~GENMASK(51, 12) )
> +    {
> +        free_xenheap_pages(buffer, 0);
> +        return NULL;
> +    }
> +    memset(buffer, 0, ITS_CMD_QUEUE_SZ);
> +
> +    reg |= GITS_VALID_BIT | virt_to_maddr(buffer);
> +    reg |= ((ITS_CMD_QUEUE_SZ / SZ_4K) - 1) & GITS_CBASER_SIZE_MASK;
> +    writeq_relaxed(reg, cbasereg);
> +    reg = readq_relaxed(cbasereg);
> +
> +    /* If the ITS dropped shareability, drop cacheability as well. */
> +    if ( (reg & GITS_BASER_SHAREABILITY_MASK) == 0 )
> +    {
> +        reg &= ~GITS_BASER_INNER_CACHEABILITY_MASK;
> +        writeq_relaxed(reg, cbasereg);
> +    }
> +
> +    /*
> +     * If the command queue memory is mapped as uncached, we need to flush
> +     * it on every access.
> +     */
> +    if ( !(reg & GITS_BASER_INNER_CACHEABILITY_MASK) )
> +    {
> +        its->flags |= HOST_ITS_FLUSH_CMD_QUEUE;
> +        dprintk(XENLOG_WARNING, "using non-cacheable ITS command queue\n");
> +    }
> +
> +    return buffer;
> +}
> +
>  /* The ITS BASE registers work with page sizes of 4K, 16K or 64K. */
>  #define BASER_PAGE_BITS(sz) ((sz) * 2 + 12)
>  
> @@ -175,6 +227,11 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
>          }
>      }
>  
> +    hw_its->cmd_buf = its_map_cbaser(hw_its);
> +    if ( !hw_its->cmd_buf )
> +        return -ENOMEM;
> +    writeq_relaxed(0, hw_its->its_base + GITS_CWRITER);
> +
>      return 0;
>  }
>  
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index a6c0acc..d5facf0 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -74,8 +74,12 @@
>  #define GITS_BASER_OUTER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)
>  #define GITS_BASER_INNER_CACHEABILITY_MASK   (0x7ULL << GITS_BASER_INNER_CACHEABILITY_SHIFT)
>  
> +#define GITS_CBASER_SIZE_MASK           0xff
> +
>  #include <xen/device_tree.h>
>  
> +#define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
> +
>  /* data structure for each hardware ITS */
>  struct host_its {
>      struct list_head entry;
> @@ -83,6 +87,8 @@ struct host_its {
>      paddr_t addr;
>      paddr_t size;
>      void __iomem *its_base;
> +    void *cmd_buf;
> +    unsigned int flags;
>  };
>  
>  
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-03-16 11:20 ` [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling Andre Przywara
  2017-03-16 15:05   ` Shanker Donthineni
@ 2017-03-22  0:02   ` Stefano Stabellini
  2017-03-22 15:59   ` Julien Grall
  2 siblings, 0 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-22  0:02 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> To be able to easily send commands to the ITS, create the respective
> wrapper functions, which take care of the ring buffer.
> The first two commands we implement provide methods to map a collection
> to a redistributor (aka host core) and to flush the command queue (SYNC).
> Start using these commands for mapping one collection to each host CPU.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c         | 181 +++++++++++++++++++++++++++++++++++++-
>  xen/arch/arm/gic-v3-lpi.c         |  22 +++++
>  xen/arch/arm/gic-v3.c             |  19 +++-
>  xen/include/asm-arm/gic_v3_defs.h |   2 +
>  xen/include/asm-arm/gic_v3_its.h  |  38 ++++++++
>  5 files changed, 260 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index e5601ed..5c11b0d 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -19,11 +19,14 @@
>   */
>  
>  #include <xen/lib.h>
> +#include <xen/delay.h>
>  #include <xen/mm.h>
>  #include <xen/sizes.h>
> +#include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
> +#include <asm/page.h>
>  
>  #define ITS_CMD_QUEUE_SZ                SZ_64K
>  
> @@ -34,6 +37,145 @@ bool gicv3_its_host_has_its(void)
>      return !list_empty(&host_its_list);
>  }
>  
> +#define BUFPTR_MASK                     GENMASK(19, 5)
> +static int its_send_command(struct host_its *hw_its, const void *its_cmd)
> +{
> +    s_time_t deadline = NOW() + MILLISECS(1);
> +    uint64_t readp, writep;
> +    int ret = -EBUSY;
> +
> +    /* No ITS commands from an interrupt handler (at the moment). */
> +    ASSERT(!in_irq());
> +
> +    spin_lock(&hw_its->cmd_lock);
> +
> +    do {
> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
> +
> +        if ( ((writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ) != readp )
> +        {
> +            ret = 0;
> +            break;
> +        }
> +
> +        /*
> +         * If the command queue is full, wait for a bit in the hope it drains
> +         * before giving up.
> +         */
> +        spin_unlock(&hw_its->cmd_lock);
> +        cpu_relax();
> +        udelay(1);
> +        spin_lock(&hw_its->cmd_lock);
> +    } while ( NOW() <= deadline );
> +
> +    if ( ret )
> +    {
> +        spin_unlock(&hw_its->cmd_lock);
> +        printk(XENLOG_WARNING "ITS: command queue full.\n");
> +        return ret;
> +    }
> +
> +    memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE);
> +    if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
> +        clean_and_invalidate_dcache_va_range(hw_its->cmd_buf + writep,
> +                                             ITS_CMD_SIZE);
> +    else
> +        dsb(ishst);
> +
> +    writep = (writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ;
> +    writeq_relaxed(writep & BUFPTR_MASK, hw_its->its_base + GITS_CWRITER);
> +
> +    spin_unlock(&hw_its->cmd_lock);
> +
> +    return 0;
> +}
> +
> +/* Wait for an ITS to finish processing all commands. */
> +static int gicv3_its_wait_commands(struct host_its *hw_its)
> +{
> +    s_time_t deadline = NOW() + MILLISECS(100);
> +    uint64_t readp, writep;
> +
> +    do {
> +        spin_lock(&hw_its->cmd_lock);
> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
> +        spin_unlock(&hw_its->cmd_lock);
> +
> +        if ( readp == writep )
> +            return 0;
> +
> +        cpu_relax();
> +        udelay(1);
> +    } while ( NOW() <= deadline );
> +
> +    return -ETIMEDOUT;
> +}
> +
> +static uint64_t encode_rdbase(struct host_its *hw_its, unsigned int cpu,
> +                              uint64_t reg)
> +{
> +    reg &= ~GENMASK(51, 16);
> +
> +    reg |= gicv3_get_redist_address(cpu, hw_its->flags & HOST_ITS_USES_PTA);
> +
> +    return reg;
> +}
> +
> +static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_SYNC;
> +    cmd[1] = 0x00;
> +    cmd[2] = encode_rdbase(its, cpu, 0x0);
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
> +static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
> +                             unsigned int cpu)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_MAPC;
> +    cmd[1] = 0x00;
> +    cmd[2] = encode_rdbase(its, cpu, (collection_id & GENMASK(15, 0)));
> +    cmd[2] |= GITS_VALID_BIT;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
> +/* Set up the (1:1) collection mapping for the given host CPU. */
> +int gicv3_its_setup_collection(unsigned int cpu)
> +{
> +    struct host_its *its;
> +    int ret;
> +
> +    list_for_each_entry(its, &host_its_list, entry)
> +    {
> +        if ( !its->cmd_buf )
> +            continue;
> +
> +        ret = its_send_cmd_mapc(its, cpu, cpu);
> +        if ( ret )
> +            return ret;
> +
> +        ret = its_send_cmd_sync(its, cpu);
> +        if ( ret )
> +            return ret;
> +
> +        ret = gicv3_its_wait_commands(its);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    return 0;
> +}
> +
>  #define BASER_ATTR_MASK                                           \
>          ((0x3UL << GITS_BASER_SHAREABILITY_SHIFT)               | \
>           (0x7UL << GITS_BASER_OUTER_CACHEABILITY_SHIFT)         | \
> @@ -184,22 +326,59 @@ retry:
>      return -EINVAL;
>  }
>  
> +/*
> + * Before an ITS gets initialized, it should be in a quiescent state, where
> + * all outstanding commands and transactions have finished.
> + * So if the ITS is already enabled, turn it off and wait for all outstanding
> + * operations to get processed by polling the QUIESCENT bit.
> + */
> +static int gicv3_disable_its(struct host_its *hw_its)
> +{
> +    uint32_t reg;
> +    s_time_t deadline = NOW() + MILLISECS(100);
> +
> +    reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
> +    if ( (reg & GITS_CTLR_QUIESCENT) && !(reg & GITS_CTLR_ENABLE) )
> +        return 0;
> +
> +    writel_relaxed(reg & ~GITS_CTLR_ENABLE, hw_its->its_base + GITS_CTLR);
> +
> +    do {
> +        reg = readl_relaxed(hw_its->its_base + GITS_CTLR);
> +        if ( reg & GITS_CTLR_QUIESCENT )
> +            return 0;
> +
> +        cpu_relax();
> +        udelay(1);
> +    } while ( NOW() <= deadline );
> +
> +    dprintk(XENLOG_ERR, "ITS not quiescent.\n");
> +
> +    return -ETIMEDOUT;
> +}
> +
>  static unsigned int max_its_device_bits = CONFIG_MAX_PHYS_ITS_DEVICE_BITS;
>  integer_param("max_its_device_bits", max_its_device_bits);
>  
>  static int gicv3_its_init_single_its(struct host_its *hw_its)
>  {
>      uint64_t reg;
> -    int i;
> +    int i ,ret;
>      unsigned int devid_bits;
>  
>      hw_its->its_base = ioremap_nocache(hw_its->addr, hw_its->size);
>      if ( !hw_its->its_base )
>          return -ENOMEM;
>  
> +    ret = gicv3_disable_its(hw_its);
> +    if ( ret )
> +        return ret;
> +
>      reg = readq_relaxed(hw_its->its_base + GITS_TYPER);
>      devid_bits = GITS_TYPER_DEVICE_ID_BITS(reg);
>      devid_bits = min(devid_bits, max_its_device_bits);
> +    if ( reg & GITS_TYPER_PTA )
> +        hw_its->flags |= HOST_ITS_USES_PTA;
>  
>      for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>      {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 4f8414b..51d7425 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -42,6 +42,8 @@ static struct {
>  } lpi_data;
>  
>  struct lpi_redist_data {
> +    paddr_t             redist_addr;
> +    unsigned int        redist_id;
>      void                *pending_table;
>  };
>  
> @@ -49,6 +51,26 @@ static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>  
>  #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
>  
> +/* Stores this redistributor's physical address and ID in a per-CPU variable */
> +void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
> +{
> +    this_cpu(lpi_redist).redist_addr = address;
> +    this_cpu(lpi_redist).redist_id = redist_id;
> +}
> +
> +/*
> + * Returns a redistributor's ID (either as an address or as an ID).
> + * This must be (and is) called only after it has been setup by the above
> + * function.
> + */
> +uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta)
> +{
> +    if ( use_pta )
> +        return per_cpu(lpi_redist, cpu).redist_addr & GENMASK(51, 16);
> +    else
> +        return per_cpu(lpi_redist, cpu).redist_id << 16;
> +}
> +
>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>  {
>      uint64_t val;
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index cc1e219..38dafe7 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -666,7 +666,21 @@ static int __init gicv3_populate_rdist(void)
>  
>                  if ( typer & GICR_TYPER_PLPIS )
>                  {
> -                    int ret;
> +                    paddr_t rdist_addr;
> +                    int procnum, ret;
> +
> +                    /*
> +                     * The ITS refers to redistributors either by their physical
> +                     * address or by their ID. Determine those two values and
> +                     * let the ITS code store them in per host CPU variables to
> +                     * later be able to address those redistributors.
> +                     */
> +                    rdist_addr = gicv3.rdist_regions[i].base;
> +                    rdist_addr += ptr - gicv3.rdist_regions[i].map_base;
> +                    procnum = (typer & GICR_TYPER_PROC_NUM_MASK);
> +                    procnum >>= GICR_TYPER_PROC_NUM_SHIFT;
> +
> +                    gicv3_set_redist_address(rdist_addr, procnum);
>  
>                      ret = gicv3_lpi_init_rdist(ptr);
>                      if ( ret && ret != -ENODEV )
> @@ -715,6 +729,9 @@ static int gicv3_cpu_init(void)
>      if ( gicv3_enable_redist() )
>          return -ENODEV;
>  
> +    if ( gicv3_its_host_has_its() )
> +        gicv3_its_setup_collection(smp_processor_id());

Check for return errors?


>      /* Set priority on PPI and SGI interrupts */
>      priority = (GIC_PRI_IPI << 24 | GIC_PRI_IPI << 16 | GIC_PRI_IPI << 8 |
>                  GIC_PRI_IPI);
> diff --git a/xen/include/asm-arm/gic_v3_defs.h b/xen/include/asm-arm/gic_v3_defs.h
> index b307322..878bae2 100644
> --- a/xen/include/asm-arm/gic_v3_defs.h
> +++ b/xen/include/asm-arm/gic_v3_defs.h
> @@ -101,6 +101,8 @@
>  #define GICR_TYPER_PLPIS             (1U << 0)
>  #define GICR_TYPER_VLPIS             (1U << 1)
>  #define GICR_TYPER_LAST              (1U << 4)
> +#define GICR_TYPER_PROC_NUM_SHIFT    8
> +#define GICR_TYPER_PROC_NUM_MASK     (0xffff << GICR_TYPER_PROC_NUM_SHIFT)
>  
>  /* For specifying the inner cacheability type only */
>  #define GIC_BASER_CACHE_nCnB         0ULL
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index d5facf0..8b493fb 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -42,10 +42,12 @@
>  #define GITS_CTLR_QUIESCENT             BIT(31)
>  #define GITS_CTLR_ENABLE                BIT(0)
>  
> +#define GITS_TYPER_PTA                  BIT_ULL(19)
>  #define GITS_TYPER_DEVIDS_SHIFT         13
>  #define GITS_TYPER_DEVIDS_MASK          (0x1fUL << GITS_TYPER_DEVIDS_SHIFT)
>  #define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
>                                                 GITS_TYPER_DEVIDS_SHIFT) + 1)
> +#define GITS_TYPER_IDBITS_SHIFT         8
>  
>  #define GITS_IIDR_VALUE                 0x34c
>  
> @@ -76,9 +78,26 @@
>  
>  #define GITS_CBASER_SIZE_MASK           0xff
>  
> +/* ITS command definitions */
> +#define ITS_CMD_SIZE                    32
> +
> +#define GITS_CMD_MOVI                   0x01
> +#define GITS_CMD_INT                    0x03
> +#define GITS_CMD_CLEAR                  0x04
> +#define GITS_CMD_SYNC                   0x05
> +#define GITS_CMD_MAPD                   0x08
> +#define GITS_CMD_MAPC                   0x09
> +#define GITS_CMD_MAPTI                  0x0a
> +#define GITS_CMD_MAPI                   0x0b
> +#define GITS_CMD_INV                    0x0c
> +#define GITS_CMD_INVALL                 0x0d
> +#define GITS_CMD_MOVALL                 0x0e
> +#define GITS_CMD_DISCARD                0x0f
> +
>  #include <xen/device_tree.h>
>  
>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
> +#define HOST_ITS_USES_PTA               (1U << 1)
>  
>  /* data structure for each hardware ITS */
>  struct host_its {
> @@ -87,6 +106,7 @@ struct host_its {
>      paddr_t addr;
>      paddr_t size;
>      void __iomem *its_base;
> +    spinlock_t cmd_lock;
>      void *cmd_buf;
>      unsigned int flags;
>  };
> @@ -107,6 +127,13 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base);
>  int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
>  int gicv3_its_init(void);
>  
> +/* Store the physical address and ID for each redistributor as read from DT. */
> +void gicv3_set_redist_address(paddr_t address, unsigned int redist_id);
> +uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
> +
> +/* Map a collection for this host CPU to each host ITS. */
> +int gicv3_its_setup_collection(unsigned int cpu);
> +
>  #else
>  
>  static LIST_HEAD(host_its_list);
> @@ -134,6 +161,17 @@ static inline int gicv3_its_init(void)
>  {
>      return 0;
>  }
> +
> +static inline void gicv3_set_redist_address(paddr_t address,
> +                                            unsigned int redist_id)
> +{
> +}
> +
> +static inline int gicv3_its_setup_collection(unsigned int cpu)
> +{
> +    return 0;
> +}
> +
>  #endif /* CONFIG_HAS_ITS */
>  
>  #endif
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table
  2017-03-16 11:20 ` [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table Andre Przywara
  2017-03-21 23:29   ` Stefano Stabellini
@ 2017-03-22 13:52   ` Julien Grall
  2017-03-22 16:08     ` André Przywara
  1 sibling, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-22 13:52 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> Each ITS maps a pair of a DeviceID (for instance derived from a PCI
> b/d/f triplet) and an EventID (the MSI payload or interrupt ID) to a
> pair of LPI number and collection ID, which points to the target CPU.
> This mapping is stored in the device and collection tables, which software
> has to provide for the ITS to use.
> Allocate the required memory and hand it to the ITS.
> The maximum number of devices is limited to a compile-time constant
> exposed in Kconfig.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  docs/misc/xen-command-line.markdown |   8 ++
>  xen/arch/arm/Kconfig                |  14 ++++
>  xen/arch/arm/gic-v3-its.c           | 163 ++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/gic-v3.c               |   3 +
>  xen/include/asm-arm/gic_v3_its.h    |  63 +++++++++++++-
>  5 files changed, 250 insertions(+), 1 deletion(-)
>
> diff --git a/docs/misc/xen-command-line.markdown b/docs/misc/xen-command-line.markdown
> index 619016d..068d116 100644
> --- a/docs/misc/xen-command-line.markdown
> +++ b/docs/misc/xen-command-line.markdown
> @@ -1158,6 +1158,14 @@ based interrupts. Any higher IRQs will be available for use via PCI MSI.
>  ### maxcpus
>  > `= <integer>`
>
> +### max\_its\_device\_bits
> +> `= <integer>`
> +
> +Specifies the maximum number of devices using MSIs on the ARM GICv3 ITS
> +controller to allocate table entries for. Each table entry uses a hardware
> +specific size, typically 8 or 16 bytes.
> +Defaults to 10 bits (to cover at most 1024 devices).

The description is misleading. Even if the platform has less than 1024
devices, 10 may not be enough if the Device ID are not contiguous.

However, I don't think this is useful. A user may not know the DevIDs in
use of the platform and hence will not be able to choose a sensible value.

I still think that we should allocate what the hardware asked us and if
it is necessary to limit this should be done per-platform.

> +
>  ### max\_lpi\_bits
>  > `= <integer>`
>
> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> index 86f7b53..0d50156 100644
> --- a/xen/arch/arm/Kconfig
> +++ b/xen/arch/arm/Kconfig
> @@ -64,6 +64,20 @@ config MAX_PHYS_LPI_BITS
>            This can be overridden on the command line with the max_lpi_bits
>            parameter.
>
> +config MAX_PHYS_ITS_DEVICE_BITS
> +        depends on HAS_ITS
> +        int "Number of device bits the ITS supports"
> +        range 1 32
> +        default "10"
> +        help
> +          Specifies the maximum number of devices which want to use the ITS.
> +          Xen needs to allocates memory for the whole range very early.
> +          The allocation scheme may be sparse, so a much larger number must
> +          be supported to cover devices with a high bus number or those on
> +          separate bus segments.
> +          This can be overridden on the command line with the
> +          max_its_device_bits parameter.

 From what I understand the default you suggested fit neither Cavium nor
Qualcomm platform. It is also hard to see how a Distribution will be
able to choose a sensible value here.

> +
>  endmenu
>
>  menu "ARM errata workaround via the alternative framework"
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 4056e5b..9982fe9 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c

[...]

> +static int its_map_baser(void __iomem *basereg, uint64_t regc,
> +                         unsigned int nr_items)
> +{
> +    uint64_t attr, reg;
> +    unsigned int entry_size = GITS_BASER_ENTRY_SIZE(regc);
> +    unsigned int pagesz = 2, order, table_size;

Please document what mean 2.

> +    void *buffer;
> +
> +    attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
> +    attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
> +    attr |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
> +
> +    /*
> +     * Setup the BASE register with the attributes that we like. Then read
> +     * it back and see what sticks (page size, cacheability and shareability
> +     * attributes), retrying if necessary.
> +     */
> +retry:
> +    table_size = ROUNDUP(nr_items * entry_size, BIT(BASER_PAGE_BITS(pagesz)));
> +    /* The BASE registers support at most 256 pages. */
> +    table_size = min(table_size, 256U << BASER_PAGE_BITS(pagesz));
> +    /* The memory block must be aligned to the requested page size. */
> +    order = max(get_order_from_bytes(table_size), pagesz * 2);
> +
> +    buffer = alloc_xenheap_pages(order, 0);

Why don't you use _zalloc(...)? This would avoid to compute the order
and drop few lines.

> +    if ( !buffer )
> +        return -ENOMEM;
> +
> +    if ( !check_propbaser_phys_addr(buffer, BASER_PAGE_BITS(pagesz)) )

The name of the function does not make sense. You check an address for
the register BASER and not PROPBASER.

> +    {
> +        free_xenheap_pages(buffer, 0);
> +        return -ERANGE;
> +    }
> +    memset(buffer, 0, table_size);

[...]

> +int gicv3_its_init(void)
> +{
> +    struct host_its *hw_its;
> +    int ret;
> +
> +    list_for_each_entry(hw_its, &host_its_list, entry) {

Coding style:

list_for_each_entry(....)
{

> +        ret = gicv3_its_init_single_its(hw_its);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    return 0;
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index ed78363..cc1e219 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -1590,6 +1590,9 @@ static int __init gicv3_init(void)
>      spin_lock(&gicv3.lock);
>
>      gicv3_dist_init();
> +    res = gicv3_its_init();
> +    if ( res )
> +        printk(XENLOG_WARNING "GICv3: ITS: initialization failed: %d\n", res);

I would have expect a panic here because the ITS subsystem could be half
initialized and it is not safe to continue.

>      res = gicv3_cpu_init();
>      gicv3_hyp_init();
>

Cheers,

--
Julien Grall
IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer
  2017-03-16 11:20 ` [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer Andre Przywara
  2017-03-21 23:48   ` Stefano Stabellini
@ 2017-03-22 15:23   ` Julien Grall
  2017-03-22 16:31     ` André Przywara
  1 sibling, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-22 15:23 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 16/03/17 11:20, Andre Przywara wrote:
> Instead of directly manipulating the tables in memory, an ITS driver
> sends commands via a ring buffer in normal system memory to the ITS h/w
> to create or alter the LPI mappings.
> Allocate memory for that buffer and tell the ITS about it to be able
> to send ITS commands.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        | 57 ++++++++++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/gic_v3_its.h |  6 +++++
>  2 files changed, 63 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 9982fe9..e5601ed 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -20,10 +20,13 @@
>
>  #include <xen/lib.h>
>  #include <xen/mm.h>
> +#include <xen/sizes.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
>
> +#define ITS_CMD_QUEUE_SZ                SZ_64K

I thought you were planning to increase the size to 1MB?

> +
>  LIST_HEAD(host_its_list);
>
>  bool gicv3_its_host_has_its(void)
> @@ -56,6 +59,55 @@ static uint64_t encode_propbaser_phys_addr(paddr_t addr, unsigned int page_bits)
>      return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
>  }
>
> +static void *its_map_cbaser(struct host_its *its)
> +{
> +    void __iomem *cbasereg = its->its_base + GITS_CBASER;
> +    uint64_t reg;
> +    void *buffer;
> +    unsigned int order;
> +
> +    reg  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
> +    reg |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
> +    reg |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
> +
> +    /* The ITS command buffer needs to be 64K aligned. */

Looking at the spec, the command buffer does not need to be 64K aligned. 
On the previous version, you made it 4K aligned. So why this restriction?

> +    order = max(get_order_from_bytes(ITS_CMD_QUEUE_SZ), 16U - PAGE_SHIFT);
> +    buffer = alloc_xenheap_pages(order, 0);

I am not sure to understand why you move from _zalloc to 
alloc_xenheap_*. The resulting behavior will be exactly the same but the 
former result to a simpler code.

> +    if ( !buffer )
> +        return NULL;
> +
> +    if ( virt_to_maddr(buffer) & ~GENMASK(51, 12) )
> +    {
> +        free_xenheap_pages(buffer, 0);
> +        return NULL;
> +    }
> +    memset(buffer, 0, ITS_CMD_QUEUE_SZ);
> +
> +    reg |= GITS_VALID_BIT | virt_to_maddr(buffer);
> +    reg |= ((ITS_CMD_QUEUE_SZ / SZ_4K) - 1) & GITS_CBASER_SIZE_MASK;
> +    writeq_relaxed(reg, cbasereg);
> +    reg = readq_relaxed(cbasereg);
> +
> +    /* If the ITS dropped shareability, drop cacheability as well. */
> +    if ( (reg & GITS_BASER_SHAREABILITY_MASK) == 0 )
> +    {
> +        reg &= ~GITS_BASER_INNER_CACHEABILITY_MASK;
> +        writeq_relaxed(reg, cbasereg);
> +    }
> +
> +    /*
> +     * If the command queue memory is mapped as uncached, we need to flush
> +     * it on every access.
> +     */
> +    if ( !(reg & GITS_BASER_INNER_CACHEABILITY_MASK) )
> +    {
> +        its->flags |= HOST_ITS_FLUSH_CMD_QUEUE;
> +        dprintk(XENLOG_WARNING, "using non-cacheable ITS command queue\n");

Please use printk, the message is useful even for non-debug build.

> +    }
> +
> +    return buffer;
> +}
> +

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-03-16 11:20 ` [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling Andre Przywara
  2017-03-16 15:05   ` Shanker Donthineni
  2017-03-22  0:02   ` Stefano Stabellini
@ 2017-03-22 15:59   ` Julien Grall
  2017-04-03 10:58     ` Andre Przywara
  2 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-22 15:59 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 16/03/17 11:20, Andre Przywara wrote:
> To be able to easily send commands to the ITS, create the respective
> wrapper functions, which take care of the ring buffer.
> The first two commands we implement provide methods to map a collection
> to a redistributor (aka host core) and to flush the command queue (SYNC).
> Start using these commands for mapping one collection to each host CPU.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c         | 181 +++++++++++++++++++++++++++++++++++++-
>  xen/arch/arm/gic-v3-lpi.c         |  22 +++++
>  xen/arch/arm/gic-v3.c             |  19 +++-
>  xen/include/asm-arm/gic_v3_defs.h |   2 +
>  xen/include/asm-arm/gic_v3_its.h  |  38 ++++++++
>  5 files changed, 260 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index e5601ed..5c11b0d 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -19,11 +19,14 @@
>   */
>
>  #include <xen/lib.h>
> +#include <xen/delay.h>
>  #include <xen/mm.h>
>  #include <xen/sizes.h>
> +#include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
> +#include <asm/page.h>
>
>  #define ITS_CMD_QUEUE_SZ                SZ_64K
>
> @@ -34,6 +37,145 @@ bool gicv3_its_host_has_its(void)
>      return !list_empty(&host_its_list);
>  }
>
> +#define BUFPTR_MASK                     GENMASK(19, 5)
> +static int its_send_command(struct host_its *hw_its, const void *its_cmd)
> +{
> +    s_time_t deadline = NOW() + MILLISECS(1);

It would be nice to explain where does this value comes from.

> +    uint64_t readp, writep;
> +    int ret = -EBUSY;
> +
> +    /* No ITS commands from an interrupt handler (at the moment). */
> +    ASSERT(!in_irq());
> +
> +    spin_lock(&hw_its->cmd_lock);
> +
> +    do {
> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
> +
> +        if ( ((writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ) != readp )
> +        {
> +            ret = 0;
> +            break;
> +        }
> +
> +        /*
> +         * If the command queue is full, wait for a bit in the hope it drains
> +         * before giving up.
> +         */
> +        spin_unlock(&hw_its->cmd_lock);
> +        cpu_relax();
> +        udelay(1);
> +        spin_lock(&hw_its->cmd_lock);
> +    } while ( NOW() <= deadline );
> +
> +    if ( ret )
> +    {
> +        spin_unlock(&hw_its->cmd_lock);
> +        printk(XENLOG_WARNING "ITS: command queue full.\n");

This function could be called from a domain. So please ratelimit the 
message (see printk_ratelimit).

> +        return ret;
> +    }
> +
> +    memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE);
> +    if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
> +        clean_and_invalidate_dcache_va_range(hw_its->cmd_buf + writep,
> +                                             ITS_CMD_SIZE);
> +    else
> +        dsb(ishst);
> +
> +    writep = (writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ;
> +    writeq_relaxed(writep & BUFPTR_MASK, hw_its->its_base + GITS_CWRITER);
> +
> +    spin_unlock(&hw_its->cmd_lock);
> +
> +    return 0;
> +}
> +
> +/* Wait for an ITS to finish processing all commands. */
> +static int gicv3_its_wait_commands(struct host_its *hw_its)
> +{
> +    s_time_t deadline = NOW() + MILLISECS(100);

Same remark as above. Why 1ms and 100ms here?

> +    uint64_t readp, writep;
> +
> +    do {
> +        spin_lock(&hw_its->cmd_lock);
> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) & BUFPTR_MASK;
> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) & BUFPTR_MASK;
> +        spin_unlock(&hw_its->cmd_lock);
> +
> +        if ( readp == writep )
> +            return 0;
> +
> +        cpu_relax();
> +        udelay(1);
> +    } while ( NOW() <= deadline );
> +
> +    return -ETIMEDOUT;
> +}

[...]

> +static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
> +                             unsigned int cpu)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_MAPC;
> +    cmd[1] = 0x00;
> +    cmd[2] = encode_rdbase(its, cpu, (collection_id & GENMASK(15, 0)));

I requested to drop the mask here and I didn't see any answer explaining 
why you would not do it. So this should be dropped unless you give me a 
reason to keep it.

> +    cmd[2] |= GITS_VALID_BIT;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
> +/* Set up the (1:1) collection mapping for the given host CPU. */
> +int gicv3_its_setup_collection(unsigned int cpu)
> +{
> +    struct host_its *its;
> +    int ret;
> +
> +    list_for_each_entry(its, &host_its_list, entry)
> +    {
> +        if ( !its->cmd_buf )
> +            continue;

Why this check?

[...]

> +/*
> + * Before an ITS gets initialized, it should be in a quiescent state, where
> + * all outstanding commands and transactions have finished.
> + * So if the ITS is already enabled, turn it off and wait for all outstanding
> + * operations to get processed by polling the QUIESCENT bit.
> + */
> +static int gicv3_disable_its(struct host_its *hw_its)
> +{
> +    uint32_t reg;
> +    s_time_t deadline = NOW() + MILLISECS(100);

Why 100ms?

[...]

>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>  {
>      uint64_t val;
> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
> index cc1e219..38dafe7 100644
> --- a/xen/arch/arm/gic-v3.c
> +++ b/xen/arch/arm/gic-v3.c
> @@ -666,7 +666,21 @@ static int __init gicv3_populate_rdist(void)
>
>                  if ( typer & GICR_TYPER_PLPIS )
>                  {
> -                    int ret;
> +                    paddr_t rdist_addr;
> +                    int procnum, ret;

As mentioned in v1, procnum should be unsigned. If you disagree, please 
explain.

> +
> +                    /*
> +                     * The ITS refers to redistributors either by their physical
> +                     * address or by their ID. Determine those two values and
> +                     * let the ITS code store them in per host CPU variables to
> +                     * later be able to address those redistributors.
> +                     */

Again, this comment does not look useful and is misleading as the code 
to get/set the redistributor information is living in gic-v3-lpi.c and 
not gic-v3-its.c.

[...]

> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index d5facf0..8b493fb 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h

[...]

>  #include <xen/device_tree.h>
>
>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
> +#define HOST_ITS_USES_PTA               (1U << 1)
>
>  /* data structure for each hardware ITS */
>  struct host_its {
> @@ -87,6 +106,7 @@ struct host_its {
>      paddr_t addr;
>      paddr_t size;
>      void __iomem *its_base;
> +    spinlock_t cmd_lock;

Again, initialization, clean-up of a field should be done in the same 
that added the field. Otherwise, this is a call to miss a bit of the 
code and makes more difficult for the reviewer.

So please initialize cmd_lock in this patch and patch #6.

>      void *cmd_buf;
>      unsigned int flags;
>  };
> @@ -107,6 +127,13 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base);
>  int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
>  int gicv3_its_init(void);
>
> +/* Store the physical address and ID for each redistributor as read from DT. */
> +void gicv3_set_redist_address(paddr_t address, unsigned int redist_id);
> +uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
> +
> +/* Map a collection for this host CPU to each host ITS. */
> +int gicv3_its_setup_collection(unsigned int cpu);
> +
>  #else
>
>  static LIST_HEAD(host_its_list);
> @@ -134,6 +161,17 @@ static inline int gicv3_its_init(void)
>  {
>      return 0;
>  }
> +
> +static inline void gicv3_set_redist_address(paddr_t address,
> +                                            unsigned int redist_id)
> +{
> +}
> +
> +static inline int gicv3_its_setup_collection(unsigned int cpu)
> +{

This function should never be called as it is gated by the presence of 
ITS. I would add a BUG() with a comment to ensure this is the case.

> +    return 0;
> +}
> +
>  #endif /* CONFIG_HAS_ITS */
>
>  #endif
>

Cheers

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table
  2017-03-22 13:52   ` Julien Grall
@ 2017-03-22 16:08     ` André Przywara
  2017-03-22 16:33       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: André Przywara @ 2017-03-22 16:08 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 22/03/17 13:52, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> Each ITS maps a pair of a DeviceID (for instance derived from a PCI
>> b/d/f triplet) and an EventID (the MSI payload or interrupt ID) to a
>> pair of LPI number and collection ID, which points to the target CPU.
>> This mapping is stored in the device and collection tables, which
>> software
>> has to provide for the ITS to use.
>> Allocate the required memory and hand it to the ITS.
>> The maximum number of devices is limited to a compile-time constant
>> exposed in Kconfig.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  docs/misc/xen-command-line.markdown |   8 ++
>>  xen/arch/arm/Kconfig                |  14 ++++
>>  xen/arch/arm/gic-v3-its.c           | 163
>> ++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/gic-v3.c               |   3 +
>>  xen/include/asm-arm/gic_v3_its.h    |  63 +++++++++++++-
>>  5 files changed, 250 insertions(+), 1 deletion(-)
>>
>> diff --git a/docs/misc/xen-command-line.markdown
>> b/docs/misc/xen-command-line.markdown
>> index 619016d..068d116 100644
>> --- a/docs/misc/xen-command-line.markdown
>> +++ b/docs/misc/xen-command-line.markdown
>> @@ -1158,6 +1158,14 @@ based interrupts. Any higher IRQs will be
>> available for use via PCI MSI.
>>  ### maxcpus
>>  > `= <integer>`
>>
>> +### max\_its\_device\_bits
>> +> `= <integer>`
>> +
>> +Specifies the maximum number of devices using MSIs on the ARM GICv3 ITS
>> +controller to allocate table entries for. Each table entry uses a
>> hardware
>> +specific size, typically 8 or 16 bytes.
>> +Defaults to 10 bits (to cover at most 1024 devices).
> 
> The description is misleading. Even if the platform has less than 1024
> devices, 10 may not be enough if the Device ID are not contiguous.

Right.

> However, I don't think this is useful. A user may not know the DevIDs in
> use of the platform and hence will not be able to choose a sensible value.
> 
> I still think that we should allocate what the hardware asked us and if
> it is necessary to limit this should be done per-platform.

I think you are right, a configurable limit does not make much sense. I
was wondering if we should have a hard coded *memory* limit instead, to
prevent ridiculously high allocations, say entry_size=8 and 32 bits of
device IDs, which would result in 32GB of memory for a flat device table.

Or if we don't want to arbitrarily limit this, we always print the
actual allocated size, maybe with a extra WARNING if it exceeds sane levels.

>> +
>>  ### max\_lpi\_bits
>>  > `= <integer>`
>>
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 86f7b53..0d50156 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -64,6 +64,20 @@ config MAX_PHYS_LPI_BITS
>>            This can be overridden on the command line with the
>> max_lpi_bits
>>            parameter.
>>
>> +config MAX_PHYS_ITS_DEVICE_BITS
>> +        depends on HAS_ITS
>> +        int "Number of device bits the ITS supports"
>> +        range 1 32
>> +        default "10"
>> +        help
>> +          Specifies the maximum number of devices which want to use
>> the ITS.
>> +          Xen needs to allocates memory for the whole range very early.
>> +          The allocation scheme may be sparse, so a much larger
>> number must
>> +          be supported to cover devices with a high bus number or
>> those on
>> +          separate bus segments.
>> +          This can be overridden on the command line with the
>> +          max_its_device_bits parameter.
> 
> From what I understand the default you suggested fit neither Cavium nor
> Qualcomm platform. It is also hard to see how a Distribution will be
> able to choose a sensible value here.

But it works for me (TM) ;-)
I was hoping that people would scream and suggest usable values instead
(because I don't know their limits). But indeed lets just remove that in
favor of something more future proof.

>> +
>>  endmenu
>>
>>  menu "ARM errata workaround via the alternative framework"
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 4056e5b..9982fe9 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
> 
> [...]
> 
>> +static int its_map_baser(void __iomem *basereg, uint64_t regc,
>> +                         unsigned int nr_items)
>> +{
>> +    uint64_t attr, reg;
>> +    unsigned int entry_size = GITS_BASER_ENTRY_SIZE(regc);
>> +    unsigned int pagesz = 2, order, table_size;
> 
> Please document what mean 2.
> 
>> +    void *buffer;
>> +
>> +    attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
>> +    attr |= GIC_BASER_CACHE_SameAsInner <<
>> GITS_BASER_OUTER_CACHEABILITY_SHIFT;
>> +    attr |= GIC_BASER_CACHE_RaWaWb <<
>> GITS_BASER_INNER_CACHEABILITY_SHIFT;
>> +
>> +    /*
>> +     * Setup the BASE register with the attributes that we like. Then
>> read
>> +     * it back and see what sticks (page size, cacheability and
>> shareability
>> +     * attributes), retrying if necessary.
>> +     */
>> +retry:
>> +    table_size = ROUNDUP(nr_items * entry_size,
>> BIT(BASER_PAGE_BITS(pagesz)));
>> +    /* The BASE registers support at most 256 pages. */
>> +    table_size = min(table_size, 256U << BASER_PAGE_BITS(pagesz));
>> +    /* The memory block must be aligned to the requested page size. */
>> +    order = max(get_order_from_bytes(table_size), pagesz * 2);
>> +
>> +    buffer = alloc_xenheap_pages(order, 0);
> 
> Why don't you use _zalloc(...)? This would avoid to compute the order
> and drop few lines.

Because they need to be physically contiguous.
Please correct me if I am wrong here, but the normal *alloc functions do
not guarantee this (from checking the implementation).

>> +    if ( !buffer )
>> +        return -ENOMEM;
>> +
>> +    if ( !check_propbaser_phys_addr(buffer, BASER_PAGE_BITS(pagesz)) )
> 
> The name of the function does not make sense. You check an address for
> the register BASER and not PROPBASER.

Indeed.

>> +    {
>> +        free_xenheap_pages(buffer, 0);
>> +        return -ERANGE;
>> +    }
>> +    memset(buffer, 0, table_size);
> 
> [...]
> 
>> +int gicv3_its_init(void)
>> +{
>> +    struct host_its *hw_its;
>> +    int ret;
>> +
>> +    list_for_each_entry(hw_its, &host_its_list, entry) {
> 
> Coding style:
> 
> list_for_each_entry(....)
> {

Yeah, I was looking at the Linux driver at the same time ;-)

>> +        ret = gicv3_its_init_single_its(hw_its);
>> +        if ( ret )
>> +            return ret;
>> +    }
>> +
>> +    return 0;
>> +}
>> +
>>  /* Scan the DT for any ITS nodes and create a list of host ITSes out
>> of it. */
>>  void gicv3_its_dt_init(const struct dt_device_node *node)
>>  {
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index ed78363..cc1e219 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -1590,6 +1590,9 @@ static int __init gicv3_init(void)
>>      spin_lock(&gicv3.lock);
>>
>>      gicv3_dist_init();
>> +    res = gicv3_its_init();
>> +    if ( res )
>> +        printk(XENLOG_WARNING "GICv3: ITS: initialization failed:
>> %d\n", res);
> 
> I would have expect a panic here because the ITS subsystem could be half
> initialized and it is not safe to continue.

OK, let me check what actually happens here if there is no ITS ;-)

Cheers,
Andre

> 
>>      res = gicv3_cpu_init();
>>      gicv3_hyp_init();
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer
  2017-03-22 15:23   ` Julien Grall
@ 2017-03-22 16:31     ` André Przywara
  2017-03-22 16:41       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: André Przywara @ 2017-03-22 16:31 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

On 22/03/17 15:23, Julien Grall wrote:
> Hi Andre,
> 
> On 16/03/17 11:20, Andre Przywara wrote:
>> Instead of directly manipulating the tables in memory, an ITS driver
>> sends commands via a ring buffer in normal system memory to the ITS h/w
>> to create or alter the LPI mappings.
>> Allocate memory for that buffer and tell the ITS about it to be able
>> to send ITS commands.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c        | 57
>> ++++++++++++++++++++++++++++++++++++++++
>>  xen/include/asm-arm/gic_v3_its.h |  6 +++++
>>  2 files changed, 63 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 9982fe9..e5601ed 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -20,10 +20,13 @@
>>
>>  #include <xen/lib.h>
>>  #include <xen/mm.h>
>> +#include <xen/sizes.h>
>>  #include <asm/gic_v3_defs.h>
>>  #include <asm/gic_v3_its.h>
>>  #include <asm/io.h>
>>
>> +#define ITS_CMD_QUEUE_SZ                SZ_64K
> 
> I thought you were planning to increase the size to 1MB?

Good, you noticed ;-) Indeed forgot to change it ...

>> +
>>  LIST_HEAD(host_its_list);
>>
>>  bool gicv3_its_host_has_its(void)
>> @@ -56,6 +59,55 @@ static uint64_t encode_propbaser_phys_addr(paddr_t
>> addr, unsigned int page_bits)
>>      return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
>>  }
>>
>> +static void *its_map_cbaser(struct host_its *its)
>> +{
>> +    void __iomem *cbasereg = its->its_base + GITS_CBASER;
>> +    uint64_t reg;
>> +    void *buffer;
>> +    unsigned int order;
>> +
>> +    reg  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
>> +    reg |= GIC_BASER_CACHE_SameAsInner <<
>> GITS_BASER_OUTER_CACHEABILITY_SHIFT;
>> +    reg |= GIC_BASER_CACHE_RaWaWb <<
>> GITS_BASER_INNER_CACHEABILITY_SHIFT;
>> +
>> +    /* The ITS command buffer needs to be 64K aligned. */
> 
> Looking at the spec, the command buffer does not need to be 64K aligned.
> On the previous version, you made it 4K aligned. So why this restriction?

As you have already learnt, the GIC is more subtle sometimes ;-) Read
the description at "Physical_Address, bits [51:12]" in the CBASER
paragraph and tell me what you take from it. I decided to read it as
"has to be 64K aligned".
Happy to correct this otherwise.

>> +    order = max(get_order_from_bytes(ITS_CMD_QUEUE_SZ), 16U -
>> PAGE_SHIFT);
>> +    buffer = alloc_xenheap_pages(order, 0);
> 
> I am not sure to understand why you move from _zalloc to
> alloc_xenheap_*. The resulting behavior will be exactly the same but the
> former result to a simpler code.
                                                  ^^^^^^^^^^^^^^^^
Really?
I checked and found that alloc_xenheap_pages gives me physically
contiguous allocation in the given order, where *alloc just guarantees
virtually contiguous pages.

> 
>> +    if ( !buffer )
>> +        return NULL;
>> +
>> +    if ( virt_to_maddr(buffer) & ~GENMASK(51, 12) )
>> +    {
>> +        free_xenheap_pages(buffer, 0);
>> +        return NULL;
>> +    }
>> +    memset(buffer, 0, ITS_CMD_QUEUE_SZ);
>> +
>> +    reg |= GITS_VALID_BIT | virt_to_maddr(buffer);
>> +    reg |= ((ITS_CMD_QUEUE_SZ / SZ_4K) - 1) & GITS_CBASER_SIZE_MASK;
>> +    writeq_relaxed(reg, cbasereg);
>> +    reg = readq_relaxed(cbasereg);
>> +
>> +    /* If the ITS dropped shareability, drop cacheability as well. */
>> +    if ( (reg & GITS_BASER_SHAREABILITY_MASK) == 0 )
>> +    {
>> +        reg &= ~GITS_BASER_INNER_CACHEABILITY_MASK;
>> +        writeq_relaxed(reg, cbasereg);
>> +    }
>> +
>> +    /*
>> +     * If the command queue memory is mapped as uncached, we need to
>> flush
>> +     * it on every access.
>> +     */
>> +    if ( !(reg & GITS_BASER_INNER_CACHEABILITY_MASK) )
>> +    {
>> +        its->flags |= HOST_ITS_FLUSH_CMD_QUEUE;
>> +        dprintk(XENLOG_WARNING, "using non-cacheable ITS command
>> queue\n");
> 
> Please use printk, the message is useful even for non-debug build.

OK.

Cheers,
Andre.

> 
>> +    }
>> +
>> +    return buffer;
>> +}
>> +
> 
> Cheers,
> 


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table
  2017-03-22 16:08     ` André Przywara
@ 2017-03-22 16:33       ` Julien Grall
  2017-03-29 13:58         ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-22 16:33 UTC (permalink / raw)
  To: André Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari



On 22/03/17 16:08, André Przywara wrote:
> Hi,

Hi Andre,

>
> On 22/03/17 13:52, Julien Grall wrote:
>> Hi Andre,
>>
>> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>>> Each ITS maps a pair of a DeviceID (for instance derived from a PCI
>>> b/d/f triplet) and an EventID (the MSI payload or interrupt ID) to a
>>> pair of LPI number and collection ID, which points to the target CPU.
>>> This mapping is stored in the device and collection tables, which
>>> software
>>> has to provide for the ITS to use.
>>> Allocate the required memory and hand it to the ITS.
>>> The maximum number of devices is limited to a compile-time constant
>>> exposed in Kconfig.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>> ---
>>>  docs/misc/xen-command-line.markdown |   8 ++
>>>  xen/arch/arm/Kconfig                |  14 ++++
>>>  xen/arch/arm/gic-v3-its.c           | 163
>>> ++++++++++++++++++++++++++++++++++++
>>>  xen/arch/arm/gic-v3.c               |   3 +
>>>  xen/include/asm-arm/gic_v3_its.h    |  63 +++++++++++++-
>>>  5 files changed, 250 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/docs/misc/xen-command-line.markdown
>>> b/docs/misc/xen-command-line.markdown
>>> index 619016d..068d116 100644
>>> --- a/docs/misc/xen-command-line.markdown
>>> +++ b/docs/misc/xen-command-line.markdown
>>> @@ -1158,6 +1158,14 @@ based interrupts. Any higher IRQs will be
>>> available for use via PCI MSI.
>>>  ### maxcpus
>>>  > `= <integer>`
>>>
>>> +### max\_its\_device\_bits
>>> +> `= <integer>`
>>> +
>>> +Specifies the maximum number of devices using MSIs on the ARM GICv3 ITS
>>> +controller to allocate table entries for. Each table entry uses a
>>> hardware
>>> +specific size, typically 8 or 16 bytes.
>>> +Defaults to 10 bits (to cover at most 1024 devices).
>>
>> The description is misleading. Even if the platform has less than 1024
>> devices, 10 may not be enough if the Device ID are not contiguous.
>
> Right.
>
>> However, I don't think this is useful. A user may not know the DevIDs in
>> use of the platform and hence will not be able to choose a sensible value.
>>
>> I still think that we should allocate what the hardware asked us and if
>> it is necessary to limit this should be done per-platform.
>
> I think you are right, a configurable limit does not make much sense. I
> was wondering if we should have a hard coded *memory* limit instead, to
> prevent ridiculously high allocations, say entry_size=8 and 32 bits of
> device IDs, which would result in 32GB of memory for a flat device table.
>
> Or if we don't want to arbitrarily limit this, we always print the
> actual allocated size, maybe with a extra WARNING if it exceeds sane levels.

But what is a sane level? To be fair, I don't think this is the business 
of the common code to care about high allocations. If the platform ask 
32bits and not able to cope, then it should be a workaround for the 
platform.

[...]

>>> +    void *buffer;
>>> +
>>> +    attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
>>> +    attr |= GIC_BASER_CACHE_SameAsInner <<
>>> GITS_BASER_OUTER_CACHEABILITY_SHIFT;
>>> +    attr |= GIC_BASER_CACHE_RaWaWb <<
>>> GITS_BASER_INNER_CACHEABILITY_SHIFT;
>>> +
>>> +    /*
>>> +     * Setup the BASE register with the attributes that we like. Then
>>> read
>>> +     * it back and see what sticks (page size, cacheability and
>>> shareability
>>> +     * attributes), retrying if necessary.
>>> +     */
>>> +retry:
>>> +    table_size = ROUNDUP(nr_items * entry_size,
>>> BIT(BASER_PAGE_BITS(pagesz)));
>>> +    /* The BASE registers support at most 256 pages. */
>>> +    table_size = min(table_size, 256U << BASER_PAGE_BITS(pagesz));
>>> +    /* The memory block must be aligned to the requested page size. */
>>> +    order = max(get_order_from_bytes(table_size), pagesz * 2);
>>> +
>>> +    buffer = alloc_xenheap_pages(order, 0);
>>
>> Why don't you use _zalloc(...)? This would avoid to compute the order
>> and drop few lines.
>
> Because they need to be physically contiguous.
> Please correct me if I am wrong here, but the normal *alloc functions do
> not guarantee this (from checking the implementation).

_x*alloc will work the same way as k*alloc in Linux. The memory will be 
contiguous. Regarding the implementation details, _x*alloc is an overlay 
of alloc_xenheap_pages. For small size (i.e < PAGE_SIZE), it will 
coalesce in the same page. For bigger size, alloc_xenheap_pages will be 
used. Although, the resulting memory usage will differ because _x*alloc 
will free unused pages.

This is because alloc_xenheap_pages is working in term of order. So if 
you request 12K, alloc_xenheap_pages will allocate 16K. The 
implementation of _x*alloc will free the last 4K to avoid wasting space.

So effectively, _x*alloc will result in lower memory usage.

>>>      gicv3_dist_init();
>>> +    res = gicv3_its_init();
>>> +    if ( res )
>>> +        printk(XENLOG_WARNING "GICv3: ITS: initialization failed:
>>> %d\n", res);
>>
>> I would have expect a panic here because the ITS subsystem could be half
>> initialized and it is not safe to continue.
>
> OK, let me check what actually happens here if there is no ITS ;-)

Technically, this message should not happen when there is no ITS because 
it is not mandatory to have one on the platform.

So this would be an coding error for me.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer
  2017-03-22 16:31     ` André Przywara
@ 2017-03-22 16:41       ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-22 16:41 UTC (permalink / raw)
  To: André Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 22/03/17 16:31, André Przywara wrote:
> On 22/03/17 15:23, Julien Grall wrote:
>> On 16/03/17 11:20, Andre Przywara wrote:
>>> +
>>>  LIST_HEAD(host_its_list);
>>>
>>>  bool gicv3_its_host_has_its(void)
>>> @@ -56,6 +59,55 @@ static uint64_t encode_propbaser_phys_addr(paddr_t
>>> addr, unsigned int page_bits)
>>>      return ret | ((addr & GENMASK(51, 48)) >> (48 - 12));
>>>  }
>>>
>>> +static void *its_map_cbaser(struct host_its *its)
>>> +{
>>> +    void __iomem *cbasereg = its->its_base + GITS_CBASER;
>>> +    uint64_t reg;
>>> +    void *buffer;
>>> +    unsigned int order;
>>> +
>>> +    reg  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
>>> +    reg |= GIC_BASER_CACHE_SameAsInner <<
>>> GITS_BASER_OUTER_CACHEABILITY_SHIFT;
>>> +    reg |= GIC_BASER_CACHE_RaWaWb <<
>>> GITS_BASER_INNER_CACHEABILITY_SHIFT;
>>> +
>>> +    /* The ITS command buffer needs to be 64K aligned. */
>>
>> Looking at the spec, the command buffer does not need to be 64K aligned.
>> On the previous version, you made it 4K aligned. So why this restriction?
>
> As you have already learnt, the GIC is more subtle sometimes ;-) Read
> the description at "Physical_Address, bits [51:12]" in the CBASER
> paragraph and tell me what you take from it. I decided to read it as
> "has to be 64K aligned".
> Happy to correct this otherwise.

Hmmm. I think you are right. This is a little bit weird, but fine. Lets 
keep like that.

>
>>> +    order = max(get_order_from_bytes(ITS_CMD_QUEUE_SZ), 16U -
>>> PAGE_SHIFT);
>>> +    buffer = alloc_xenheap_pages(order, 0);
>>
>> I am not sure to understand why you move from _zalloc to
>> alloc_xenheap_*. The resulting behavior will be exactly the same but the
>> former result to a simpler code.
>                                                   ^^^^^^^^^^^^^^^^
> Really?
> I checked and found that alloc_xenheap_pages gives me physically
> contiguous allocation in the given order, where *alloc just guarantees
> virtually contiguous pages.

Yes. Give a look at my answer on patch #3.

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-03-16 11:20 ` [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping Andre Przywara
@ 2017-03-22 17:29   ` Julien Grall
  2017-04-03 20:08     ` Andre Przywara
  2017-03-22 22:45   ` Stefano Stabellini
  2017-03-30 11:17   ` Vijay Kilari
  2 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-22 17:29 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 16/03/17 11:20, Andre Przywara wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        | 207 +++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vgic-v3.c           |   3 +
>  xen/include/asm-arm/domain.h     |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  18 ++++
>  4 files changed, 231 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 5c11b0d..60b15b5 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include <xen/lib.h>
>  #include <xen/delay.h>
>  #include <xen/mm.h>
> +#include <xen/rbtree.h>
> +#include <xen/sched.h>
>  #include <xen/sizes.h>
>  #include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
> @@ -32,6 +34,17 @@
>
>  LIST_HEAD(host_its_list);
>
> +struct its_devices {
> +    struct rb_node rbnode;
> +    struct host_its *hw_its;
> +    void *itt_addr;
> +    paddr_t guest_doorbell;

I think it would be worth to explain in the commit message why you need 
the guest_doorbell in the struct its_devices and how you plan to use it.

> +    uint32_t host_devid;
> +    uint32_t guest_devid;
> +    uint32_t eventids;
> +    uint32_t *host_lpis;
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>      return !list_empty(&host_its_list);
> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>      return its_send_command(its, cmd);
>  }
>
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> +                             uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +    uint64_t cmd[4];
> +
> +    if ( valid )
> +    {
> +        ASSERT(size_bits < 32);

It would be better if you do the check against the real number in 
hardware (i.e GITS_TYPER.ID_bits).


> +        ASSERT(!(itt_addr & ~GENMASK(51, 8)));
> +    }
> +    cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +    cmd[1] = valid ? size_bits : 0x00;

This is really confusing. The check was not on the previous version. So 
why do you need that?

Also, it would have been better to hide the "size - 1" in the helper 
avoiding to really on the caller to do the right thing.

> +    cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;

Ditto about "valid? ...".

[...]

> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +    struct host_its *hw_its;
> +
> +    list_for_each_entry(hw_its, &host_its_list, entry)
> +    {
> +        if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )

Why not storing the ITS address rather than the doorbell to avoid this 
check?


[...]

> +int gicv3_its_map_guest_device(struct domain *d,
> +                               paddr_t host_doorbell, uint32_t host_devid,
> +                               paddr_t guest_doorbell, uint32_t guest_devid,
> +                               uint32_t nr_events, bool valid)
> +{
> +    void *itt_addr = NULL;
> +    struct host_its *hw_its;
> +    struct its_devices *dev = NULL, *temp;
> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> +    int ret = -ENOENT;
> +
> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> +    if ( !hw_its )
> +        return ret;
> +
> +    /* check for already existing mappings */
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    while ( *new )
> +    {
> +        temp = rb_entry(*new, struct its_devices, rbnode);
> +
> +        parent = *new;
> +        if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
> +        {
> +            if ( !valid )
> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
> +
> +            spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +            if ( valid )

Again, a printk(XENLOG_GUEST...) here would be useful to know which host 
DeviceID was associated to the guest DeviceID.

> +                return -EBUSY;
> +
> +            return remove_mapped_guest_device(temp);

Just above you removed the device from the RB-tree but this function may 
fail and never free the memory. This means that memory will be leaked 
leading to a potential denial of service.

> +        }
> +
> +        if ( compare_its_guest_devices(temp, guest_doorbell, guest_devid) > 0 )
> +            new = &((*new)->rb_left);
> +        else
> +            new = &((*new)->rb_right);
> +    }
> +
> +    if ( !valid )
> +        goto out_unlock;
> +
> +    ret = -ENOMEM;
> +
> +    /* An Interrupt Translation Table needs to be 256-byte aligned. */
> +    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
> +    if ( !itt_addr )
> +        goto out_unlock;
> +
> +    dev = xzalloc(struct its_devices);
> +    if ( !dev )
> +        goto out_unlock;
> +
> +    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,

I don't understand why nr_events - 1. Can you explain?

[...]

> +/* Removing any connections a domain had to any ITS in the system. */
> +void gicv3_its_unmap_all_devices(struct domain *d)
> +{
> +    struct rb_node *victim;
> +    struct its_devices *dev;
> +
> +    /*
> +     * This is an easily readable, yet inefficient implementation.
> +     * It uses the provided iteration wrapper and erases each node, which
> +     * possibly triggers rebalancing.
> +     * This seems overkill since we are going to abolish the whole tree, but
> +     * avoids an open-coded re-implementation of the traversal functions with
> +     * some recursive function calls.
> +     * Performance does not matter here, since we are destroying a domain.

Again, this is slightly untrue. Performance matter when destroying a 
domain as Xen cannot be preempted. So if it takes too long, you will 
have an impact on the overall system.

However, I think it would be fair to assume that all device will be 
deassigned before the ITS is destroyed. So I would just drop this 
function. Note that we have the same assumption in the SMMU driver.

> +     */
> +restart:
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    if ( (victim = rb_first(&d->arch.vgic.its_devices)) )
> +    {
> +        dev = rb_entry(victim, struct its_devices, rbnode);
> +        rb_erase(victim, &d->arch.vgic.its_devices);
> +
> +        spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +        remove_mapped_guest_device(dev);
> +
> +        goto restart;
> +    }
> +
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> @@ -455,6 +661,7 @@ void gicv3_its_dt_init(const struct dt_device_node *node)
>          its_data->addr = addr;
>          its_data->size = size;
>          its_data->dt_node = its;
> +        spin_lock_init(&its_data->cmd_lock);

This should be in patch #5.

>
>          printk("GICv3: Found ITS @0x%lx\n", addr);
>
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index d61479d..1fadb00 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
>      d->arch.vgic.nr_regions = rdist_count;
>      d->arch.vgic.rdist_regions = rdist_regions;
>
> +    spin_lock_init(&d->arch.vgic.its_devices_lock);
> +    d->arch.vgic.its_devices = RB_ROOT;

Again, the placement of those 2 lines are likely wrong. This should 
belong to the vITS and not the vgic-v3.

I think it would make sense to get a patch that introduces a skeleton 
for the vITS before this patch and start plumbing through.

> +
>      /*
>       * Domain 0 gets the hardware address.
>       * Guests get the virtual platform layout.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors
  2017-03-16 11:20 ` [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors Andre Przywara
@ 2017-03-22 17:30   ` Julien Grall
  2017-03-22 22:49     ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-22 17:30 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 16/03/17 11:20, Andre Przywara wrote:
> For some reason (probably because there was no user before) the 64-bit
> atomic access wrappers were commented out so far.
> As we will need them in the next patch, active (and fix) them now.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Reviewed-by: Julien Grall <julien.grall@arm.com>

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-03-16 11:20 ` [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping Andre Przywara
  2017-03-22 17:29   ` Julien Grall
@ 2017-03-22 22:45   ` Stefano Stabellini
  2017-04-03 19:45     ` Andre Przywara
  2017-03-30 11:17   ` Vijay Kilari
  2 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-22 22:45 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        | 207 +++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vgic-v3.c           |   3 +
>  xen/include/asm-arm/domain.h     |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  18 ++++
>  4 files changed, 231 insertions(+)
> 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 5c11b0d..60b15b5 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include <xen/lib.h>
>  #include <xen/delay.h>
>  #include <xen/mm.h>
> +#include <xen/rbtree.h>
> +#include <xen/sched.h>
>  #include <xen/sizes.h>
>  #include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
> @@ -32,6 +34,17 @@
>  
>  LIST_HEAD(host_its_list);
>  
> +struct its_devices {
> +    struct rb_node rbnode;
> +    struct host_its *hw_its;
> +    void *itt_addr;
> +    paddr_t guest_doorbell;
> +    uint32_t host_devid;
> +    uint32_t guest_devid;
> +    uint32_t eventids;
> +    uint32_t *host_lpis;
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>      return !list_empty(&host_its_list);
> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>      return its_send_command(its, cmd);
>  }
>  
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> +                             uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +    uint64_t cmd[4];
> +
> +    if ( valid )
> +    {
> +        ASSERT(size_bits < 32);
> +        ASSERT(!(itt_addr & ~GENMASK(51, 8)));
> +    }
> +    cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +    cmd[1] = valid ? size_bits : 0x00;
> +    cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
>      devid_bits = min(devid_bits, max_its_device_bits);
>      if ( reg & GITS_TYPER_PTA )
>          hw_its->flags |= HOST_ITS_USES_PTA;
> +    hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>  
>      for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>      {
> @@ -428,6 +460,180 @@ int gicv3_its_init(void)
>      return 0;
>  }
>  
> +static int remove_mapped_guest_device(struct its_devices *dev)
> +{
> +    int ret;
> +
> +    if ( dev->hw_its )
> +    {
> +        int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    ret = gicv3_its_wait_commands(dev->hw_its);
> +    if ( ret )
> +        return ret;
> +
> +    xfree(dev->itt_addr);
> +    xfree(dev);
> +
> +    return 0;
> +}
> +
> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +    struct host_its *hw_its;
> +
> +    list_for_each_entry(hw_its, &host_its_list, entry)

Does this need to take a spinlock to protect host_its_list? I guess not
because the list is not modified after boot?


> +    {
> +        if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> +            return hw_its;
> +    }
> +
> +    return NULL;
> +}
> +
> +static int compare_its_guest_devices(struct its_devices *dev,
> +                                     paddr_t doorbell, uint32_t devid)
> +{
> +    if ( dev->guest_doorbell < doorbell )
> +        return -1;
> +
> +    if ( dev->guest_doorbell > doorbell )
> +        return 1;
> +
> +    if ( dev->guest_devid < devid )
> +        return -1;
> +
> +    if ( dev->guest_devid > devid )
> +        return 1;
> +
> +    return 0;
> +}
> +
> +/*
> + * Map a hardware device, identified by a certain host ITS and its device ID
> + * to domain d, a guest ITS (identified by its doorbell address) and device ID.
> + * Also provide the number of events (MSIs) needed for that device.
> + * This does not check if this particular hardware device is already mapped
> + * at another domain, it is expected that this would be done by the caller.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +                               paddr_t host_doorbell, uint32_t host_devid,
> +                               paddr_t guest_doorbell, uint32_t guest_devid,
> +                               uint32_t nr_events, bool valid)
> +{
> +    void *itt_addr = NULL;
> +    struct host_its *hw_its;
> +    struct its_devices *dev = NULL, *temp;
> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> +    int ret = -ENOENT;
> +
> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> +    if ( !hw_its )
> +        return ret;
> +
> +    /* check for already existing mappings */
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    while ( *new )
> +    {
> +        temp = rb_entry(*new, struct its_devices, rbnode);
> +
> +        parent = *new;
> +        if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
> +        {
> +            if ( !valid )
> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
> +
> +            spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +            if ( valid )
> +                return -EBUSY;
> +
> +            return remove_mapped_guest_device(temp);
> +        }
> +

Plese don't run compare_its_guest_devices twice with the same arguments,
just store the return value.


> +        if ( compare_its_guest_devices(temp, guest_doorbell, guest_devid) > 0 )
> +            new = &((*new)->rb_left);
> +        else
> +            new = &((*new)->rb_right);
> +    }
> +
> +    if ( !valid )
> +        goto out_unlock;
> +
> +    ret = -ENOMEM;
> +
> +    /* An Interrupt Translation Table needs to be 256-byte aligned. */
> +    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
> +    if ( !itt_addr )
> +        goto out_unlock;
> +
> +    dev = xzalloc(struct its_devices);
> +    if ( !dev )
> +        goto out_unlock;
> +
> +    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
> +                            virt_to_maddr(itt_addr), true);
> +    if ( ret )
> +        goto out_unlock;
> +
> +    dev->itt_addr = itt_addr;
> +    dev->hw_its = hw_its;
> +    dev->guest_doorbell = guest_doorbell;
> +    dev->guest_devid = guest_devid;
> +    dev->host_devid = host_devid;
> +    dev->eventids = nr_events;
> +
> +    rb_link_node(&dev->rbnode, parent, new);
> +    rb_insert_color(&dev->rbnode, &d->arch.vgic.its_devices);
> +
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +    return 0;
> +
> +out_unlock:
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +    if ( dev )
> +        xfree(dev->host_lpis);

Where is host_lpis allocated? Why is it freed here?


> +    xfree(itt_addr);
> +    xfree(dev);
> +    return ret;
> +}
> +
> +/* Removing any connections a domain had to any ITS in the system. */
> +void gicv3_its_unmap_all_devices(struct domain *d)
> +{
> +    struct rb_node *victim;
> +    struct its_devices *dev;
> +
> +    /*
> +     * This is an easily readable, yet inefficient implementation.
> +     * It uses the provided iteration wrapper and erases each node, which
> +     * possibly triggers rebalancing.
> +     * This seems overkill since we are going to abolish the whole tree, but
> +     * avoids an open-coded re-implementation of the traversal functions with
> +     * some recursive function calls.
> +     * Performance does not matter here, since we are destroying a domain.
> +     */
> +restart:
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    if ( (victim = rb_first(&d->arch.vgic.its_devices)) )
> +    {
> +        dev = rb_entry(victim, struct its_devices, rbnode);
> +        rb_erase(victim, &d->arch.vgic.its_devices);
> +
> +        spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +        remove_mapped_guest_device(dev);
> +
> +        goto restart;
> +    }

There is no need to use goto here, a simple while loop should suffice.


> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> @@ -455,6 +661,7 @@ void gicv3_its_dt_init(const struct dt_device_node *node)
>          its_data->addr = addr;
>          its_data->size = size;
>          its_data->dt_node = its;
> +        spin_lock_init(&its_data->cmd_lock);

This change should be in the previous patch


>          printk("GICv3: Found ITS @0x%lx\n", addr);
>  
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index d61479d..1fadb00 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
>      d->arch.vgic.nr_regions = rdist_count;
>      d->arch.vgic.rdist_regions = rdist_regions;
>  
> +    spin_lock_init(&d->arch.vgic.its_devices_lock);
> +    d->arch.vgic.its_devices = RB_ROOT;
> +
>      /*
>       * Domain 0 gets the hardware address.
>       * Guests get the virtual platform layout.
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 2d6fbb1..00b9c1a 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -11,6 +11,7 @@
>  #include <asm/gic.h>
>  #include <public/hvm/params.h>
>  #include <xen/serial.h>
> +#include <xen/rbtree.h>
>  
>  struct hvm_domain
>  {
> @@ -109,6 +110,8 @@ struct arch_domain
>          } *rdist_regions;
>          int nr_regions;                     /* Number of rdist regions */
>          uint32_t rdist_stride;              /* Re-Distributor stride */
> +        struct rb_root its_devices;         /* devices mapped to an ITS */
> +        spinlock_t its_devices_lock;        /* protects the its_devices tree */
>  #endif
>      } vgic;
>  
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 8b493fb..3421ea0 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -48,6 +48,10 @@
>  #define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
>                                                 GITS_TYPER_DEVIDS_SHIFT) + 1)
>  #define GITS_TYPER_IDBITS_SHIFT         8
> +#define GITS_TYPER_ITT_SIZE_SHIFT       4
> +#define GITS_TYPER_ITT_SIZE_MASK        (0xfUL << GITS_TYPER_ITT_SIZE_SHIFT)
> +#define GITS_TYPER_ITT_SIZE(r)          ((((r) & GITS_TYPER_ITT_SIZE_MASK) >> \
> +                                                GITS_TYPER_ITT_SIZE_SHIFT) + 1)
>  
>  #define GITS_IIDR_VALUE                 0x34c
>  
> @@ -94,7 +98,10 @@
>  #define GITS_CMD_MOVALL                 0x0e
>  #define GITS_CMD_DISCARD                0x0f
>  
> +#define ITS_DOORBELL_OFFSET             0x10040
> +
>  #include <xen/device_tree.h>
> +#include <xen/rbtree.h>
>  
>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>  #define HOST_ITS_USES_PTA               (1U << 1)
> @@ -108,6 +115,7 @@ struct host_its {
>      void __iomem *its_base;
>      spinlock_t cmd_lock;
>      void *cmd_buf;
> +    unsigned int itte_size;
>      unsigned int flags;
>  };

I would move itte_size and its initialization to the patch that
introduced struct host_its.


> @@ -134,6 +142,16 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
>  /* Map a collection for this host CPU to each host ITS. */
>  int gicv3_its_setup_collection(unsigned int cpu);
>  
> +/*
> + * Map a device on the host by allocating an ITT on the host (ITS).
> + * "nr_event" specifies how many events (interrupts) this device will need.
> + * Setting "valid" to false deallocates the device.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +                               paddr_t host_doorbell, uint32_t host_devid,
> +                               paddr_t guest_doorbell, uint32_t guest_devid,
> +                               uint32_t nr_events, bool valid);
> +
>  #else
>  
>  static LIST_HEAD(host_its_list);
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors
  2017-03-22 17:30   ` Julien Grall
@ 2017-03-22 22:49     ` Stefano Stabellini
  0 siblings, 0 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-22 22:49 UTC (permalink / raw)
  To: Julien Grall
  Cc: Andre Przywara, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari, xen-devel

On Wed, 22 Mar 2017, Julien Grall wrote:
> Hi Andre,
> 
> On 16/03/17 11:20, Andre Przywara wrote:
> > For some reason (probably because there was no user before) the 64-bit
> > atomic access wrappers were commented out so far.
> > As we will need them in the next patch, active (and fix) them now.
> > 
> > Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> 
> Reviewed-by: Julien Grall <julien.grall@arm.com>

Reviewed-by: Stefano Stabellini <sstabellini@kernel.org>

I committed this patch.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-16 11:20 ` [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array Andre Przywara
@ 2017-03-22 23:38   ` Stefano Stabellini
  2017-03-23  8:48     ` Julien Grall
  2017-03-23 10:21     ` Andre Przywara
  2017-03-23 19:08   ` Julien Grall
  1 sibling, 2 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-22 23:38 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> The number of LPIs on a host can be potentially huge (millions),
> although in practise will be mostly reasonable. So prematurely allocating
> an array of struct irq_desc's for each LPI is not an option.
> However Xen itself does not care about LPIs, as every LPI will be injected
> into a guest (Dom0 for now).
> Create a dense data structure (8 Bytes) for each LPI which holds just
> enough information to determine the virtual IRQ number and the VCPU into
> which the LPI needs to be injected.
> Also to not artificially limit the number of LPIs, we create a 2-level
> table for holding those structures.
> This patch introduces functions to initialize these tables and to
> create, lookup and destroy entries for a given LPI.
> By using the naturally atomic access guarantee the native uint64_t data
> type gives us, we allocate and access LPI information in a way that does
> not require a lock.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        |  90 ++++++++++++++++++-
>  xen/arch/arm/gic-v3-lpi.c        | 188 +++++++++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/gic.h        |   5 ++
>  xen/include/asm-arm/gic_v3_its.h |  11 +++
>  4 files changed, 292 insertions(+), 2 deletions(-)
> 
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 60b15b5..ed14d95 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -148,6 +148,20 @@ static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
>      return its_send_command(its, cmd);
>  }
>  
> +static int its_send_cmd_mapti(struct host_its *its,
> +                              uint32_t deviceid, uint32_t eventid,
> +                              uint32_t pintid, uint16_t icid)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_MAPTI | ((uint64_t)deviceid << 32);
> +    cmd[1] = eventid | ((uint64_t)pintid << 32);
> +    cmd[2] = icid;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
>  static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>                               unsigned int cpu)
>  {
> @@ -180,6 +194,19 @@ static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>      return its_send_command(its, cmd);
>  }
>  
> +static int its_send_cmd_inv(struct host_its *its,
> +                            uint32_t deviceid, uint32_t eventid)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_INV | ((uint64_t)deviceid << 32);
> +    cmd[1] = eventid;
> +    cmd[2] = 0x00;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -462,7 +489,7 @@ int gicv3_its_init(void)
>  
>  static int remove_mapped_guest_device(struct its_devices *dev)
>  {
> -    int ret;
> +    int ret, i;
>  
>      if ( dev->hw_its )
>      {
> @@ -471,11 +498,19 @@ static int remove_mapped_guest_device(struct its_devices *dev)
>              return ret;
>      }
>  
> +    /*
> +     * The only error the function below would return is -ENOENT, in which
> +     * case there is nothing to free here. So we just ignore it.
> +     */
> +    for ( i = 0; i < DIV_ROUND_UP(dev->eventids, LPI_BLOCK); i++ )
> +        gicv3_free_host_lpi_block(dev->hw_its, dev->host_lpis[i]);
> +
>      ret = gicv3_its_wait_commands(dev->hw_its);
>      if ( ret )
>          return ret;
>  
>      xfree(dev->itt_addr);
> +    xfree(dev->host_lpis);
>      xfree(dev);
>  
>      return 0;
> @@ -513,6 +548,36 @@ static int compare_its_guest_devices(struct its_devices *dev,
>  }
>  
>  /*
> + * On the host ITS @its, map @nr_events consecutive LPIs.
> + * The mapping connects a device @devid and event @eventid pair to LPI @lpi,
> + * increasing both @eventid and @lpi to cover the number of requested LPIs.
> + */
> +int gicv3_its_map_host_events(struct host_its *its,
> +                              uint32_t devid, uint32_t eventid, uint32_t lpi,
> +                              uint32_t nr_events)
> +{
> +    uint32_t i;
> +    int ret;
> +
> +    for ( i = 0; i < nr_events; i++ )
> +    {
> +        /* For now we map every host LPI to host CPU 0 */
> +        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
> +        if ( ret )
> +            return ret;
> +        ret = its_send_cmd_inv(its, devid, eventid + i);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    ret = its_send_cmd_sync(its, 0);
> +    if ( ret )
> +        return ret;
> +
> +    return gicv3_its_wait_commands(its);
> +}
> +
> +/*
>   * Map a hardware device, identified by a certain host ITS and its device ID
>   * to domain d, a guest ITS (identified by its doorbell address) and device ID.
>   * Also provide the number of events (MSIs) needed for that device.
> @@ -528,7 +593,7 @@ int gicv3_its_map_guest_device(struct domain *d,
>      struct host_its *hw_its;
>      struct its_devices *dev = NULL, *temp;
>      struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> -    int ret = -ENOENT;
> +    int ret = -ENOENT, i;
>  
>      hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>      if ( !hw_its )
> @@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
>      if ( !dev )
>          goto out_unlock;
>  
> +    dev->host_lpis = xzalloc_array(uint32_t,
> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
> +    if ( !dev->host_lpis )
> +        goto out_unlock;

The expectation is that we are going to allocate far more than 32 lpis
for one device, possibly thousands, right?
If so, dev->host_lpis could be quite large. I am thinking we should
consider switching to a sparse data structure to save memory, as in most
cases the allocated blocks are going to be consecutive.

I would probably store the first lpi and the number of consecutive lpis,
rather than the first lpi in each block. For example, if the allocated
blocks are:

 0-31, 32-63, 64-95, 96-127, 128-159, 256-287

Now we store: 0, 32, 64, 96, 128, 256
I would store [0, 160], [256, 32]

In the case of a thousand consecutive block allocations, it would
shorten the array from 1000/32 to 1 element.

Does it make sense? Is it worth doing? This last question is mostly for
the Qualcomm and Cavium guys on the list because it depends on the
numbers of events on a given platform.


>      ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
>                              virt_to_maddr(itt_addr), true);
>      if ( ret )
> @@ -591,10 +661,26 @@ int gicv3_its_map_guest_device(struct domain *d,
>  
>      spin_unlock(&d->arch.vgic.its_devices_lock);
>  
> +    /*
> +     * Map all host LPIs within this device already. We can't afford to queue
> +     * any host ITS commands later on during the guest's runtime.
> +     */
> +    for ( i = 0; i < DIV_ROUND_UP(nr_events, LPI_BLOCK); i++ )
> +    {
> +        ret = gicv3_allocate_host_lpi_block(hw_its, d, host_devid,
> +                                            i * LPI_BLOCK);
> +        if ( ret < 0 )
> +            goto out;
> +
> +        dev->host_lpis[i] = ret;
> +    }
> +
>      return 0;
>  
>  out_unlock:
>      spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +out:
>      if ( dev )
>          xfree(dev->host_lpis);
>      xfree(itt_addr);
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 51d7425..59d3ba4 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -20,24 +20,44 @@
>  
>  #include <xen/lib.h>
>  #include <xen/mm.h>
> +#include <xen/sched.h>
> +#include <asm/atomic.h>
> +#include <asm/domain.h>
>  #include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
>  #include <asm/page.h>
>  
> +/* LPIs on the host always go to a guest, so no struct irq_desc for them. */
> +union host_lpi {
> +    uint64_t data;
> +    struct {
> +        uint32_t virt_lpi;
> +        uint16_t dom_id;
> +        uint16_t vcpu_id;
> +    };
> +};
>
>  #define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
>  /* Global state */
>  static struct {
>      /* The global LPI property table, shared by all redistributors. */
>      uint8_t *lpi_property;
>      /*
> +     * A two-level table to lookup LPIs firing on the host and look up the
> +     * domain and virtual LPI number to inject.
> +     */
> +    union host_lpi **host_lpis;
> +    /*
>       * Number of physical LPIs the host supports. This is a property of
>       * the GIC hardware. We depart from the habit of naming these things
>       * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
>       * in a different context to differentiate them from "virtual LPIs".
>       */
>      unsigned long int nr_host_lpis;
> +    /* Protects allocation and deallocation of host LPIs, but not the access */

OK. What does protect the access to host LPIs?


> +    spinlock_t host_lpis_lock;
>      unsigned int flags;
>  } lpi_data;
>  
> @@ -50,6 +70,19 @@ struct lpi_redist_data {
>  static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>  
>  #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
> +#define HOST_LPIS_PER_PAGE      (PAGE_SIZE / sizeof(union host_lpi))
> +
> +static union host_lpi *gic_get_host_lpi(uint32_t plpi)
> +{
> +    if ( !is_lpi(plpi) || plpi >= MAX_PHYS_LPIS + LPI_OFFSET )

Arguably if (plpi >= MAX_PHYS_LPIS + LPI_OFFSET) is not really an lpi. I
would add the plpi >= MAX_PHYS_LPIS + LPI_OFFSET check to is_lpi.


> +        return NULL;
> +
> +    plpi -= LPI_OFFSET;
> +    if ( !lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE] )
> +        return NULL;
> +
> +    return &lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE][plpi % HOST_LPIS_PER_PAGE];

I am almost sure this line is longer than 80 characters.. Please double
check the coding style throughout the series.


> +}
>  
>  /* Stores this redistributor's physical address and ID in a per-CPU variable */
>  void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
> @@ -204,15 +237,170 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base)
>  static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
>  integer_param("max_lpi_bits", max_lpi_bits);
>  
> +/*
> + * Allocate the 2nd level array for host LPIs. This one holds pointers
> + * to the page with the actual "union host_lpi" entries. Our LPI limit
> + * avoids excessive memory usage.
> + */
>  int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
>  {
> +    int nr_lpi_ptrs;
> +
> +    /* We rely on the data structure being atomically accessible. */
> +    BUILD_BUG_ON(sizeof(union host_lpi) > sizeof(unsigned long));

Technically, I think that arm32 is able to access uint64_t atomically
(ldrexd and strexd).


>      lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
>  
> +    spin_lock_init(&lpi_data.host_lpis_lock);
> +
> +    nr_lpi_ptrs = MAX_PHYS_LPIS / (PAGE_SIZE / sizeof(union host_lpi));
> +    lpi_data.host_lpis = xzalloc_array(union host_lpi *, nr_lpi_ptrs);
> +    if ( !lpi_data.host_lpis )
> +        return -ENOMEM;
> +
>      printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);
>  
>      return 0;
>  }
>  
> +/* Must be called with host_lpis_lock held. */
> +static int find_unused_host_lpi(int start, uint32_t *index)
> +{
> +    int chunk;
> +    uint32_t i = *index;
> +
> +    for ( chunk = start; chunk < MAX_PHYS_LPIS / HOST_LPIS_PER_PAGE; chunk++ )
> +    {
> +        /* If we hit an unallocated chunk, use entry 0 in that one. */
> +        if ( !lpi_data.host_lpis[chunk] )
> +        {
> +            *index = 0;
> +            return chunk;
> +        }
> +
> +        /* Find an unallocated entry in this chunk. */
> +        for ( ; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
> +        {
> +            if ( lpi_data.host_lpis[chunk][i].dom_id == INVALID_DOMID )
> +            {
> +                *index = i;
> +                return chunk;
> +            }
> +        }
> +        i = 0;
> +    }
> +
> +    return -1;
> +}
> +
> +/*
> + * Allocate a block of 32 LPIs on the given host ITS for device "devid",
> + * starting with "eventid". Put them into the respective ITT by issuing a
> + * MAPTI command for each of them.
> + */
> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
> +                                  uint32_t host_devid, uint32_t eventid)
> +{
> +    static uint32_t next_lpi = 0;
> +    uint32_t lpi, lpi_idx = next_lpi % HOST_LPIS_PER_PAGE;
> +    int chunk;
> +    int i;
> +
> +    spin_lock(&lpi_data.host_lpis_lock);
> +    chunk = find_unused_host_lpi(next_lpi / HOST_LPIS_PER_PAGE, &lpi_idx);
> +
> +    if ( chunk == - 1 )          /* rescan for a hole from the beginning */
> +    {
> +        lpi_idx = 0;
> +        chunk = find_unused_host_lpi(0, &lpi_idx);
> +        if ( chunk == -1 )
> +        {
> +            spin_unlock(&lpi_data.host_lpis_lock);
> +            return -ENOSPC;
> +        }
> +    }
> +
> +    /* If we hit an unallocated chunk, we initialize it and use entry 0. */
> +    if ( !lpi_data.host_lpis[chunk] )
> +    {
> +        union host_lpi *new_chunk;
> +
> +        new_chunk = alloc_xenheap_pages(0, 0);
> +        if ( !new_chunk )
> +        {
> +            spin_unlock(&lpi_data.host_lpis_lock);
> +            return -ENOMEM;
> +        }
> +
> +        for ( i = 0; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
> +            new_chunk[i].dom_id = INVALID_DOMID;
> +
> +        lpi_data.host_lpis[chunk] = new_chunk;
> +        lpi_idx = 0;
> +    }
> +
> +    lpi = chunk * HOST_LPIS_PER_PAGE + lpi_idx;
> +
> +    for ( i = 0; i < LPI_BLOCK; i++ )
> +    {
> +        union host_lpi hlpi;
> +
> +        /*
> +         * Mark this host LPI as belonging to the domain, but don't assign
> +         * any virtual LPI or a VCPU yet.
> +         */
> +        hlpi.virt_lpi = INVALID_LPI;
> +        hlpi.dom_id = d->domain_id;
> +        hlpi.vcpu_id = INVALID_DOMID;
> +        write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
> +                         hlpi.data);
> +
> +        /*
> +         * Enable this host LPI, so we don't have to do this during the
> +         * guest's runtime.
> +         */
> +        lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
> +    }
> +
> +    /*
> +     * We have allocated and initialized the host LPI entries, so it's safe
> +     * to drop the lock now. Access to the structures can be done concurrently
> +     * as it involves only an atomic uint64_t access.
> +     */
> +    spin_unlock(&lpi_data.host_lpis_lock);
> +
> +    if ( lpi_data.flags & LPI_PROPTABLE_NEEDS_FLUSHING )
> +        clean_and_invalidate_dcache_va_range(&lpi_data.lpi_property[lpi],
> +                                             LPI_BLOCK);

I would move this right below 
  lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;


> +    gicv3_its_map_host_events(its, host_devid, eventid, lpi + LPI_OFFSET,
> +                              LPI_BLOCK);
> +
> +    next_lpi = lpi + LPI_BLOCK;
> +    return lpi + LPI_OFFSET;
> +}
> +
> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi)
> +{
> +    union host_lpi *hlpi, empty_lpi = { .dom_id = INVALID_DOMID };
> +    int i;
> +
> +    hlpi = gic_get_host_lpi(lpi);

I think you need to make sure that "lpi" is the first lpi of a LPI_BLOCK.
Either that, or gic_get_host_lpi should make sure to always return the
first lpi in a block. 


> +    if ( !hlpi )
> +        return -ENOENT;
> +
> +    spin_lock(&lpi_data.host_lpis_lock);
> +
> +    for ( i = 0; i < LPI_BLOCK; i++ )
> +        write_u64_atomic(&hlpi[i].data, empty_lpi.data);
> +
> +    /* TODO: Call a function in gic-v3-its.c to send DISCARDs */
> +
> +    spin_unlock(&lpi_data.host_lpis_lock);
> +
> +    return 0;
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 12bd155..7825575 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -220,7 +220,12 @@ enum gic_version {
>      GIC_V3,
>  };
>  
> +#define INVALID_LPI     0
>  #define LPI_OFFSET      8192
> +static inline bool is_lpi(unsigned int irq)
> +{
> +    return irq >= LPI_OFFSET;
> +}
>  
>  extern enum gic_version gic_hw_version(void);
>  
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 3421ea0..2387972 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -106,6 +106,11 @@
>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>  #define HOST_ITS_USES_PTA               (1U << 1)
>  
> +#define INVALID_DOMID ((uint16_t)~0)

We already have DOMID_INVALID


> +/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
> +#define LPI_BLOCK                       32
> +
>  /* data structure for each hardware ITS */
>  struct host_its {
>      struct list_head entry;
> @@ -151,6 +156,12 @@ int gicv3_its_map_guest_device(struct domain *d,
>                                 paddr_t host_doorbell, uint32_t host_devid,
>                                 paddr_t guest_doorbell, uint32_t guest_devid,
>                                 uint32_t nr_events, bool valid);
> +int gicv3_its_map_host_events(struct host_its *its,
> +                              uint32_t host_devid, uint32_t eventid,
> +                              uint32_t lpi, uint32_t nrevents);
> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
> +                                  uint32_t host_devid, uint32_t eventid);
> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
>  
>  #else
>  
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-16 11:20 ` [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs Andre Przywara
@ 2017-03-22 23:44   ` Stefano Stabellini
  2017-03-23 20:08     ` André Przywara
  2017-03-24 11:40   ` Julien Grall
  1 sibling, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-22 23:44 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 16 Mar 2017, Andre Przywara wrote:
> For the same reason that allocating a struct irq_desc for each
> possible LPI is not an option, having a struct pending_irq for each LPI
> is also not feasible. However we actually only need those when an
> interrupt is on a vCPU (or is about to be injected).
> Maintain a list of those structs that we can use for the lifecycle of
> a guest LPI. We allocate new entries if necessary, however reuse
> pre-owned entries whenever possible.
> I added some locking around this list here, however my gut feeling is
> that we don't need one because this a per-VCPU structure anyway.
> If someone could confirm this, I'd be grateful.
> Teach the existing VGIC functions to find the right pointer when being
> given a virtual LPI number.
> 
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Still all past comments are unaddress ;-(

It makes very hard to continue doing reviews. I am sorry but I'll have
to stop reviewing until I see a series with my past comments addressed.
I encourage Julien to do the same.


> ---
>  xen/arch/arm/gic.c           |  3 +++
>  xen/arch/arm/vgic-v3.c       |  3 +++
>  xen/arch/arm/vgic.c          | 64 +++++++++++++++++++++++++++++++++++++++++---
>  xen/include/asm-arm/domain.h |  2 ++
>  xen/include/asm-arm/vgic.h   | 14 ++++++++++
>  5 files changed, 83 insertions(+), 3 deletions(-)
> 
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index a5348f2..bd3c032 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -509,6 +509,9 @@ static void gic_update_one_lr(struct vcpu *v, int i)
>                  struct vcpu *v_target = vgic_get_target_vcpu(v, irq);
>                  irq_set_affinity(p->desc, cpumask_of(v_target->processor));
>              }
> +            /* If this was an LPI, mark this struct as available again. */
> +            if ( is_lpi(p->irq) )
> +                p->irq = 0;
>          }
>      }
>  }
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index 1fadb00..b0653c2 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -1426,6 +1426,9 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
>      if ( v->vcpu_id == last_cpu || (v->vcpu_id == (d->max_vcpus - 1)) )
>          v->arch.vgic.flags |= VGIC_V3_RDIST_LAST;
>  
> +    spin_lock_init(&v->arch.vgic.pending_lpi_list_lock);
> +    INIT_LIST_HEAD(&v->arch.vgic.pending_lpi_list);
> +
>      return 0;
>  }
>  
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 364d5f0..e5cfa54 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -30,6 +30,8 @@
>  
>  #include <asm/mmio.h>
>  #include <asm/gic.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
>  #include <asm/vgic.h>
>  
>  static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v, int rank)
> @@ -61,7 +63,7 @@ struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq)
>      return vgic_get_rank(v, rank);
>  }
>  
> -static void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
> +void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
>  {
>      INIT_LIST_HEAD(&p->inflight);
>      INIT_LIST_HEAD(&p->lr_queue);
> @@ -244,10 +246,14 @@ struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq)
>  
>  static int vgic_get_virq_priority(struct vcpu *v, unsigned int virq)
>  {
> -    struct vgic_irq_rank *rank = vgic_rank_irq(v, virq);
> +    struct vgic_irq_rank *rank;
>      unsigned long flags;
>      int priority;
>  
> +    if ( is_lpi(virq) )
> +        return vgic_lpi_get_priority(v->domain, virq);
> +
> +    rank = vgic_rank_irq(v, virq);
>      vgic_lock_rank(v, rank, flags);
>      priority = rank->priority[virq & INTERRUPT_RANK_MASK];
>      vgic_unlock_rank(v, rank, flags);
> @@ -446,13 +452,63 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
>      return true;
>  }
>  
> +/*
> + * Holding struct pending_irq's for each possible virtual LPI in each domain
> + * requires too much Xen memory, also a malicious guest could potentially
> + * spam Xen with LPI map requests. We cannot cover those with (guest allocated)
> + * ITS memory, so we use a dynamic scheme of allocating struct pending_irq's
> + * on demand.
> + */
> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
> +                                   bool allocate)
> +{
> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
> +
> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
> +    {
> +        if ( lpi_irq->pirq.irq == lpi )
> +        {
> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> +            return &lpi_irq->pirq;
> +        }
> +
> +        if ( lpi_irq->pirq.irq == 0 && !empty )
> +            empty = lpi_irq;
> +    }
> +
> +    if ( !allocate )
> +    {
> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> +        return NULL;
> +    }
> +
> +    if ( !empty )
> +    {
> +        empty = xzalloc(struct lpi_pending_irq);
> +        vgic_init_pending_irq(&empty->pirq, lpi);
> +        list_add_tail(&empty->entry, &v->arch.vgic.pending_lpi_list);
> +    } else
> +    {
> +        empty->pirq.status = 0;
> +        empty->pirq.irq = lpi;
> +    }
> +
> +    spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> +
> +    return &empty->pirq;
> +}
> +
>  struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>  {
>      struct pending_irq *n;
> +
>      /* Pending irqs allocation strategy: the first vgic.nr_spis irqs
>       * are used for SPIs; the rests are used for per cpu irqs */
>      if ( irq < 32 )
>          n = &v->arch.vgic.pending_irqs[irq];
> +    else if ( is_lpi(irq) )
> +        n = lpi_to_pending(v, irq, true);
>      else
>          n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>      return n;
> @@ -480,7 +536,7 @@ void vgic_clear_pending_irqs(struct vcpu *v)
>  void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>  {
>      uint8_t priority;
> -    struct pending_irq *iter, *n = irq_to_pending(v, virq);
> +    struct pending_irq *iter, *n;
>      unsigned long flags;
>      bool running;
>  
> @@ -488,6 +544,8 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>  
>      spin_lock_irqsave(&v->arch.vgic.lock, flags);
>  
> +    n = irq_to_pending(v, virq);
> +
>      /* vcpu offline */
>      if ( test_bit(_VPF_down, &v->pause_flags) )
>      {
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 00b9c1a..f44a84b 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -257,6 +257,8 @@ struct arch_vcpu
>          paddr_t rdist_base;
>  #define VGIC_V3_RDIST_LAST  (1 << 0)        /* last vCPU of the rdist */
>          uint8_t flags;
> +        struct list_head pending_lpi_list;
> +        spinlock_t pending_lpi_list_lock;   /* protects the pending_lpi_list */
>      } vgic;
>  
>      /* Timer registers  */
> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index 467333c..8f1099c 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -83,6 +83,12 @@ struct pending_irq
>      struct list_head lr_queue;
>  };
>  
> +struct lpi_pending_irq
> +{
> +    struct list_head entry;
> +    struct pending_irq pirq;
> +};
> +
>  #define NR_INTERRUPT_PER_RANK   32
>  #define INTERRUPT_RANK_MASK (NR_INTERRUPT_PER_RANK - 1)
>  
> @@ -296,13 +302,21 @@ extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
>  extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
>  extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
>  extern void vgic_clear_pending_irqs(struct vcpu *v);
> +extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
>  extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
>  extern struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq);
> +extern struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int irq,
> +                                          bool allocate);
>  extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v, int b, int n, int s);
>  extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
>  extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
>  extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
>  extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
> +/* placeholder function until the property table gets introduced */
> +static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
> +{
> +    return 0xa;
> +}
>  extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
>  int vgic_v2_init(struct domain *d, int *mmio_count);
>  int vgic_v3_init(struct domain *d, int *mmio_count);
> -- 
> 2.9.0
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-22 23:38   ` Stefano Stabellini
@ 2017-03-23  8:48     ` Julien Grall
  2017-03-23 10:21     ` Andre Przywara
  1 sibling, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-23  8:48 UTC (permalink / raw)
  To: Stefano Stabellini, Andre Przywara
  Cc: xen-devel, nd, Shanker Donthineni, Vijay Kilari

Hi Stefano,

On 22/03/2017 23:38, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Andre Przywara wrote:

[...]

>> +    dev->host_lpis = xzalloc_array(uint32_t,
>> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
>> +    if ( !dev->host_lpis )
>> +        goto out_unlock;
>
> The expectation is that we are going to allocate far more than 32 lpis
> for one device, possibly thousands, right?
> If so, dev->host_lpis could be quite large. I am thinking we should
> consider switching to a sparse data structure to save memory, as in most
> cases the allocated blocks are going to be consecutive.
>
> I would probably store the first lpi and the number of consecutive lpis,
> rather than the first lpi in each block. For example, if the allocated
> blocks are:
>
>  0-31, 32-63, 64-95, 96-127, 128-159, 256-287
>
> Now we store: 0, 32, 64, 96, 128, 256
> I would store [0, 160], [256, 32]
>
> In the case of a thousand consecutive block allocations, it would
> shorten the array from 1000/32 to 1 element.
>
> Does it make sense? Is it worth doing? This last question is mostly for
> the Qualcomm and Cavium guys on the list because it depends on the
> numbers of events on a given platform.

Not answering directly to the question. But I think this is an 
improvement and not necessary for a first version.

I would like to see this series merged in Xen 4.9 as a tech preview, so 
I would suggest to gather this list of improvement (maybe on Jira?) and 
defer them for Xen 4.10. They would need to be addressed before making 
ITS stable. Stefano, does it sound good to you?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-22 23:38   ` Stefano Stabellini
  2017-03-23  8:48     ` Julien Grall
@ 2017-03-23 10:21     ` Andre Przywara
  2017-03-23 17:52       ` Stefano Stabellini
  1 sibling, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-23 10:21 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

Hi,

On 22/03/17 23:38, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Andre Przywara wrote:
>> The number of LPIs on a host can be potentially huge (millions),
>> although in practise will be mostly reasonable. So prematurely allocating
>> an array of struct irq_desc's for each LPI is not an option.
>> However Xen itself does not care about LPIs, as every LPI will be injected
>> into a guest (Dom0 for now).
>> Create a dense data structure (8 Bytes) for each LPI which holds just
>> enough information to determine the virtual IRQ number and the VCPU into
>> which the LPI needs to be injected.
>> Also to not artificially limit the number of LPIs, we create a 2-level
>> table for holding those structures.
>> This patch introduces functions to initialize these tables and to
>> create, lookup and destroy entries for a given LPI.
>> By using the naturally atomic access guarantee the native uint64_t data
>> type gives us, we allocate and access LPI information in a way that does
>> not require a lock.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c        |  90 ++++++++++++++++++-
>>  xen/arch/arm/gic-v3-lpi.c        | 188 +++++++++++++++++++++++++++++++++++++++
>>  xen/include/asm-arm/gic.h        |   5 ++
>>  xen/include/asm-arm/gic_v3_its.h |  11 +++
>>  4 files changed, 292 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 60b15b5..ed14d95 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -148,6 +148,20 @@ static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
>>      return its_send_command(its, cmd);
>>  }
>>  
>> +static int its_send_cmd_mapti(struct host_its *its,
>> +                              uint32_t deviceid, uint32_t eventid,
>> +                              uint32_t pintid, uint16_t icid)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    cmd[0] = GITS_CMD_MAPTI | ((uint64_t)deviceid << 32);
>> +    cmd[1] = eventid | ((uint64_t)pintid << 32);
>> +    cmd[2] = icid;
>> +    cmd[3] = 0x00;
>> +
>> +    return its_send_command(its, cmd);
>> +}
>> +
>>  static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>>                               unsigned int cpu)
>>  {
>> @@ -180,6 +194,19 @@ static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>>      return its_send_command(its, cmd);
>>  }
>>  
>> +static int its_send_cmd_inv(struct host_its *its,
>> +                            uint32_t deviceid, uint32_t eventid)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    cmd[0] = GITS_CMD_INV | ((uint64_t)deviceid << 32);
>> +    cmd[1] = eventid;
>> +    cmd[2] = 0x00;
>> +    cmd[3] = 0x00;
>> +
>> +    return its_send_command(its, cmd);
>> +}
>> +
>>  /* Set up the (1:1) collection mapping for the given host CPU. */
>>  int gicv3_its_setup_collection(unsigned int cpu)
>>  {
>> @@ -462,7 +489,7 @@ int gicv3_its_init(void)
>>  
>>  static int remove_mapped_guest_device(struct its_devices *dev)
>>  {
>> -    int ret;
>> +    int ret, i;
>>  
>>      if ( dev->hw_its )
>>      {
>> @@ -471,11 +498,19 @@ static int remove_mapped_guest_device(struct its_devices *dev)
>>              return ret;
>>      }
>>  
>> +    /*
>> +     * The only error the function below would return is -ENOENT, in which
>> +     * case there is nothing to free here. So we just ignore it.
>> +     */
>> +    for ( i = 0; i < DIV_ROUND_UP(dev->eventids, LPI_BLOCK); i++ )
>> +        gicv3_free_host_lpi_block(dev->hw_its, dev->host_lpis[i]);
>> +
>>      ret = gicv3_its_wait_commands(dev->hw_its);
>>      if ( ret )
>>          return ret;
>>  
>>      xfree(dev->itt_addr);
>> +    xfree(dev->host_lpis);
>>      xfree(dev);
>>  
>>      return 0;
>> @@ -513,6 +548,36 @@ static int compare_its_guest_devices(struct its_devices *dev,
>>  }
>>  
>>  /*
>> + * On the host ITS @its, map @nr_events consecutive LPIs.
>> + * The mapping connects a device @devid and event @eventid pair to LPI @lpi,
>> + * increasing both @eventid and @lpi to cover the number of requested LPIs.
>> + */
>> +int gicv3_its_map_host_events(struct host_its *its,
>> +                              uint32_t devid, uint32_t eventid, uint32_t lpi,
>> +                              uint32_t nr_events)
>> +{
>> +    uint32_t i;
>> +    int ret;
>> +
>> +    for ( i = 0; i < nr_events; i++ )
>> +    {
>> +        /* For now we map every host LPI to host CPU 0 */
>> +        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
>> +        if ( ret )
>> +            return ret;
>> +        ret = its_send_cmd_inv(its, devid, eventid + i);
>> +        if ( ret )
>> +            return ret;
>> +    }
>> +
>> +    ret = its_send_cmd_sync(its, 0);
>> +    if ( ret )
>> +        return ret;
>> +
>> +    return gicv3_its_wait_commands(its);
>> +}
>> +
>> +/*
>>   * Map a hardware device, identified by a certain host ITS and its device ID
>>   * to domain d, a guest ITS (identified by its doorbell address) and device ID.
>>   * Also provide the number of events (MSIs) needed for that device.
>> @@ -528,7 +593,7 @@ int gicv3_its_map_guest_device(struct domain *d,
>>      struct host_its *hw_its;
>>      struct its_devices *dev = NULL, *temp;
>>      struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
>> -    int ret = -ENOENT;
>> +    int ret = -ENOENT, i;
>>  
>>      hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>>      if ( !hw_its )
>> @@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
>>      if ( !dev )
>>          goto out_unlock;
>>  
>> +    dev->host_lpis = xzalloc_array(uint32_t,
>> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
>> +    if ( !dev->host_lpis )
>> +        goto out_unlock;
> 
> The expectation is that we are going to allocate far more than 32 lpis
> for one device, possibly thousands, right?

Mmh, what makes you think so?
My understanding is that you have one MSI per interrupt "reason" (TX,
RX, error, for instance). And even if you have multiple queues, each
connected to separate MSIs, that won't go into the thousands. MSI-X for
instance is *limited* to 2048 MSIs.

But in reality I dont' see nearly that many MSIs, mostly less than a
dozen per device. And since 32 is the old PCI MSI limit, I thought this
would be a good starting point. So most of the time we just need one block.

I believe this could add up with virtual functions, but those are
separate devices, right?

Cheers,
Andre.

> If so, dev->host_lpis could be quite large. I am thinking we should
> consider switching to a sparse data structure to save memory, as in most
> cases the allocated blocks are going to be consecutive.
> 
> I would probably store the first lpi and the number of consecutive lpis,
> rather than the first lpi in each block. For example, if the allocated
> blocks are:
> 
>  0-31, 32-63, 64-95, 96-127, 128-159, 256-287
> 
> Now we store: 0, 32, 64, 96, 128, 256
> I would store [0, 160], [256, 32]
> 
> In the case of a thousand consecutive block allocations, it would
> shorten the array from 1000/32 to 1 element.
> 
> Does it make sense? Is it worth doing? This last question is mostly for
> the Qualcomm and Cavium guys on the list because it depends on the
> numbers of events on a given platform.
> 
> 
>>      ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
>>                              virt_to_maddr(itt_addr), true);
>>      if ( ret )
>> @@ -591,10 +661,26 @@ int gicv3_its_map_guest_device(struct domain *d,
>>  
>>      spin_unlock(&d->arch.vgic.its_devices_lock);
>>  
>> +    /*
>> +     * Map all host LPIs within this device already. We can't afford to queue
>> +     * any host ITS commands later on during the guest's runtime.
>> +     */
>> +    for ( i = 0; i < DIV_ROUND_UP(nr_events, LPI_BLOCK); i++ )
>> +    {
>> +        ret = gicv3_allocate_host_lpi_block(hw_its, d, host_devid,
>> +                                            i * LPI_BLOCK);
>> +        if ( ret < 0 )
>> +            goto out;
>> +
>> +        dev->host_lpis[i] = ret;
>> +    }
>> +
>>      return 0;
>>  
>>  out_unlock:
>>      spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +out:
>>      if ( dev )
>>          xfree(dev->host_lpis);
>>      xfree(itt_addr);
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>> index 51d7425..59d3ba4 100644
>> --- a/xen/arch/arm/gic-v3-lpi.c
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -20,24 +20,44 @@
>>  
>>  #include <xen/lib.h>
>>  #include <xen/mm.h>
>> +#include <xen/sched.h>
>> +#include <asm/atomic.h>
>> +#include <asm/domain.h>
>>  #include <asm/gic.h>
>>  #include <asm/gic_v3_defs.h>
>>  #include <asm/gic_v3_its.h>
>>  #include <asm/io.h>
>>  #include <asm/page.h>
>>  
>> +/* LPIs on the host always go to a guest, so no struct irq_desc for them. */
>> +union host_lpi {
>> +    uint64_t data;
>> +    struct {
>> +        uint32_t virt_lpi;
>> +        uint16_t dom_id;
>> +        uint16_t vcpu_id;
>> +    };
>> +};
>>
>>  #define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
>>  /* Global state */
>>  static struct {
>>      /* The global LPI property table, shared by all redistributors. */
>>      uint8_t *lpi_property;
>>      /*
>> +     * A two-level table to lookup LPIs firing on the host and look up the
>> +     * domain and virtual LPI number to inject.
>> +     */
>> +    union host_lpi **host_lpis;
>> +    /*
>>       * Number of physical LPIs the host supports. This is a property of
>>       * the GIC hardware. We depart from the habit of naming these things
>>       * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
>>       * in a different context to differentiate them from "virtual LPIs".
>>       */
>>      unsigned long int nr_host_lpis;
>> +    /* Protects allocation and deallocation of host LPIs, but not the access */
> 
> OK. What does protect the access to host LPIs?
> 
> 
>> +    spinlock_t host_lpis_lock;
>>      unsigned int flags;
>>  } lpi_data;
>>  
>> @@ -50,6 +70,19 @@ struct lpi_redist_data {
>>  static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>>  
>>  #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
>> +#define HOST_LPIS_PER_PAGE      (PAGE_SIZE / sizeof(union host_lpi))
>> +
>> +static union host_lpi *gic_get_host_lpi(uint32_t plpi)
>> +{
>> +    if ( !is_lpi(plpi) || plpi >= MAX_PHYS_LPIS + LPI_OFFSET )
> 
> Arguably if (plpi >= MAX_PHYS_LPIS + LPI_OFFSET) is not really an lpi. I
> would add the plpi >= MAX_PHYS_LPIS + LPI_OFFSET check to is_lpi.
> 
> 
>> +        return NULL;
>> +
>> +    plpi -= LPI_OFFSET;
>> +    if ( !lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE] )
>> +        return NULL;
>> +
>> +    return &lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE][plpi % HOST_LPIS_PER_PAGE];
> 
> I am almost sure this line is longer than 80 characters.. Please double
> check the coding style throughout the series.
> 
> 
>> +}
>>  
>>  /* Stores this redistributor's physical address and ID in a per-CPU variable */
>>  void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
>> @@ -204,15 +237,170 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base)
>>  static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
>>  integer_param("max_lpi_bits", max_lpi_bits);
>>  
>> +/*
>> + * Allocate the 2nd level array for host LPIs. This one holds pointers
>> + * to the page with the actual "union host_lpi" entries. Our LPI limit
>> + * avoids excessive memory usage.
>> + */
>>  int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
>>  {
>> +    int nr_lpi_ptrs;
>> +
>> +    /* We rely on the data structure being atomically accessible. */
>> +    BUILD_BUG_ON(sizeof(union host_lpi) > sizeof(unsigned long));
> 
> Technically, I think that arm32 is able to access uint64_t atomically
> (ldrexd and strexd).
> 
> 
>>      lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
>>  
>> +    spin_lock_init(&lpi_data.host_lpis_lock);
>> +
>> +    nr_lpi_ptrs = MAX_PHYS_LPIS / (PAGE_SIZE / sizeof(union host_lpi));
>> +    lpi_data.host_lpis = xzalloc_array(union host_lpi *, nr_lpi_ptrs);
>> +    if ( !lpi_data.host_lpis )
>> +        return -ENOMEM;
>> +
>>      printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);
>>  
>>      return 0;
>>  }
>>  
>> +/* Must be called with host_lpis_lock held. */
>> +static int find_unused_host_lpi(int start, uint32_t *index)
>> +{
>> +    int chunk;
>> +    uint32_t i = *index;
>> +
>> +    for ( chunk = start; chunk < MAX_PHYS_LPIS / HOST_LPIS_PER_PAGE; chunk++ )
>> +    {
>> +        /* If we hit an unallocated chunk, use entry 0 in that one. */
>> +        if ( !lpi_data.host_lpis[chunk] )
>> +        {
>> +            *index = 0;
>> +            return chunk;
>> +        }
>> +
>> +        /* Find an unallocated entry in this chunk. */
>> +        for ( ; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
>> +        {
>> +            if ( lpi_data.host_lpis[chunk][i].dom_id == INVALID_DOMID )
>> +            {
>> +                *index = i;
>> +                return chunk;
>> +            }
>> +        }
>> +        i = 0;
>> +    }
>> +
>> +    return -1;
>> +}
>> +
>> +/*
>> + * Allocate a block of 32 LPIs on the given host ITS for device "devid",
>> + * starting with "eventid". Put them into the respective ITT by issuing a
>> + * MAPTI command for each of them.
>> + */
>> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
>> +                                  uint32_t host_devid, uint32_t eventid)
>> +{
>> +    static uint32_t next_lpi = 0;
>> +    uint32_t lpi, lpi_idx = next_lpi % HOST_LPIS_PER_PAGE;
>> +    int chunk;
>> +    int i;
>> +
>> +    spin_lock(&lpi_data.host_lpis_lock);
>> +    chunk = find_unused_host_lpi(next_lpi / HOST_LPIS_PER_PAGE, &lpi_idx);
>> +
>> +    if ( chunk == - 1 )          /* rescan for a hole from the beginning */
>> +    {
>> +        lpi_idx = 0;
>> +        chunk = find_unused_host_lpi(0, &lpi_idx);
>> +        if ( chunk == -1 )
>> +        {
>> +            spin_unlock(&lpi_data.host_lpis_lock);
>> +            return -ENOSPC;
>> +        }
>> +    }
>> +
>> +    /* If we hit an unallocated chunk, we initialize it and use entry 0. */
>> +    if ( !lpi_data.host_lpis[chunk] )
>> +    {
>> +        union host_lpi *new_chunk;
>> +
>> +        new_chunk = alloc_xenheap_pages(0, 0);
>> +        if ( !new_chunk )
>> +        {
>> +            spin_unlock(&lpi_data.host_lpis_lock);
>> +            return -ENOMEM;
>> +        }
>> +
>> +        for ( i = 0; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
>> +            new_chunk[i].dom_id = INVALID_DOMID;
>> +
>> +        lpi_data.host_lpis[chunk] = new_chunk;
>> +        lpi_idx = 0;
>> +    }
>> +
>> +    lpi = chunk * HOST_LPIS_PER_PAGE + lpi_idx;
>> +
>> +    for ( i = 0; i < LPI_BLOCK; i++ )
>> +    {
>> +        union host_lpi hlpi;
>> +
>> +        /*
>> +         * Mark this host LPI as belonging to the domain, but don't assign
>> +         * any virtual LPI or a VCPU yet.
>> +         */
>> +        hlpi.virt_lpi = INVALID_LPI;
>> +        hlpi.dom_id = d->domain_id;
>> +        hlpi.vcpu_id = INVALID_DOMID;
>> +        write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
>> +                         hlpi.data);
>> +
>> +        /*
>> +         * Enable this host LPI, so we don't have to do this during the
>> +         * guest's runtime.
>> +         */
>> +        lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
>> +    }
>> +
>> +    /*
>> +     * We have allocated and initialized the host LPI entries, so it's safe
>> +     * to drop the lock now. Access to the structures can be done concurrently
>> +     * as it involves only an atomic uint64_t access.
>> +     */
>> +    spin_unlock(&lpi_data.host_lpis_lock);
>> +
>> +    if ( lpi_data.flags & LPI_PROPTABLE_NEEDS_FLUSHING )
>> +        clean_and_invalidate_dcache_va_range(&lpi_data.lpi_property[lpi],
>> +                                             LPI_BLOCK);
> 
> I would move this right below 
>   lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
> 
> 
>> +    gicv3_its_map_host_events(its, host_devid, eventid, lpi + LPI_OFFSET,
>> +                              LPI_BLOCK);
>> +
>> +    next_lpi = lpi + LPI_BLOCK;
>> +    return lpi + LPI_OFFSET;
>> +}
>> +
>> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi)
>> +{
>> +    union host_lpi *hlpi, empty_lpi = { .dom_id = INVALID_DOMID };
>> +    int i;
>> +
>> +    hlpi = gic_get_host_lpi(lpi);
> 
> I think you need to make sure that "lpi" is the first lpi of a LPI_BLOCK.
> Either that, or gic_get_host_lpi should make sure to always return the
> first lpi in a block. 
> 
> 
>> +    if ( !hlpi )
>> +        return -ENOENT;
>> +
>> +    spin_lock(&lpi_data.host_lpis_lock);
>> +
>> +    for ( i = 0; i < LPI_BLOCK; i++ )
>> +        write_u64_atomic(&hlpi[i].data, empty_lpi.data);
>> +
>> +    /* TODO: Call a function in gic-v3-its.c to send DISCARDs */
>> +
>> +    spin_unlock(&lpi_data.host_lpis_lock);
>> +
>> +    return 0;
>> +}
>> +
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>> index 12bd155..7825575 100644
>> --- a/xen/include/asm-arm/gic.h
>> +++ b/xen/include/asm-arm/gic.h
>> @@ -220,7 +220,12 @@ enum gic_version {
>>      GIC_V3,
>>  };
>>  
>> +#define INVALID_LPI     0
>>  #define LPI_OFFSET      8192
>> +static inline bool is_lpi(unsigned int irq)
>> +{
>> +    return irq >= LPI_OFFSET;
>> +}
>>  
>>  extern enum gic_version gic_hw_version(void);
>>  
>> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
>> index 3421ea0..2387972 100644
>> --- a/xen/include/asm-arm/gic_v3_its.h
>> +++ b/xen/include/asm-arm/gic_v3_its.h
>> @@ -106,6 +106,11 @@
>>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>>  #define HOST_ITS_USES_PTA               (1U << 1)
>>  
>> +#define INVALID_DOMID ((uint16_t)~0)
> 
> We already have DOMID_INVALID
> 
> 
>> +/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
>> +#define LPI_BLOCK                       32
>> +
>>  /* data structure for each hardware ITS */
>>  struct host_its {
>>      struct list_head entry;
>> @@ -151,6 +156,12 @@ int gicv3_its_map_guest_device(struct domain *d,
>>                                 paddr_t host_doorbell, uint32_t host_devid,
>>                                 paddr_t guest_doorbell, uint32_t guest_devid,
>>                                 uint32_t nr_events, bool valid);
>> +int gicv3_its_map_host_events(struct host_its *its,
>> +                              uint32_t host_devid, uint32_t eventid,
>> +                              uint32_t lpi, uint32_t nrevents);
>> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
>> +                                  uint32_t host_devid, uint32_t eventid);
>> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
>>  
>>  #else
>>  
>> -- 
>> 2.9.0
>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-21 23:27       ` Stefano Stabellini
@ 2017-03-23 10:50         ` Andre Przywara
  2017-03-23 17:47           ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-23 10:50 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

Hi,

On 21/03/17 23:27, Stefano Stabellini wrote:
> On Tue, 21 Mar 2017, André Przywara wrote:
>> On 21/03/17 22:57, Stefano Stabellini wrote:
>>> On Thu, 16 Mar 2017, Andre Przywara wrote:
>>>> The ARM GICv3 provides a new kind of interrupt called LPIs.
>>>> The pending bits and the configuration data (priority, enable bits) for
>>>> those LPIs are stored in tables in normal memory, which software has to
>>>> provide to the hardware.
>>>> Allocate the required memory, initialize it and hand it over to each
>>>> redistributor. The maximum number of LPIs to be used can be adjusted with
>>>> the command line option "max_lpi_bits", which defaults to a compile time
>>>> constant exposed in Kconfig.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>
>> ....
>>
>>>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>>>> new file mode 100644
>>>> index 0000000..4f8414b
>>>> --- /dev/null
>>>> +++ b/xen/arch/arm/gic-v3-lpi.c
>>
>> ....
>>
>>>> +/*
>>>> + * Tell a redistributor about the (shared) property table, allocating one
>>>> + * if not already done.
>>>> + */
>>>> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
>>>> +{
>>>> +    uint64_t reg;
>>>> +
>>>> +    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
>>>> +    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
>>>> +    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
>>>> +
>>>> +    /*
>>>> +     * The property table is shared across all redistributors, so allocate
>>>> +     * this only once, but return the same value on subsequent calls.
>>>> +     */
>>>> +    if ( !lpi_data.lpi_property )
>>>> +    {
>>>> +        /* The property table holds one byte per LPI. */
>>>> +        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
>>>> +        void *table = alloc_xenheap_pages(order, 0);
>>>
>>> you investigated the possibility of using _xmalloc to 64K align it?
>>
>> Yes, _xmalloc gives me the alignment, but AFAICT not the physical
>> contiguous memory guarantee the table needs. If there are other ways to
>> satisfy those requirements, I am all ears. Otherwise I'd just add a
>> comment to motivate this.
> 
> OK, but the problem is that I am not sure whether alloc_xenheap_pages
> makes any guarantees on the alignment of the allocated pages. Is it
> possible to allocate 1<<4 pages that are not 64K aligned with it?

My understanding of the code is that it naturally aligns the allocation
to the order you requested: So calling it with (4, 0) would give you (4K
<< 4) = 64K alignment (because it's taken from the 64K bucket).

But anyway the property table needs only to be 4K aligned, isn't it?

I know that the GICv3 is amazing in this aspect: every table has
slightly different requirements when it comes to alignment, support of
physical address space, page sizes and so on...

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
  2017-03-21 20:17   ` Julien Grall
@ 2017-03-23 10:57     ` Andre Przywara
  2017-03-23 17:32       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-23 10:57 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 21/03/17 20:17, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index 2e023d1..bf64c61 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -45,6 +45,10 @@ config ACPI
>>  config HAS_GICV3
>>      bool
>>
>> +config HAS_ITS
>> +        bool "GICv3 ITS MSI controller support"
>> +        depends on HAS_GICV3
> 
> IIRC, we discussed that GICv3 ITS will be a technical preview for Xen
> 4.9. I think case I think it should depends on EXPERT mode to avoid been
> shipped by default.
> 
>> +
>>  endmenu
>>
>>  menu "ARM errata workaround via the alternative framework"
> 
> [...]
> 
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> new file mode 100644
>> index 0000000..4056e5b
>> --- /dev/null
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -0,0 +1,73 @@
>> +/*
>> + * xen/arch/arm/gic-v3-its.c
>> + *
>> + * ARM GICv3 Interrupt Translation Service (ITS) support
>> + *
>> + * Copyright (C) 2016,2017 - ARM Ltd
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; under version 2 of the License.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/lib.h>
>> +#include <asm/gic_v3_defs.h>
>> +#include <asm/gic_v3_its.h>
>> +
>> +LIST_HEAD(host_its_list);
> 
> On the previous version we agreed that host_its_list should be confined
> to gic-v3-its.c. So why is it still exported?

Because I need it later on in vgic-v3.c, when creating virtual ITSes
that mimic the host ITSes, see patch 25/27. I haven't found a neat
solution for that yet.

For the rest of the code you will see that every other reference to the
host_its_list has been removed, so it is *mostly* confined to
gic-v3-its.c by now.

> 
> [...]
> 
>> diff --git a/xen/include/asm-arm/gic_v3_its.h
>> b/xen/include/asm-arm/gic_v3_its.h
>> new file mode 100644
>> index 0000000..765a655
>> --- /dev/null
>> +++ b/xen/include/asm-arm/gic_v3_its.h
>> @@ -0,0 +1,67 @@
> 
> 
> [...]
> 
>> +#else
>> +
>> +static LIST_HEAD(host_its_list);
> 
> This is quite ugly and could really be avoided if the host_its_list was
> confined as we agreed in the previous version.

I think you said that you are fine with an export if it cannot easily be
avoided.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-21 21:23   ` Julien Grall
@ 2017-03-23 14:40     ` Andre Przywara
  2017-03-23 17:42       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-23 14:40 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 21/03/17 21:23, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>> index bf64c61..86f7b53 100644
>> --- a/xen/arch/arm/Kconfig
>> +++ b/xen/arch/arm/Kconfig
>> @@ -49,6 +49,21 @@ config HAS_ITS
>>          bool "GICv3 ITS MSI controller support"
>>          depends on HAS_GICV3
>>
>> +config MAX_PHYS_LPI_BITS
>> +        depends on HAS_ITS
>> +        int "Maximum bits for GICv3 host LPIs (14-32)"
>> +        range 14 32
>> +        default "20"
>> +        help
>> +          Specifies the maximum number of LPIs (in bits) Xen should take
>> +          care of. The host ITS may provide support for a very large
>> number
>> +          of supported LPIs, for all of which we may not want to
>> allocate
>> +          memory, so this number here allows to limit this.
>> +          Xen itself does not know how many LPIs domains will ever need
>> +          beforehand.
>> +          This can be overridden on the command line with the
>> max_lpi_bits
>> +          parameter.
> 
> I continue to disagree on this Kconfig option. There is no sensible
> default value and I don't see how a distribution will be able to pick-up
> one value here.
> 
> If the number of LPIs have to be restricted, this should be done via the
> command line or a per-platform option.

So are you opting for taking the hardware provided value, unless there
is a command line option or a per-platform limit?

Please keep in mind that the situation here is different from the device
table:
- There is no indirect table option for the property and pending table.
  Any redistributor supporting 32 bits worth of LPIs would lead to a
  4GB property table and 512MB pending table allocation.
- An OS like Linux can make sensible assumptions about the number of
  LPIs needed and only allocate as much LPIs as required. Linux for
  instance uses at most 65536. Xen cannot easily make this decision.
  So we would need to go as high as possible, but not too high to not
  waste memory. I see different tradeoffs here between a distribution
  targeting servers or another one aiming at some embedded devices, for
  instance. So I would expect exactly this decision to be covered by a
  Kconfig option.
- In contrast to the device ID, which is guest provided, the host LPI
  number is chosen by Xen. We allocate them in chunks of 32, starting
  at the beginning and just counting up. So we can limit them without
  loosing too much flexibility, because it's not that sparse as device
  IDs, for instance.

> I have already made my point on previous e-mails so I am not going to
> argue more.

So as I mentioned before, I am happy to loose the Kconfig option, but
then we need some sensible default value. In our case we could be cheeky
here for now and just use the Linux value, because a Linux Dom0 would be
the only user. But that doesn't sound very future proof, though this may
not matter for 4.9.

>> +
>>  endmenu
>>
>>  menu "ARM errata workaround via the alternative framework"
>> diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
>> index 54860e0..02a8737 100644
>> --- a/xen/arch/arm/Makefile
>> +++ b/xen/arch/arm/Makefile
>> @@ -19,6 +19,7 @@ obj-y += gic.o
>>  obj-y += gic-v2.o
>>  obj-$(CONFIG_HAS_GICV3) += gic-v3.o
>>  obj-$(CONFIG_HAS_ITS) += gic-v3-its.o
>> +obj-$(CONFIG_HAS_ITS) += gic-v3-lpi.o
>>  obj-y += guestcopy.o
>>  obj-y += hvm.o
>>  obj-y += io.o
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>> new file mode 100644
>> index 0000000..4f8414b
>> --- /dev/null
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -0,0 +1,201 @@
>> +/*
>> + * xen/arch/arm/gic-v3-lpi.c
>> + *
>> + * ARM GICv3 Locality-specific Peripheral Interrupts (LPI) support
>> + *
>> + * Copyright (C) 2016,2017 - ARM Ltd
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License as published by
>> + * the Free Software Foundation; under version 2 of the License.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + *
>> + * You should have received a copy of the GNU General Public License
>> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
>> + */
>> +
>> +#include <xen/lib.h>
>> +#include <xen/mm.h>
>> +#include <asm/gic.h>
>> +#include <asm/gic_v3_defs.h>
>> +#include <asm/gic_v3_its.h>
>> +#include <asm/io.h>
>> +#include <asm/page.h>
>> +
>> +#define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
> 
> NIT: Newline here please.
> 
>> +/* Global state */
>> +static struct {
>> +    /* The global LPI property table, shared by all redistributors. */
>> +    uint8_t *lpi_property;
>> +    /*
>> +     * Number of physical LPIs the host supports. This is a property of
>> +     * the GIC hardware. We depart from the habit of naming these things
>> +     * "physical" in Xen, as the GICv3/4 spec uses the term "physical
>> LPI"
>> +     * in a different context to differentiate them from "virtual LPIs".
>> +     */
>> +    unsigned long int nr_host_lpis;
> 
> Why unsigned long?

Because someone requested that I store the actual number of LPIs, not
the number of bits required to encode them. So I need to cover the case
of exactly 32 bits worth of LPIs, which u32 cannot hold.

>> +    unsigned int flags;
>> +} lpi_data;
>> +
>> +struct lpi_redist_data {
>> +    void                *pending_table;
>> +};
>> +
>> +static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>> +
>> +#define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
> 
> This is fairly confusing. When I read "nr_host_lpis" I understand that
> you store the number of LPIs. But in fact you store the maximum LPI ID.

One of the reasons I argued for storing the number of bits ....

> Also it is very unclear the difference between MAX_PHYS_LPIS and
> nr_host_lpis and will likely cause trouble in the future.
> 
> So it looks like to me that nr_host_lpis should be named
> max_host_lpis_ids and MAX_PHYS_LPIS should be MAX_NR_PHYS_LPIS.
> 
> Also, please stay consistent with the naming. If you use host then use
> host everywhere and not a mix between phys and host for the same purpose.

OK, can do.

>> +
>> +static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>> +{
>> +    uint64_t val;
>> +    unsigned int order;
>> +    void *pendtable;
>> +
>> +    if ( this_cpu(lpi_redist).pending_table )
>> +        return -EBUSY;
>> +
>> +    val  = GIC_BASER_CACHE_RaWaWb <<
>> GICR_PENDBASER_INNER_CACHEABILITY_SHIFT;
>> +    val |= GIC_BASER_CACHE_SameAsInner <<
>> GICR_PENDBASER_OUTER_CACHEABILITY_SHIFT;
>> +    val |= GIC_BASER_InnerShareable <<
>> GICR_PENDBASER_SHAREABILITY_SHIFT;
>> +
>> +    /*
>> +     * The pending table holds one bit per LPI and even covers bits for
>> +     * interrupt IDs below 8192, so we allocate the full range.
>> +     * The GICv3 imposes a 64KB alignment requirement, also requires
>> +     * physically contigious memory.
> 
> The bit after the comma seems quite obvious. Both xalloc and
> alloc_xenheap_pages will ensure that the region will be contiguous in
> the physical address space.

Mmh, somehow I missed that when I tried to decide which allocation
function is the right one. But indeed xmalloc() eventually calls
alloc_xenheap_pages() with the order of alignment, so we get the same
guarantees.

> 
> [...]
> 
>> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
>> +{
>> +    uint64_t reg;
>> +
>> +    reg  = GIC_BASER_CACHE_RaWaWb <<
>> GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
>> +    reg |= GIC_BASER_CACHE_SameAsInner <<
>> GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
>> +    reg |= GIC_BASER_InnerShareable <<
>> GICR_PROPBASER_SHAREABILITY_SHIFT;
>> +
>> +    /*
>> +     * The property table is shared across all redistributors, so
>> allocate
>> +     * this only once, but return the same value on subsequent calls.
>> +     */
>> +    if ( !lpi_data.lpi_property )
>> +    {
>> +        /* The property table holds one byte per LPI. */
>> +        unsigned int order =
>> get_order_from_bytes(lpi_data.nr_host_lpis);
>> +        void *table = alloc_xenheap_pages(order, 0);
>> +
>> +        if ( !table )
>> +            return -ENOMEM;
>> +
>> +        /* Make sure the physical address can be encoded in the
>> register. */
>> +        if ( (virt_to_maddr(table) & ~GENMASK(51, 12)) )
>> +        {
>> +            free_xenheap_pages(table, 0);
>> +            return -ERANGE;
>> +        }
>> +        memset(table, GIC_PRI_IRQ | LPI_PROP_RES1, MAX_PHYS_LPIS);
>> +        clean_and_invalidate_dcache_va_range(table, MAX_PHYS_LPIS);
>> +        lpi_data.lpi_property = table;
>> +    }
>> +
>> +    /* Encode the number of bits needed, minus one */
>> +    reg |= ((fls(lpi_data.nr_host_lpis - 1) - 1) << 0);
> 
> As said on the previous version, please avoid hardcoded shift.
> 
> [...]
> 
>> +int gicv3_lpi_init_rdist(void __iomem * rdist_base)
>> +{
>> +    uint32_t reg;
>> +    uint64_t table_reg;
>> +    int ret;
>> +
>> +    /* We don't support LPIs without an ITS. */
>> +    if ( !gicv3_its_host_has_its() )
>> +        return -ENODEV;
>> +
>> +    /* Make sure LPIs are disabled before setting up the tables. */
>> +    reg = readl_relaxed(rdist_base + GICR_CTLR);
>> +    if ( reg & GICR_CTLR_ENABLE_LPIS )
>> +        return -EBUSY;
>> +
>> +    ret = gicv3_lpi_allocate_pendtable(&table_reg);
>> +    if (ret)
>> +        return ret;
>> +    writeq_relaxed(table_reg, rdist_base + GICR_PENDBASER);
> 
> Same question as on the previous version. The cacheability and
> shareability may not stick. This was a bug fix in Linux (see commit
> 241a386) and I am struggling to understand why Xen would not be affected?

Oh, OK, we need to remove cacheability if shareability is denied, though
that sounds like a hardware bug to me if this is not observed by the
redistributor.
The reason I didn't care about it here was that software doesn't use the
pending table, so there would be nothing on the code side to be done in
case the redistributor shoots down one of our requests.
The mentioned Linux patch covers all ITS tables and the property table
as well.

>> +
>> +    return gicv3_lpi_set_proptable(rdist_base);
>> +}
>> +
>> +static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
>> +integer_param("max_lpi_bits", max_lpi_bits);
>> +
>> +int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
>> +{
> 
> 
> Please add a comment to explain why you don't sanitize the value from
> the command line.
> 
>> +    lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
> 
> nr_host_lpis is "unsigned long" so why are you using BIT_ULL?

Right, creeping over from Linux where the ITS is supported by ARM(32) as
well.

>> +
>> +    printk("GICv3: using at most %ld LPIs on the host.\n",
>> MAX_PHYS_LPIS);
> 
> s/%ld/%lu/
> 
> [...]
> 
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index 1512521..ed78363 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -548,6 +548,9 @@ static void __init gicv3_dist_init(void)
>>      type = readl_relaxed(GICD + GICD_TYPER);
>>      nr_lines = 32 * ((type & GICD_TYPE_LINES) + 1);
>>
>> +    if ( type & GICD_TYPE_LPIS )
>> +        gicv3_lpi_init_host_lpis(((type >> GICD_TYPE_ID_BITS_SHIFT) &
>> 0x1f) + 1);
> 
> Again, a macro has been suggested on in the last 2 versions to avoid
> hardcoded value. You said you will fix it and it is not done. Please do it.

Sorry for that, I did this for GITS_TYPER_DEVICE_ID_BITS, so mentally
ticked this one off as well.
Will be fixed.

> 
> [...]
> 
>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>> index 836a103..12bd155 100644
>> --- a/xen/include/asm-arm/gic.h
>> +++ b/xen/include/asm-arm/gic.h
>> @@ -220,6 +220,8 @@ enum gic_version {
>>      GIC_V3,
>>  };
>>
>> +#define LPI_OFFSET      8192
> 
> My comments on the previous version about LPI_OFFSET are not addressed.
> And one question was left unanswered.

Do you mean this?
>> It would make much sense to have this definition moved in irq.h
>> close to NR_IRQS.

Admittedly I missed that over the next question...
I think I wasn't convinced in the first place to put this GIC specific
detail into the generic irq.h, but I see that it makes sense since we
have the do_LPI() prototype in there as well.

>> Also, I am a bit surprised that NR_IRQS & co has not been modified.
>> Is there any reason for that?

But I answered on this one.
Is there anything else you want to have answered?

> 
>> +
>>  extern enum gic_version gic_hw_version(void);
>>
>>  /* Program the IRQ type into the GIC */
> 
> [...]
> 
>> diff --git a/xen/include/asm-arm/gic_v3_its.h
>> b/xen/include/asm-arm/gic_v3_its.h
>> index 765a655..219d109 100644
>> --- a/xen/include/asm-arm/gic_v3_its.h
>> +++ b/xen/include/asm-arm/gic_v3_its.h
>> @@ -40,6 +40,11 @@ void gicv3_its_dt_init(const struct dt_device_node
>> *node);
>>
>>  bool gicv3_its_host_has_its(void);
>>
>> +int gicv3_lpi_init_rdist(void __iomem * rdist_base);
>> +
>> +/* Initialize the host structures for LPIs. */
>> +int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
> 
> In the implementation, the parameter is called "hw_lpi_bits". Please
> stay consistent.

Yeah, this header file is a constant source of merge conflicts when
squashing fixes and reworks, which makes it a pain to work with. Sorry
for that.

Cheers,
Andre.

>> +
>>  #else
>>
>>  static LIST_HEAD(host_its_list);
>> @@ -53,6 +58,15 @@ static inline bool gicv3_its_host_has_its(void)
>>      return false;
>>  }
>>
>> +static inline int gicv3_lpi_init_rdist(void __iomem * rdist_base)
>> +{
>> +    return -ENODEV;
>> +}
>> +
>> +static inline int gicv3_lpi_init_host_lpis(unsigned int nr_lpis)
> 
> Ditto.
> 
>> +{
>> +    return 0;
>> +}
>>  #endif /* CONFIG_HAS_ITS */
>>
>>  #endif
>>
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT
  2017-03-23 10:57     ` Andre Przywara
@ 2017-03-23 17:32       ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-23 17:32 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 23/03/17 10:57, Andre Przywara wrote:
> On 21/03/17 20:17, Julien Grall wrote:
>> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index 2e023d1..bf64c61 100644
>>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>>> new file mode 100644
>>> index 0000000..4056e5b
>>> --- /dev/null
>>> +++ b/xen/arch/arm/gic-v3-its.c
>>> @@ -0,0 +1,73 @@
>>> +/*
>>> + * xen/arch/arm/gic-v3-its.c
>>> + *
>>> + * ARM GICv3 Interrupt Translation Service (ITS) support
>>> + *
>>> + * Copyright (C) 2016,2017 - ARM Ltd
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU General Public License as published by
>>> + * the Free Software Foundation; under version 2 of the License.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> + * GNU General Public License for more details.
>>> + *
>>> + * You should have received a copy of the GNU General Public License
>>> + * along with this program; If not, see <http://www.gnu.org/licenses/>.
>>> + */
>>> +
>>> +#include <xen/lib.h>
>>> +#include <asm/gic_v3_defs.h>
>>> +#include <asm/gic_v3_its.h>
>>> +
>>> +LIST_HEAD(host_its_list);
>>
>> On the previous version we agreed that host_its_list should be confined
>> to gic-v3-its.c. So why is it still exported?
>
> Because I need it later on in vgic-v3.c, when creating virtual ITSes
> that mimic the host ITSes, see patch 25/27. I haven't found a neat
> solution for that yet.

I will have a look.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 14:40     ` Andre Przywara
@ 2017-03-23 17:42       ` Julien Grall
  2017-03-23 17:45         ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-23 17:42 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari



On 23/03/17 14:40, Andre Przywara wrote:
> Hi,

Hi Andre,

>
> On 21/03/17 21:23, Julien Grall wrote:
>> Hi Andre,
>>
>> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>>> diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
>>> index bf64c61..86f7b53 100644
>>> --- a/xen/arch/arm/Kconfig
>>> +++ b/xen/arch/arm/Kconfig
>>> @@ -49,6 +49,21 @@ config HAS_ITS
>>>          bool "GICv3 ITS MSI controller support"
>>>          depends on HAS_GICV3
>>>
>>> +config MAX_PHYS_LPI_BITS
>>> +        depends on HAS_ITS
>>> +        int "Maximum bits for GICv3 host LPIs (14-32)"
>>> +        range 14 32
>>> +        default "20"
>>> +        help
>>> +          Specifies the maximum number of LPIs (in bits) Xen should take
>>> +          care of. The host ITS may provide support for a very large
>>> number
>>> +          of supported LPIs, for all of which we may not want to
>>> allocate
>>> +          memory, so this number here allows to limit this.
>>> +          Xen itself does not know how many LPIs domains will ever need
>>> +          beforehand.
>>> +          This can be overridden on the command line with the
>>> max_lpi_bits
>>> +          parameter.
>>
>> I continue to disagree on this Kconfig option. There is no sensible
>> default value and I don't see how a distribution will be able to pick-up
>> one value here.
>>
>> If the number of LPIs have to be restricted, this should be done via the
>> command line or a per-platform option.
>
> So are you opting for taking the hardware provided value, unless there
> is a command line option or a per-platform limit?
>
> Please keep in mind that the situation here is different from the device
> table:
> - There is no indirect table option for the property and pending table.
>   Any redistributor supporting 32 bits worth of LPIs would lead to a
>   4GB property table and 512MB pending table allocation.
> - An OS like Linux can make sensible assumptions about the number of
>   LPIs needed and only allocate as much LPIs as required. Linux for
>   instance uses at most 65536. Xen cannot easily make this decision.
>   So we would need to go as high as possible, but not too high to not
>   waste memory. I see different tradeoffs here between a distribution
>   targeting servers or another one aiming at some embedded devices, for
>   instance. So I would expect exactly this decision to be covered by a
>   Kconfig option.

Hence why a parameter on the command line is the better place for that. 
The decision is left to the user.

>> I have already made my point on previous e-mails so I am not going to
>> argue more.
>
> So as I mentioned before, I am happy to loose the Kconfig option, but
> then we need some sensible default value. In our case we could be cheeky
> here for now and just use the Linux value, because a Linux Dom0 would be
> the only user. But that doesn't sound very future proof, though this may
> not matter for 4.9.

I don't think we need a sensible default value and IHMO there is none. I 
would left the user to decide the exact number.

>>> +/* Global state */
>>> +static struct {
>>> +    /* The global LPI property table, shared by all redistributors. */
>>> +    uint8_t *lpi_property;
>>> +    /*
>>> +     * Number of physical LPIs the host supports. This is a property of
>>> +     * the GIC hardware. We depart from the habit of naming these things
>>> +     * "physical" in Xen, as the GICv3/4 spec uses the term "physical
>>> LPI"
>>> +     * in a different context to differentiate them from "virtual LPIs".
>>> +     */
>>> +    unsigned long int nr_host_lpis;
>>
>> Why unsigned long?
>
> Because someone requested that I store the actual number of LPIs, not
> the number of bits required to encode them. So I need to cover the case
> of exactly 32 bits worth of LPIs, which u32 cannot hold.

Fair enough.

[...]

>>> +int gicv3_lpi_init_rdist(void __iomem * rdist_base)
>>> +{
>>> +    uint32_t reg;
>>> +    uint64_t table_reg;
>>> +    int ret;
>>> +
>>> +    /* We don't support LPIs without an ITS. */
>>> +    if ( !gicv3_its_host_has_its() )
>>> +        return -ENODEV;
>>> +
>>> +    /* Make sure LPIs are disabled before setting up the tables. */
>>> +    reg = readl_relaxed(rdist_base + GICR_CTLR);
>>> +    if ( reg & GICR_CTLR_ENABLE_LPIS )
>>> +        return -EBUSY;
>>> +
>>> +    ret = gicv3_lpi_allocate_pendtable(&table_reg);
>>> +    if (ret)
>>> +        return ret;
>>> +    writeq_relaxed(table_reg, rdist_base + GICR_PENDBASER);
>>
>> Same question as on the previous version. The cacheability and
>> shareability may not stick. This was a bug fix in Linux (see commit
>> 241a386) and I am struggling to understand why Xen would not be affected?
>
> Oh, OK, we need to remove cacheability if shareability is denied, though
> that sounds like a hardware bug to me if this is not observed by the
> redistributor.
> The reason I didn't care about it here was that software doesn't use the
> pending table, so there would be nothing on the code side to be done in
> case the redistributor shoots down one of our requests.
> The mentioned Linux patch covers all ITS tables and the property table
> as well.

The patch was sent by Marc and I trust him to do the right things. 
Looking at the commit message, it is to avoid cacheable traffic. Unless 
you give me a good reason for not doing that, please do the change.

I don't want to end-up digging into weird bug later on because we 
decided that "the hardware is not supposed to do that".

[...]

>>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>>> index 836a103..12bd155 100644
>>> --- a/xen/include/asm-arm/gic.h
>>> +++ b/xen/include/asm-arm/gic.h
>>> @@ -220,6 +220,8 @@ enum gic_version {
>>>      GIC_V3,
>>>  };
>>>
>>> +#define LPI_OFFSET      8192
>>
>> My comments on the previous version about LPI_OFFSET are not addressed.
>> And one question was left unanswered.
>
> Do you mean this?
>>> It would make much sense to have this definition moved in irq.h
>>> close to NR_IRQS.
>
> Admittedly I missed that over the next question...
> I think I wasn't convinced in the first place to put this GIC specific
> detail into the generic irq.h, but I see that it makes sense since we
> have the do_LPI() prototype in there as well.
>
>>> Also, I am a bit surprised that NR_IRQS & co has not been modified.
>>> Is there any reason for that?
>
> But I answered on this one.
> Is there anything else you want to have answered?

Yes, the question in  <dae3afdb-d64d-a81b-7df3-ed8a61e00ad1@arm.com>.

"Can we have a comment on top of NR_IRQS explaining why LPIs are not 
taken into account?"

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 17:42       ` Julien Grall
@ 2017-03-23 17:45         ` Stefano Stabellini
  2017-03-23 17:49           ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-23 17:45 UTC (permalink / raw)
  To: Julien Grall
  Cc: Andre Przywara, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari, xen-devel

On Thu, 23 Mar 2017, Julien Grall wrote:
> On 23/03/17 14:40, Andre Przywara wrote:
> > Hi,
> 
> Hi Andre,
> 
> > 
> > On 21/03/17 21:23, Julien Grall wrote:
> > > Hi Andre,
> > > 
> > > On 03/16/2017 11:20 AM, Andre Przywara wrote:
> > > > diff --git a/xen/arch/arm/Kconfig b/xen/arch/arm/Kconfig
> > > > index bf64c61..86f7b53 100644
> > > > --- a/xen/arch/arm/Kconfig
> > > > +++ b/xen/arch/arm/Kconfig
> > > > @@ -49,6 +49,21 @@ config HAS_ITS
> > > >          bool "GICv3 ITS MSI controller support"
> > > >          depends on HAS_GICV3
> > > > 
> > > > +config MAX_PHYS_LPI_BITS
> > > > +        depends on HAS_ITS
> > > > +        int "Maximum bits for GICv3 host LPIs (14-32)"
> > > > +        range 14 32
> > > > +        default "20"
> > > > +        help
> > > > +          Specifies the maximum number of LPIs (in bits) Xen should
> > > > take
> > > > +          care of. The host ITS may provide support for a very large
> > > > number
> > > > +          of supported LPIs, for all of which we may not want to
> > > > allocate
> > > > +          memory, so this number here allows to limit this.
> > > > +          Xen itself does not know how many LPIs domains will ever need
> > > > +          beforehand.
> > > > +          This can be overridden on the command line with the
> > > > max_lpi_bits
> > > > +          parameter.
> > > 
> > > I continue to disagree on this Kconfig option. There is no sensible
> > > default value and I don't see how a distribution will be able to pick-up
> > > one value here.
> > > 
> > > If the number of LPIs have to be restricted, this should be done via the
> > > command line or a per-platform option.
> > 
> > So are you opting for taking the hardware provided value, unless there
> > is a command line option or a per-platform limit?
> > 
> > Please keep in mind that the situation here is different from the device
> > table:
> > - There is no indirect table option for the property and pending table.
> >   Any redistributor supporting 32 bits worth of LPIs would lead to a
> >   4GB property table and 512MB pending table allocation.
> > - An OS like Linux can make sensible assumptions about the number of
> >   LPIs needed and only allocate as much LPIs as required. Linux for
> >   instance uses at most 65536. Xen cannot easily make this decision.
> >   So we would need to go as high as possible, but not too high to not
> >   waste memory. I see different tradeoffs here between a distribution
> >   targeting servers or another one aiming at some embedded devices, for
> >   instance. So I would expect exactly this decision to be covered by a
> >   Kconfig option.
> 
> Hence why a parameter on the command line is the better place for that. The
> decision is left to the user.
> 
> > > I have already made my point on previous e-mails so I am not going to
> > > argue more.
> > 
> > So as I mentioned before, I am happy to loose the Kconfig option, but
> > then we need some sensible default value. In our case we could be cheeky
> > here for now and just use the Linux value, because a Linux Dom0 would be
> > the only user. But that doesn't sound very future proof, though this may
> > not matter for 4.9.
> 
> I don't think we need a sensible default value and IHMO there is none. I would
> left the user to decide the exact number.

In that case, the command line parameter becomes mandatory: we need to
force the user to specify it as we do for dom0_mem today.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 10:50         ` Andre Przywara
@ 2017-03-23 17:47           ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-23 17:47 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari



On 23/03/17 10:50, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 21/03/17 23:27, Stefano Stabellini wrote:
>> On Tue, 21 Mar 2017, André Przywara wrote:
>>> On 21/03/17 22:57, Stefano Stabellini wrote:
>>>> On Thu, 16 Mar 2017, Andre Przywara wrote:
>>>>> The ARM GICv3 provides a new kind of interrupt called LPIs.
>>>>> The pending bits and the configuration data (priority, enable bits) for
>>>>> those LPIs are stored in tables in normal memory, which software has to
>>>>> provide to the hardware.
>>>>> Allocate the required memory, initialize it and hand it over to each
>>>>> redistributor. The maximum number of LPIs to be used can be adjusted with
>>>>> the command line option "max_lpi_bits", which defaults to a compile time
>>>>> constant exposed in Kconfig.
>>>>>
>>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>
>>> ....
>>>
>>>>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>>>>> new file mode 100644
>>>>> index 0000000..4f8414b
>>>>> --- /dev/null
>>>>> +++ b/xen/arch/arm/gic-v3-lpi.c
>>>
>>> ....
>>>
>>>>> +/*
>>>>> + * Tell a redistributor about the (shared) property table, allocating one
>>>>> + * if not already done.
>>>>> + */
>>>>> +static int gicv3_lpi_set_proptable(void __iomem * rdist_base)
>>>>> +{
>>>>> +    uint64_t reg;
>>>>> +
>>>>> +    reg  = GIC_BASER_CACHE_RaWaWb << GICR_PROPBASER_INNER_CACHEABILITY_SHIFT;
>>>>> +    reg |= GIC_BASER_CACHE_SameAsInner << GICR_PROPBASER_OUTER_CACHEABILITY_SHIFT;
>>>>> +    reg |= GIC_BASER_InnerShareable << GICR_PROPBASER_SHAREABILITY_SHIFT;
>>>>> +
>>>>> +    /*
>>>>> +     * The property table is shared across all redistributors, so allocate
>>>>> +     * this only once, but return the same value on subsequent calls.
>>>>> +     */
>>>>> +    if ( !lpi_data.lpi_property )
>>>>> +    {
>>>>> +        /* The property table holds one byte per LPI. */
>>>>> +        unsigned int order = get_order_from_bytes(lpi_data.nr_host_lpis);
>>>>> +        void *table = alloc_xenheap_pages(order, 0);
>>>>
>>>> you investigated the possibility of using _xmalloc to 64K align it?
>>>
>>> Yes, _xmalloc gives me the alignment, but AFAICT not the physical
>>> contiguous memory guarantee the table needs. If there are other ways to
>>> satisfy those requirements, I am all ears. Otherwise I'd just add a
>>> comment to motivate this.

As already said numerous times, _x*alloc will guarantee contiguous 
physical memory. The same as kzalloc.

>>
>> OK, but the problem is that I am not sure whether alloc_xenheap_pages
>> makes any guarantees on the alignment of the allocated pages. Is it
>> possible to allocate 1<<4 pages that are not 64K aligned with it?
>
> My understanding of the code is that it naturally aligns the allocation
> to the order you requested: So calling it with (4, 0) would give you (4K
> << 4) = 64K alignment (because it's taken from the 64K bucket).
>
> But anyway the property table needs only to be 4K aligned, isn't it?
>
> I know that the GICv3 is amazing in this aspect: every table has
> slightly different requirements when it comes to alignment, support of
> physical address space, page sizes and so on...

alloc_xenheap_pages will guarantee allocation aligned to the (1 << 
order) * PAGE_SIZE.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 17:45         ` Stefano Stabellini
@ 2017-03-23 17:49           ` Julien Grall
  2017-03-23 18:01             ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-23 17:49 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Andre Przywara, Shanker Donthineni, Vijay Kilari, xen-devel

Hi Stefano,

On 23/03/17 17:45, Stefano Stabellini wrote:
> On Thu, 23 Mar 2017, Julien Grall wrote:
>>> So as I mentioned before, I am happy to loose the Kconfig option, but
>>> then we need some sensible default value. In our case we could be cheeky
>>> here for now and just use the Linux value, because a Linux Dom0 would be
>>> the only user. But that doesn't sound very future proof, though this may
>>> not matter for 4.9.
>>
>> I don't think we need a sensible default value and IHMO there is none. I would
>> left the user to decide the exact number.
>
> In that case, the command line parameter becomes mandatory: we need to
> force the user to specify it as we do for dom0_mem today.

Not really. We should use the hardware value by default. If a user 
thinks the number allocated is too big for his use case, then it can 
limit using the command line.

Anyway, I will not oppose to make this command option mandatory when ITS 
is in use.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-23 10:21     ` Andre Przywara
@ 2017-03-23 17:52       ` Stefano Stabellini
  2017-03-24 11:54         ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-23 17:52 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Thu, 23 Mar 2017, Andre Przywara wrote:
> Hi,
> 
> On 22/03/17 23:38, Stefano Stabellini wrote:
> > On Thu, 16 Mar 2017, Andre Przywara wrote:
> >> The number of LPIs on a host can be potentially huge (millions),
> >> although in practise will be mostly reasonable. So prematurely allocating
> >> an array of struct irq_desc's for each LPI is not an option.
> >> However Xen itself does not care about LPIs, as every LPI will be injected
> >> into a guest (Dom0 for now).
> >> Create a dense data structure (8 Bytes) for each LPI which holds just
> >> enough information to determine the virtual IRQ number and the VCPU into
> >> which the LPI needs to be injected.
> >> Also to not artificially limit the number of LPIs, we create a 2-level
> >> table for holding those structures.
> >> This patch introduces functions to initialize these tables and to
> >> create, lookup and destroy entries for a given LPI.
> >> By using the naturally atomic access guarantee the native uint64_t data
> >> type gives us, we allocate and access LPI information in a way that does
> >> not require a lock.
> >>
> >> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> >> ---
> >>  xen/arch/arm/gic-v3-its.c        |  90 ++++++++++++++++++-
> >>  xen/arch/arm/gic-v3-lpi.c        | 188 +++++++++++++++++++++++++++++++++++++++
> >>  xen/include/asm-arm/gic.h        |   5 ++
> >>  xen/include/asm-arm/gic_v3_its.h |  11 +++
> >>  4 files changed, 292 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> >> index 60b15b5..ed14d95 100644
> >> --- a/xen/arch/arm/gic-v3-its.c
> >> +++ b/xen/arch/arm/gic-v3-its.c
> >> @@ -148,6 +148,20 @@ static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
> >>      return its_send_command(its, cmd);
> >>  }
> >>  
> >> +static int its_send_cmd_mapti(struct host_its *its,
> >> +                              uint32_t deviceid, uint32_t eventid,
> >> +                              uint32_t pintid, uint16_t icid)
> >> +{
> >> +    uint64_t cmd[4];
> >> +
> >> +    cmd[0] = GITS_CMD_MAPTI | ((uint64_t)deviceid << 32);
> >> +    cmd[1] = eventid | ((uint64_t)pintid << 32);
> >> +    cmd[2] = icid;
> >> +    cmd[3] = 0x00;
> >> +
> >> +    return its_send_command(its, cmd);
> >> +}
> >> +
> >>  static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
> >>                               unsigned int cpu)
> >>  {
> >> @@ -180,6 +194,19 @@ static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> >>      return its_send_command(its, cmd);
> >>  }
> >>  
> >> +static int its_send_cmd_inv(struct host_its *its,
> >> +                            uint32_t deviceid, uint32_t eventid)
> >> +{
> >> +    uint64_t cmd[4];
> >> +
> >> +    cmd[0] = GITS_CMD_INV | ((uint64_t)deviceid << 32);
> >> +    cmd[1] = eventid;
> >> +    cmd[2] = 0x00;
> >> +    cmd[3] = 0x00;
> >> +
> >> +    return its_send_command(its, cmd);
> >> +}
> >> +
> >>  /* Set up the (1:1) collection mapping for the given host CPU. */
> >>  int gicv3_its_setup_collection(unsigned int cpu)
> >>  {
> >> @@ -462,7 +489,7 @@ int gicv3_its_init(void)
> >>  
> >>  static int remove_mapped_guest_device(struct its_devices *dev)
> >>  {
> >> -    int ret;
> >> +    int ret, i;
> >>  
> >>      if ( dev->hw_its )
> >>      {
> >> @@ -471,11 +498,19 @@ static int remove_mapped_guest_device(struct its_devices *dev)
> >>              return ret;
> >>      }
> >>  
> >> +    /*
> >> +     * The only error the function below would return is -ENOENT, in which
> >> +     * case there is nothing to free here. So we just ignore it.
> >> +     */
> >> +    for ( i = 0; i < DIV_ROUND_UP(dev->eventids, LPI_BLOCK); i++ )
> >> +        gicv3_free_host_lpi_block(dev->hw_its, dev->host_lpis[i]);
> >> +
> >>      ret = gicv3_its_wait_commands(dev->hw_its);
> >>      if ( ret )
> >>          return ret;
> >>  
> >>      xfree(dev->itt_addr);
> >> +    xfree(dev->host_lpis);
> >>      xfree(dev);
> >>  
> >>      return 0;
> >> @@ -513,6 +548,36 @@ static int compare_its_guest_devices(struct its_devices *dev,
> >>  }
> >>  
> >>  /*
> >> + * On the host ITS @its, map @nr_events consecutive LPIs.
> >> + * The mapping connects a device @devid and event @eventid pair to LPI @lpi,
> >> + * increasing both @eventid and @lpi to cover the number of requested LPIs.
> >> + */
> >> +int gicv3_its_map_host_events(struct host_its *its,
> >> +                              uint32_t devid, uint32_t eventid, uint32_t lpi,
> >> +                              uint32_t nr_events)
> >> +{
> >> +    uint32_t i;
> >> +    int ret;
> >> +
> >> +    for ( i = 0; i < nr_events; i++ )
> >> +    {
> >> +        /* For now we map every host LPI to host CPU 0 */
> >> +        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
> >> +        if ( ret )
> >> +            return ret;
> >> +        ret = its_send_cmd_inv(its, devid, eventid + i);
> >> +        if ( ret )
> >> +            return ret;
> >> +    }
> >> +
> >> +    ret = its_send_cmd_sync(its, 0);
> >> +    if ( ret )
> >> +        return ret;
> >> +
> >> +    return gicv3_its_wait_commands(its);
> >> +}
> >> +
> >> +/*
> >>   * Map a hardware device, identified by a certain host ITS and its device ID
> >>   * to domain d, a guest ITS (identified by its doorbell address) and device ID.
> >>   * Also provide the number of events (MSIs) needed for that device.
> >> @@ -528,7 +593,7 @@ int gicv3_its_map_guest_device(struct domain *d,
> >>      struct host_its *hw_its;
> >>      struct its_devices *dev = NULL, *temp;
> >>      struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> >> -    int ret = -ENOENT;
> >> +    int ret = -ENOENT, i;
> >>  
> >>      hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> >>      if ( !hw_its )
> >> @@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
> >>      if ( !dev )
> >>          goto out_unlock;
> >>  
> >> +    dev->host_lpis = xzalloc_array(uint32_t,
> >> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
> >> +    if ( !dev->host_lpis )
> >> +        goto out_unlock;
> > 
> > The expectation is that we are going to allocate far more than 32 lpis
> > for one device, possibly thousands, right?
> 
> Mmh, what makes you think so?
> My understanding is that you have one MSI per interrupt "reason" (TX,
> RX, error, for instance). And even if you have multiple queues, each
> connected to separate MSIs, that won't go into the thousands. MSI-X for
> instance is *limited* to 2048 MSIs.
> 
> But in reality I dont' see nearly that many MSIs, mostly less than a
> dozen per device. And since 32 is the old PCI MSI limit, I thought this
> would be a good starting point. So most of the time we just need one block.
> 
> I believe this could add up with virtual functions, but those are
> separate devices, right?

Yes, you are right about that, but even 2048 can make a difference when
taken 32 at a time: an array of 64 entries vs. 1 entry (if the
allocation is fully contiguous), and that for each device, in the case
of SR-IOV they can be hundreds.


On Thu, 23 Mar 2017, Julien Grall wrote:
> Not answering directly to the question. But I think this is an improvement and
> not necessary for a first version.
> 
> I would like to see this series merged in Xen 4.9 as a tech preview, so I
> would suggest to gather this list of improvement (maybe on Jira?) and defer
> them for Xen 4.10. They would need to be addressed before making ITS stable.
> Stefano, does it sound good to you?

Yes, I am OK with not having this in 4.9, but we need to keep track
(Jira or otherwise) of these things.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 17:49           ` Julien Grall
@ 2017-03-23 18:01             ` Stefano Stabellini
  2017-03-23 18:21               ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-23 18:01 UTC (permalink / raw)
  To: Julien Grall
  Cc: Andre Przywara, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari, xen-devel

On Thu, 23 Mar 2017, Julien Grall wrote:
> Hi Stefano,
> 
> On 23/03/17 17:45, Stefano Stabellini wrote:
> > On Thu, 23 Mar 2017, Julien Grall wrote:
> > > > So as I mentioned before, I am happy to loose the Kconfig option, but
> > > > then we need some sensible default value. In our case we could be cheeky
> > > > here for now and just use the Linux value, because a Linux Dom0 would be
> > > > the only user. But that doesn't sound very future proof, though this may
> > > > not matter for 4.9.
> > > 
> > > I don't think we need a sensible default value and IHMO there is none. I
> > > would
> > > left the user to decide the exact number.
> > 
> > In that case, the command line parameter becomes mandatory: we need to
> > force the user to specify it as we do for dom0_mem today.
> 
> Not really. We should use the hardware value by default. If a user thinks the
> number allocated is too big for his use case, then it can limit using the
> command line.
>
> Anyway, I will not oppose to make this command option mandatory when ITS is in
> use.

Andre wrote:

  Any redistributor supporting 32 bits worth of LPIs would lead to a
  4GB property table and 512MB pending table allocation.

Let's assume that such scenario is realistic (if it is not, then this
discussion is fruitless), in this case the user most surely is not going
to want to use the hardware provided value. But how can she knows it?
How can she find out that she is wasting too much memory on her system?
Is there an easy and obvious way to know?

She could find out if Xen printed a big warning such as:

  USING 4G OF MEMORY FOR ITS PROPTABLE, CONSIDER PASSING max_lpi_bits to XEN 

but to do that, we need a threshold value in Xen, above which the
hypervisor prints the warning. But if we have a threshold value in Xen,
then we might as well consider making it the default ceiling: Xen uses
the hardware provided value, unless it's greater than threshold, in that
case it uses threshold and prints a warning, for example:

  LIMITING ITS PROPTABLE MEMORY TO 1G, CHANGE IT WITH max_lpi_bits PARAMETER


Does it make sense?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 18:01             ` Stefano Stabellini
@ 2017-03-23 18:21               ` Andre Przywara
  2017-03-24 11:45                 ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-23 18:21 UTC (permalink / raw)
  To: Stefano Stabellini, Julien Grall
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 23/03/17 18:01, Stefano Stabellini wrote:
> On Thu, 23 Mar 2017, Julien Grall wrote:
>> Hi Stefano,
>>
>> On 23/03/17 17:45, Stefano Stabellini wrote:
>>> On Thu, 23 Mar 2017, Julien Grall wrote:
>>>>> So as I mentioned before, I am happy to loose the Kconfig option, but
>>>>> then we need some sensible default value. In our case we could be cheeky
>>>>> here for now and just use the Linux value, because a Linux Dom0 would be
>>>>> the only user. But that doesn't sound very future proof, though this may
>>>>> not matter for 4.9.
>>>>
>>>> I don't think we need a sensible default value and IHMO there is none. I
>>>> would
>>>> left the user to decide the exact number.
>>>
>>> In that case, the command line parameter becomes mandatory: we need to
>>> force the user to specify it as we do for dom0_mem today.
>>
>> Not really. We should use the hardware value by default. If a user thinks the
>> number allocated is too big for his use case, then it can limit using the
>> command line.
>>
>> Anyway, I will not oppose to make this command option mandatory when ITS is in
>> use.
> 
> Andre wrote:
> 
>   Any redistributor supporting 32 bits worth of LPIs would lead to a
>   4GB property table and 512MB pending table allocation.
> 
> Let's assume that such scenario is realistic (if it is not, then this
> discussion is fruitless), in this case the user most surely is not going
> to want to use the hardware provided value. But how can she knows it?
> How can she find out that she is wasting too much memory on her system?
> Is there an easy and obvious way to know?
> 
> She could find out if Xen printed a big warning such as:
> 
>   USING 4G OF MEMORY FOR ITS PROPTABLE, CONSIDER PASSING max_lpi_bits to XEN 
> 
> but to do that, we need a threshold value in Xen, above which the
> hypervisor prints the warning. But if we have a threshold value in Xen,
> then we might as well consider making it the default ceiling: Xen uses
> the hardware provided value, unless it's greater than threshold, in that
> case it uses threshold and prints a warning, for example:
> 
>   LIMITING ITS PROPTABLE MEMORY TO 1G, CHANGE IT WITH max_lpi_bits PARAMETER
> 
> 
> Does it make sense?

Yes, that is exactly what I was after.
In contrast to the number of device IDs I think the number of LPI bits
is _not_ a value that software should use directly, it's more a limit,
actually a GICv3 implementation choice (how wide the LPI ID fields
internally are, for instance).

So do we want to limit to 1GB and warn starting at 256MB?

Cheers,
Andre.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-16 11:20 ` [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array Andre Przywara
  2017-03-22 23:38   ` Stefano Stabellini
@ 2017-03-23 19:08   ` Julien Grall
  2017-04-03 19:30     ` Andre Przywara
  1 sibling, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-23 19:08 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 16/03/17 11:20, Andre Przywara wrote:
> The number of LPIs on a host can be potentially huge (millions),
> although in practise will be mostly reasonable. So prematurely allocating
> an array of struct irq_desc's for each LPI is not an option.
> However Xen itself does not care about LPIs, as every LPI will be injected
> into a guest (Dom0 for now).
> Create a dense data structure (8 Bytes) for each LPI which holds just
> enough information to determine the virtual IRQ number and the VCPU into
> which the LPI needs to be injected.
> Also to not artificially limit the number of LPIs, we create a 2-level
> table for holding those structures.
> This patch introduces functions to initialize these tables and to
> create, lookup and destroy entries for a given LPI.
> By using the naturally atomic access guarantee the native uint64_t data
> type gives us, we allocate and access LPI information in a way that does
> not require a lock.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        |  90 ++++++++++++++++++-
>  xen/arch/arm/gic-v3-lpi.c        | 188 +++++++++++++++++++++++++++++++++++++++
>  xen/include/asm-arm/gic.h        |   5 ++
>  xen/include/asm-arm/gic_v3_its.h |  11 +++
>  4 files changed, 292 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 60b15b5..ed14d95 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -148,6 +148,20 @@ static int its_send_cmd_sync(struct host_its *its, unsigned int cpu)
>      return its_send_command(its, cmd);
>  }
>
> +static int its_send_cmd_mapti(struct host_its *its,
> +                              uint32_t deviceid, uint32_t eventid,
> +                              uint32_t pintid, uint16_t icid)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_MAPTI | ((uint64_t)deviceid << 32);
> +    cmd[1] = eventid | ((uint64_t)pintid << 32);
> +    cmd[2] = icid;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
>  static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>                               unsigned int cpu)
>  {
> @@ -180,6 +194,19 @@ static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>      return its_send_command(its, cmd);
>  }
>
> +static int its_send_cmd_inv(struct host_its *its,
> +                            uint32_t deviceid, uint32_t eventid)
> +{
> +    uint64_t cmd[4];
> +
> +    cmd[0] = GITS_CMD_INV | ((uint64_t)deviceid << 32);
> +    cmd[1] = eventid;
> +    cmd[2] = 0x00;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -462,7 +489,7 @@ int gicv3_its_init(void)
>
>  static int remove_mapped_guest_device(struct its_devices *dev)
>  {
> -    int ret;
> +    int ret, i;
>
>      if ( dev->hw_its )
>      {
> @@ -471,11 +498,19 @@ static int remove_mapped_guest_device(struct its_devices *dev)
>              return ret;
>      }
>
> +    /*
> +     * The only error the function below would return is -ENOENT, in which
> +     * case there is nothing to free here. So we just ignore it.
> +     */

The function gicv3_free_host_lpi_block will only be used here. And to be 
fair, if you try to free something that does not exist then it is not 
really an error...

So I would prefer to see the function to be void.

> +    for ( i = 0; i < DIV_ROUND_UP(dev->eventids, LPI_BLOCK); i++ )
> +        gicv3_free_host_lpi_block(dev->hw_its, dev->host_lpis[i]);

Again, without looking at the implementation of 
gicv3_free_host_lpi_block, I think the usage of the function is very 
confusing. When I read host_lpis, I expect to see one LPI per event. But 
instead it be the first LPI of a block. The lack of documentation of the 
field in its_devices does not help to understand what's going on.

So please add some documentation and probably renaming some fields.

Lastly, you have not answered to my question: should not we discard the 
LPIs before removing the device? Or does MAPD take care for you?"

> +
>      ret = gicv3_its_wait_commands(dev->hw_its);
>      if ( ret )
>          return ret;

I know I asked to wait the command, thank you for addressing it. But 
now, if the function fail you will end-up to leak memory. This is not 
better than failing to wait commands.

>
>      xfree(dev->itt_addr);
> +    xfree(dev->host_lpis);
>      xfree(dev);
>
>      return 0;
> @@ -513,6 +548,36 @@ static int compare_its_guest_devices(struct its_devices *dev,
>  }
>
>  /*
> + * On the host ITS @its, map @nr_events consecutive LPIs.
> + * The mapping connects a device @devid and event @eventid pair to LPI @lpi,
> + * increasing both @eventid and @lpi to cover the number of requested LPIs.
> + */
> +int gicv3_its_map_host_events(struct host_its *its,
> +                              uint32_t devid, uint32_t eventid, uint32_t lpi,
> +                              uint32_t nr_events)
> +{
> +    uint32_t i;
> +    int ret;
> +
> +    for ( i = 0; i < nr_events; i++ )
> +    {
> +        /* For now we map every host LPI to host CPU 0 */
> +        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
> +        if ( ret )
> +            return ret;
> +        ret = its_send_cmd_inv(its, devid, eventid + i);


So the spec allows up to 32KB event per device. As all the LPIs will be 
routed to CPU0 (e.g collection 0), it would be more efficient to do an 
INVALL. I would be happy to defer that to post Xen 4.9, but the rest 
needs to be fixed.

Furthermore, what if the queue is full? AFAIU, you will return an error 
but it is not propagate. So Xen will think the device has been mapped 
even if it is not true.

I think we need to have a plan here as this may likely happen if a 
device has many MSI and/or the queue is nearly full.

> +        if ( ret )
> +            return ret;
> +    }
> +
> +    ret = its_send_cmd_sync(its, 0);
> +    if ( ret )
> +        return ret;
> +
> +    return gicv3_its_wait_commands(its);
> +}
> +
> +/*
>   * Map a hardware device, identified by a certain host ITS and its device ID
>   * to domain d, a guest ITS (identified by its doorbell address) and device ID.
>   * Also provide the number of events (MSIs) needed for that device.
> @@ -528,7 +593,7 @@ int gicv3_its_map_guest_device(struct domain *d,
>      struct host_its *hw_its;
>      struct its_devices *dev = NULL, *temp;
>      struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> -    int ret = -ENOENT;
> +    int ret = -ENOENT, i;

Again, i should be unsigned.

>
>      hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>      if ( !hw_its )
> @@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
>      if ( !dev )
>          goto out_unlock;
>
> +    dev->host_lpis = xzalloc_array(uint32_t,
> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));

Rather than having DIV_ROUND_UP spread everywhere. Would not be easier 
to round_up nr_events once for all?

> +    if ( !dev->host_lpis )
> +        goto out_unlock;
> +
>      ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
>                              virt_to_maddr(itt_addr), true);
>      if ( ret )
> @@ -591,10 +661,26 @@ int gicv3_its_map_guest_device(struct domain *d,
>
>      spin_unlock(&d->arch.vgic.its_devices_lock);
>
> +    /*
> +     * Map all host LPIs within this device already. We can't afford to queue
> +     * any host ITS commands later on during the guest's runtime.
> +     */
> +    for ( i = 0; i < DIV_ROUND_UP(nr_events, LPI_BLOCK); i++ )
> +    {
> +        ret = gicv3_allocate_host_lpi_block(hw_its, d, host_devid,
> +                                            i * LPI_BLOCK);
> +        if ( ret < 0 )
> +            goto out;

This sounds quite wrong to me. If you fail to allocate a block, you will 
end-up with half of the device configured. However, you are freeing the 
memory but not reverting the ITS command. This is going to end up very 
badly.

Please be really careful with the error path. Most of them are not 
correct and I don't want to end up dealing with security issue later on.

> +
> +        dev->host_lpis[i] = ret;

Hmmm... as I said on the previous version you should not use the return 
for both error code and LPIs. LPIs are encoded on 32-bit, sooner or 
later there will be a clash between the error and the LPI.

> +    }
> +
>      return 0;
>
>  out_unlock:
>      spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +out:
>      if ( dev )
>          xfree(dev->host_lpis);
>      xfree(itt_addr);
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 51d7425..59d3ba4 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -20,24 +20,44 @@
>
>  #include <xen/lib.h>
>  #include <xen/mm.h>
> +#include <xen/sched.h>
> +#include <asm/atomic.h>
> +#include <asm/domain.h>
>  #include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
>  #include <asm/gic_v3_its.h>
>  #include <asm/io.h>
>  #include <asm/page.h>
>
> +/* LPIs on the host always go to a guest, so no struct irq_desc for them. */
> +union host_lpi {
> +    uint64_t data;
> +    struct {
> +        uint32_t virt_lpi;
> +        uint16_t dom_id;
> +        uint16_t vcpu_id;
> +    };
> +};
> +
>  #define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
>  /* Global state */
>  static struct {
>      /* The global LPI property table, shared by all redistributors. */
>      uint8_t *lpi_property;
>      /*
> +     * A two-level table to lookup LPIs firing on the host and look up the
> +     * domain and virtual LPI number to inject.
> +     */
> +    union host_lpi **host_lpis;
> +    /*
>       * Number of physical LPIs the host supports. This is a property of
>       * the GIC hardware. We depart from the habit of naming these things
>       * "physical" in Xen, as the GICv3/4 spec uses the term "physical LPI"
>       * in a different context to differentiate them from "virtual LPIs".
>       */
>      unsigned long int nr_host_lpis;
> +    /* Protects allocation and deallocation of host LPIs, but not the access */
> +    spinlock_t host_lpis_lock;
>      unsigned int flags;
>  } lpi_data;
>
> @@ -50,6 +70,19 @@ struct lpi_redist_data {
>  static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>
>  #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
> +#define HOST_LPIS_PER_PAGE      (PAGE_SIZE / sizeof(union host_lpi))
> +
> +static union host_lpi *gic_get_host_lpi(uint32_t plpi)
> +{
> +    if ( !is_lpi(plpi) || plpi >= MAX_PHYS_LPIS + LPI_OFFSET )
> +        return NULL;
> +
> +    plpi -= LPI_OFFSET;
> +    if ( !lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE] )
> +        return NULL;
> +
> +    return &lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE][plpi % HOST_LPIS_PER_PAGE];
> +}
>
>  /* Stores this redistributor's physical address and ID in a per-CPU variable */
>  void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
> @@ -204,15 +237,170 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base)
>  static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
>  integer_param("max_lpi_bits", max_lpi_bits);
>
> +/*
> + * Allocate the 2nd level array for host LPIs. This one holds pointers
> + * to the page with the actual "union host_lpi" entries. Our LPI limit
> + * avoids excessive memory usage.
> + */
>  int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
>  {
> +    int nr_lpi_ptrs;

unsigned int please.

> +
> +    /* We rely on the data structure being atomically accessible. */
> +    BUILD_BUG_ON(sizeof(union host_lpi) > sizeof(unsigned long));
> +
>      lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
>
> +    spin_lock_init(&lpi_data.host_lpis_lock);
> +
> +    nr_lpi_ptrs = MAX_PHYS_LPIS / (PAGE_SIZE / sizeof(union host_lpi));
> +    lpi_data.host_lpis = xzalloc_array(union host_lpi *, nr_lpi_ptrs);
> +    if ( !lpi_data.host_lpis )
> +        return -ENOMEM;
> +
>      printk("GICv3: using at most %ld LPIs on the host.\n", MAX_PHYS_LPIS);
>
>      return 0;
>  }
>
> +/* Must be called with host_lpis_lock held. */

Again, this is a call for adding an ASSERT in the function.

> +static int find_unused_host_lpi(int start, uint32_t *index)

start should probably be unsigned.

> +{
> +    int chunk;

Ditto.

> +    uint32_t i = *index;
> +
> +    for ( chunk = start; chunk < MAX_PHYS_LPIS / HOST_LPIS_PER_PAGE; chunk++ )
> +    {
> +        /* If we hit an unallocated chunk, use entry 0 in that one. */
> +        if ( !lpi_data.host_lpis[chunk] )
> +        {
> +            *index = 0;
> +            return chunk;
> +        }
> +
> +        /* Find an unallocated entry in this chunk. */
> +        for ( ; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
> +        {
> +            if ( lpi_data.host_lpis[chunk][i].dom_id == INVALID_DOMID )
> +            {
> +                *index = i;
> +                return chunk;
> +            }
> +        }
> +        i = 0;
> +    }
> +
> +    return -1;
> +}
> +
> +/*
> + * Allocate a block of 32 LPIs on the given host ITS for device "devid",
> + * starting with "eventid". Put them into the respective ITT by issuing a
> + * MAPTI command for each of them.
> + */
> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
> +                                  uint32_t host_devid, uint32_t eventid)

Again, most of the parameters here is just for calling back ITS and the 
caller should all in hand. So I would much prefer to avoid a call chain 
ITS -> LPI -> ITS and make the LPI code ITS agnostic.

Furthermore, the return value is both used to return an error and the 
LPI. Given that an LPI is encoded on 32-bit, sooner or later there will 
be a clash with between the error and the LPI. So it would be wise to 
dissociate the error code and the LPIs.

The prototype of this function would like:

int gicv3_allocate_host_lpi_block(struct domain *domain, uint32_t 
*first_lpi);

> +{
> +    static uint32_t next_lpi = 0;
> +    uint32_t lpi, lpi_idx = next_lpi % HOST_LPIS_PER_PAGE;
> +    int chunk;
> +    int i;
> +
> +    spin_lock(&lpi_data.host_lpis_lock);
> +    chunk = find_unused_host_lpi(next_lpi / HOST_LPIS_PER_PAGE, &lpi_idx);
> +
> +    if ( chunk == - 1 )          /* rescan for a hole from the beginning */
> +    {
> +        lpi_idx = 0;
> +        chunk = find_unused_host_lpi(0, &lpi_idx);
> +        if ( chunk == -1 )
> +        {
> +            spin_unlock(&lpi_data.host_lpis_lock);
> +            return -ENOSPC;
> +        }

The algo does not seem to have changed since the previous version. 
Looking at it again, my understanding is you will always try to allocate 
forward. So if the current chunk is full, you will allocate the next one 
rather than looking whether a previous chunk has space available. This 
will result to allocate more memory than necessary.

Similarly unused chunk could be freed to save memory.

To be clear, I am asking for confirmation, documentation, and listing as 
an improvement.

> +    }
> +
> +    /* If we hit an unallocated chunk, we initialize it and use entry 0. */
> +    if ( !lpi_data.host_lpis[chunk] )
> +    {
> +        union host_lpi *new_chunk;
> +
> +        new_chunk = alloc_xenheap_pages(0, 0);

As said on the previous version, please use alloc_xenheap_page as you 
only allocate one page.

Also, when NUMA support will be added we may want to take into account 
the node associated to the device saving us some time when reading the 
memory. You don't need to handle that now, but a TODO would be quite 
helpful.

> +        if ( !new_chunk )
> +        {
> +            spin_unlock(&lpi_data.host_lpis_lock);
> +            return -ENOMEM;
> +        }
> +
> +        for ( i = 0; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
> +            new_chunk[i].dom_id = INVALID_DOMID;
> +
> +        lpi_data.host_lpis[chunk] = new_chunk;
> +        lpi_idx = 0;
> +    }
> +
> +    lpi = chunk * HOST_LPIS_PER_PAGE + lpi_idx;
> +
> +    for ( i = 0; i < LPI_BLOCK; i++ )
> +    {
> +        union host_lpi hlpi;
> +
> +        /*
> +         * Mark this host LPI as belonging to the domain, but don't assign
> +         * any virtual LPI or a VCPU yet.
> +         */
> +        hlpi.virt_lpi = INVALID_LPI;
> +        hlpi.dom_id = d->domain_id;
> +        hlpi.vcpu_id = INVALID_DOMID;

Again, please don't mix vcpu and domain. If INVALID_VCPUID does not 
exist then it might be worth adding one.

> +        write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
> +                         hlpi.data);
> +
> +        /*
> +         * Enable this host LPI, so we don't have to do this during the
> +         * guest's runtime.
> +         */
> +        lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
> +    }
> +
> +    /*
> +     * We have allocated and initialized the host LPI entries, so it's safe
> +     * to drop the lock now. Access to the structures can be done concurrently
> +     * as it involves only an atomic uint64_t access.
> +     */
> +    spin_unlock(&lpi_data.host_lpis_lock);
> +
> +    if ( lpi_data.flags & LPI_PROPTABLE_NEEDS_FLUSHING )
> +        clean_and_invalidate_dcache_va_range(&lpi_data.lpi_property[lpi],
> +                                             LPI_BLOCK);
> +
> +    gicv3_its_map_host_events(its, host_devid, eventid, lpi + LPI_OFFSET,
> +                              LPI_BLOCK);

See my comment at the top of the function.

> +
> +    next_lpi = lpi + LPI_BLOCK;

Newline here please.

> +    return lpi + LPI_OFFSET;
> +}
> +
> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi)
> +{
> +    union host_lpi *hlpi, empty_lpi = { .dom_id = INVALID_DOMID };
> +    int i;
> +
> +    hlpi = gic_get_host_lpi(lpi);
> +    if ( !hlpi )
> +        return -ENOENT;
> +
> +    spin_lock(&lpi_data.host_lpis_lock);
> +
> +    for ( i = 0; i < LPI_BLOCK; i++ )
> +        write_u64_atomic(&hlpi[i].data, empty_lpi.data);
> +
> +    /* TODO: Call a function in gic-v3-its.c to send DISCARDs */

It is quite worrying to see a TODO in a code that is supposed to be 
merged really soon.

I think this should be done by the caller and not here. You also need to 
disable the interrupts which has not been done here.

> +
> +    spin_unlock(&lpi_data.host_lpis_lock);
> +
> +    return 0;
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
> index 12bd155..7825575 100644
> --- a/xen/include/asm-arm/gic.h
> +++ b/xen/include/asm-arm/gic.h
> @@ -220,7 +220,12 @@ enum gic_version {
>      GIC_V3,
>  };
>
> +#define INVALID_LPI     0
>  #define LPI_OFFSET      8192

Again, newline here please.

> +static inline bool is_lpi(unsigned int irq)
> +{
> +    return irq >= LPI_OFFSET;
> +}

Again, I think both INVALID_LPI and is_lpi should be moved in irq.h.

>
>  extern enum gic_version gic_hw_version(void);
>
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 3421ea0..2387972 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -106,6 +106,11 @@
>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>  #define HOST_ITS_USES_PTA               (1U << 1)
>
> +#define INVALID_DOMID ((uint16_t)~0)

Again, rather than defining your own invalid domid, it would be better 
to use the one defined in xen.h (see DOMID_INVALID).

> +
> +/* We allocate LPIs on the hosts in chunks of 32 to reduce handling overhead. */
> +#define LPI_BLOCK                       32
> +
>  /* data structure for each hardware ITS */
>  struct host_its {
>      struct list_head entry;
> @@ -151,6 +156,12 @@ int gicv3_its_map_guest_device(struct domain *d,
>                                 paddr_t host_doorbell, uint32_t host_devid,
>                                 paddr_t guest_doorbell, uint32_t guest_devid,
>                                 uint32_t nr_events, bool valid);
> +int gicv3_its_map_host_events(struct host_its *its,
> +                              uint32_t host_devid, uint32_t eventid,
> +                              uint32_t lpi, uint32_t nrevents);
> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain *d,
> +                                  uint32_t host_devid, uint32_t eventid);
> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
>
>  #else
>
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-22 23:44   ` Stefano Stabellini
@ 2017-03-23 20:08     ` André Przywara
  2017-03-24 10:59       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: André Przywara @ 2017-03-23 20:08 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

On 22/03/17 23:44, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Andre Przywara wrote:
>> For the same reason that allocating a struct irq_desc for each
>> possible LPI is not an option, having a struct pending_irq for each LPI
>> is also not feasible. However we actually only need those when an
>> interrupt is on a vCPU (or is about to be injected).
>> Maintain a list of those structs that we can use for the lifecycle of
>> a guest LPI. We allocate new entries if necessary, however reuse
>> pre-owned entries whenever possible.
>> I added some locking around this list here, however my gut feeling is
>> that we don't need one because this a per-VCPU structure anyway.
>> If someone could confirm this, I'd be grateful.
>> Teach the existing VGIC functions to find the right pointer when being
>> given a virtual LPI number.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> 
> Still all past comments are unaddress ;-(

Sorry for that. I just see that the reviews for those and later emails
were not in the same thread as the other ones, so they were swamped in
my inbox and didn't show up in my TODO thread. This might be due to
races between direct email and list bounces. Interestingly this seems to
affect previous emails as well, which isn't really an excuse, but
explains my perceived rudeness in not addressing your comments (simply
because I didn't see them).
I will try to sort this mess out in the next few days and send a new
version early next week.
If you know from the top of your head any serious architectural change
requests you raised, it would be nice if you could repeat them now, so
that I can address them early enough.
I will now sort my mails by author name to find any comment I might have
missed before.

Sorry again for the mess!

Cheers,
Andre.

> It makes very hard to continue doing reviews. I am sorry but I'll have
> to stop reviewing until I see a series with my past comments addressed.
> I encourage Julien to do the same.
> 
> 
>> ---
>>  xen/arch/arm/gic.c           |  3 +++
>>  xen/arch/arm/vgic-v3.c       |  3 +++
>>  xen/arch/arm/vgic.c          | 64 +++++++++++++++++++++++++++++++++++++++++---
>>  xen/include/asm-arm/domain.h |  2 ++
>>  xen/include/asm-arm/vgic.h   | 14 ++++++++++
>>  5 files changed, 83 insertions(+), 3 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
>> index a5348f2..bd3c032 100644
>> --- a/xen/arch/arm/gic.c
>> +++ b/xen/arch/arm/gic.c
>> @@ -509,6 +509,9 @@ static void gic_update_one_lr(struct vcpu *v, int i)
>>                  struct vcpu *v_target = vgic_get_target_vcpu(v, irq);
>>                  irq_set_affinity(p->desc, cpumask_of(v_target->processor));
>>              }
>> +            /* If this was an LPI, mark this struct as available again. */
>> +            if ( is_lpi(p->irq) )
>> +                p->irq = 0;
>>          }
>>      }
>>  }
>> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
>> index 1fadb00..b0653c2 100644
>> --- a/xen/arch/arm/vgic-v3.c
>> +++ b/xen/arch/arm/vgic-v3.c
>> @@ -1426,6 +1426,9 @@ static int vgic_v3_vcpu_init(struct vcpu *v)
>>      if ( v->vcpu_id == last_cpu || (v->vcpu_id == (d->max_vcpus - 1)) )
>>          v->arch.vgic.flags |= VGIC_V3_RDIST_LAST;
>>  
>> +    spin_lock_init(&v->arch.vgic.pending_lpi_list_lock);
>> +    INIT_LIST_HEAD(&v->arch.vgic.pending_lpi_list);
>> +
>>      return 0;
>>  }
>>  
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 364d5f0..e5cfa54 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -30,6 +30,8 @@
>>  
>>  #include <asm/mmio.h>
>>  #include <asm/gic.h>
>> +#include <asm/gic_v3_defs.h>
>> +#include <asm/gic_v3_its.h>
>>  #include <asm/vgic.h>
>>  
>>  static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v, int rank)
>> @@ -61,7 +63,7 @@ struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq)
>>      return vgic_get_rank(v, rank);
>>  }
>>  
>> -static void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
>> +void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
>>  {
>>      INIT_LIST_HEAD(&p->inflight);
>>      INIT_LIST_HEAD(&p->lr_queue);
>> @@ -244,10 +246,14 @@ struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq)
>>  
>>  static int vgic_get_virq_priority(struct vcpu *v, unsigned int virq)
>>  {
>> -    struct vgic_irq_rank *rank = vgic_rank_irq(v, virq);
>> +    struct vgic_irq_rank *rank;
>>      unsigned long flags;
>>      int priority;
>>  
>> +    if ( is_lpi(virq) )
>> +        return vgic_lpi_get_priority(v->domain, virq);
>> +
>> +    rank = vgic_rank_irq(v, virq);
>>      vgic_lock_rank(v, rank, flags);
>>      priority = rank->priority[virq & INTERRUPT_RANK_MASK];
>>      vgic_unlock_rank(v, rank, flags);
>> @@ -446,13 +452,63 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
>>      return true;
>>  }
>>  
>> +/*
>> + * Holding struct pending_irq's for each possible virtual LPI in each domain
>> + * requires too much Xen memory, also a malicious guest could potentially
>> + * spam Xen with LPI map requests. We cannot cover those with (guest allocated)
>> + * ITS memory, so we use a dynamic scheme of allocating struct pending_irq's
>> + * on demand.
>> + */
>> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
>> +                                   bool allocate)
>> +{
>> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
>> +
>> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
>> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
>> +    {
>> +        if ( lpi_irq->pirq.irq == lpi )
>> +        {
>> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>> +            return &lpi_irq->pirq;
>> +        }
>> +
>> +        if ( lpi_irq->pirq.irq == 0 && !empty )
>> +            empty = lpi_irq;
>> +    }
>> +
>> +    if ( !allocate )
>> +    {
>> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>> +        return NULL;
>> +    }
>> +
>> +    if ( !empty )
>> +    {
>> +        empty = xzalloc(struct lpi_pending_irq);
>> +        vgic_init_pending_irq(&empty->pirq, lpi);
>> +        list_add_tail(&empty->entry, &v->arch.vgic.pending_lpi_list);
>> +    } else
>> +    {
>> +        empty->pirq.status = 0;
>> +        empty->pirq.irq = lpi;
>> +    }
>> +
>> +    spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>> +
>> +    return &empty->pirq;
>> +}
>> +
>>  struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>>  {
>>      struct pending_irq *n;
>> +
>>      /* Pending irqs allocation strategy: the first vgic.nr_spis irqs
>>       * are used for SPIs; the rests are used for per cpu irqs */
>>      if ( irq < 32 )
>>          n = &v->arch.vgic.pending_irqs[irq];
>> +    else if ( is_lpi(irq) )
>> +        n = lpi_to_pending(v, irq, true);
>>      else
>>          n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>>      return n;
>> @@ -480,7 +536,7 @@ void vgic_clear_pending_irqs(struct vcpu *v)
>>  void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>>  {
>>      uint8_t priority;
>> -    struct pending_irq *iter, *n = irq_to_pending(v, virq);
>> +    struct pending_irq *iter, *n;
>>      unsigned long flags;
>>      bool running;
>>  
>> @@ -488,6 +544,8 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>>  
>>      spin_lock_irqsave(&v->arch.vgic.lock, flags);
>>  
>> +    n = irq_to_pending(v, virq);
>> +
>>      /* vcpu offline */
>>      if ( test_bit(_VPF_down, &v->pause_flags) )
>>      {
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 00b9c1a..f44a84b 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -257,6 +257,8 @@ struct arch_vcpu
>>          paddr_t rdist_base;
>>  #define VGIC_V3_RDIST_LAST  (1 << 0)        /* last vCPU of the rdist */
>>          uint8_t flags;
>> +        struct list_head pending_lpi_list;
>> +        spinlock_t pending_lpi_list_lock;   /* protects the pending_lpi_list */
>>      } vgic;
>>  
>>      /* Timer registers  */
>> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
>> index 467333c..8f1099c 100644
>> --- a/xen/include/asm-arm/vgic.h
>> +++ b/xen/include/asm-arm/vgic.h
>> @@ -83,6 +83,12 @@ struct pending_irq
>>      struct list_head lr_queue;
>>  };
>>  
>> +struct lpi_pending_irq
>> +{
>> +    struct list_head entry;
>> +    struct pending_irq pirq;
>> +};
>> +
>>  #define NR_INTERRUPT_PER_RANK   32
>>  #define INTERRUPT_RANK_MASK (NR_INTERRUPT_PER_RANK - 1)
>>  
>> @@ -296,13 +302,21 @@ extern struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq);
>>  extern void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq);
>>  extern void vgic_vcpu_inject_spi(struct domain *d, unsigned int virq);
>>  extern void vgic_clear_pending_irqs(struct vcpu *v);
>> +extern void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq);
>>  extern struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq);
>>  extern struct pending_irq *spi_to_pending(struct domain *d, unsigned int irq);
>> +extern struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int irq,
>> +                                          bool allocate);
>>  extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v, int b, int n, int s);
>>  extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
>>  extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
>>  extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
>>  extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
>> +/* placeholder function until the property table gets introduced */
>> +static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
>> +{
>> +    return 0xa;
>> +}
>>  extern void register_vgic_ops(struct domain *d, const struct vgic_ops *ops);
>>  int vgic_v2_init(struct domain *d, int *mmio_count);
>>  int vgic_v3_init(struct domain *d, int *mmio_count);
>> -- 
>> 2.9.0
>>


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-23 20:08     ` André Przywara
@ 2017-03-24 10:59       ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 10:59 UTC (permalink / raw)
  To: André Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/23/2017 08:08 PM, André Przywara wrote:
> On 22/03/17 23:44, Stefano Stabellini wrote:
>> On Thu, 16 Mar 2017, Andre Przywara wrote:
>>> For the same reason that allocating a struct irq_desc for each
>>> possible LPI is not an option, having a struct pending_irq for each LPI
>>> is also not feasible. However we actually only need those when an
>>> interrupt is on a vCPU (or is about to be injected).
>>> Maintain a list of those structs that we can use for the lifecycle of
>>> a guest LPI. We allocate new entries if necessary, however reuse
>>> pre-owned entries whenever possible.
>>> I added some locking around this list here, however my gut feeling is
>>> that we don't need one because this a per-VCPU structure anyway.
>>> If someone could confirm this, I'd be grateful.
>>> Teach the existing VGIC functions to find the right pointer when being
>>> given a virtual LPI number.
>>>
>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>
>> Still all past comments are unaddress ;-(
>
> Sorry for that. I just see that the reviews for those and later emails
> were not in the same thread as the other ones, so they were swamped in
> my inbox and didn't show up in my TODO thread. This might be due to
> races between direct email and list bounces. Interestingly this seems to
> affect previous emails as well, which isn't really an excuse, but
> explains my perceived rudeness in not addressing your comments (simply
> because I didn't see them).
> I will try to sort this mess out in the next few days and send a new
> version early next week.
> If you know from the top of your head any serious architectural change
> requests you raised, it would be nice if you could repeat them now, so
> that I can address them early enough.

I am planning to have a look at the full series today. And will try to 
point out major architectural changes.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-16 11:20 ` [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs Andre Przywara
  2017-03-22 23:44   ` Stefano Stabellini
@ 2017-03-24 11:40   ` Julien Grall
  2017-03-24 15:50     ` Andre Przywara
  1 sibling, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-24 11:40 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
> index 364d5f0..e5cfa54 100644
> --- a/xen/arch/arm/vgic.c
> +++ b/xen/arch/arm/vgic.c
> @@ -30,6 +30,8 @@
>
>  #include <asm/mmio.h>
>  #include <asm/gic.h>
> +#include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>

I really don't want to see gic_v3_* headers included in common code. Why 
do you need them?

>  #include <asm/vgic.h>
>
>  static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v, int rank)
> @@ -61,7 +63,7 @@ struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq)
>      return vgic_get_rank(v, rank);
>  }
>
> -static void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
> +void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
>  {
>      INIT_LIST_HEAD(&p->inflight);
>      INIT_LIST_HEAD(&p->lr_queue);
> @@ -244,10 +246,14 @@ struct vcpu *vgic_get_target_vcpu(struct vcpu *v, unsigned int virq)
>
>  static int vgic_get_virq_priority(struct vcpu *v, unsigned int virq)
>  {
> -    struct vgic_irq_rank *rank = vgic_rank_irq(v, virq);
> +    struct vgic_irq_rank *rank;
>      unsigned long flags;
>      int priority;
>
> +    if ( is_lpi(virq) )
> +        return vgic_lpi_get_priority(v->domain, virq);

This would benefits some comments to explain why LPI pending handling is 
a different path.

> +
> +    rank = vgic_rank_irq(v, virq);
>      vgic_lock_rank(v, rank, flags);
>      priority = rank->priority[virq & INTERRUPT_RANK_MASK];
>      vgic_unlock_rank(v, rank, flags);
> @@ -446,13 +452,63 @@ bool vgic_to_sgi(struct vcpu *v, register_t sgir, enum gic_sgi_mode irqmode,
>      return true;
>  }
>
> +/*
> + * Holding struct pending_irq's for each possible virtual LPI in each domain
> + * requires too much Xen memory, also a malicious guest could potentially
> + * spam Xen with LPI map requests. We cannot cover those with (guest allocated)
> + * ITS memory, so we use a dynamic scheme of allocating struct pending_irq's
> + * on demand.
> + */

I am afraid this will not prevent a guest to use too much Xen memory. 
Let's imagine the guest is mapping thousands of LPIs but decides to 
never handle them or is slow. You would allocate pending_irq for each 
LPI, and never release the memory.

If we use dynamic allocation, we need a way to limit the number of LPIs 
received by a guest to avoid memory exhaustion. The only idea I have is 
an artificial limit, but I don't think it is good. Any other ideas?

> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
> +                                   bool allocate)
> +{
> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
> +
> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
> +    {
> +        if ( lpi_irq->pirq.irq == lpi )
> +        {
> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> +            return &lpi_irq->pirq;
> +        }
> +
> +        if ( lpi_irq->pirq.irq == 0 && !empty )
> +            empty = lpi_irq;
> +    }
> +
> +    if ( !allocate )
> +    {
> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> +        return NULL;
> +    }
> +
> +    if ( !empty )
> +    {
> +        empty = xzalloc(struct lpi_pending_irq);

xzalloc will return NULL if memory is exhausted. There is a general lack 
of error checking within this series. Any missing error could be a 
potential target from a guest, leading to security issue. Stefano and I 
already spot some, it does not mean we found all. So Before sending the 
next version, please go through the series and verify *every* return.

Also, I can't find the code which release LPIs neither in this patch nor 
in this series. A general rule is too have allocation and free within 
the same patch. It is much easier to spot missing free.

> +        vgic_init_pending_irq(&empty->pirq, lpi);
> +        list_add_tail(&empty->entry, &v->arch.vgic.pending_lpi_list);
> +    } else
> +    {
> +        empty->pirq.status = 0;
> +        empty->pirq.irq = lpi;
> +    }
> +
> +    spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> +
> +    return &empty->pirq;
> +}
> +
>  struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>  {
>      struct pending_irq *n;
> +

Spurious change.

>      /* Pending irqs allocation strategy: the first vgic.nr_spis irqs
>       * are used for SPIs; the rests are used for per cpu irqs */
>      if ( irq < 32 )
>          n = &v->arch.vgic.pending_irqs[irq];
> +    else if ( is_lpi(irq) )
> +        n = lpi_to_pending(v, irq, true);
>      else
>          n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>      return n;
> @@ -480,7 +536,7 @@ void vgic_clear_pending_irqs(struct vcpu *v)
>  void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>  {
>      uint8_t priority;
> -    struct pending_irq *iter, *n = irq_to_pending(v, virq);
> +    struct pending_irq *iter, *n;
>      unsigned long flags;
>      bool running;
>
> @@ -488,6 +544,8 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>
>      spin_lock_irqsave(&v->arch.vgic.lock, flags);
>
> +    n = irq_to_pending(v, virq);

Why did you move this code?

> +
>      /* vcpu offline */
>      if ( test_bit(_VPF_down, &v->pause_flags) )
>      {
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 00b9c1a..f44a84b 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -257,6 +257,8 @@ struct arch_vcpu
>          paddr_t rdist_base;
>  #define VGIC_V3_RDIST_LAST  (1 << 0)        /* last vCPU of the rdist */
>          uint8_t flags;
> +        struct list_head pending_lpi_list;
> +        spinlock_t pending_lpi_list_lock;   /* protects the pending_lpi_list */


It would be better to have this structure per-domain limiting the amount 
of memory allocating. Also, Stefano was suggesting to use a more 
efficient data structure, such as an hashtable or a tree.

I would be ok to focus on the correctness so far, but this would need to 
be address before ITS is marked as stable.

>  extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v, int b, int n, int s);
>  extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned int irq);
>  extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
>  extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
>  extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
> +/* placeholder function until the property table gets introduced */
> +static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
> +{
> +    return 0xa;
> +}

To be fair, you can avoid this function by re-ordering the patches. As 
suggested earlier, I would introduce some bits of the vITS before. This 
would also make the series easier to read.

Also, looking how it has been implemented later. I would prefer to see a 
new callback get_priority in vgic_ops which will return the correct 
priority.

I agree this would introduce a bit more of duplicated code. But it would 
limit the use of is_lpi in the common code.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-23 18:21               ` Andre Przywara
@ 2017-03-24 11:45                 ` Julien Grall
  2017-03-24 17:22                   ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-24 11:45 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/23/2017 06:21 PM, Andre Przywara wrote:
> Hi,
>
> On 23/03/17 18:01, Stefano Stabellini wrote:
>> On Thu, 23 Mar 2017, Julien Grall wrote:
>>> Hi Stefano,
>>>
>>> On 23/03/17 17:45, Stefano Stabellini wrote:
>>>> On Thu, 23 Mar 2017, Julien Grall wrote:
>>>>>> So as I mentioned before, I am happy to loose the Kconfig option, but
>>>>>> then we need some sensible default value. In our case we could be cheeky
>>>>>> here for now and just use the Linux value, because a Linux Dom0 would be
>>>>>> the only user. But that doesn't sound very future proof, though this may
>>>>>> not matter for 4.9.
>>>>>
>>>>> I don't think we need a sensible default value and IHMO there is none. I
>>>>> would
>>>>> left the user to decide the exact number.
>>>>
>>>> In that case, the command line parameter becomes mandatory: we need to
>>>> force the user to specify it as we do for dom0_mem today.
>>>
>>> Not really. We should use the hardware value by default. If a user thinks the
>>> number allocated is too big for his use case, then it can limit using the
>>> command line.
>>>
>>> Anyway, I will not oppose to make this command option mandatory when ITS is in
>>> use.
>>
>> Andre wrote:
>>
>>   Any redistributor supporting 32 bits worth of LPIs would lead to a
>>   4GB property table and 512MB pending table allocation.
>>
>> Let's assume that such scenario is realistic (if it is not, then this
>> discussion is fruitless), in this case the user most surely is not going
>> to want to use the hardware provided value. But how can she knows it?
>> How can she find out that she is wasting too much memory on her system?
>> Is there an easy and obvious way to know?
>>
>> She could find out if Xen printed a big warning such as:
>>
>>   USING 4G OF MEMORY FOR ITS PROPTABLE, CONSIDER PASSING max_lpi_bits to XEN
>>
>> but to do that, we need a threshold value in Xen, above which the
>> hypervisor prints the warning. But if we have a threshold value in Xen,
>> then we might as well consider making it the default ceiling: Xen uses
>> the hardware provided value, unless it's greater than threshold, in that
>> case it uses threshold and prints a warning, for example:
>>
>>   LIMITING ITS PROPTABLE MEMORY TO 1G, CHANGE IT WITH max_lpi_bits PARAMETER
>>
>>
>> Does it make sense?
>
> Yes, that is exactly what I was after.
> In contrast to the number of device IDs I think the number of LPI bits
> is _not_ a value that software should use directly, it's more a limit,
> actually a GICv3 implementation choice (how wide the LPI ID fields
> internally are, for instance).
>
> So do we want to limit to 1GB and warn starting at 256MB?

I think Stefano was suggesting to make the command line mandatory.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-23 17:52       ` Stefano Stabellini
@ 2017-03-24 11:54         ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 11:54 UTC (permalink / raw)
  To: Stefano Stabellini, Andre Przywara
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Stefano,

On 03/23/2017 05:52 PM, Stefano Stabellini wrote:
> On Thu, 23 Mar 2017, Andre Przywara wrote:
> On Thu, 23 Mar 2017, Julien Grall wrote:
>> Not answering directly to the question. But I think this is an improvement and
>> not necessary for a first version.
>>
>> I would like to see this series merged in Xen 4.9 as a tech preview, so I
>> would suggest to gather this list of improvement (maybe on Jira?) and defer
>> them for Xen 4.10. They would need to be addressed before making ITS stable.
>> Stefano, does it sound good to you?
>
> Yes, I am OK with not having this in 4.9, but we need to keep track
> (Jira or otherwise) of these things.

I have created XEN-60 for this item and linked to XEN-2 (the Xen ITS task).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests
  2017-03-16 11:20 ` [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests Andre Przywara
@ 2017-03-24 12:03   ` Julien Grall
  2017-04-03 14:18     ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-24 12:03 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> Upon receiving an LPI, we need to find the right VCPU and virtual IRQ
> number to get this IRQ injected.
> Iterate our two-level LPI table to find this information quickly when
> the host takes an LPI. Call the existing injection function to let the
> GIC emulation deal with this interrupt.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-lpi.c | 41 +++++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/gic.c        |  6 ++++--
>  xen/include/asm-arm/irq.h |  8 ++++++++
>  3 files changed, 53 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 59d3ba4..0579976 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -104,6 +104,47 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta)
>          return per_cpu(lpi_redist, cpu).redist_id << 16;
>  }
>
> +/*
> + * Handle incoming LPIs, which are a bit special, because they are potentially
> + * numerous and also only get injected into guests. Treat them specially here,
> + * by just looking up their target vCPU and virtual LPI number and hand it
> + * over to the injection function.
> + */
> +void do_LPI(unsigned int lpi)
> +{
> +    struct domain *d;
> +    union host_lpi *hlpip, hlpi;
> +    struct vcpu *vcpu;
> +
> +    WRITE_SYSREG32(lpi, ICC_EOIR1_EL1);
> +
> +    hlpip = gic_get_host_lpi(lpi);
> +    if ( !hlpip )
> +        return;
> +
> +    hlpi.data = read_u64_atomic(&hlpip->data);
> +
> +    /* We may have mapped more host LPIs than the guest actually asked for. */

Another way, is the interrupt has been received at the same time the 
guest is configuring it. What will happen if the interrupt is lost?

> +    if ( !hlpi.virt_lpi )
> +        return;
> +
> +    d = get_domain_by_id(hlpi.dom_id);

It would be enough to use rcu_lock_domain_by_id here.

> +    if ( !d )
> +        return;
> +
> +    if ( hlpi.vcpu_id >= d->max_vcpus )

A comment would be certainly useful here to explain why this check.

> +    {
> +        put_domain(d);
> +        return;
> +    }
> +
> +    vcpu = d->vcpu[hlpi.vcpu_id];
> +
> +    put_domain(d);
> +
> +    vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
> +}
> +
>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>  {
>      uint64_t val;
> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
> index bd3c032..7286e5d 100644
> --- a/xen/arch/arm/gic.c
> +++ b/xen/arch/arm/gic.c
> @@ -700,8 +700,10 @@ void gic_interrupt(struct cpu_user_regs *regs, int is_fiq)
>              local_irq_enable();
>              do_IRQ(regs, irq, is_fiq);
>              local_irq_disable();
> -        }
> -        else if (unlikely(irq < 16))
> +        } else if ( is_lpi(irq) )
> +        {
> +            do_LPI(irq);

I really don't want to see GICv3 specific code called in common code. 
Please introduce a specific callback in gic_hw_operations.

> +        } else if ( unlikely(irq < 16) )

Coding style:

}
else if (...)
{
}
else if (...)

>          {
>              do_sgi(regs, irq);
>          }
> diff --git a/xen/include/asm-arm/irq.h b/xen/include/asm-arm/irq.h
> index 8f7a167..ee47de8 100644
> --- a/xen/include/asm-arm/irq.h
> +++ b/xen/include/asm-arm/irq.h
> @@ -34,6 +34,14 @@ struct irq_desc *__irq_to_desc(int irq);
>
>  void do_IRQ(struct cpu_user_regs *regs, unsigned int irq, int is_fiq);
>
> +#ifdef CONFIG_HAS_ITS
> +void do_LPI(unsigned int irq);
> +#else
> +static inline void do_LPI(unsigned int irq)
> +{
> +}
> +#endif
> +

This would avoid such ugly hack where do_LPI is define in gic-v3-its.c 
but declared in irq.h.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 12/27] ARM: vGICv3: handle virtual LPI pending and property tables
  2017-03-16 11:20 ` [PATCH v2 12/27] ARM: vGICv3: handle virtual LPI pending and property tables Andre Przywara
@ 2017-03-24 12:09   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 12:09 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> Allow a guest to provide the address and size for the memory regions
> it has reserved for the GICv3 pending and property tables.
> We sanitise the various fields of the respective redistributor
> registers and map those pages into Xen's address space to have easy
> access.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Please address comments from: 
alpine.DEB.2.10.1610281619240.9978@sstabellini-ThinkPad-X260 and 
7eedd317-1238-96ed-620a-408af606e1ad@arm.com

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 13/27] ARM: vGICv3: Handle disabled LPIs
  2017-03-16 11:20 ` [PATCH v2 13/27] ARM: vGICv3: Handle disabled LPIs Andre Przywara
@ 2017-03-24 12:20   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 12:20 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> If a guest disables an LPI, we do not forward this to the associated
> host LPI to avoid queueing commands to the host ITS command queue.
> So it may happen that an LPI fires nevertheless on the host. In this
> case we can bail out early, but have to save the pending state on the
> virtual side.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>

Please see alpine.DEB.2.10.1701051422020.2866@sstabellini-ThinkPad-X260

> ---
>  xen/arch/arm/gic-v3-lpi.c  |  8 ++++++++
>  xen/arch/arm/vgic-v3.c     | 12 ++++++++++++
>  xen/include/asm-arm/vgic.h |  6 ++++++
>  3 files changed, 26 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 0579976..994698e 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -142,6 +142,14 @@ void do_LPI(unsigned int lpi)
>
>      put_domain(d);
>
> +    /*
> +     * We keep all host LPIs enabled, so check if it's disabled on the guest
> +     * side and just record this LPI in the virtual pending table in this case.
> +     * The guest picks it up once it gets enabled again.
> +     */
> +    if ( !vgic_can_inject_lpi(vcpu, hlpi.virt_lpi) )
> +        return;
> +
>      vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
>  }
>
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index c6db2d7..de625bf 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -498,6 +498,18 @@ bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi)
>      return d->arch.vgic.proptable[vlpi - LPI_OFFSET] & LPI_PROP_ENABLED;
>  }
>
> +bool vgic_can_inject_lpi(struct vcpu *vcpu, uint32_t vlpi)
> +{
> +    if ( vlpi >= vcpu->domain->arch.vgic.nr_lpis )
> +        return false;
> +
> +    if ( vgic_lpi_is_enabled(vcpu->domain, vlpi) )
> +        return true;
> +
> +    set_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable);

New line here.

> +    return false;
> +}
> +
>  static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
>                                            uint32_t gicr_reg,
>                                            register_t r)
> diff --git a/xen/include/asm-arm/vgic.h b/xen/include/asm-arm/vgic.h
> index f8bccfa..302702e 100644
> --- a/xen/include/asm-arm/vgic.h
> +++ b/xen/include/asm-arm/vgic.h
> @@ -323,6 +323,7 @@ int vgic_v3_init(struct domain *d, int *mmio_count);
>  #ifdef CONFIG_HAS_GICV3
>  extern int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi);
>  extern bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi);
> +extern bool vgic_can_inject_lpi(struct vcpu *v, uint32_t vlpi);
>  #else
>  static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
>  {
> @@ -333,6 +334,11 @@ static inline bool vgic_lpi_is_enabled(struct domain *d, uint32_t vlpi)
>  {
>      return false;
>  }
> +
> +static inline bool vgic_can_inject_lpi(struct vcpu *v, uint32_t vlpi)
> +{
> +    return false;
> +}

Why do you need this dummy helper?

>  #endif
>
>  extern int domain_vgic_register(struct domain *d, int *mmio_count);
>

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits
  2017-03-16 11:20 ` [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits Andre Przywara
  2017-03-16 16:25   ` Shanker Donthineni
@ 2017-03-24 12:41   ` Julien Grall
  1 sibling, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 12:41 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> Create a new file to hold the emulation code for the ITS widget.
> For now we emulate the memory mapped ITS registers and provide a stub
> to introduce the ITS command handling framework (but without actually
> emulating any commands at this time).

All the comments Stefano and I made a year ago are still valid. Please 
have a look the answer on <20160928182457.12433-13-andre.przywara@arm.com>.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 15/27] ARM: vITS: introduce translation table walks
  2017-03-16 11:20 ` [PATCH v2 15/27] ARM: vITS: introduce translation table walks Andre Przywara
@ 2017-03-24 13:00   ` Julien Grall
  2017-04-03 18:25     ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-24 13:00 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> The ITS stores the target (v)CPU and the (virtual) LPI number in tables.
> Introduce functions to walk those tables and translate an device ID -
> event ID pair into a pair of virtual LPI and vCPU.
> Since the final interrupt translation tables can be smaller than a page,
> we map them on demand (which is cheap on arm64). Also we take care of
> the locking on the way, since we can't easily protect those ITTs from
> being altered by the guest.
>
> To allow compiling without warnings, we declare two functions as
> non-static for the moment.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3-its.c | 135 +++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 135 insertions(+)
>
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 5337638..267a573 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -62,6 +62,141 @@ struct vits_itte
>      uint16_t collection;
>  };
>
> +#define UNMAPPED_COLLECTION      ((uint16_t)~0)
> +
> +/* Must be called with the ITS lock held. */

This is a call for an ASSERT in the function.

> +static struct vcpu *get_vcpu_from_collection(struct virt_its *its, int collid)

s/int/unsigned int/

> +{
> +    uint16_t vcpu_id;
> +
> +    if ( collid >= its->max_collections )
> +        return NULL;
> +
> +    vcpu_id = its->coll_table[collid];
> +    if ( vcpu_id == UNMAPPED_COLLECTION || vcpu_id >= its->d->max_vcpus )
> +        return NULL;
> +
> +    return its->d->vcpu[vcpu_id];
> +}
> +
> +#define DEV_TABLE_ITT_ADDR(x) ((x) & GENMASK(51, 8))
> +#define DEV_TABLE_ITT_SIZE(x) (BIT(((x) & GENMASK(7, 0)) + 1))
> +#define DEV_TABLE_ENTRY(addr, bits)                     \
> +        (((addr) & GENMASK(51, 8)) | (((bits) - 1) & GENMASK(7, 0)))

The layout of dev_table[...] really needs to be explained. It took me 
quite a while to understand how it works. For instance why you skip the 
first 8 bits for the address...

> +
> +static paddr_t get_itte_address(struct virt_its *its,
> +                                uint32_t devid, uint32_t evid)
> +{

I was expected to see the support of two-level page table for the vITS. 
Any plan for that?

> +    paddr_t addr;
> +
> +    if ( devid >= its->max_devices )
> +        return ~0;

Please don't hardcode invalid address and use INVALID_PADDR.

> +
> +    if ( evid >= DEV_TABLE_ITT_SIZE(its->dev_table[devid]) )
> +        return ~0;

Ditto.

> +
> +    addr = DEV_TABLE_ITT_ADDR(its->dev_table[devid]);

You read dev_table[...] multiple time. What prevents someone to modify 
dev_table after you did someone sanity check?

It would be safer to do a read_atomic(..) at the beginning and use a 
temporary variable.

> +
> +    return addr + evid * sizeof(struct vits_itte);
> +}
> +
> +/*
> + * Looks up a given deviceID/eventID pair on an ITS and returns a pointer to
> + * the corresponding ITTE. This maps the respective guest page into Xen.
> + * Once finished with handling the ITTE, call put_devid_evid() to unmap
> + * the page again.
> + * Must be called with the ITS lock held.

This is a call for an ASSERT in the code.

> + */
> +static struct vits_itte *get_devid_evid(struct virt_its *its,
> +                                        uint32_t devid, uint32_t evid)

The naming of the function is confusing. It doesn't look up a device 
ID/event ID but an IIT. So I would rename it to find_itte.

> +{
> +    paddr_t addr = get_itte_address(its, devid, evid);
> +
> +    if ( addr == ~0 )


Please use INVALID_PADDR.

> +        return NULL;
> +
> +    return map_guest_pages(its->d, addr, 1);
> +}
> +
> +/* Must be called with the ITS lock held. */
> +static void put_devid_evid(struct virt_its *its, struct vits_itte *itte)
> +{
> +    unmap_guest_pages(itte, 1);
> +}
> +
> +/*
> + * Queries the collection and device tables to get the vCPU and virtual
> + * LPI number for a given guest event. This takes care of mapping the
> + * respective tables and validating the values, since we can't efficiently
> + * protect the ITTs with their less-than-page-size granularity.
> + * Takes and drops the its_lock.

I am not sure to understand the usefulness of "takes and drops the 
its_lock".

> + */
> +bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
> +               struct vcpu **vcpu, uint32_t *vlpi)
> +{
> +    struct vits_itte *itte;
> +    int collid;
> +    uint32_t _vlpi;
> +    struct vcpu *_vcpu;
> +
> +    spin_lock(&its->its_lock);

Do we expect multiple vCPU calling read_itte at the same time? This 
needs to be clarify in the comments because this current function is not 
scalable.

> +    itte = get_devid_evid(its, devid, evid);
> +    if ( !itte )
> +    {
> +        spin_unlock(&its->its_lock);
> +        return false;
> +    }
> +    collid = itte->collection;
> +    _vlpi = itte->vlpi;
> +    put_devid_evid(its, itte);
> +
> +    _vcpu = get_vcpu_from_collection(its, collid);
> +    spin_unlock(&its->its_lock);
> +
> +    if ( !_vcpu )
> +        return false;
> +
> +    if ( collid >= its->max_collections )
> +        return false;

No need for this check. It is already done in get_vcpu_from_collection.

> +
> +    *vcpu = _vcpu;
> +    *vlpi = _vlpi;
> +
> +    return true;
> +}
> +
> +#define SKIP_LPI_UPDATE 1
> +bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
> +                uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
> +{
> +    struct vits_itte *itte;
> +
> +    if ( collid >= its->max_collections )
> +        return false;
> +
> +    /* TODO: validate vlpi */

This TODO needs to be fixed as soon as possible.

> +
> +    spin_lock(&its->its_lock);
> +    itte = get_devid_evid(its, devid, evid);
> +    if ( !itte )
> +    {
> +        spin_unlock(&its->its_lock);
> +        return false;
> +    }
> +
> +    itte->collection = collid;
> +    if ( vlpi != SKIP_LPI_UPDATE )
> +        itte->vlpi = vlpi;
> +
> +    if ( vcpu )
> +        *vcpu = get_vcpu_from_collection(its, collid);
> +
> +    put_devid_evid(its, itte);
> +    spin_unlock(&its->its_lock);
> +
> +    return true;
> +}
> +
>  /**************************************
>   * Functions that handle ITS commands *
>   **************************************/
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 16/27] ARM: vITS: handle CLEAR command
  2017-03-16 11:20 ` [PATCH v2 16/27] ARM: vITS: handle CLEAR command Andre Przywara
@ 2017-03-24 14:27   ` Julien Grall
  2017-03-24 15:53     ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-24 14:27 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> This introduces the ITS command handler for the CLEAR command, which
> clears the pending state of an LPI.
> This removes a not-yet injected, but already queued IRQ from a VCPU.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3-its.c | 35 +++++++++++++++++++++++++++++++++--
>  1 file changed, 33 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 267a573..e808f43 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -131,8 +131,8 @@ static void put_devid_evid(struct virt_its *its, struct vits_itte *itte)
>   * protect the ITTs with their less-than-page-size granularity.
>   * Takes and drops the its_lock.
>   */
> -bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
> -               struct vcpu **vcpu, uint32_t *vlpi)
> +static bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,

NIT: Please mention in the commit message why you turned those functions 
to static.

> +                      struct vcpu **vcpu, uint32_t *vlpi)
>  {
>      struct vits_itte *itte;
>      int collid;
> @@ -216,6 +216,34 @@ static uint64_t its_cmd_mask_field(uint64_t *its_cmd,
>  #define its_cmd_get_target_addr(cmd)    its_cmd_mask_field(cmd, 2, 16, 32)
>  #define its_cmd_get_validbit(cmd)       its_cmd_mask_field(cmd, 2, 63,  1)
>
> +static int its_handle_clear(struct virt_its *its, uint64_t *cmdptr)
> +{
> +    uint32_t devid = its_cmd_get_deviceid(cmdptr);
> +    uint32_t eventid = its_cmd_get_id(cmdptr);
> +    struct pending_irq *pirq;
> +    struct vcpu *vcpu;
> +    uint32_t vlpi;
> +
> +    if ( !read_itte(its, devid, eventid, &vcpu, &vlpi) )
> +        return -1;
> +
> +    /* Remove a pending, but not yet injected guest IRQ. */

Copying Stefano's comment from last year:

"We need to check that the vlpi hasn't already been added to an LR
register. We can do that with GIC_IRQ_GUEST_VISIBLE.

In case GIC_IRQ_GUEST_VISIBLE is set, we need to clear the lr
(clear_lr). If we don't handle this case, we should at least print a
warning."

> +    pirq = lpi_to_pending(vcpu, vlpi, false);
> +    if ( pirq )
> +    {
> +        clear_bit(GIC_IRQ_GUEST_QUEUED, &pirq->status);
> +        gic_remove_from_queues(vcpu, vlpi);
> +
> +        /* Mark this pending IRQ struct as availabe again. */
> +        if ( !test_bit(GIC_IRQ_GUEST_VISIBLE, &pirq->status) )
> +            pirq->irq = 0;
> +    }
> +
> +    clear_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable);
> +
> +    return 0;
> +}
> +
>  #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
>
>  static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
> @@ -236,6 +264,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
>          cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
>          switch (its_cmd_get_command(cmdptr))
>          {
> +        case GITS_CMD_CLEAR:
> +            its_handle_clear(its, cmdptr);

Should not you check the return for its_handle_clear?

> +            break;
>          case GITS_CMD_SYNC:
>              /* We handle ITS commands synchronously, so we ignore SYNC. */
>  	    break;
>

Regards,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 17/27] ARM: vITS: handle INT command
  2017-03-16 11:20 ` [PATCH v2 17/27] ARM: vITS: handle INT command Andre Przywara
@ 2017-03-24 14:38   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 14:38 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> The INT command sets a given LPI identified by a DeviceID/EventID pair
> as pending and thus triggers it to be injected.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3-its.c | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
>
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index e808f43..23476f9 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -244,6 +244,26 @@ static int its_handle_clear(struct virt_its *its, uint64_t *cmdptr)
>      return 0;
>  }
>
> +static int its_handle_int(struct virt_its *its, uint64_t *cmdptr)
> +{
> +    uint32_t devid = its_cmd_get_deviceid(cmdptr);
> +    uint32_t eventid = its_cmd_get_id(cmdptr);
> +    struct vcpu *vcpu;
> +    uint32_t vlpi;
> +    uint8_t prop;
> +
> +    if ( !read_itte(its, devid, eventid, &vcpu, &vlpi) )
> +        return -1;
> +
> +    prop = vcpu->domain->arch.vgic.proptable[vlpi - LPI_OFFSET];

I looked at the code and cannot find any code which will ensure that the 
proptable is not NULL.

> +    if ( prop & LPI_PROP_ENABLED )
> +        vgic_vcpu_inject_irq(vcpu, vlpi);
> +    else
> +        set_bit(vlpi - LPI_OFFSET, vcpu->arch.vgic.pendtable);
> +
> +    return 0;
> +}
> +
>  #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
>
>  static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
> @@ -267,6 +287,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
>          case GITS_CMD_CLEAR:
>              its_handle_clear(its, cmdptr);
>              break;
> +        case GITS_CMD_INT:
> +            its_handle_int(its, cmdptr);

Should not you check the return value?

> +            break;
>          case GITS_CMD_SYNC:
>              /* We handle ITS commands synchronously, so we ignore SYNC. */
>  	    break;
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 19/27] ARM: vITS: handle MAPD command
  2017-03-16 11:20 ` [PATCH v2 19/27] ARM: vITS: handle MAPD command Andre Przywara
@ 2017-03-24 14:41   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 14:41 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> The MAPD command maps a device by associating a memory region for
> storing ITEs with a certain device ID.
> We store the given guest physical address in the device table, and, if
> this command comes from Dom0, tell the host ITS driver about this new
> mapping, so it can issue the corresponing host MAPD command and create
> the required tables.
> We don't map the device tables permanently, as their alignment
> requirement is only 256 Bytes, thus making mapping of several tables
> complicated. Instead we map the device tables on demand when we need
> them later.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3-its.c | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 44 insertions(+)
>
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 175a213..c26d5d4 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -38,6 +38,7 @@
>  /* Data structure to describe a virtual ITS */
>  struct virt_its {
>      struct domain *d;
> +    paddr_t doorbell_address;
>      spinlock_t vcmd_lock;       /* protects the virtual command buffer */
>      uint64_t cbaser;
>      uint64_t *cmdbuf;
> @@ -291,6 +292,46 @@ static int its_handle_mapc(struct virt_its *its, uint64_t *cmdptr)
>      return ret;
>  }
>
> +static int its_handle_mapd(struct virt_its *its, uint64_t *cmdptr)
> +{
> +    uint32_t devid = its_cmd_get_deviceid(cmdptr);
> +    int size = its_cmd_get_size(cmdptr);
> +    bool valid = its_cmd_get_validbit(cmdptr);
> +    paddr_t itt_addr = its_cmd_mask_field(cmdptr, 2, 0, 52) & GENMASK(51, 8);
> +    int ret;
> +
> +    if ( !its->dev_table )
> +        return -1;
> +
> +    /*
> +     * There is no easy and clean way for Xen to know the ITS device ID of a
> +     * particular (PCI) device, so we have to rely on the guest telling
> +     * us about it. For *now* we are just using the device ID *Dom0* uses,
> +     * because the driver there has the actual knowledge.
> +     * Eventually this will be replaced with a dedicated hypercall to
> +     * announce pass-through of devices.
> +     */
> +    if ( is_hardware_domain(its->d) )
> +    {
> +        /* Dom0's ITSes are mapped 1:1, so both address are the same. */
> +        ret = gicv3_its_map_guest_device(its->d, its->doorbell_address, devid,
> +                                         its->doorbell_address, devid,
> +                                         BIT(size + 1), valid);

The size should be satinized against the number of event support by the 
vITS.

And it should also be sanitzed in gicv3_its_map_* against the number of 
event supported by the pITS.

> +        if ( ret )
> +            return ret;
> +    }
> +
> +    spin_lock(&its->its_lock);
> +    if ( valid )
> +        its->dev_table[devid] = DEV_TABLE_ENTRY(itt_addr, size + 1);
> +    else
> +        its->dev_table[devid] = 0;
> +
> +    spin_unlock(&its->its_lock);
> +
> +    return 0;
> +}
> +
>  #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
>
>  static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
> @@ -320,6 +361,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
>          case GITS_CMD_MAPC:
>              its_handle_mapc(its, cmdptr);
>              break;
> +        case GITS_CMD_MAPD:
> +            its_handle_mapd(its, cmdptr);

Should not you check the return value?

> +	    break;
>          case GITS_CMD_SYNC:
>              /* We handle ITS commands synchronously, so we ignore SYNC. */
>  	    break;
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 20/27] ARM: vITS: handle MAPTI command
  2017-03-16 11:20 ` [PATCH v2 20/27] ARM: vITS: handle MAPTI command Andre Przywara
@ 2017-03-24 14:54   ` Julien Grall
  2017-04-03 18:47     ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-24 14:54 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> The MAPTI commands associates a DeviceID/EventID pair with a LPI/CPU
> pair and actually instantiates LPI interrupts.
> We connect the already allocated host LPI to this virtual LPI, so that
> any triggering IRQ on the host can be quickly forwarded to a guest.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        | 63 ++++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/gic-v3-lpi.c        | 18 ++++++++++++
>  xen/arch/arm/vgic-v3-its.c       | 27 +++++++++++++++--
>  xen/include/asm-arm/gic_v3_its.h |  6 ++++
>  4 files changed, 112 insertions(+), 2 deletions(-)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 5a2dbec..e2fcf50 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -724,6 +724,69 @@ restart:
>      spin_unlock(&d->arch.vgic.its_devices_lock);
>  }
>
> +/*
> + * Translates an event for a given guest device ID into the associated host
> + * LPI number. This can be used to look up the mapped guest LPI.
> + */
> +static uint32_t translate_event(struct domain *d, paddr_t doorbell,
> +                                uint32_t devid, uint32_t eventid)
> +{
> +    struct rb_node *node;
> +    struct its_devices *dev;
> +    uint32_t host_lpi = 0;
> +    int cmp;
> +
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    node = d->arch.vgic.its_devices.rb_node;
> +    while (node)
> +    {
> +        dev = rb_entry(node, struct its_devices, rbnode);
> +        cmp = compare_its_guest_devices(dev, doorbell, devid);
> +
> +        if ( !cmp )
> +        {
> +            if ( eventid >= dev->eventids )
> +                goto out;
> +
> +            host_lpi = dev->host_lpis[eventid / LPI_BLOCK] +
> +                                (eventid % LPI_BLOCK);
> +            if ( !is_lpi(host_lpi) )

Hmmm, I don't understand this check. host_lpi should always be an LPI. No?

> +                host_lpi = 0;
> +            goto out;
> +        }
> +
> +        if ( cmp > 0 )
> +            node = node->rb_left;
> +        else
> +            node = node->rb_right;
> +    }
> +
> +out:
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +    return host_lpi;
> +}
> +
> +/*
> + * Connects the event ID for an already assigned device to the given VCPU/vLPI
> + * pair. The corresponding physical LPI is already mapped on the host side
> + * (when assigning the physical device to the guest), so we just connect the
> + * target VCPU/vLPI pair to that interrupt to inject it properly if it fires.
> + */
> +int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell_address,
> +                             uint32_t devid, uint32_t eventid,

It looks like to me that devid is the virtual deviceID. If so, please 
prefix with 'v', otherwise 'p'.

> +                             struct vcpu *v, uint32_t virt_lpi)
> +{
> +    uint32_t host_lpi = translate_event(d, doorbell_address, devid, eventid);
> +
> +    if ( !host_lpi )
> +        return -ENOENT;
> +
> +    gicv3_lpi_update_host_entry(host_lpi, d->domain_id, v->vcpu_id, virt_lpi);
> +
> +    return 0;
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index 994698e..c110ec9 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -153,6 +153,24 @@ void do_LPI(unsigned int lpi)
>      vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
>  }
>
> +int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
> +                                unsigned int vcpu_id, uint32_t virt_lpi)
> +{
> +    union host_lpi *hlpip, hlpi;
> +
> +    host_lpi -= LPI_OFFSET;

I would add an ASSERT(host_lpi > LPI_OFFSET);

> +
> +    hlpip = &lpi_data.host_lpis[host_lpi / HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
> +
> +    hlpi.virt_lpi = virt_lpi;
> +    hlpi.dom_id = domain_id;
> +    hlpi.vcpu_id = vcpu_id;
> +
> +    write_u64_atomic(&hlpip->data, hlpi.data);
> +
> +    return 0;
> +}
> +
>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>  {
>      uint64_t val;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index c26d5d4..600ff69 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -167,8 +167,8 @@ static bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
>  }
>
>  #define SKIP_LPI_UPDATE 1
> -bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
> -                uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
> +static bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
> +                       uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)

Please explain in the commit message why you move to static.

>  {
>      struct vits_itte *itte;
>
> @@ -332,6 +332,25 @@ static int its_handle_mapd(struct virt_its *its, uint64_t *cmdptr)
>      return 0;
>  }
>
> +static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
> +{
> +    uint32_t devid = its_cmd_get_deviceid(cmdptr);
> +    uint32_t eventid = its_cmd_get_id(cmdptr);
> +    uint32_t intid = its_cmd_get_physical_id(cmdptr);
> +    int collid = its_cmd_get_collection(cmdptr);

This should be unsigned.

> +    struct vcpu *vcpu;
> +
> +    if ( its_cmd_get_command(cmdptr) == GITS_CMD_MAPI )
> +        intid = eventid;
> +
> +    if ( !write_itte(its, devid, eventid, collid, intid, &vcpu) )
> +        return -1;
> +
> +    gicv3_assign_guest_event(its->d, its->doorbell_address, devid, eventid, vcpu, intid);

If you have a function returning an error, then you should check it and 
not ignoring it.


Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 21/27] ARM: vITS: handle MOVI command
  2017-03-16 11:20 ` [PATCH v2 21/27] ARM: vITS: handle MOVI command Andre Przywara
@ 2017-03-24 15:00   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 15:00 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> The MOVI command moves the interrupt affinity from one redistributor
> (read: VCPU) to another.
> For now migration of "live" LPIs is not yet implemented, but we store
> the changed affinity in the host LPI structure and in our virtual ITTE.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        | 15 +++++++++++++++
>  xen/arch/arm/gic-v3-lpi.c        | 13 +++++++++++++
>  xen/arch/arm/vgic-v3-its.c       | 24 ++++++++++++++++++++++++
>  xen/include/asm-arm/gic_v3_its.h |  4 ++++
>  4 files changed, 56 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index e2fcf50..aa9b1b2 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -787,6 +787,21 @@ int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell_address,
>      return 0;
>  }
>
> +/* Changes the target VCPU for a given host LPI assigned to a domain. */
> +int gicv3_lpi_change_vcpu(struct domain *d, paddr_t doorbell,
> +                          uint32_t devid, uint32_t eventid,
> +                          unsigned int vcpu_id)
> +{
> +    uint32_t host_lpi = translate_event(d, doorbell, devid, eventid);
> +
> +    if ( !host_lpi )
> +        return -ENOENT;
> +
> +    gicv3_lpi_update_host_vcpuid(host_lpi, vcpu_id);
> +
> +    return 0;
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
> index c110ec9..8c7cea1 100644
> --- a/xen/arch/arm/gic-v3-lpi.c
> +++ b/xen/arch/arm/gic-v3-lpi.c
> @@ -171,6 +171,19 @@ int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
>      return 0;
>  }
>
> +int gicv3_lpi_update_host_vcpuid(uint32_t host_lpi, unsigned int vcpu_id)
> +{
> +    union host_lpi *hlpip;
> +
> +    host_lpi -= LPI_OFFSET;
> +
> +    hlpip = &lpi_data.host_lpis[host_lpi / HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
> +
> +    write_u16_atomic(&hlpip->vcpu_id, vcpu_id);
> +
> +    return 0;
> +}
> +
>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>  {
>      uint64_t val;
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 600ff69..bb573de 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -351,6 +351,24 @@ static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
>      return 0;
>  }
>
> +static int its_handle_movi(struct virt_its *its, uint64_t *cmdptr)
> +{
> +    uint32_t devid = its_cmd_get_deviceid(cmdptr);
> +    uint32_t eventid = its_cmd_get_id(cmdptr);
> +    int collid = its_cmd_get_collection(cmdptr);
> +    struct vcpu *vcpu;
> +
> +    if ( !write_itte(its, devid, eventid, collid, SKIP_LPI_UPDATE, &vcpu) )
> +        return -1;
> +
> +    /* TODO: lookup currently-in-guest virtual IRQs and migrate them */

Looking at the code, you record the vcpuID which means that the next LPI 
will be direct to this vCPU. However, the previous LPI may still be 
inflight on a different vCPU.

How the guest will behave in this case?

> +
> +    gicv3_lpi_change_vcpu(its->d,
> +                          its->doorbell_address, devid, eventid, vcpu->vcpu_id);
> +
> +    return 0;
> +}
> +
>  #define ITS_CMD_BUFFER_SIZE(baser)      ((((baser) & 0xff) + 1) << 12)
>
>  static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
> @@ -387,6 +405,12 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
>          case GITS_CMD_MAPTI:
>              its_handle_mapti(its, cmdptr);
>              break;
> +        case GITS_CMD_MOVALL:
> +            gdprintk(XENLOG_G_INFO, "ITS: ignoring MOVALL command\n");

Why MOVALL is not implemented?

> +            break;
> +        case GITS_CMD_MOVI:
> +            its_handle_movi(its, cmdptr);
> +            break;
>          case GITS_CMD_SYNC:
>              /* We handle ITS commands synchronously, so we ignore SYNC. */
>  	    break;
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 1ea00e9..25b6b3c 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -166,8 +166,12 @@ int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
>  int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell,
>                               uint32_t devid, uint32_t eventid,
>                               struct vcpu *v, uint32_t virt_lpi);
> +int gicv3_lpi_change_vcpu(struct domain *d, paddr_t doorbell,
> +                          uint32_t devid, uint32_t eventid,
> +                          unsigned int vcpu_id);
>  int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
>                                  unsigned int vcpu_id, uint32_t virt_lpi);
> +int gicv3_lpi_update_host_vcpuid(uint32_t host_lpi, unsigned int vcpu_id);
>
>  #else
>
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 24/27] ARM: vITS: handle INVALL command
  2017-03-16 11:20 ` [PATCH v2 24/27] ARM: vITS: handle INVALL command Andre Przywara
@ 2017-03-24 15:12   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 15:12 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> The INVALL command instructs an ITS to invalidate the configuration
> data for all LPIs associated with a given redistributor (read: VCPU).
> This is nasty to emulate exactly with our architecture, so we just scan
> the pending table and inject _every_ LPI found there that got enabled.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3-its.c | 37 +++++++++++++++++++++++++++++++++++++
>  1 file changed, 37 insertions(+)
>
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index 8af06ac..cc12c1c 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -319,6 +319,40 @@ static int its_handle_inv(struct virt_its *its, uint64_t *cmdptr)
>      return 0;
>  }
>
> +/*
> + * INVALL updates the per-LPI configuration status for every LPI mapped to
> + * a particular redistributor. Since our property table is referenced when
> + * needed, we don't need to sync anything, really. But we have to take care
> + * of LPIs getting enabled if there is an interrupt pending.
> + * To catch every LPI without iterating through the device table we just
> + * look for set bits in our virtual pending table and check the status of
> + * the enabled bit in the respective property table entry.
> + * This actually covers every (pending) LPI from every redistributor,
> + * but update_lpi_enabled_status() is a NOP for LPIs not being mapped
> + * to the redistributor/VCPU we are interested in.
> + */
> +static int its_handle_invall(struct virt_its *its, uint64_t *cmdptr)
> +{
> +    uint32_t collid = its_cmd_get_collection(cmdptr);
> +    struct vcpu *vcpu;
> +    int vlpi = 0;
> +
> +    spin_lock(&its->its_lock);
> +    vcpu = get_vcpu_from_collection(its, collid);
> +    spin_unlock(&its->its_lock);
> +
> +    do {
> +        vlpi = find_next_bit(vcpu->arch.vgic.pendtable,
> +                             its->d->arch.vgic.nr_lpis, vlpi);
> +        if ( vlpi >= its->d->arch.vgic.nr_lpis )
> +            break;
> +
> +        update_lpi_enabled_status(its, vcpu, vlpi);
> +    } while (1);
> +
> +    return 0;
> +}

I still don't like the implementation of INVALL. You rely on the guest 
to behave well and not making sure that pendtable is full of 1 and the 
associate LPIs not enabled.

This could fairly be easy to achieve by sending INT command and modify 
the property table to have all IRQs enabled.

> +
>  static int its_handle_mapc(struct virt_its *its, uint64_t *cmdptr)
>  {
>      uint32_t collid = its_cmd_get_collection(cmdptr);
> @@ -485,6 +519,9 @@ static int vgic_its_handle_cmds(struct domain *d, struct virt_its *its,
>          case GITS_CMD_INV:
>              its_handle_inv(its, cmdptr);
>  	    break;
> +        case GITS_CMD_INVALL:
> +            its_handle_invall(its, cmdptr);
> +	    break;
>          case GITS_CMD_MAPC:
>              its_handle_mapc(its, cmdptr);
>              break;
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 25/27] ARM: vITS: create and initialize virtual ITSes for Dom0
  2017-03-16 11:20 ` [PATCH v2 25/27] ARM: vITS: create and initialize virtual ITSes for Dom0 Andre Przywara
@ 2017-03-24 15:18   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 15:18 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> For each hardware ITS create and initialize a virtual ITS for Dom0.
> We use the same memory mapped address to keep the doorbell working.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3-its.c       | 29 +++++++++++++++++++++++++++++
>  xen/arch/arm/vgic-v3.c           | 12 ++++++++++++
>  xen/include/asm-arm/domain.h     |  1 +
>  xen/include/asm-arm/gic_v3_its.h | 12 ++++++++++++
>  4 files changed, 54 insertions(+)
>
> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> index cc12c1c..caa7354 100644
> --- a/xen/arch/arm/vgic-v3-its.c
> +++ b/xen/arch/arm/vgic-v3-its.c
> @@ -914,6 +914,35 @@ static const struct mmio_handler_ops vgic_its_mmio_handler = {
>      .write = vgic_v3_its_mmio_write,
>  };
>
> +int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr)

*All* initialization function should have a counterpart in the same 
patch to free the memory.

> +{
> +    struct virt_its *its;
> +    uint64_t base_attr;
> +
> +    its = xzalloc(struct virt_its);
> +    if ( ! its )
> +        return -ENOMEM;
> +
> +    base_attr  = GIC_BASER_InnerShareable << GITS_BASER_SHAREABILITY_SHIFT;
> +    base_attr |= GIC_BASER_CACHE_SameAsInner << GITS_BASER_OUTER_CACHEABILITY_SHIFT;
> +    base_attr |= GIC_BASER_CACHE_RaWaWb << GITS_BASER_INNER_CACHEABILITY_SHIFT;
> +
> +    its->cbaser  = base_attr;
> +    base_attr |= 0ULL << GITS_BASER_PAGE_SIZE_SHIFT;

Please explain the 0ULL.

> +    its->baser0  = GITS_BASER_TYPE_DEVICE << GITS_BASER_TYPE_SHIFT;
> +    its->baser0 |= (7ULL << GITS_BASER_ENTRY_SIZE_SHIFT) | base_attr;

Please explain 7ULL. I suspect you can use a sizeof of the device structure.

> +    its->baser1  = GITS_BASER_TYPE_COLLECTION << GITS_BASER_TYPE_SHIFT;
> +    its->baser1 |= (1ULL << GITS_BASER_ENTRY_SIZE_SHIFT) | base_attr;

Please explain 1ULL. I suspect you can use a sizeof of the collection 
structure.

> +    its->d = d;
> +    its->doorbell_address = guest_addr + ITS_DOORBELL_OFFSET;
> +    spin_lock_init(&its->vcmd_lock);
> +    spin_lock_init(&its->its_lock);
> +
> +    register_mmio_handler(d, &vgic_its_mmio_handler, guest_addr, SZ_64K, its);
> +
> +    return 0;
> +}
> +
>  /*
>   * Local variables:
>   * mode: C
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index d1382be..8faec95 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -31,6 +31,7 @@
>  #include <asm/current.h>
>  #include <asm/mmio.h>
>  #include <asm/gic_v3_defs.h>
> +#include <asm/gic_v3_its.h>
>  #include <asm/vgic.h>
>  #include <asm/vgic-emul.h>
>  #include <asm/vreg.h>
> @@ -1651,6 +1652,7 @@ static int vgic_v3_domain_init(struct domain *d)
>       */
>      if ( is_hardware_domain(d) )
>      {
> +        struct host_its *hw_its;
>          unsigned int first_cpu = 0;
>
>          d->arch.vgic.dbase = vgic_v3_hw.dbase;
> @@ -1676,6 +1678,16 @@ static int vgic_v3_domain_init(struct domain *d)
>
>              first_cpu += size / d->arch.vgic.rdist_stride;
>          }
> +        d->arch.vgic.nr_regions = vgic_v3_hw.nr_rdist_regions;

Why did you add that?

> +
> +        list_for_each_entry(hw_its, &host_its_list, entry)

This could be done in vgic-v3-its.c. Also, I would prefer to see 
something similar to vgic_v3_setup_hw for ITS to make the code agnostic.

> +        {
> +            /* Emulate the control registers frame (lower 64K). */
> +            vgic_v3_its_init_virtual(d, hw_its->addr);

You likely need to pass more information to the vITS such as the number 
of deviceID, EventID...

Also, please check the return value.

> +
> +            d->arch.vgic.has_its = true;
> +        }
> +
>      }
>      else
>      {
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 33c1851..27cc310 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -115,6 +115,7 @@ struct arch_domain
>          uint8_t *proptable;
>          struct rb_root its_devices;         /* devices mapped to an ITS */
>          spinlock_t its_devices_lock;        /* protects the its_devices tree */
> +        bool has_its;
>  #endif
>      } vgic;
>
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 25b6b3c..55ef143 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -148,6 +148,13 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
>  int gicv3_its_setup_collection(unsigned int cpu);
>
>  /*
> + * Create and register a virtual ITS at the given guest address.
> + * If a host ITS is specified, a hardware domain can reach out to that host
> + * ITS to deal with devices and LPI mappings and can enable/disable LPIs.
> + */
> +int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr);
> +
> +/*
>   * Map a device on the host by allocating an ITT on the host (ITS).
>   * "nr_event" specifies how many events (interrupts) this device will need.
>   * Setting "valid" to false deallocates the device.
> @@ -211,6 +218,11 @@ static inline int gicv3_its_setup_collection(unsigned int cpu)
>      return 0;
>  }
>
> +static inline int vgic_v3_its_init_virtual(struct domain *d, paddr_t guest_addr)
> +{
> +    return 0;
> +}
> +
>  #endif /* CONFIG_HAS_ITS */
>
>  #endif
>

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 27/27] ARM: vGIC: advertise LPI support
  2017-03-16 11:20 ` [PATCH v2 27/27] ARM: vGIC: advertise LPI support Andre Przywara
@ 2017-03-24 15:25   ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 15:25 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/16/2017 11:20 AM, Andre Przywara wrote:
> To let a guest know about the availability of virtual LPIs, set the
> respective bits in the virtual GIC registers and let a guest control
> the LPI enable bit.
> Only report the LPI capability if the host has initialized at least
> one ITS.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/vgic-v3.c | 88 +++++++++++++++++++++++++++++++++++++++++++++++---
>  1 file changed, 83 insertions(+), 5 deletions(-)
>
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index 8faec95..6d5b7f4 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -169,8 +169,10 @@ static int __vgic_v3_rdistr_rd_mmio_read(struct vcpu *v, mmio_info_t *info,
>      switch ( gicr_reg )
>      {
>      case VREG32(GICR_CTLR):
> -        /* We have not implemented LPI's, read zero */
> -        goto read_as_zero_32;
> +        if ( dabt.size != DABT_WORD ) goto bad_width;
> +        *r = vgic_reg32_extract(!!(v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED),
> +                                info);

I would be more readable to have:

uint32_t ctlr;

ctlr = !!(v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED);
*r = vgic_reg32_extract(ctlr, info);

> +        return 1;
>
>      case VREG32(GICR_IIDR):
>          if ( dabt.size != DABT_WORD ) goto bad_width;
> @@ -182,16 +184,19 @@ static int __vgic_v3_rdistr_rd_mmio_read(struct vcpu *v, mmio_info_t *info,
>          uint64_t typer, aff;
>
>          if ( !vgic_reg64_check_access(dabt) ) goto bad_width;
> -        /* TBD: Update processor id in [23:8] when ITS support is added */
>          aff = (MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 3) << 56 |
>                 MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 2) << 48 |
>                 MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 1) << 40 |
>                 MPIDR_AFFINITY_LEVEL(v->arch.vmpidr, 0) << 32);
>          typer = aff;
> +        typer |= (v->vcpu_id & 0xffff) << 8;

Please explain in the commit message why this change. At first glance, 
this should be a separate patch.

>
>          if ( v->arch.vgic.flags & VGIC_V3_RDIST_LAST )
>              typer |= GICR_TYPER_LAST;
>
> +        if ( v->domain->arch.vgic.has_its )
> +            typer |= GICR_TYPER_PLPIS;
> +
>          *r = vgic_reg64_extract(typer, info);
>
>          return 1;
> @@ -502,6 +507,40 @@ bool vgic_can_inject_lpi(struct vcpu *vcpu, uint32_t vlpi)
>      return false;
>  }
>
> +static void vgic_vcpu_enable_lpis(struct vcpu *v)
> +{
> +    uint64_t reg = v->domain->arch.vgic.rdist_propbase;
> +    unsigned int nr_lpis = BIT((reg & 0x1f) + 1) - LPI_OFFSET;
> +    int nr_pages;
> +
> +    /* The first VCPU to enable LPIs maps the property table. */
> +    if ( !v->domain->arch.vgic.proptable )
> +    {
> +        v->domain->arch.vgic.nr_lpis = nr_lpis;
> +        nr_pages = DIV_ROUND_UP(nr_lpis, PAGE_SIZE);
> +
> +        get_guest_pages(v->domain, reg & GENMASK(51, 12), nr_pages);
> +        v->domain->arch.vgic.proptable = map_guest_pages(v->domain,
> +                                                         reg & GENMASK(51, 12),
> +                                                         nr_pages);

What is map_guest_pages fail?

> +        printk("VGIC-v3: VCPU%d mapped %d pages for property table\n",
> +               v->vcpu_id, nr_pages);

Please don't use printk in function called by a guest, as a guest could 
potentially DOS xen.

I think this should be gdprintk(XENLOG_DEBUG,...)

> +    }
> +    nr_pages = DIV_ROUND_UP(((nr_lpis + LPI_OFFSET) / 8), PAGE_SIZE);
> +    reg = v->arch.vgic.rdist_pendbase;
> +
> +    get_guest_pages(v->domain, reg & GENMASK(51, 12), nr_pages);
> +    v->arch.vgic.pendtable = map_guest_pages(v->domain,
> +                                             reg & GENMASK(51, 12), nr_pages);
> +
> +    printk("VGIC-v3: VCPU%d mapped %d pages for pending table\n",
> +           v->vcpu_id, nr_pages);

Ditto.

> +
> +    v->arch.vgic.flags |= VGIC_V3_LPIS_ENABLED;
> +
> +    printk("VGICv3: enabled %d LPIs for VCPU%d\n", nr_lpis, v->vcpu_id);

Ditto.

> +}
> +
>  static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
>                                            uint32_t gicr_reg,
>                                            register_t r)
> @@ -512,8 +551,18 @@ static int __vgic_v3_rdistr_rd_mmio_write(struct vcpu *v, mmio_info_t *info,
>      switch ( gicr_reg )
>      {
>      case VREG32(GICR_CTLR):
> -        /* LPI's not implemented */
> -        goto write_ignore_32;
> +        if ( dabt.size != DABT_WORD ) goto bad_width;
> +        if ( !v->domain->arch.vgic.has_its )
> +            return 1;
> +
> +        /* LPIs can only be enabled once, but never disabled again. */
> +        if ( !(r & GICR_CTLR_ENABLE_LPIS) ||
> +             (v->arch.vgic.flags & VGIC_V3_LPIS_ENABLED) )

This is fragile. A re-distributor can be updated from any vCPU. So you 
would end up to call vgic_vcpu_enable_lpis twice. You like want to use a 
lock protecting the GICR_CTLR emulation.

> +            return 1;
> +
> +        vgic_vcpu_enable_lpis(v);
> +
> +        return 1;
>
>      case VREG32(GICR_IIDR):
>          /* RO */
> @@ -1113,6 +1162,11 @@ static int vgic_v3_distr_mmio_read(struct vcpu *v, mmio_info_t *info,
>          typer = ((ncpus - 1) << GICD_TYPE_CPUS_SHIFT |
>                   DIV_ROUND_UP(v->domain->arch.vgic.nr_spis, 32));
>
> +        if ( v->domain->arch.vgic.has_its )
> +        {
> +            typer |= GICD_TYPE_LPIS;
> +            irq_bits = 16;

Why 16?

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-24 11:40   ` Julien Grall
@ 2017-03-24 15:50     ` Andre Przywara
  2017-03-24 16:19       ` Julien Grall
  2017-03-24 17:26       ` Stefano Stabellini
  0 siblings, 2 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-24 15:50 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 11:40, Julien Grall wrote:
> Hi Andre
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> diff --git a/xen/arch/arm/vgic.c b/xen/arch/arm/vgic.c
>> index 364d5f0..e5cfa54 100644
>> --- a/xen/arch/arm/vgic.c
>> +++ b/xen/arch/arm/vgic.c
>> @@ -30,6 +30,8 @@
>>
>>  #include <asm/mmio.h>
>>  #include <asm/gic.h>
>> +#include <asm/gic_v3_defs.h>
>> +#include <asm/gic_v3_its.h>
> 
> I really don't want to see gic_v3_* headers included in common code. Why
> do you need them?
> 
>>  #include <asm/vgic.h>
>>
>>  static inline struct vgic_irq_rank *vgic_get_rank(struct vcpu *v, int
>> rank)
>> @@ -61,7 +63,7 @@ struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v,
>> unsigned int irq)
>>      return vgic_get_rank(v, rank);
>>  }
>>
>> -static void vgic_init_pending_irq(struct pending_irq *p, unsigned int
>> virq)
>> +void vgic_init_pending_irq(struct pending_irq *p, unsigned int virq)
>>  {
>>      INIT_LIST_HEAD(&p->inflight);
>>      INIT_LIST_HEAD(&p->lr_queue);
>> @@ -244,10 +246,14 @@ struct vcpu *vgic_get_target_vcpu(struct vcpu
>> *v, unsigned int virq)
>>
>>  static int vgic_get_virq_priority(struct vcpu *v, unsigned int virq)
>>  {
>> -    struct vgic_irq_rank *rank = vgic_rank_irq(v, virq);
>> +    struct vgic_irq_rank *rank;
>>      unsigned long flags;
>>      int priority;
>>
>> +    if ( is_lpi(virq) )
>> +        return vgic_lpi_get_priority(v->domain, virq);
> 
> This would benefits some comments to explain why LPI pending handling is
> a different path.
> 
>> +
>> +    rank = vgic_rank_irq(v, virq);
>>      vgic_lock_rank(v, rank, flags);
>>      priority = rank->priority[virq & INTERRUPT_RANK_MASK];
>>      vgic_unlock_rank(v, rank, flags);
>> @@ -446,13 +452,63 @@ bool vgic_to_sgi(struct vcpu *v, register_t
>> sgir, enum gic_sgi_mode irqmode,
>>      return true;
>>  }
>>
>> +/*
>> + * Holding struct pending_irq's for each possible virtual LPI in each
>> domain
>> + * requires too much Xen memory, also a malicious guest could
>> potentially
>> + * spam Xen with LPI map requests. We cannot cover those with (guest
>> allocated)
>> + * ITS memory, so we use a dynamic scheme of allocating struct
>> pending_irq's
>> + * on demand.
>> + */
> 
> I am afraid this will not prevent a guest to use too much Xen memory.
> Let's imagine the guest is mapping thousands of LPIs but decides to
> never handle them or is slow. You would allocate pending_irq for each
> LPI, and never release the memory.

So what would be the alternative here? In the current GICv2/3 scheme a
pending_irq for each (SPI) interrupt is allocated up front. This
approach here tries just to be a bit smarter for LPIs, since the
expectation is that we by far don't need so many. In the worst case the
memory allocation would converge to the upper limit (number of mapped
LPIs), which would probably be used otherwise for the static allocation
scheme. Which means we would never be worse.
What actually is missing is the freeing the memory upon domain
destruction, which can't be tested for Dom0.

> If we use dynamic allocation, we need a way to limit the number of LPIs
> received by a guest to avoid memory exhaustion. The only idea I have is
> an artificial limit, but I don't think it is good. Any other ideas?

I think if we are concerned about this, we shouldn't allow *DomUs* to
allocate too many LPIs, which are bounded by the DeviceID/EventID
combinations. This will be under full control by Dom0, which will
communicate the number of MSIs (events) for each passthrough-ed device,
so we can easily impose a policy limit upon creation.

And just to be sure: we are at most allocating one structure per event.
An MSI interrupt storm would still converge into one pending LPI
(struct), so wouldn't trigger any further allocations.
Remapping LPIs won't increase that as well, since we only can assign one
LPI to a DevID/EvID pair at any given point in time. Remapping requires
the pending bit to be cleared. And the structures are not indexed by the
LPI number, but just take a free slot.

For Dom0 I don't see any good ways of limiting the number of LPIs, but
we shouldn't need any.

>> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
>> +                                   bool allocate)
>> +{
>> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
>> +
>> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
>> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
>> +    {
>> +        if ( lpi_irq->pirq.irq == lpi )
>> +        {
>> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>> +            return &lpi_irq->pirq;
>> +        }
>> +
>> +        if ( lpi_irq->pirq.irq == 0 && !empty )
>> +            empty = lpi_irq;
>> +    }
>> +
>> +    if ( !allocate )
>> +    {
>> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>> +        return NULL;
>> +    }
>> +
>> +    if ( !empty )
>> +    {
>> +        empty = xzalloc(struct lpi_pending_irq);
> 
> xzalloc will return NULL if memory is exhausted. There is a general lack
> of error checking within this series. Any missing error could be a
> potential target from a guest, leading to security issue. Stefano and I
> already spot some, it does not mean we found all. So Before sending the
> next version, please go through the series and verify *every* return.
> 
> Also, I can't find the code which release LPIs neither in this patch nor
> in this series. A general rule is too have allocation and free within
> the same patch. It is much easier to spot missing free.

There is no such code, on purpose. We only grow the number, but never
shrink it (to what?, where to stop?, what if we need more again?). As
said above, in the worst case this ends up at something where a static
allocation would have started with from the beginning.

And as mentioned, I only skipped the "free the whole list upon domain
destruction" part.

Cheers,
Andre.

>> +        vgic_init_pending_irq(&empty->pirq, lpi);
>> +        list_add_tail(&empty->entry, &v->arch.vgic.pending_lpi_list);
>> +    } else
>> +    {
>> +        empty->pirq.status = 0;
>> +        empty->pirq.irq = lpi;
>> +    }
>> +
>> +    spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>> +
>> +    return &empty->pirq;
>> +}
>> +
>>  struct pending_irq *irq_to_pending(struct vcpu *v, unsigned int irq)
>>  {
>>      struct pending_irq *n;
>> +
> 
> Spurious change.
> 
>>      /* Pending irqs allocation strategy: the first vgic.nr_spis irqs
>>       * are used for SPIs; the rests are used for per cpu irqs */
>>      if ( irq < 32 )
>>          n = &v->arch.vgic.pending_irqs[irq];
>> +    else if ( is_lpi(irq) )
>> +        n = lpi_to_pending(v, irq, true);
>>      else
>>          n = &v->domain->arch.vgic.pending_irqs[irq - 32];
>>      return n;
>> @@ -480,7 +536,7 @@ void vgic_clear_pending_irqs(struct vcpu *v)
>>  void vgic_vcpu_inject_irq(struct vcpu *v, unsigned int virq)
>>  {
>>      uint8_t priority;
>> -    struct pending_irq *iter, *n = irq_to_pending(v, virq);
>> +    struct pending_irq *iter, *n;
>>      unsigned long flags;
>>      bool running;
>>
>> @@ -488,6 +544,8 @@ void vgic_vcpu_inject_irq(struct vcpu *v, unsigned
>> int virq)
>>
>>      spin_lock_irqsave(&v->arch.vgic.lock, flags);
>>
>> +    n = irq_to_pending(v, virq);
> 
> Why did you move this code?
> 
>> +
>>      /* vcpu offline */
>>      if ( test_bit(_VPF_down, &v->pause_flags) )
>>      {
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 00b9c1a..f44a84b 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -257,6 +257,8 @@ struct arch_vcpu
>>          paddr_t rdist_base;
>>  #define VGIC_V3_RDIST_LAST  (1 << 0)        /* last vCPU of the rdist */
>>          uint8_t flags;
>> +        struct list_head pending_lpi_list;
>> +        spinlock_t pending_lpi_list_lock;   /* protects the
>> pending_lpi_list */
> 
> 
> It would be better to have this structure per-domain limiting the amount
> of memory allocating. Also, Stefano was suggesting to use a more
> efficient data structure, such as an hashtable or a tree.
> 
> I would be ok to focus on the correctness so far, but this would need to
> be address before ITS is marked as stable.
> 
>>  extern struct vgic_irq_rank *vgic_rank_offset(struct vcpu *v, int b,
>> int n, int s);
>>  extern struct vgic_irq_rank *vgic_rank_irq(struct vcpu *v, unsigned
>> int irq);
>>  extern bool vgic_emulate(struct cpu_user_regs *regs, union hsr hsr);
>>  extern void vgic_disable_irqs(struct vcpu *v, uint32_t r, int n);
>>  extern void vgic_enable_irqs(struct vcpu *v, uint32_t r, int n);
>> +/* placeholder function until the property table gets introduced */
>> +static inline int vgic_lpi_get_priority(struct domain *d, uint32_t vlpi)
>> +{
>> +    return 0xa;
>> +}
> 
> To be fair, you can avoid this function by re-ordering the patches. As
> suggested earlier, I would introduce some bits of the vITS before. This
> would also make the series easier to read.
> 
> Also, looking how it has been implemented later. I would prefer to see a
> new callback get_priority in vgic_ops which will return the correct
> priority.
> 
> I agree this would introduce a bit more of duplicated code. But it would
> limit the use of is_lpi in the common code.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 16/27] ARM: vITS: handle CLEAR command
  2017-03-24 14:27   ` Julien Grall
@ 2017-03-24 15:53     ` Andre Przywara
  2017-03-24 17:17       ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-24 15:53 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 14:27, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> This introduces the ITS command handler for the CLEAR command, which
>> clears the pending state of an LPI.
>> This removes a not-yet injected, but already queued IRQ from a VCPU.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/vgic-v3-its.c | 35 +++++++++++++++++++++++++++++++++--
>>  1 file changed, 33 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
>> index 267a573..e808f43 100644
>> --- a/xen/arch/arm/vgic-v3-its.c
>> +++ b/xen/arch/arm/vgic-v3-its.c

[....]

>> @@ -236,6 +264,9 @@ static int vgic_its_handle_cmds(struct domain *d,
>> struct virt_its *its,
>>          cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
>>          switch (its_cmd_get_command(cmdptr))
>>          {
>> +        case GITS_CMD_CLEAR:
>> +            its_handle_clear(its, cmdptr);
> 
> Should not you check the return for its_handle_clear?

That sounds obvious, but actually I don't know of a good way of handling
this. Blame the architecture, if you like. Passing the error value up
would end up in the MMIO handler, where it is not correct to return an
error (since the CWRITER MMIO access itself was successful).

So I picked one of the behaviors described in 6.3.2 "Command errors",
which is simply to ignore the command.
If we have a nice way of injecting an SError (do we?), I _could_ check
GITS_TYPER.SEIS and then inject it. But effectively this would be
untested code, since Linux does not use this feature.

So any idea here? I don't think the typical Xen answer of "Crash the
guest!" is compliant with the architecture, which leaves three choices,
and setting the box on fire is not one of them.

That's why I chose to ignore the return value at this point, but at
least generate the error condition internally and bail out early. At
least for the Tech Preview Dom0 edition of the ITS emulation.
If we later gain a good method of handling (and testing!) command
errors, we can easily add them.

Cheers,
Andre.

> 
>> +            break;
>>          case GITS_CMD_SYNC:
>>              /* We handle ITS commands synchronously, so we ignore
>> SYNC. */
>>          break;
>>
> 
> Regards,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-24 15:50     ` Andre Przywara
@ 2017-03-24 16:19       ` Julien Grall
  2017-03-24 17:26       ` Stefano Stabellini
  1 sibling, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-24 16:19 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/24/2017 03:50 PM, Andre Przywara wrote:
> On 24/03/17 11:40, Julien Grall wrote:
>>> +/*
>>> + * Holding struct pending_irq's for each possible virtual LPI in each
>>> domain
>>> + * requires too much Xen memory, also a malicious guest could
>>> potentially
>>> + * spam Xen with LPI map requests. We cannot cover those with (guest
>>> allocated)
>>> + * ITS memory, so we use a dynamic scheme of allocating struct
>>> pending_irq's
>>> + * on demand.
>>> + */
>>
>> I am afraid this will not prevent a guest to use too much Xen memory.
>> Let's imagine the guest is mapping thousands of LPIs but decides to
>> never handle them or is slow. You would allocate pending_irq for each
>> LPI, and never release the memory.
>
> So what would be the alternative here? In the current GICv2/3 scheme a
> pending_irq for each (SPI) interrupt is allocated up front. This
> approach here tries just to be a bit smarter for LPIs, since the
> expectation is that we by far don't need so many. In the worst case the
> memory allocation would converge to the upper limit (number of mapped
> LPIs), which would probably be used otherwise for the static allocation
> scheme. Which means we would never be worse.

It is getting worse in the current version because you have the list per 
vCPU. So you may get nLPIs * nvCPUs pending_irq allocated.

> What actually is missing is the freeing the memory upon domain
> destruction, which can't be tested for Dom0.

As I already said multiple time, forgetting to add the domain 
destruction is much worse because this is a call to miss something when 
we will need it. I prefer to see the code now so we can review.

>
>> If we use dynamic allocation, we need a way to limit the number of LPIs
>> received by a guest to avoid memory exhaustion. The only idea I have is
>> an artificial limit, but I don't think it is good. Any other ideas?
>
> I think if we are concerned about this, we shouldn't allow *DomUs* to
> allocate too many LPIs, which are bounded by the DeviceID/EventID
> combinations. This will be under full control by Dom0, which will
> communicate the number of MSIs (events) for each passthrough-ed device,
> so we can easily impose a policy limit upon creation.

If DOM0 is able to control the number of DeviceID/EventID, then why do 
we need to allocate on the fly?

Also, I don't think the spec prevents to populate Device Table for a 
deviceID that has no Device associated. It could be used by a domain to 
inject fake LPI. For instance this could replace polling mode for the 
command queue.

My main concern is memory allocation can fail. Then what will you do? 
You will penalize a well-behave domain that because someone else was nasty.

And you have no way to report to the guest: "I was not able to allocate 
memory, please try later" because this is an interrupt.

>>> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
>>> +                                   bool allocate)
>>> +{
>>> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
>>> +
>>> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
>>> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
>>> +    {
>>> +        if ( lpi_irq->pirq.irq == lpi )
>>> +        {
>>> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>>> +            return &lpi_irq->pirq;
>>> +        }
>>> +
>>> +        if ( lpi_irq->pirq.irq == 0 && !empty )
>>> +            empty = lpi_irq;
>>> +    }
>>> +
>>> +    if ( !allocate )
>>> +    {
>>> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    if ( !empty )
>>> +    {
>>> +        empty = xzalloc(struct lpi_pending_irq);
>>
>> xzalloc will return NULL if memory is exhausted. There is a general lack
>> of error checking within this series. Any missing error could be a
>> potential target from a guest, leading to security issue. Stefano and I
>> already spot some, it does not mean we found all. So Before sending the
>> next version, please go through the series and verify *every* return.
>>
>> Also, I can't find the code which release LPIs neither in this patch nor
>> in this series. A general rule is too have allocation and free within
>> the same patch. It is much easier to spot missing free.
>
> There is no such code, on purpose. We only grow the number, but never
> shrink it (to what?, where to stop?, what if we need more again?). As
> said above, in the worst case this ends up at something where a static
> allocation would have started with from the beginning.
>
> And as mentioned, I only skipped the "free the whole list upon domain
> destruction" part.

A general rule is to explain in the commit message what is done. So you 
avoid argument on the ML.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 16/27] ARM: vITS: handle CLEAR command
  2017-03-24 15:53     ` Andre Przywara
@ 2017-03-24 17:17       ` Stefano Stabellini
  2017-03-27  8:44         ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-24 17:17 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Fri, 24 Mar 2017, Andre Przywara wrote:
> Hi,
> 
> On 24/03/17 14:27, Julien Grall wrote:
> > Hi Andre,
> > 
> > On 03/16/2017 11:20 AM, Andre Przywara wrote:
> >> This introduces the ITS command handler for the CLEAR command, which
> >> clears the pending state of an LPI.
> >> This removes a not-yet injected, but already queued IRQ from a VCPU.
> >>
> >> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> >> ---
> >>  xen/arch/arm/vgic-v3-its.c | 35 +++++++++++++++++++++++++++++++++--
> >>  1 file changed, 33 insertions(+), 2 deletions(-)
> >>
> >> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
> >> index 267a573..e808f43 100644
> >> --- a/xen/arch/arm/vgic-v3-its.c
> >> +++ b/xen/arch/arm/vgic-v3-its.c
> 
> [....]
> 
> >> @@ -236,6 +264,9 @@ static int vgic_its_handle_cmds(struct domain *d,
> >> struct virt_its *its,
> >>          cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
> >>          switch (its_cmd_get_command(cmdptr))
> >>          {
> >> +        case GITS_CMD_CLEAR:
> >> +            its_handle_clear(its, cmdptr);
> > 
> > Should not you check the return for its_handle_clear?
> 
> That sounds obvious, but actually I don't know of a good way of handling
> this. Blame the architecture, if you like. Passing the error value up
> would end up in the MMIO handler, where it is not correct to return an
> error (since the CWRITER MMIO access itself was successful).
> 
> So I picked one of the behaviors described in 6.3.2 "Command errors",
> which is simply to ignore the command.
> If we have a nice way of injecting an SError (do we?),

We do with Wei's series, which is very likely to go in before this one:

http://marc.info/?l=xen-devel&m=148940261914581

In particular, see 1489402563-4978-7-git-send-email-Wei.Chen@arm.com.


> I _could_ check GITS_TYPER.SEIS and then inject it. But effectively
> this would be untested code, since Linux does not use this feature.

Keep in mind that Xen supports a wide range of OSes. GITS_TYPER is
emulated by Xen in the virtual ITS, right? If so, it doesn't matter the
hardware value of SEIS, we can set the virtual value to 1.


> So any idea here? I don't think the typical Xen answer of "Crash the
> guest!" is compliant with the architecture, which leaves three choices,
> and setting the box on fire is not one of them.
> 
> That's why I chose to ignore the return value at this point, but at
> least generate the error condition internally and bail out early. At
> least for the Tech Preview Dom0 edition of the ITS emulation.
> If we later gain a good method of handling (and testing!) command
> errors, we can easily add them.

At the very least, we need to print a warning (rate limited, so probably
gdprintk). All errors that cannot be handled otherwise need to result in
a warning.

Sending an SError would be fine, I think.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table
  2017-03-24 11:45                 ` Julien Grall
@ 2017-03-24 17:22                   ` Stefano Stabellini
  0 siblings, 0 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-24 17:22 UTC (permalink / raw)
  To: Julien Grall
  Cc: Andre Przywara, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari, xen-devel

On Fri, 24 Mar 2017, Julien Grall wrote:
> Hi Andre,
> 
> On 03/23/2017 06:21 PM, Andre Przywara wrote:
> > Hi,
> > 
> > On 23/03/17 18:01, Stefano Stabellini wrote:
> > > On Thu, 23 Mar 2017, Julien Grall wrote:
> > > > Hi Stefano,
> > > > 
> > > > On 23/03/17 17:45, Stefano Stabellini wrote:
> > > > > On Thu, 23 Mar 2017, Julien Grall wrote:
> > > > > > > So as I mentioned before, I am happy to loose the Kconfig option,
> > > > > > > but
> > > > > > > then we need some sensible default value. In our case we could be
> > > > > > > cheeky
> > > > > > > here for now and just use the Linux value, because a Linux Dom0
> > > > > > > would be
> > > > > > > the only user. But that doesn't sound very future proof, though
> > > > > > > this may
> > > > > > > not matter for 4.9.
> > > > > > 
> > > > > > I don't think we need a sensible default value and IHMO there is
> > > > > > none. I
> > > > > > would
> > > > > > left the user to decide the exact number.
> > > > > 
> > > > > In that case, the command line parameter becomes mandatory: we need to
> > > > > force the user to specify it as we do for dom0_mem today.
> > > > 
> > > > Not really. We should use the hardware value by default. If a user
> > > > thinks the
> > > > number allocated is too big for his use case, then it can limit using
> > > > the
> > > > command line.
> > > > 
> > > > Anyway, I will not oppose to make this command option mandatory when ITS
> > > > is in
> > > > use.
> > > 
> > > Andre wrote:
> > > 
> > >   Any redistributor supporting 32 bits worth of LPIs would lead to a
> > >   4GB property table and 512MB pending table allocation.
> > > 
> > > Let's assume that such scenario is realistic (if it is not, then this
> > > discussion is fruitless), in this case the user most surely is not going
> > > to want to use the hardware provided value. But how can she knows it?
> > > How can she find out that she is wasting too much memory on her system?
> > > Is there an easy and obvious way to know?
> > > 
> > > She could find out if Xen printed a big warning such as:
> > > 
> > >   USING 4G OF MEMORY FOR ITS PROPTABLE, CONSIDER PASSING max_lpi_bits to
> > > XEN
> > > 
> > > but to do that, we need a threshold value in Xen, above which the
> > > hypervisor prints the warning. But if we have a threshold value in Xen,
> > > then we might as well consider making it the default ceiling: Xen uses
> > > the hardware provided value, unless it's greater than threshold, in that
> > > case it uses threshold and prints a warning, for example:
> > > 
> > >   LIMITING ITS PROPTABLE MEMORY TO 1G, CHANGE IT WITH max_lpi_bits
> > > PARAMETER
> > > 
> > > 
> > > Does it make sense?
> > 
> > Yes, that is exactly what I was after.
> > In contrast to the number of device IDs I think the number of LPI bits
> > is _not_ a value that software should use directly, it's more a limit,
> > actually a GICv3 implementation choice (how wide the LPI ID fields
> > internally are, for instance).
> > 
> > So do we want to limit to 1GB and warn starting at 256MB?
> 
> I think Stefano was suggesting to make the command line mandatory.

Either way works, but the idea is that the user should not be surprised
by the amount of memory utilized because either

1) she chose max_lpi_bits
2) Xen printed a WARNING at boot impossible to miss

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-24 15:50     ` Andre Przywara
  2017-03-24 16:19       ` Julien Grall
@ 2017-03-24 17:26       ` Stefano Stabellini
  2017-03-27  9:02         ` Andre Przywara
  1 sibling, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-24 17:26 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari

On Fri, 24 Mar 2017, Andre Przywara wrote:
> >> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
> >> +                                   bool allocate)
> >> +{
> >> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
> >> +
> >> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
> >> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
> >> +    {
> >> +        if ( lpi_irq->pirq.irq == lpi )
> >> +        {
> >> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> >> +            return &lpi_irq->pirq;
> >> +        }
> >> +
> >> +        if ( lpi_irq->pirq.irq == 0 && !empty )
> >> +            empty = lpi_irq;
> >> +    }
> >> +
> >> +    if ( !allocate )
> >> +    {
> >> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
> >> +        return NULL;
> >> +    }
> >> +
> >> +    if ( !empty )
> >> +    {
> >> +        empty = xzalloc(struct lpi_pending_irq);
> > 
> > xzalloc will return NULL if memory is exhausted. There is a general lack
> > of error checking within this series. Any missing error could be a
> > potential target from a guest, leading to security issue. Stefano and I
> > already spot some, it does not mean we found all. So Before sending the
> > next version, please go through the series and verify *every* return.
> > 
> > Also, I can't find the code which release LPIs neither in this patch nor
> > in this series. A general rule is too have allocation and free within
> > the same patch. It is much easier to spot missing free.
> 
> There is no such code, on purpose. We only grow the number, but never
> shrink it (to what?, where to stop?, what if we need more again?). As
> said above, in the worst case this ends up at something where a static
> allocation would have started with from the beginning.

Dellocate struct pending_irq when the domain is destroyed or when an LPI
is disabled?

> to what?, where to stop?

We don't stop, why stop? Let's go back to 0 if we can.

> what if we need more again?

We allocate them again?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 16/27] ARM: vITS: handle CLEAR command
  2017-03-24 17:17       ` Stefano Stabellini
@ 2017-03-27  8:44         ` Andre Przywara
  2017-03-27 14:12           ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-27  8:44 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 17:17, Stefano Stabellini wrote:
> On Fri, 24 Mar 2017, Andre Przywara wrote:
>> Hi,
>>
>> On 24/03/17 14:27, Julien Grall wrote:
>>> Hi Andre,
>>>
>>> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>>>> This introduces the ITS command handler for the CLEAR command, which
>>>> clears the pending state of an LPI.
>>>> This removes a not-yet injected, but already queued IRQ from a VCPU.
>>>>
>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>> ---
>>>>  xen/arch/arm/vgic-v3-its.c | 35 +++++++++++++++++++++++++++++++++--
>>>>  1 file changed, 33 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
>>>> index 267a573..e808f43 100644
>>>> --- a/xen/arch/arm/vgic-v3-its.c
>>>> +++ b/xen/arch/arm/vgic-v3-its.c
>>
>> [....]
>>
>>>> @@ -236,6 +264,9 @@ static int vgic_its_handle_cmds(struct domain *d,
>>>> struct virt_its *its,
>>>>          cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
>>>>          switch (its_cmd_get_command(cmdptr))
>>>>          {
>>>> +        case GITS_CMD_CLEAR:
>>>> +            its_handle_clear(its, cmdptr);
>>>
>>> Should not you check the return for its_handle_clear?
>>
>> That sounds obvious, but actually I don't know of a good way of handling
>> this. Blame the architecture, if you like. Passing the error value up
>> would end up in the MMIO handler, where it is not correct to return an
>> error (since the CWRITER MMIO access itself was successful).
>>
>> So I picked one of the behaviors described in 6.3.2 "Command errors",
>> which is simply to ignore the command.
>> If we have a nice way of injecting an SError (do we?),
> 
> We do with Wei's series, which is very likely to go in before this one:
> 
> http://marc.info/?l=xen-devel&m=148940261914581
> 
> In particular, see 1489402563-4978-7-git-send-email-Wei.Chen@arm.com.
> 
> 
>> I _could_ check GITS_TYPER.SEIS and then inject it. But effectively
>> this would be untested code, since Linux does not use this feature.
> 
> Keep in mind that Xen supports a wide range of OSes.

I clearly understand that and don't question it. What I wanted to point
out is that using an SError to signal ITS errors is mostly uncharted
territory (see the current discussion about handling SErrors in
Linux[1]). So we would add code that cannot be tested. And given the
current situation and the tech preview status of the ITS support I'd
prefer to not go there at the moment.

I would offer to annotate the error returns with the actual ITS error
codes (as in the KVM code, for instance [2]).
Then put a comment in the code explaining the missing error signalling
situation, and we create a ticket to notify ourselves of fixing this in
the future.
Does that make sense?

> GITS_TYPER is
> emulated by Xen in the virtual ITS, right? If so, it doesn't matter the
> hardware value of SEIS, we can set the virtual value to 1.
> 
> 
>> So any idea here? I don't think the typical Xen answer of "Crash the
>> guest!" is compliant with the architecture, which leaves three choices,
>> and setting the box on fire is not one of them.
>>
>> That's why I chose to ignore the return value at this point, but at
>> least generate the error condition internally and bail out early. At
>> least for the Tech Preview Dom0 edition of the ITS emulation.
>> If we later gain a good method of handling (and testing!) command
>> errors, we can easily add them.
> 
> At the very least, we need to print a warning (rate limited, so probably
> gdprintk). All errors that cannot be handled otherwise need to result in
> a warning.

OK, I will do this.

> Sending an SError would be fine, I think.

As mentioned above, I'd refrain from it at the moment.

Cheers,
Andre.

[1]
http://lists.infradead.org/pipermail/linux-arm-kernel/2017-March/496722.html
[2]
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/virt/kvm/arm/vgic/vgic-its.c#n561

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-24 17:26       ` Stefano Stabellini
@ 2017-03-27  9:02         ` Andre Przywara
  2017-03-27 14:01           ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-03-27  9:02 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 17:26, Stefano Stabellini wrote:
> On Fri, 24 Mar 2017, Andre Przywara wrote:
>>>> +struct pending_irq *lpi_to_pending(struct vcpu *v, unsigned int lpi,
>>>> +                                   bool allocate)
>>>> +{
>>>> +    struct lpi_pending_irq *lpi_irq, *empty = NULL;
>>>> +
>>>> +    spin_lock(&v->arch.vgic.pending_lpi_list_lock);
>>>> +    list_for_each_entry(lpi_irq, &v->arch.vgic.pending_lpi_list, entry)
>>>> +    {
>>>> +        if ( lpi_irq->pirq.irq == lpi )
>>>> +        {
>>>> +            spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>>>> +            return &lpi_irq->pirq;
>>>> +        }
>>>> +
>>>> +        if ( lpi_irq->pirq.irq == 0 && !empty )
>>>> +            empty = lpi_irq;
>>>> +    }
>>>> +
>>>> +    if ( !allocate )
>>>> +    {
>>>> +        spin_unlock(&v->arch.vgic.pending_lpi_list_lock);
>>>> +        return NULL;
>>>> +    }
>>>> +
>>>> +    if ( !empty )
>>>> +    {
>>>> +        empty = xzalloc(struct lpi_pending_irq);
>>>
>>> xzalloc will return NULL if memory is exhausted. There is a general lack
>>> of error checking within this series. Any missing error could be a
>>> potential target from a guest, leading to security issue. Stefano and I
>>> already spot some, it does not mean we found all. So Before sending the
>>> next version, please go through the series and verify *every* return.
>>>
>>> Also, I can't find the code which release LPIs neither in this patch nor
>>> in this series. A general rule is too have allocation and free within
>>> the same patch. It is much easier to spot missing free.
>>
>> There is no such code, on purpose. We only grow the number, but never
>> shrink it (to what?, where to stop?, what if we need more again?). As
>> said above, in the worst case this ends up at something where a static
>> allocation would have started with from the beginning.
> 
> Dellocate struct pending_irq when the domain is destroyed

Sure, I think this is what is missing at the moment.

> or when an LPI is disabled?

A certain struct pending_irq is not assigned to a particular LPI, it's
more a member of a pool to allocate from when in need.
I consider them like shadow list registers, which can in addition be
used as spill-overs.
Very much like an LR the pending_irq is only assigned to a certain LPI
for the lifetime of a *pending* LPI on a particular VCPU. As mentioned
earlier, under normal conditions this is very short for always edge
triggered LPIs which don't have an active state.

>> to what?, where to stop?
> 
> We don't stop, why stop? Let's go back to 0 if we can.
> 
>> what if we need more again?
> 
> We allocate them again?

I am afraid that this would lead to situations where we needlessly
allocate and deallocate pending_irqs. Under normal load I'd expect to
have something like zero to three LPIs pending at any given point in
time (mostly zero, to be honest).
So this will lead to a situation where *every* LPI that becomes pending
triggers a memory allocation - in the hot path. That's why the pool
idea. So if we are going to shrink the pool, I'd stop at something like
five entries, to not penalize the common case.
Does that sound useful?

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-27  9:02         ` Andre Przywara
@ 2017-03-27 14:01           ` Julien Grall
  2017-03-27 17:44             ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-27 14:01 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 27/03/17 10:02, Andre Przywara wrote:
> On 24/03/17 17:26, Stefano Stabellini wrote:
>> On Fri, 24 Mar 2017, Andre Przywara wrote:
> I am afraid that this would lead to situations where we needlessly
> allocate and deallocate pending_irqs. Under normal load I'd expect to
> have something like zero to three LPIs pending at any given point in
> time (mostly zero, to be honest).
> So this will lead to a situation where *every* LPI that becomes pending
> triggers a memory allocation - in the hot path. That's why the pool
> idea. So if we are going to shrink the pool, I'd stop at something like
> five entries, to not penalize the common case.
> Does that sound useful?

Not answering directly to the question here. I will summarize the face 
to face discussion I had with Andre this morning.

So allocating the pending_irq in the IRQ path is not a solution because 
memory allocation should not happen in IRQ context, see 
ASSERT(!in_irq()) in _xmalloc.

Regardless the ASSERT, it will also increase the time to handle and 
forward an interrupt when there are no pending_irq free because it is 
necessary to allocate a new one. Lastly, we have no way to tell the 
guest: "Try again" if it Xen is running out of memory.

The outcome of the discussion is to pre-allocate the pending_irq when a 
device is assigned to a domain. We know the maximum number of event 
supported by a device and that 1 event = 1 LPI.

This may allocate more memory (a pending_irq is 56 bytes), but at least 
we don't need allocation on the fly and can report error.

One could argue that we could allocate on MAPTI to limit the allocation. 
However, as we are not able to rate-limit/defer the execution of the 
command queue so far, a guest could potentially flood with MAPTI and 
monopolize the pCPU for a long time.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 16/27] ARM: vITS: handle CLEAR command
  2017-03-27  8:44         ` Andre Przywara
@ 2017-03-27 14:12           ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-27 14:12 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 27/03/17 09:44, Andre Przywara wrote:
> Hi,
>
> On 24/03/17 17:17, Stefano Stabellini wrote:
>> On Fri, 24 Mar 2017, Andre Przywara wrote:
>>> Hi,
>>>
>>> On 24/03/17 14:27, Julien Grall wrote:
>>>> Hi Andre,
>>>>
>>>> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>>>>> This introduces the ITS command handler for the CLEAR command, which
>>>>> clears the pending state of an LPI.
>>>>> This removes a not-yet injected, but already queued IRQ from a VCPU.
>>>>>
>>>>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>>>>> ---
>>>>>  xen/arch/arm/vgic-v3-its.c | 35 +++++++++++++++++++++++++++++++++--
>>>>>  1 file changed, 33 insertions(+), 2 deletions(-)
>>>>>
>>>>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
>>>>> index 267a573..e808f43 100644
>>>>> --- a/xen/arch/arm/vgic-v3-its.c
>>>>> +++ b/xen/arch/arm/vgic-v3-its.c
>>>
>>> [....]
>>>
>>>>> @@ -236,6 +264,9 @@ static int vgic_its_handle_cmds(struct domain *d,
>>>>> struct virt_its *its,
>>>>>          cmdptr = its->cmdbuf + (its->creadr / sizeof(*its->cmdbuf));
>>>>>          switch (its_cmd_get_command(cmdptr))
>>>>>          {
>>>>> +        case GITS_CMD_CLEAR:
>>>>> +            its_handle_clear(its, cmdptr);
>>>>
>>>> Should not you check the return for its_handle_clear?
>>>
>>> That sounds obvious, but actually I don't know of a good way of handling
>>> this. Blame the architecture, if you like. Passing the error value up
>>> would end up in the MMIO handler, where it is not correct to return an
>>> error (since the CWRITER MMIO access itself was successful).
>>>
>>> So I picked one of the behaviors described in 6.3.2 "Command errors",
>>> which is simply to ignore the command.
>>> If we have a nice way of injecting an SError (do we?),
>>
>> We do with Wei's series, which is very likely to go in before this one:
>>
>> http://marc.info/?l=xen-devel&m=148940261914581
>>
>> In particular, see 1489402563-4978-7-git-send-email-Wei.Chen@arm.com.
>>
>>
>>> I _could_ check GITS_TYPER.SEIS and then inject it. But effectively
>>> this would be untested code, since Linux does not use this feature.
>>
>> Keep in mind that Xen supports a wide range of OSes.
>
> I clearly understand that and don't question it. What I wanted to point
> out is that using an SError to signal ITS errors is mostly uncharted
> territory (see the current discussion about handling SErrors in
> Linux[1]). So we would add code that cannot be tested.

The SError is sent or not, it is not important whether the OS is able to 
handle it or not.

And justifying with "the code cannot be tested" is not argument as I 
would have expected you to hack Linux to exercise the vITS.

> And given the
> current situation and the tech preview status of the ITS support I'd
> prefer to not go there at the moment.
>
> I would offer to annotate the error returns with the actual ITS error
> codes (as in the KVM code, for instance [2]).

I am happy with that as a first step.

> Then put a comment in the code explaining the missing error signalling
> situation, and we create a ticket to notify ourselves of fixing this in
> the future.

I think we still need to log the error. It is useful for the user to 
know what's going on. You could imagine someone trying to implement an 
OS using Xen and KVM. He would be really grateful to know what's going on.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-27 14:01           ` Julien Grall
@ 2017-03-27 17:44             ` Stefano Stabellini
  2017-03-27 17:49               ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-27 17:44 UTC (permalink / raw)
  To: Julien Grall
  Cc: Andre Przywara, Stefano Stabellini, Shanker Donthineni,
	Vijay Kilari, xen-devel

On Mon, 27 Mar 2017, Julien Grall wrote:
> Hi,
> 
> On 27/03/17 10:02, Andre Przywara wrote:
> > On 24/03/17 17:26, Stefano Stabellini wrote:
> > > On Fri, 24 Mar 2017, Andre Przywara wrote:
> > I am afraid that this would lead to situations where we needlessly
> > allocate and deallocate pending_irqs. Under normal load I'd expect to
> > have something like zero to three LPIs pending at any given point in
> > time (mostly zero, to be honest).
> > So this will lead to a situation where *every* LPI that becomes pending
> > triggers a memory allocation - in the hot path. That's why the pool
> > idea. So if we are going to shrink the pool, I'd stop at something like
> > five entries, to not penalize the common case.
> > Does that sound useful?
> 
> Not answering directly to the question here. I will summarize the face to face
> discussion I had with Andre this morning.
> 
> So allocating the pending_irq in the IRQ path is not a solution because memory
> allocation should not happen in IRQ context, see ASSERT(!in_irq()) in
> _xmalloc.
> 
> Regardless the ASSERT, it will also increase the time to handle and forward an
> interrupt when there are no pending_irq free because it is necessary to
> allocate a new one. Lastly, we have no way to tell the guest: "Try again" if
> it Xen is running out of memory.
> 
> The outcome of the discussion is to pre-allocate the pending_irq when a device
> is assigned to a domain. We know the maximum number of event supported by a
> device and that 1 event = 1 LPI.
> 
> This may allocate more memory (a pending_irq is 56 bytes), but at least we
> don't need allocation on the fly and can report error.
> 
> One could argue that we could allocate on MAPTI to limit the allocation.
> However, as we are not able to rate-limit/defer the execution of the command
> queue so far, a guest could potentially flood with MAPTI and monopolize the
> pCPU for a long time.

It makes a lot of sense to keep the allocation out of the irq path.
However, I am wondering if the allocation/deallocation of pending_irq
structs could be done at the point the vLPIs are enabled/disabled,
instead of device assignment time.

In any case, there should be code for the deallocation. If we keep the
allocation at device assignment time, there should be code for the
deallocation when a device is remove from the guest (even if we cannot
test that case well now).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-27 17:44             ` Stefano Stabellini
@ 2017-03-27 17:49               ` Julien Grall
  2017-03-27 18:39                 ` Stefano Stabellini
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-27 17:49 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Andre Przywara, Shanker Donthineni, Vijay Kilari, xen-devel

Hi Stefano,

On 27/03/17 18:44, Stefano Stabellini wrote:
> On Mon, 27 Mar 2017, Julien Grall wrote:
>> Hi,
>>
>> On 27/03/17 10:02, Andre Przywara wrote:
>>> On 24/03/17 17:26, Stefano Stabellini wrote:
>>>> On Fri, 24 Mar 2017, Andre Przywara wrote:
>>> I am afraid that this would lead to situations where we needlessly
>>> allocate and deallocate pending_irqs. Under normal load I'd expect to
>>> have something like zero to three LPIs pending at any given point in
>>> time (mostly zero, to be honest).
>>> So this will lead to a situation where *every* LPI that becomes pending
>>> triggers a memory allocation - in the hot path. That's why the pool
>>> idea. So if we are going to shrink the pool, I'd stop at something like
>>> five entries, to not penalize the common case.
>>> Does that sound useful?
>>
>> Not answering directly to the question here. I will summarize the face to face
>> discussion I had with Andre this morning.
>>
>> So allocating the pending_irq in the IRQ path is not a solution because memory
>> allocation should not happen in IRQ context, see ASSERT(!in_irq()) in
>> _xmalloc.
>>
>> Regardless the ASSERT, it will also increase the time to handle and forward an
>> interrupt when there are no pending_irq free because it is necessary to
>> allocate a new one. Lastly, we have no way to tell the guest: "Try again" if
>> it Xen is running out of memory.
>>
>> The outcome of the discussion is to pre-allocate the pending_irq when a device
>> is assigned to a domain. We know the maximum number of event supported by a
>> device and that 1 event = 1 LPI.
>>
>> This may allocate more memory (a pending_irq is 56 bytes), but at least we
>> don't need allocation on the fly and can report error.
>>
>> One could argue that we could allocate on MAPTI to limit the allocation.
>> However, as we are not able to rate-limit/defer the execution of the command
>> queue so far, a guest could potentially flood with MAPTI and monopolize the
>> pCPU for a long time.
>
> It makes a lot of sense to keep the allocation out of the irq path.
> However, I am wondering if the allocation/deallocation of pending_irq
> structs could be done at the point the vLPIs are enabled/disabled,
> instead of device assignment time.

I am not sure what you mean by enabling/disabling vLPIS. Do you mean 
when the guest is enabling/disabling them or the guest will assign a 
vLPI to a (deviceID, event) via MAPTI.

For both guest could potentially flood us. It would take us a lot of 
time to allocate/free memory for each vLPIs modified. Hence, why I 
didn't suggest it and said: "One could argue that we could allocate on 
MAPTI to limit the allocation...".

>
> In any case, there should be code for the deallocation. If we keep the
> allocation at device assignment time, there should be code for the
> deallocation when a device is remove from the guest (even if we cannot
> test that case well now).

It was implied in my answered :).

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-27 17:49               ` Julien Grall
@ 2017-03-27 18:39                 ` Stefano Stabellini
  2017-03-27 21:24                   ` Julien Grall
  2017-03-28  7:58                   ` Jan Beulich
  0 siblings, 2 replies; 119+ messages in thread
From: Stefano Stabellini @ 2017-03-27 18:39 UTC (permalink / raw)
  To: Julien Grall
  Cc: Stefano Stabellini, Vijay Kilari, Andre Przywara, george.dunlap,
	jbeulich, andrew.cooper3, xen-devel, Shanker Donthineni

CC'ing Andrew, Jan and George to get more feedback on the security
impact of this patch.

I'll make a quick summary for you: we need to allocate a 56 bytes struct
(called pending_irq) for each potential interrupt injected to guests
(dom0 and domUs). With the new ARM interrupt controller there could be
thousands.

We could do the allocation upfront, which requires more memory, or we
could do the allocation dynamically at run time when each interrupt is
enabled, and deallocate the struct when an interrupt is disabled.

However, there is a concern that doing xmalloc/xfree in response to an
unprivileged DomU request could end up becoming a potential vector of
denial of service attacks. The guest could enable a thousand interrupts,
then disable a thousand interrupts and so on, monopolizing the usage of
one physical cpu. It only takes the write of 1 bit in memory for a guest
to enable/disable an interrupt.

See below.


On Mon, 27 Mar 2017, Julien Grall wrote:
> Hi Stefano,
> 
> On 27/03/17 18:44, Stefano Stabellini wrote:
> > On Mon, 27 Mar 2017, Julien Grall wrote:
> > > Hi,
> > > 
> > > On 27/03/17 10:02, Andre Przywara wrote:
> > > > On 24/03/17 17:26, Stefano Stabellini wrote:
> > > > > On Fri, 24 Mar 2017, Andre Przywara wrote:
> > > > I am afraid that this would lead to situations where we needlessly
> > > > allocate and deallocate pending_irqs. Under normal load I'd expect to
> > > > have something like zero to three LPIs pending at any given point in
> > > > time (mostly zero, to be honest).
> > > > So this will lead to a situation where *every* LPI that becomes pending
> > > > triggers a memory allocation - in the hot path. That's why the pool
> > > > idea. So if we are going to shrink the pool, I'd stop at something like
> > > > five entries, to not penalize the common case.
> > > > Does that sound useful?
> > > 
> > > Not answering directly to the question here. I will summarize the face to
> > > face
> > > discussion I had with Andre this morning.
> > > 
> > > So allocating the pending_irq in the IRQ path is not a solution because
> > > memory
> > > allocation should not happen in IRQ context, see ASSERT(!in_irq()) in
> > > _xmalloc.
> > > 
> > > Regardless the ASSERT, it will also increase the time to handle and
> > > forward an
> > > interrupt when there are no pending_irq free because it is necessary to
> > > allocate a new one. Lastly, we have no way to tell the guest: "Try again"
> > > if
> > > it Xen is running out of memory.
> > > 
> > > The outcome of the discussion is to pre-allocate the pending_irq when a
> > > device
> > > is assigned to a domain. We know the maximum number of event supported by
> > > a
> > > device and that 1 event = 1 LPI.
> > > 
> > > This may allocate more memory (a pending_irq is 56 bytes), but at least we
> > > don't need allocation on the fly and can report error.
> > > 
> > > One could argue that we could allocate on MAPTI to limit the allocation.
> > > However, as we are not able to rate-limit/defer the execution of the
> > > command
> > > queue so far, a guest could potentially flood with MAPTI and monopolize
> > > the
> > > pCPU for a long time.
> > 
> > It makes a lot of sense to keep the allocation out of the irq path.
> > However, I am wondering if the allocation/deallocation of pending_irq
> > structs could be done at the point the vLPIs are enabled/disabled,
> > instead of device assignment time.
> 
> I am not sure what you mean by enabling/disabling vLPIS. Do you mean when the
> guest is enabling/disabling them or the guest will assign a vLPI to a
> (deviceID, event) via MAPTI.

I was thinking enable/disable on the LPI Configuration table. However,
it would have the same issues you wrote for MAPTI.


> For both guest could potentially flood us. It would take us a lot of time to
> allocate/free memory for each vLPIs modified. Hence, why I didn't suggest it
> and said: "One could argue that we could allocate on MAPTI to limit the
> allocation...".

Given that Xen wouldn't allocate the same pending_irq twice, at most Xen
would allocate the same amount of memory and the same number of
pending_irq that it would otherwise allocate if we did it at assignment
time, right?

Therefore, we are not concerned about memory utilization. We are
concerned about the CPU time to do the allocation itself, right?

However, the CPU time to run xmalloc/xfree should be small and in any
case the scheduler has always the chance to deschedule the vcpu if it
wants to. This is no different than issuing any of the hypercalls that
require some work on the hypervisor side.

I don't think we have reasons to be concerned. Any opinions?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-27 18:39                 ` Stefano Stabellini
@ 2017-03-27 21:24                   ` Julien Grall
  2017-03-28  7:58                   ` Jan Beulich
  1 sibling, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-03-27 21:24 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: Vijay Kilari, Andre Przywara, george.dunlap, jbeulich,
	andrew.cooper3, xen-devel, nd, Shanker Donthineni

Hi Stefano,

On 27/03/2017 19:39, Stefano Stabellini wrote:
> On Mon, 27 Mar 2017, Julien Grall wrote:
>> For both guest could potentially flood us. It would take us a lot of time to
>> allocate/free memory for each vLPIs modified. Hence, why I didn't suggest it
>> and said: "One could argue that we could allocate on MAPTI to limit the
>> allocation...".
>
> Given that Xen wouldn't allocate the same pending_irq twice, at most Xen
> would allocate the same amount of memory and the same number of
> pending_irq that it would otherwise allocate if we did it at assignment
> time, right?
>
> Therefore, we are not concerned about memory utilization. We are
> concerned about the CPU time to do the allocation itself, right?
>
> However, the CPU time to run xmalloc/xfree should be small and in any
> case the scheduler has always the chance to deschedule the vcpu if it
> wants to. This is no different than issuing any of the hypercalls that
> require some work on the hypervisor side.
>
> I don't think we have reasons to be concerned. Any opinions?

Well, I have already said on the IRQ version but forgot to mention here. 
How do you report error if xmalloc has failed? This could happen if 
memory resource has been exhausted even for a well-behaved guest.

If you do the allocation on enable, this means on a memory trap, you 
would have to inject a data abort to the guest. If you do on the MAPTI 
command, it would be a SError with a IMPLEMENTATION DEFINED error code. 
In both case the guest will likely not able interpret it and crash. This 
could be a vector attack by exhausting the resource.

Furthermore, in the case of enable/disable vLPI solution, you can 
disable an LPI that are still inflight. So you would not be able to free 
here. Furthermore enable/disable can happen often if you want to mask 
the interrupt temporarily (all MSI are edge-interrupt).

Lastly, in the case of MAPTI it is not possible to preempt the command 
queue that can contain ~32K of commands. So we may have to do 32K of 
xmalloc/xfree. Even one MAPTI is fast, likely 32K will be slow and 
monopolize a pCPU for more than expected.

But can we focus on getting a sensible design which can be easily 
extended and without any flaw? This already is quite a difficult tasks 
with the ITS. Asking also for performance from the first draft is making 
much worse.

If you recall, a lot of ARM subsystems (P2M, vGIC...) has been built in 
multiple stage. Each stage improved the performance. It sounds quite 
unfair to me to require the same level of performance as the current 
vGIC just because the code was merged later one.

This is raising the bar to contribute on Xen on ARM very high and may 
discourage someone to push something upstream.

To be clear, I do care about performance. But I think this has to come 
in a second step when it is too complicate to address as a first step. 
The first step is to be able to use Xen on the hardware without any flaw 
and allowing us to extend the code easily.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-27 18:39                 ` Stefano Stabellini
  2017-03-27 21:24                   ` Julien Grall
@ 2017-03-28  7:58                   ` Jan Beulich
  2017-03-28 13:12                     ` Julien Grall
  1 sibling, 1 reply; 119+ messages in thread
From: Jan Beulich @ 2017-03-28  7:58 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: Vijay Kilari, Andre Przywara, george.dunlap, andrew.cooper3,
	xen-devel, Shanker Donthineni

>>> On 27.03.17 at 20:39, <sstabellini@kernel.org> wrote:
> CC'ing Andrew, Jan and George to get more feedback on the security
> impact of this patch.
> 
> I'll make a quick summary for you: we need to allocate a 56 bytes struct
> (called pending_irq) for each potential interrupt injected to guests
> (dom0 and domUs). With the new ARM interrupt controller there could be
> thousands.
> 
> We could do the allocation upfront, which requires more memory, or we
> could do the allocation dynamically at run time when each interrupt is
> enabled, and deallocate the struct when an interrupt is disabled.
> 
> However, there is a concern that doing xmalloc/xfree in response to an
> unprivileged DomU request could end up becoming a potential vector of
> denial of service attacks. The guest could enable a thousand interrupts,
> then disable a thousand interrupts and so on, monopolizing the usage of
> one physical cpu. It only takes the write of 1 bit in memory for a guest
> to enable/disable an interrupt.

Well, I think doing the allocations at device assign time would be
the least problematic approach: The tool stack could account for
the needed memory (ballooning Dom0 if necessary). (We still
have the open work item of introducing accounting of guest-
associated, but guest-inaccessible memory in the hypervisor.)

What I don't really understand the background of is the pCPU
monopolization concern. Is there anything here that's long-running
inside the hypervisor? Otherwise, the scheduler should be taking
care of everything.

Jan


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-28  7:58                   ` Jan Beulich
@ 2017-03-28 13:12                     ` Julien Grall
  2017-03-28 13:34                       ` Jan Beulich
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-03-28 13:12 UTC (permalink / raw)
  To: Jan Beulich, Stefano Stabellini
  Cc: Vijay Kilari, Andre Przywara, george.dunlap, andrew.cooper3,
	xen-devel, Shanker Donthineni

Hi Jan,

On 28/03/17 08:58, Jan Beulich wrote:
>>>> On 27.03.17 at 20:39, <sstabellini@kernel.org> wrote:
>> CC'ing Andrew, Jan and George to get more feedback on the security
>> impact of this patch.
>>
>> I'll make a quick summary for you: we need to allocate a 56 bytes struct
>> (called pending_irq) for each potential interrupt injected to guests
>> (dom0 and domUs). With the new ARM interrupt controller there could be
>> thousands.
>>
>> We could do the allocation upfront, which requires more memory, or we
>> could do the allocation dynamically at run time when each interrupt is
>> enabled, and deallocate the struct when an interrupt is disabled.
>>
>> However, there is a concern that doing xmalloc/xfree in response to an
>> unprivileged DomU request could end up becoming a potential vector of
>> denial of service attacks. The guest could enable a thousand interrupts,
>> then disable a thousand interrupts and so on, monopolizing the usage of
>> one physical cpu. It only takes the write of 1 bit in memory for a guest
>> to enable/disable an interrupt.
>
> Well, I think doing the allocations at device assign time would be
> the least problematic approach: The tool stack could account for
> the needed memory (ballooning Dom0 if necessary). (We still
> have the open work item of introducing accounting of guest-
> associated, but guest-inaccessible memory in the hypervisor.)
>
> What I don't really understand the background of is the pCPU
> monopolization concern. Is there anything here that's long-running
> inside the hypervisor? Otherwise, the scheduler should be taking
> care of everything.

Let me give you some background before answering to the question. The 
ITS is an interrupt controller widget which provides a sophisticated way
of dealing with MSIs in a scalable manner. A command queue is used to 
manage the ITS. It is a memory region provided by the guest can be up to 
1MB. Each command is composed of 4 double-word (e.g 32 bytes) which make 
the possibly for a guest to queue up 32768 commands.

In order to control the command queue there are 2 registers:
	- GITS_CWRITER that points to the end of the queue and updated by the 
software when a new command is written
	- GITS_CREADR that points to the beginning of the queue and updated by 
the forware when a new command has been executed.

The ITS will process command until the queue is empty (u.e GITS_CWRITER 
== GITS_CREADR). A software has 2 solutions to wait the completion of 
the command:
	- Polling on GITS_CREADR
	- Adding a command INT which will send a interrupt when executed

Usually the polling mode also includes a timeout in the code.

A guest could potentially queue up to 32768 commands before updating 
GITS_CWRITER. In the current approach, the command queue will be handled 
synchronously when the software wrote into GITS_CWRITER and trapped by 
the hypervisor. This means that we will not return from the hypervisor 
until the ITS executed the command between GITS_CREADER and GITS_CWRITER.

All the command have to be executed quickly to avoid the guest 
monopolizing the pCPU by flooding the command queue.

We thought using different alternative such as tasklet or breaking down 
the command queue in batch. But none of them fit well.

If you have a better idea, I am happy to consider it.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs
  2017-03-28 13:12                     ` Julien Grall
@ 2017-03-28 13:34                       ` Jan Beulich
  0 siblings, 0 replies; 119+ messages in thread
From: Jan Beulich @ 2017-03-28 13:34 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: Vijay Kilari, Andre Przywara, george.dunlap, andrew.cooper3,
	xen-devel, Shanker Donthineni

>>> On 28.03.17 at 15:12, <julien.grall@arm.com> wrote:
> Hi Jan,
> 
> On 28/03/17 08:58, Jan Beulich wrote:
>>>>> On 27.03.17 at 20:39, <sstabellini@kernel.org> wrote:
>>> CC'ing Andrew, Jan and George to get more feedback on the security
>>> impact of this patch.
>>>
>>> I'll make a quick summary for you: we need to allocate a 56 bytes struct
>>> (called pending_irq) for each potential interrupt injected to guests
>>> (dom0 and domUs). With the new ARM interrupt controller there could be
>>> thousands.
>>>
>>> We could do the allocation upfront, which requires more memory, or we
>>> could do the allocation dynamically at run time when each interrupt is
>>> enabled, and deallocate the struct when an interrupt is disabled.
>>>
>>> However, there is a concern that doing xmalloc/xfree in response to an
>>> unprivileged DomU request could end up becoming a potential vector of
>>> denial of service attacks. The guest could enable a thousand interrupts,
>>> then disable a thousand interrupts and so on, monopolizing the usage of
>>> one physical cpu. It only takes the write of 1 bit in memory for a guest
>>> to enable/disable an interrupt.
>>
>> Well, I think doing the allocations at device assign time would be
>> the least problematic approach: The tool stack could account for
>> the needed memory (ballooning Dom0 if necessary). (We still
>> have the open work item of introducing accounting of guest-
>> associated, but guest-inaccessible memory in the hypervisor.)
>>
>> What I don't really understand the background of is the pCPU
>> monopolization concern. Is there anything here that's long-running
>> inside the hypervisor? Otherwise, the scheduler should be taking
>> care of everything.
> 
> Let me give you some background before answering to the question. The 
> ITS is an interrupt controller widget which provides a sophisticated way
> of dealing with MSIs in a scalable manner. A command queue is used to 
> manage the ITS. It is a memory region provided by the guest can be up to 
> 1MB. Each command is composed of 4 double-word (e.g 32 bytes) which make 
> the possibly for a guest to queue up 32768 commands.
> 
> In order to control the command queue there are 2 registers:
> 	- GITS_CWRITER that points to the end of the queue and updated by the 
> software when a new command is written
> 	- GITS_CREADR that points to the beginning of the queue and updated by 
> the forware when a new command has been executed.
> 
> The ITS will process command until the queue is empty (u.e GITS_CWRITER 
> == GITS_CREADR). A software has 2 solutions to wait the completion of 
> the command:
> 	- Polling on GITS_CREADR
> 	- Adding a command INT which will send a interrupt when executed
> 
> Usually the polling mode also includes a timeout in the code.
> 
> A guest could potentially queue up to 32768 commands before updating 
> GITS_CWRITER. In the current approach, the command queue will be handled 
> synchronously when the software wrote into GITS_CWRITER and trapped by 
> the hypervisor. This means that we will not return from the hypervisor 
> until the ITS executed the command between GITS_CREADER and GITS_CWRITER.
> 
> All the command have to be executed quickly to avoid the guest 
> monopolizing the pCPU by flooding the command queue.
> 
> We thought using different alternative such as tasklet or breaking down 
> the command queue in batch. But none of them fit well.

I guess you want something continuation-like then? Does the
instruction doing the GITS_CWRITER write have any other side
effects? If not, wouldn't the approach used on x86 for forwarding
requests to qemu work here too? I.e. retry the instruction as long
as you're not ready to actually complete it, with a continuation
check put in the handling code you describe every so many
iterations. Of course there are dependencies here on the
cross-CPU visibility of the register - if all guest vCPU-s can access
it, things would require more care (as intermediate reads as well
as successive writes from multiple parties would need taking into
consideration).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table
  2017-03-22 16:33       ` Julien Grall
@ 2017-03-29 13:58         ` Andre Przywara
  0 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-03-29 13:58 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 22/03/17 16:33, Julien Grall wrote:
[ ... ]
>>>>      gicv3_dist_init();
>>>> +    res = gicv3_its_init();
>>>> +    if ( res )
>>>> +        printk(XENLOG_WARNING "GICv3: ITS: initialization failed:
>>>> %d\n", res);
>>>
>>> I would have expect a panic here because the ITS subsystem could be half
>>> initialized and it is not safe to continue.
>>
>> OK, let me check what actually happens here if there is no ITS ;-)
> 
> Technically, this message should not happen when there is no ITS because
> it is not mandatory to have one on the platform.
> 
> So this would be an coding error for me.

Having no ITS (node in the DT) would result in a empty host_its_list and
a "0" return, so doesn't raise any issues.

Technically we could cope with one or all ITSes to not initialize (by
not propagating them to any guests).
But for now I can just panic here, I guess, because it should point to
some serious issue.
If there is a use case, we can always add a more relaxed behavior later.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-03-16 11:20 ` [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping Andre Przywara
  2017-03-22 17:29   ` Julien Grall
  2017-03-22 22:45   ` Stefano Stabellini
@ 2017-03-30 11:17   ` Vijay Kilari
  2 siblings, 0 replies; 119+ messages in thread
From: Vijay Kilari @ 2017-03-30 11:17 UTC (permalink / raw)
  To: Andre Przywara
  Cc: xen-devel, Julien Grall, Stefano Stabellini, Shanker Donthineni

Hi Andre,

On Thu, Mar 16, 2017 at 4:50 PM, Andre Przywara <andre.przywara@arm.com> wrote:
> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
> those IDs, which we directly pass on to the host.
> For this we have to map each device that Dom0 may request to a host
> ITS device with the same identifier.
> Allocate the respective memory and enter each device into an rbtree to
> later be able to iterate over it or to easily teardown guests.
>
> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
> ---
>  xen/arch/arm/gic-v3-its.c        | 207 +++++++++++++++++++++++++++++++++++++++
>  xen/arch/arm/vgic-v3.c           |   3 +
>  xen/include/asm-arm/domain.h     |   3 +
>  xen/include/asm-arm/gic_v3_its.h |  18 ++++
>  4 files changed, 231 insertions(+)
>
> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
> index 5c11b0d..60b15b5 100644
> --- a/xen/arch/arm/gic-v3-its.c
> +++ b/xen/arch/arm/gic-v3-its.c
> @@ -21,6 +21,8 @@
>  #include <xen/lib.h>
>  #include <xen/delay.h>
>  #include <xen/mm.h>
> +#include <xen/rbtree.h>
> +#include <xen/sched.h>
>  #include <xen/sizes.h>
>  #include <asm/gic.h>
>  #include <asm/gic_v3_defs.h>
> @@ -32,6 +34,17 @@
>
>  LIST_HEAD(host_its_list);
>
> +struct its_devices {
> +    struct rb_node rbnode;
> +    struct host_its *hw_its;
> +    void *itt_addr;
> +    paddr_t guest_doorbell;
> +    uint32_t host_devid;
> +    uint32_t guest_devid;
> +    uint32_t eventids;
> +    uint32_t *host_lpis;
> +};
> +
>  bool gicv3_its_host_has_its(void)
>  {
>      return !list_empty(&host_its_list);
> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>      return its_send_command(its, cmd);
>  }
>
> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
> +                             uint8_t size_bits, paddr_t itt_addr, bool valid)
> +{
> +    uint64_t cmd[4];
> +
> +    if ( valid )
> +    {
> +        ASSERT(size_bits < 32);
> +        ASSERT(!(itt_addr & ~GENMASK(51, 8)));
> +    }
> +    cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
> +    cmd[1] = valid ? size_bits : 0x00;
> +    cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
> +    cmd[3] = 0x00;
> +
> +    return its_send_command(its, cmd);
> +}
> +
>  /* Set up the (1:1) collection mapping for the given host CPU. */
>  int gicv3_its_setup_collection(unsigned int cpu)
>  {
> @@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
>      devid_bits = min(devid_bits, max_its_device_bits);
>      if ( reg & GITS_TYPER_PTA )
>          hw_its->flags |= HOST_ITS_USES_PTA;
> +    hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>
>      for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>      {
> @@ -428,6 +460,180 @@ int gicv3_its_init(void)
>      return 0;
>  }
>
> +static int remove_mapped_guest_device(struct its_devices *dev)
> +{
> +    int ret;
> +
> +    if ( dev->hw_its )
> +    {
> +        int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);
> +        if ( ret )
> +            return ret;
> +    }
> +
> +    ret = gicv3_its_wait_commands(dev->hw_its);
> +    if ( ret )
> +        return ret;
> +
> +    xfree(dev->itt_addr);
> +    xfree(dev);
> +
> +    return 0;
> +}
> +
> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
> +{
> +    struct host_its *hw_its;
> +
> +    list_for_each_entry(hw_its, &host_its_list, entry)
> +    {
> +        if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> +            return hw_its;
> +    }
> +
> +    return NULL;
> +}
> +
> +static int compare_its_guest_devices(struct its_devices *dev,
> +                                     paddr_t doorbell, uint32_t devid)
> +{
> +    if ( dev->guest_doorbell < doorbell )
> +        return -1;
> +
> +    if ( dev->guest_doorbell > doorbell )
> +        return 1;
> +
> +    if ( dev->guest_devid < devid )
> +        return -1;
> +
> +    if ( dev->guest_devid > devid )
> +        return 1;
> +
> +    return 0;
> +}
> +
> +/*
> + * Map a hardware device, identified by a certain host ITS and its device ID
> + * to domain d, a guest ITS (identified by its doorbell address) and device ID.
> + * Also provide the number of events (MSIs) needed for that device.
> + * This does not check if this particular hardware device is already mapped
> + * at another domain, it is expected that this would be done by the caller.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +                               paddr_t host_doorbell, uint32_t host_devid,
> +                               paddr_t guest_doorbell, uint32_t guest_devid,
> +                               uint32_t nr_events, bool valid)
> +{
> +    void *itt_addr = NULL;
> +    struct host_its *hw_its;
> +    struct its_devices *dev = NULL, *temp;
> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
> +    int ret = -ENOENT;
> +
> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
> +    if ( !hw_its )
> +        return ret;
> +
> +    /* check for already existing mappings */
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    while ( *new )
> +    {
> +        temp = rb_entry(*new, struct its_devices, rbnode);
> +
> +        parent = *new;
> +        if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
> +        {
> +            if ( !valid )
> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
> +
> +            spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +            if ( valid )
> +                return -EBUSY;
> +
> +            return remove_mapped_guest_device(temp);
> +        }
> +
> +        if ( compare_its_guest_devices(temp, guest_doorbell, guest_devid) > 0 )
> +            new = &((*new)->rb_left);
> +        else
> +            new = &((*new)->rb_right);
> +    }
> +
> +    if ( !valid )
> +        goto out_unlock;
> +
> +    ret = -ENOMEM;
> +
> +    /* An Interrupt Translation Table needs to be 256-byte aligned. */
> +    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);

Here itt_addr should be calculated based number of IDs mapped with MAPTI.
Should be LPI_BLOCK size. Ex: If guest sends MAPD with nr_events 2 and
later MAPTI
is sent for Xen for mapping 32 events, causes memory corruption.

> +    if ( !itt_addr )
> +        goto out_unlock;
> +
> +    dev = xzalloc(struct its_devices);
> +    if ( !dev )
> +        goto out_unlock;
> +
> +    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
> +                            virt_to_maddr(itt_addr), true);

     Here MAPD is sent with Size (nr_events) set by domain. However
MAPTI is sent for LPI block (32). ITS HW will throw 0xa05 (MAPVI_ID_OOR)
error indicating ID out of range.

> +    if ( ret )
> +        goto out_unlock;
> +
> +    dev->itt_addr = itt_addr;
> +    dev->hw_its = hw_its;
> +    dev->guest_doorbell = guest_doorbell;
> +    dev->guest_devid = guest_devid;
> +    dev->host_devid = host_devid;
> +    dev->eventids = nr_events;
> +
> +    rb_link_node(&dev->rbnode, parent, new);
> +    rb_insert_color(&dev->rbnode, &d->arch.vgic.its_devices);
> +
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +    return 0;
> +
> +out_unlock:
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +    if ( dev )
> +        xfree(dev->host_lpis);
> +    xfree(itt_addr);
> +    xfree(dev);
> +    return ret;
> +}
> +
> +/* Removing any connections a domain had to any ITS in the system. */
> +void gicv3_its_unmap_all_devices(struct domain *d)
> +{
> +    struct rb_node *victim;
> +    struct its_devices *dev;
> +
> +    /*
> +     * This is an easily readable, yet inefficient implementation.
> +     * It uses the provided iteration wrapper and erases each node, which
> +     * possibly triggers rebalancing.
> +     * This seems overkill since we are going to abolish the whole tree, but
> +     * avoids an open-coded re-implementation of the traversal functions with
> +     * some recursive function calls.
> +     * Performance does not matter here, since we are destroying a domain.
> +     */
> +restart:
> +    spin_lock(&d->arch.vgic.its_devices_lock);
> +    if ( (victim = rb_first(&d->arch.vgic.its_devices)) )
> +    {
> +        dev = rb_entry(victim, struct its_devices, rbnode);
> +        rb_erase(victim, &d->arch.vgic.its_devices);
> +
> +        spin_unlock(&d->arch.vgic.its_devices_lock);
> +
> +        remove_mapped_guest_device(dev);
> +
> +        goto restart;
> +    }
> +
> +    spin_unlock(&d->arch.vgic.its_devices_lock);
> +}
> +
>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>  void gicv3_its_dt_init(const struct dt_device_node *node)
>  {
> @@ -455,6 +661,7 @@ void gicv3_its_dt_init(const struct dt_device_node *node)
>          its_data->addr = addr;
>          its_data->size = size;
>          its_data->dt_node = its;
> +        spin_lock_init(&its_data->cmd_lock);
>
>          printk("GICv3: Found ITS @0x%lx\n", addr);
>
> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
> index d61479d..1fadb00 100644
> --- a/xen/arch/arm/vgic-v3.c
> +++ b/xen/arch/arm/vgic-v3.c
> @@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
>      d->arch.vgic.nr_regions = rdist_count;
>      d->arch.vgic.rdist_regions = rdist_regions;
>
> +    spin_lock_init(&d->arch.vgic.its_devices_lock);
> +    d->arch.vgic.its_devices = RB_ROOT;
> +
>      /*
>       * Domain 0 gets the hardware address.
>       * Guests get the virtual platform layout.
> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
> index 2d6fbb1..00b9c1a 100644
> --- a/xen/include/asm-arm/domain.h
> +++ b/xen/include/asm-arm/domain.h
> @@ -11,6 +11,7 @@
>  #include <asm/gic.h>
>  #include <public/hvm/params.h>
>  #include <xen/serial.h>
> +#include <xen/rbtree.h>
>
>  struct hvm_domain
>  {
> @@ -109,6 +110,8 @@ struct arch_domain
>          } *rdist_regions;
>          int nr_regions;                     /* Number of rdist regions */
>          uint32_t rdist_stride;              /* Re-Distributor stride */
> +        struct rb_root its_devices;         /* devices mapped to an ITS */
> +        spinlock_t its_devices_lock;        /* protects the its_devices tree */
>  #endif
>      } vgic;
>
> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
> index 8b493fb..3421ea0 100644
> --- a/xen/include/asm-arm/gic_v3_its.h
> +++ b/xen/include/asm-arm/gic_v3_its.h
> @@ -48,6 +48,10 @@
>  #define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
>                                                 GITS_TYPER_DEVIDS_SHIFT) + 1)
>  #define GITS_TYPER_IDBITS_SHIFT         8
> +#define GITS_TYPER_ITT_SIZE_SHIFT       4
> +#define GITS_TYPER_ITT_SIZE_MASK        (0xfUL << GITS_TYPER_ITT_SIZE_SHIFT)
> +#define GITS_TYPER_ITT_SIZE(r)          ((((r) & GITS_TYPER_ITT_SIZE_MASK) >> \
> +                                                GITS_TYPER_ITT_SIZE_SHIFT) + 1)
>
>  #define GITS_IIDR_VALUE                 0x34c
>
> @@ -94,7 +98,10 @@
>  #define GITS_CMD_MOVALL                 0x0e
>  #define GITS_CMD_DISCARD                0x0f
>
> +#define ITS_DOORBELL_OFFSET             0x10040
> +
>  #include <xen/device_tree.h>
> +#include <xen/rbtree.h>
>
>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>  #define HOST_ITS_USES_PTA               (1U << 1)
> @@ -108,6 +115,7 @@ struct host_its {
>      void __iomem *its_base;
>      spinlock_t cmd_lock;
>      void *cmd_buf;
> +    unsigned int itte_size;
>      unsigned int flags;
>  };
>
> @@ -134,6 +142,16 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
>  /* Map a collection for this host CPU to each host ITS. */
>  int gicv3_its_setup_collection(unsigned int cpu);
>
> +/*
> + * Map a device on the host by allocating an ITT on the host (ITS).
> + * "nr_event" specifies how many events (interrupts) this device will need.
> + * Setting "valid" to false deallocates the device.
> + */
> +int gicv3_its_map_guest_device(struct domain *d,
> +                               paddr_t host_doorbell, uint32_t host_devid,
> +                               paddr_t guest_doorbell, uint32_t guest_devid,
> +                               uint32_t nr_events, bool valid);
> +
>  #else
>
>  static LIST_HEAD(host_its_list);
> --
> 2.9.0
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-03-22 15:59   ` Julien Grall
@ 2017-04-03 10:58     ` Andre Przywara
  2017-04-03 11:23       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 10:58 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 22/03/17 15:59, Julien Grall wrote:
> Hi Andre,
> 
> On 16/03/17 11:20, Andre Przywara wrote:
>> To be able to easily send commands to the ITS, create the respective
>> wrapper functions, which take care of the ring buffer.
>> The first two commands we implement provide methods to map a collection
>> to a redistributor (aka host core) and to flush the command queue (SYNC).
>> Start using these commands for mapping one collection to each host CPU.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c         | 181
>> +++++++++++++++++++++++++++++++++++++-
>>  xen/arch/arm/gic-v3-lpi.c         |  22 +++++
>>  xen/arch/arm/gic-v3.c             |  19 +++-
>>  xen/include/asm-arm/gic_v3_defs.h |   2 +
>>  xen/include/asm-arm/gic_v3_its.h  |  38 ++++++++
>>  5 files changed, 260 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index e5601ed..5c11b0d 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -19,11 +19,14 @@
>>   */
>>
>>  #include <xen/lib.h>
>> +#include <xen/delay.h>
>>  #include <xen/mm.h>
>>  #include <xen/sizes.h>
>> +#include <asm/gic.h>
>>  #include <asm/gic_v3_defs.h>
>>  #include <asm/gic_v3_its.h>
>>  #include <asm/io.h>
>> +#include <asm/page.h>
>>
>>  #define ITS_CMD_QUEUE_SZ                SZ_64K
>>
>> @@ -34,6 +37,145 @@ bool gicv3_its_host_has_its(void)
>>      return !list_empty(&host_its_list);
>>  }
>>
>> +#define BUFPTR_MASK                     GENMASK(19, 5)
>> +static int its_send_command(struct host_its *hw_its, const void
>> *its_cmd)
>> +{
>> +    s_time_t deadline = NOW() + MILLISECS(1);
> 
> It would be nice to explain where does this value comes from.

In lack of any real data on this, one millisecond is just a guess.
In v3 I added some comment stating that this is a "short" period to wait
for. We don't want to stall for a long time for a single command, but
also don't want to give up too easily. The command queue is handled by
the ITS, which is hardware, so the only thing it does is handling
commands. If we are unlucky enough to hit a full command queue due to a
sudden spike in host commands, chances are that we get a free slot very
soon afterwards. If not, we are screwed anyway.

>> +    uint64_t readp, writep;
>> +    int ret = -EBUSY;
>> +
>> +    /* No ITS commands from an interrupt handler (at the moment). */
>> +    ASSERT(!in_irq());
>> +
>> +    spin_lock(&hw_its->cmd_lock);
>> +
>> +    do {
>> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) &
>> BUFPTR_MASK;
>> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) &
>> BUFPTR_MASK;
>> +
>> +        if ( ((writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ) != readp )
>> +        {
>> +            ret = 0;
>> +            break;
>> +        }
>> +
>> +        /*
>> +         * If the command queue is full, wait for a bit in the hope
>> it drains
>> +         * before giving up.
>> +         */
>> +        spin_unlock(&hw_its->cmd_lock);
>> +        cpu_relax();
>> +        udelay(1);
>> +        spin_lock(&hw_its->cmd_lock);
>> +    } while ( NOW() <= deadline );
>> +
>> +    if ( ret )
>> +    {
>> +        spin_unlock(&hw_its->cmd_lock);
>> +        printk(XENLOG_WARNING "ITS: command queue full.\n");
> 
> This function could be called from a domain. So please ratelimit the
> message (see printk_ratelimit).

Fixed in v4.

> 
>> +        return ret;
>> +    }
>> +
>> +    memcpy(hw_its->cmd_buf + writep, its_cmd, ITS_CMD_SIZE);
>> +    if ( hw_its->flags & HOST_ITS_FLUSH_CMD_QUEUE )
>> +        clean_and_invalidate_dcache_va_range(hw_its->cmd_buf + writep,
>> +                                             ITS_CMD_SIZE);
>> +    else
>> +        dsb(ishst);
>> +
>> +    writep = (writep + ITS_CMD_SIZE) % ITS_CMD_QUEUE_SZ;
>> +    writeq_relaxed(writep & BUFPTR_MASK, hw_its->its_base +
>> GITS_CWRITER);
>> +
>> +    spin_unlock(&hw_its->cmd_lock);
>> +
>> +    return 0;
>> +}
>> +
>> +/* Wait for an ITS to finish processing all commands. */
>> +static int gicv3_its_wait_commands(struct host_its *hw_its)
>> +{
>> +    s_time_t deadline = NOW() + MILLISECS(100);
> 
> Same remark as above. Why 1ms and 100ms here?

I made this period longer, because it covers multiple commands and we
are somewhat expected to wait here. Still it should be shorter than the
Linux timeout, which is one second.

>> +    uint64_t readp, writep;
>> +
>> +    do {
>> +        spin_lock(&hw_its->cmd_lock);
>> +        readp = readq_relaxed(hw_its->its_base + GITS_CREADR) &
>> BUFPTR_MASK;
>> +        writep = readq_relaxed(hw_its->its_base + GITS_CWRITER) &
>> BUFPTR_MASK;
>> +        spin_unlock(&hw_its->cmd_lock);
>> +
>> +        if ( readp == writep )
>> +            return 0;
>> +
>> +        cpu_relax();
>> +        udelay(1);
>> +    } while ( NOW() <= deadline );
>> +
>> +    return -ETIMEDOUT;
>> +}
> 
> [...]
> 
>> +static int its_send_cmd_mapc(struct host_its *its, uint32_t
>> collection_id,
>> +                             unsigned int cpu)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    cmd[0] = GITS_CMD_MAPC;
>> +    cmd[1] = 0x00;
>> +    cmd[2] = encode_rdbase(its, cpu, (collection_id & GENMASK(15, 0)));
> 
> I requested to drop the mask here and I didn't see any answer explaining
> why you would not do it. So this should be dropped unless you give me a
> reason to keep it.

Fixed in v3.

> 
>> +    cmd[2] |= GITS_VALID_BIT;
>> +    cmd[3] = 0x00;
>> +
>> +    return its_send_command(its, cmd);
>> +}
>> +
>> +/* Set up the (1:1) collection mapping for the given host CPU. */
>> +int gicv3_its_setup_collection(unsigned int cpu)
>> +{
>> +    struct host_its *its;
>> +    int ret;
>> +
>> +    list_for_each_entry(its, &host_its_list, entry)
>> +    {
>> +        if ( !its->cmd_buf )
>> +            continue;
> 
> Why this check?

Indeed a leftover from the older code. Given the current code flow, we
shouldn't get here without the cmd_buf being initialized. Still I liked
the idea of having some check here in case the code flow changes.
Will drop this is v4.

> [...]
> 
>> +/*
>> + * Before an ITS gets initialized, it should be in a quiescent state,
>> where
>> + * all outstanding commands and transactions have finished.
>> + * So if the ITS is already enabled, turn it off and wait for all
>> outstanding
>> + * operations to get processed by polling the QUIESCENT bit.
>> + */
>> +static int gicv3_disable_its(struct host_its *hw_its)
>> +{
>> +    uint32_t reg;
>> +    s_time_t deadline = NOW() + MILLISECS(100);
> 
> Why 100ms?

Same rationale as above: We are dealing with multiple commands here,
also are expected to take a longer time.

>>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>>  {
>>      uint64_t val;
>> diff --git a/xen/arch/arm/gic-v3.c b/xen/arch/arm/gic-v3.c
>> index cc1e219..38dafe7 100644
>> --- a/xen/arch/arm/gic-v3.c
>> +++ b/xen/arch/arm/gic-v3.c
>> @@ -666,7 +666,21 @@ static int __init gicv3_populate_rdist(void)
>>
>>                  if ( typer & GICR_TYPER_PLPIS )
>>                  {
>> -                    int ret;
>> +                    paddr_t rdist_addr;
>> +                    int procnum, ret;
> 
> As mentioned in v1, procnum should be unsigned. If you disagree, please
> explain.

Fixed in v4.

>> +
>> +                    /*
>> +                     * The ITS refers to redistributors either by
>> their physical
>> +                     * address or by their ID. Determine those two
>> values and
>> +                     * let the ITS code store them in per host CPU
>> variables to
>> +                     * later be able to address those redistributors.
>> +                     */
> 
> Again, this comment does not look useful and is misleading as the code
> to get/set the redistributor information is living in gic-v3-lpi.c and
> not gic-v3-its.c.

But we need to prepare those values here (and only here) and it does
make a lot of sense to explain this, since the ability to deal with both
ways of representing a redistributor is not obvious.
And this is one of the connections between the ITS and the (GICv3)
redistributors, where the line between them is not easy to draw.
So in v4 I now reworded the comment to say that we hand this over to the
ITS (instead of spoiling ITS implementation details).
Hope you are OK with this.

> 
> [...]
> 
>> diff --git a/xen/include/asm-arm/gic_v3_its.h
>> b/xen/include/asm-arm/gic_v3_its.h
>> index d5facf0..8b493fb 100644
>> --- a/xen/include/asm-arm/gic_v3_its.h
>> +++ b/xen/include/asm-arm/gic_v3_its.h
> 
> [...]
> 
>>  #include <xen/device_tree.h>
>>
>>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>> +#define HOST_ITS_USES_PTA               (1U << 1)
>>
>>  /* data structure for each hardware ITS */
>>  struct host_its {
>> @@ -87,6 +106,7 @@ struct host_its {
>>      paddr_t addr;
>>      paddr_t size;
>>      void __iomem *its_base;
>> +    spinlock_t cmd_lock;
> 
> Again, initialization, clean-up of a field should be done in the same
> that added the field. Otherwise, this is a call to miss a bit of the
> code and makes more difficult for the reviewer.
> 
> So please initialize cmd_lock in this patch and patch #6.

Fixed in v4.

> 
>>      void *cmd_buf;
>>      unsigned int flags;
>>  };
>> @@ -107,6 +127,13 @@ int gicv3_lpi_init_rdist(void __iomem * rdist_base);
>>  int gicv3_lpi_init_host_lpis(unsigned int nr_lpis);
>>  int gicv3_its_init(void);
>>
>> +/* Store the physical address and ID for each redistributor as read
>> from DT. */
>> +void gicv3_set_redist_address(paddr_t address, unsigned int redist_id);
>> +uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
>> +
>> +/* Map a collection for this host CPU to each host ITS. */
>> +int gicv3_its_setup_collection(unsigned int cpu);
>> +
>>  #else
>>
>>  static LIST_HEAD(host_its_list);
>> @@ -134,6 +161,17 @@ static inline int gicv3_its_init(void)
>>  {
>>      return 0;
>>  }
>> +
>> +static inline void gicv3_set_redist_address(paddr_t address,
>> +                                            unsigned int redist_id)
>> +{
>> +}
>> +
>> +static inline int gicv3_its_setup_collection(unsigned int cpu)
>> +{
> 
> This function should never be called as it is gated by the presence of
> ITS. I would add a BUG() with a comment to ensure this is the case.

Right, fixed that in v4.

Cheers,
Andre.

> 
>> +    return 0;
>> +}
>> +
>>  #endif /* CONFIG_HAS_ITS */
>>
>>  #endif
>>
> 
> Cheers
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling
  2017-04-03 10:58     ` Andre Przywara
@ 2017-04-03 11:23       ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-04-03 11:23 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/04/17 11:58, Andre Przywara wrote:
>>> +#define BUFPTR_MASK                     GENMASK(19, 5)
>>> +static int its_send_command(struct host_its *hw_its, const void
>>> *its_cmd)
>>> +{
>>> +    s_time_t deadline = NOW() + MILLISECS(1);
>>
>> It would be nice to explain where does this value comes from.
>
> In lack of any real data on this, one millisecond is just a guess.
> In v3 I added some comment stating that this is a "short" period to wait
> for. We don't want to stall for a long time for a single command, but
> also don't want to give up too easily. The command queue is handled by
> the ITS, which is hardware, so the only thing it does is handling
> commands. If we are unlucky enough to hit a full command queue due to a
> sudden spike in host commands, chances are that we get a free slot very
> soon afterwards. If not, we are screwed anyway.

It is not the first time I am saying that. If it is a guess, then this 
should be spelled out. A reader cannot know and we would have a question 
asking why "1 ms in Xen and 1 sec in Linux"...

>>> +/* Wait for an ITS to finish processing all commands. */
>>> +static int gicv3_its_wait_commands(struct host_its *hw_its)
>>> +{
>>> +    s_time_t deadline = NOW() + MILLISECS(100);
>>
>> Same remark as above. Why 1ms and 100ms here?
>
> I made this period longer, because it covers multiple commands and we
> are somewhat expected to wait here. Still it should be shorter than the
> Linux timeout, which is one second.

See above.

[...]

>>
>>> +/*
>>> + * Before an ITS gets initialized, it should be in a quiescent state,
>>> where
>>> + * all outstanding commands and transactions have finished.
>>> + * So if the ITS is already enabled, turn it off and wait for all
>>> outstanding
>>> + * operations to get processed by polling the QUIESCENT bit.
>>> + */
>>> +static int gicv3_disable_its(struct host_its *hw_its)
>>> +{
>>> +    uint32_t reg;
>>> +    s_time_t deadline = NOW() + MILLISECS(100);
>>
>> Why 100ms?
>
> Same rationale as above: We are dealing with multiple commands here,
> also are expected to take a longer time.

See above.

[...]

>>> +
>>> +                    /*
>>> +                     * The ITS refers to redistributors either by
>>> their physical
>>> +                     * address or by their ID. Determine those two
>>> values and
>>> +                     * let the ITS code store them in per host CPU
>>> variables to
>>> +                     * later be able to address those redistributors.
>>> +                     */
>>
>> Again, this comment does not look useful and is misleading as the code
>> to get/set the redistributor information is living in gic-v3-lpi.c and
>> not gic-v3-its.c.
>
> But we need to prepare those values here (and only here) and it does
> make a lot of sense to explain this, since the ability to deal with both
> ways of representing a redistributor is not obvious.
> And this is one of the connections between the ITS and the (GICv3)
> redistributors, where the line between them is not easy to draw.
> So in v4 I now reworded the comment to say that we hand this over to the
> ITS (instead of spoiling ITS implementation details).
> Hope you are OK with this.

Well, it makes sense to initialize the redistributor address in here and 
not in another place.

This kind of comments is more useful on the place where you initialize 
the ITS to explain why it is done there and not before.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests
  2017-03-24 12:03   ` Julien Grall
@ 2017-04-03 14:18     ` Andre Przywara
  2017-04-04 11:49       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 14:18 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 12:03, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> Upon receiving an LPI, we need to find the right VCPU and virtual IRQ
>> number to get this IRQ injected.
>> Iterate our two-level LPI table to find this information quickly when
>> the host takes an LPI. Call the existing injection function to let the
>> GIC emulation deal with this interrupt.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-lpi.c | 41 +++++++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/gic.c        |  6 ++++--
>>  xen/include/asm-arm/irq.h |  8 ++++++++
>>  3 files changed, 53 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>> index 59d3ba4..0579976 100644
>> --- a/xen/arch/arm/gic-v3-lpi.c
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -104,6 +104,47 @@ uint64_t gicv3_get_redist_address(unsigned int
>> cpu, bool use_pta)
>>          return per_cpu(lpi_redist, cpu).redist_id << 16;
>>  }
>>
>> +/*
>> + * Handle incoming LPIs, which are a bit special, because they are
>> potentially
>> + * numerous and also only get injected into guests. Treat them
>> specially here,
>> + * by just looking up their target vCPU and virtual LPI number and
>> hand it
>> + * over to the injection function.
>> + */
>> +void do_LPI(unsigned int lpi)
>> +{
>> +    struct domain *d;
>> +    union host_lpi *hlpip, hlpi;
>> +    struct vcpu *vcpu;
>> +
>> +    WRITE_SYSREG32(lpi, ICC_EOIR1_EL1);
>> +
>> +    hlpip = gic_get_host_lpi(lpi);
>> +    if ( !hlpip )
>> +        return;
>> +
>> +    hlpi.data = read_u64_atomic(&hlpip->data);
>> +
>> +    /* We may have mapped more host LPIs than the guest actually
>> asked for. */
> 
> Another way, is the interrupt has been received at the same time the
> guest is configuring it. What will happen if the interrupt is lost?

I don't see how this would be our problem. If the guest is still about
to configure an LPI, then any incoming one is pretty clearly a spurious
interrupt.
I take it that you mean "mapping an LPI using ITS commands" when you say
"configuring" here. If this is not what you mean, please correct me.

I'd expect a guest to first map the LPI, then enable it and send an INV
command. Doing that differently will not result in any sane behavior (as
in: a guest cannot expect an LPI to come through), so nothing we need to
worry about (from a lost IRQ perspective).

Or what do I miss here?

> 
>> +    if ( !hlpi.virt_lpi )
>> +        return;
>> +
>> +    d = get_domain_by_id(hlpi.dom_id);
> 
> It would be enough to use rcu_lock_domain_by_id here.

Done in v3.

>> +    if ( !d )
>> +        return;
>> +
>> +    if ( hlpi.vcpu_id >= d->max_vcpus )
> 
> A comment would be certainly useful here to explain why this check.

Done in v3, though personally I don't see what is unclear about an:
	if ( id >= max )
		return;

>> +    {
>> +        put_domain(d);
>> +        return;
>> +    }
>> +
>> +    vcpu = d->vcpu[hlpi.vcpu_id];
>> +
>> +    put_domain(d);
>> +
>> +    vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
>> +}
>> +
>>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>>  {
>>      uint64_t val;
>> diff --git a/xen/arch/arm/gic.c b/xen/arch/arm/gic.c
>> index bd3c032..7286e5d 100644
>> --- a/xen/arch/arm/gic.c
>> +++ b/xen/arch/arm/gic.c
>> @@ -700,8 +700,10 @@ void gic_interrupt(struct cpu_user_regs *regs,
>> int is_fiq)
>>              local_irq_enable();
>>              do_IRQ(regs, irq, is_fiq);
>>              local_irq_disable();
>> -        }
>> -        else if (unlikely(irq < 16))
>> +        } else if ( is_lpi(irq) )
>> +        {
>> +            do_LPI(irq);
> 
> I really don't want to see GICv3 specific code called in common code.
> Please introduce a specific callback in gic_hw_operations.

OK, you convinced me by persistence ;-)
I still don't fully believe it, but well ...

> 
>> +        } else if ( unlikely(irq < 16) )
> 
> Coding style:
> 
> }
> else if (...)
> {
> }
> else if (...)

Fixed in v3.

> 
>>          {
>>              do_sgi(regs, irq);
>>          }
>> diff --git a/xen/include/asm-arm/irq.h b/xen/include/asm-arm/irq.h
>> index 8f7a167..ee47de8 100644
>> --- a/xen/include/asm-arm/irq.h
>> +++ b/xen/include/asm-arm/irq.h
>> @@ -34,6 +34,14 @@ struct irq_desc *__irq_to_desc(int irq);
>>
>>  void do_IRQ(struct cpu_user_regs *regs, unsigned int irq, int is_fiq);
>>
>> +#ifdef CONFIG_HAS_ITS
>> +void do_LPI(unsigned int irq);
>> +#else
>> +static inline void do_LPI(unsigned int irq)
>> +{
>> +}
>> +#endif
>> +
> 
> This would avoid such ugly hack where do_LPI is define in gic-v3-its.c
> but declared in irq.h.

I agree that this doesn't look nice, that's why I came up with a neater
version in v3.
So doing the indirection comes at the price of additional code, which is
hard to chase (because of the function pointer) and probably more costly
(because it's harder to speculate on a CPU). But well, as you wish ...

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 15/27] ARM: vITS: introduce translation table walks
  2017-03-24 13:00   ` Julien Grall
@ 2017-04-03 18:25     ` Andre Przywara
  2017-04-04 15:59       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 18:25 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 13:00, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> The ITS stores the target (v)CPU and the (virtual) LPI number in tables.
>> Introduce functions to walk those tables and translate an device ID -
>> event ID pair into a pair of virtual LPI and vCPU.
>> Since the final interrupt translation tables can be smaller than a page,
>> we map them on demand (which is cheap on arm64). Also we take care of
>> the locking on the way, since we can't easily protect those ITTs from
>> being altered by the guest.
>>
>> To allow compiling without warnings, we declare two functions as
>> non-static for the moment.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/vgic-v3-its.c | 135
>> +++++++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 135 insertions(+)
>>
>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
>> index 5337638..267a573 100644
>> --- a/xen/arch/arm/vgic-v3-its.c
>> +++ b/xen/arch/arm/vgic-v3-its.c
>> @@ -62,6 +62,141 @@ struct vits_itte
>>      uint16_t collection;
>>  };
>>
>> +#define UNMAPPED_COLLECTION      ((uint16_t)~0)
>> +
>> +/* Must be called with the ITS lock held. */
> 
> This is a call for an ASSERT in the function.
> 
>> +static struct vcpu *get_vcpu_from_collection(struct virt_its *its,
>> int collid)
> 
> s/int/unsigned int/

Fixed in v4.

>> +{
>> +    uint16_t vcpu_id;
>> +
>> +    if ( collid >= its->max_collections )
>> +        return NULL;
>> +
>> +    vcpu_id = its->coll_table[collid];
>> +    if ( vcpu_id == UNMAPPED_COLLECTION || vcpu_id >=
>> its->d->max_vcpus )
>> +        return NULL;
>> +
>> +    return its->d->vcpu[vcpu_id];
>> +}
>> +
>> +#define DEV_TABLE_ITT_ADDR(x) ((x) & GENMASK(51, 8))
>> +#define DEV_TABLE_ITT_SIZE(x) (BIT(((x) & GENMASK(7, 0)) + 1))
>> +#define DEV_TABLE_ENTRY(addr, bits)                     \
>> +        (((addr) & GENMASK(51, 8)) | (((bits) - 1) & GENMASK(7, 0)))
> 
> The layout of dev_table[...] really needs to be explained. It took me
> quite a while to understand how it works. For instance why you skip the
> first 8 bits for the address...

Fixed in v3.

>> +
>> +static paddr_t get_itte_address(struct virt_its *its,
>> +                                uint32_t devid, uint32_t evid)
>> +{
> 
> I was expected to see the support of two-level page table for the vITS.
> Any plan for that?
> 
>> +    paddr_t addr;
>> +
>> +    if ( devid >= its->max_devices )
>> +        return ~0;
> 
> Please don't hardcode invalid address and use INVALID_PADDR.
> 
>> +
>> +    if ( evid >= DEV_TABLE_ITT_SIZE(its->dev_table[devid]) )
>> +        return ~0;
> 
> Ditto.

Fixed in v3.

>> +
>> +    addr = DEV_TABLE_ITT_ADDR(its->dev_table[devid]);
> 
> You read dev_table[...] multiple time. What prevents someone to modify
> dev_table after you did someone sanity check?
> 
> It would be safer to do a read_atomic(..) at the beginning and use a
> temporary variable.

Fixed in v4.

> 
>> +
>> +    return addr + evid * sizeof(struct vits_itte);
>> +}
>> +
>> +/*
>> + * Looks up a given deviceID/eventID pair on an ITS and returns a
>> pointer to
>> + * the corresponding ITTE. This maps the respective guest page into Xen.
>> + * Once finished with handling the ITTE, call put_devid_evid() to unmap
>> + * the page again.
>> + * Must be called with the ITS lock held.
> 
> This is a call for an ASSERT in the code.
> 
>> + */
>> +static struct vits_itte *get_devid_evid(struct virt_its *its,
>> +                                        uint32_t devid, uint32_t evid)
> 
> The naming of the function is confusing. It doesn't look up a device
> ID/event ID but an IIT. So I would rename it to find_itte.

Fixed in v3.

>> +{
>> +    paddr_t addr = get_itte_address(its, devid, evid);
>> +
>> +    if ( addr == ~0 )
> 
> 
> Please use INVALID_PADDR.

Fixed in v3.

>> +        return NULL;
>> +
>> +    return map_guest_pages(its->d, addr, 1);
>> +}
>> +
>> +/* Must be called with the ITS lock held. */
>> +static void put_devid_evid(struct virt_its *its, struct vits_itte *itte)
>> +{
>> +    unmap_guest_pages(itte, 1);
>> +}
>> +
>> +/*
>> + * Queries the collection and device tables to get the vCPU and virtual
>> + * LPI number for a given guest event. This takes care of mapping the
>> + * respective tables and validating the values, since we can't
>> efficiently
>> + * protect the ITTs with their less-than-page-size granularity.
>> + * Takes and drops the its_lock.
> 
> I am not sure to understand the usefulness of "takes and drops the
> its_lock".

It tells people that they should not hold the its_lock when calling this
function, as this might deadlock. Also it means that this function takes
care about locking itself.
Improved the wording in v4.

>> + */
>> +bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
>> +               struct vcpu **vcpu, uint32_t *vlpi)
>> +{
>> +    struct vits_itte *itte;
>> +    int collid;
>> +    uint32_t _vlpi;
>> +    struct vcpu *_vcpu;
>> +
>> +    spin_lock(&its->its_lock);
> 
> Do we expect multiple vCPU calling read_itte at the same time? This
> needs to be clarify in the comments because this current function is not
> scalable.

We need this lock here because this protects our data structures.
A guest locks accesses to the ITS command queue anyway (otherwrite
multiple VCPUs would race for CWRITER and CREADR), so I don't see a real
problem. And this lock is pre ITS, not per guest.

>> +    itte = get_devid_evid(its, devid, evid);
>> +    if ( !itte )
>> +    {
>> +        spin_unlock(&its->its_lock);
>> +        return false;
>> +    }
>> +    collid = itte->collection;
>> +    _vlpi = itte->vlpi;
>> +    put_devid_evid(its, itte);
>> +
>> +    _vcpu = get_vcpu_from_collection(its, collid);
>> +    spin_unlock(&its->its_lock);
>> +
>> +    if ( !_vcpu )
>> +        return false;
>> +
>> +    if ( collid >= its->max_collections )
>> +        return false;
> 
> No need for this check. It is already done in get_vcpu_from_collection.

Good point, dropped in v4.

>> +
>> +    *vcpu = _vcpu;
>> +    *vlpi = _vlpi;
>> +
>> +    return true;
>> +}
>> +
>> +#define SKIP_LPI_UPDATE 1
>> +bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
>> +                uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
>> +{
>> +    struct vits_itte *itte;
>> +
>> +    if ( collid >= its->max_collections )
>> +        return false;
>> +
>> +    /* TODO: validate vlpi */
> 
> This TODO needs to be fixed as soon as possible.

Done in v3.

Cheers,
Andre.


>> +
>> +    spin_lock(&its->its_lock);
>> +    itte = get_devid_evid(its, devid, evid);
>> +    if ( !itte )
>> +    {
>> +        spin_unlock(&its->its_lock);
>> +        return false;
>> +    }
>> +
>> +    itte->collection = collid;
>> +    if ( vlpi != SKIP_LPI_UPDATE )
>> +        itte->vlpi = vlpi;
>> +
>> +    if ( vcpu )
>> +        *vcpu = get_vcpu_from_collection(its, collid);
>> +
>> +    put_devid_evid(its, itte);
>> +    spin_unlock(&its->its_lock);
>> +
>> +    return true;
>> +}
>> +
>>  /**************************************
>>   * Functions that handle ITS commands *
>>   **************************************/
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 20/27] ARM: vITS: handle MAPTI command
  2017-03-24 14:54   ` Julien Grall
@ 2017-04-03 18:47     ` Andre Przywara
  0 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 18:47 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 24/03/17 14:54, Julien Grall wrote:
> Hi Andre,
> 
> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>> The MAPTI commands associates a DeviceID/EventID pair with a LPI/CPU
>> pair and actually instantiates LPI interrupts.
>> We connect the already allocated host LPI to this virtual LPI, so that
>> any triggering IRQ on the host can be quickly forwarded to a guest.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c        | 63
>> ++++++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/gic-v3-lpi.c        | 18 ++++++++++++
>>  xen/arch/arm/vgic-v3-its.c       | 27 +++++++++++++++--
>>  xen/include/asm-arm/gic_v3_its.h |  6 ++++
>>  4 files changed, 112 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 5a2dbec..e2fcf50 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -724,6 +724,69 @@ restart:
>>      spin_unlock(&d->arch.vgic.its_devices_lock);
>>  }
>>
>> +/*
>> + * Translates an event for a given guest device ID into the
>> associated host
>> + * LPI number. This can be used to look up the mapped guest LPI.
>> + */
>> +static uint32_t translate_event(struct domain *d, paddr_t doorbell,
>> +                                uint32_t devid, uint32_t eventid)
>> +{
>> +    struct rb_node *node;
>> +    struct its_devices *dev;
>> +    uint32_t host_lpi = 0;
>> +    int cmp;
>> +
>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>> +    node = d->arch.vgic.its_devices.rb_node;
>> +    while (node)
>> +    {
>> +        dev = rb_entry(node, struct its_devices, rbnode);
>> +        cmp = compare_its_guest_devices(dev, doorbell, devid);
>> +
>> +        if ( !cmp )
>> +        {
>> +            if ( eventid >= dev->eventids )
>> +                goto out;
>> +
>> +            host_lpi = dev->host_lpis[eventid / LPI_BLOCK] +
>> +                                (eventid % LPI_BLOCK);
>> +            if ( !is_lpi(host_lpi) )
> 
> Hmmm, I don't understand this check. host_lpi should always be an LPI. No?

Looks like. Dropped in v4.

>> +                host_lpi = 0;
>> +            goto out;
>> +        }
>> +
>> +        if ( cmp > 0 )
>> +            node = node->rb_left;
>> +        else
>> +            node = node->rb_right;
>> +    }
>> +
>> +out:
>> +    spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +    return host_lpi;
>> +}
>> +
>> +/*
>> + * Connects the event ID for an already assigned device to the given
>> VCPU/vLPI
>> + * pair. The corresponding physical LPI is already mapped on the host
>> side
>> + * (when assigning the physical device to the guest), so we just
>> connect the
>> + * target VCPU/vLPI pair to that interrupt to inject it properly if
>> it fires.
>> + */
>> +int gicv3_assign_guest_event(struct domain *d, paddr_t doorbell_address,
>> +                             uint32_t devid, uint32_t eventid,
> 
> It looks like to me that devid is the virtual deviceID. If so, please
> prefix with 'v', otherwise 'p'.

Fixed in v4.

>> +                             struct vcpu *v, uint32_t virt_lpi)
>> +{
>> +    uint32_t host_lpi = translate_event(d, doorbell_address, devid,
>> eventid);
>> +
>> +    if ( !host_lpi )
>> +        return -ENOENT;
>> +
>> +    gicv3_lpi_update_host_entry(host_lpi, d->domain_id, v->vcpu_id,
>> virt_lpi);
>> +
>> +    return 0;
>> +}
>> +
>>  /* Scan the DT for any ITS nodes and create a list of host ITSes out
>> of it. */
>>  void gicv3_its_dt_init(const struct dt_device_node *node)
>>  {
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>> index 994698e..c110ec9 100644
>> --- a/xen/arch/arm/gic-v3-lpi.c
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -153,6 +153,24 @@ void do_LPI(unsigned int lpi)
>>      vgic_vcpu_inject_irq(vcpu, hlpi.virt_lpi);
>>  }
>>
>> +int gicv3_lpi_update_host_entry(uint32_t host_lpi, int domain_id,
>> +                                unsigned int vcpu_id, uint32_t virt_lpi)
>> +{
>> +    union host_lpi *hlpip, hlpi;
>> +
>> +    host_lpi -= LPI_OFFSET;
> 
> I would add an ASSERT(host_lpi > LPI_OFFSET);

Fixed in v4.

>> +
>> +    hlpip = &lpi_data.host_lpis[host_lpi /
>> HOST_LPIS_PER_PAGE][host_lpi % HOST_LPIS_PER_PAGE];
>> +
>> +    hlpi.virt_lpi = virt_lpi;
>> +    hlpi.dom_id = domain_id;
>> +    hlpi.vcpu_id = vcpu_id;
>> +
>> +    write_u64_atomic(&hlpip->data, hlpi.data);
>> +
>> +    return 0;
>> +}
>> +
>>  static int gicv3_lpi_allocate_pendtable(uint64_t *reg)
>>  {
>>      uint64_t val;
>> diff --git a/xen/arch/arm/vgic-v3-its.c b/xen/arch/arm/vgic-v3-its.c
>> index c26d5d4..600ff69 100644
>> --- a/xen/arch/arm/vgic-v3-its.c
>> +++ b/xen/arch/arm/vgic-v3-its.c
>> @@ -167,8 +167,8 @@ static bool read_itte(struct virt_its *its,
>> uint32_t devid, uint32_t evid,
>>  }
>>
>>  #define SKIP_LPI_UPDATE 1
>> -bool write_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
>> -                uint32_t collid, uint32_t vlpi, struct vcpu **vcpu)
>> +static bool write_itte(struct virt_its *its, uint32_t devid, uint32_t
>> evid,
>> +                       uint32_t collid, uint32_t vlpi, struct vcpu
>> **vcpu)
> 
> Please explain in the commit message why you move to static.

Fixed in v4.

> 
>>  {
>>      struct vits_itte *itte;
>>
>> @@ -332,6 +332,25 @@ static int its_handle_mapd(struct virt_its *its,
>> uint64_t *cmdptr)
>>      return 0;
>>  }
>>
>> +static int its_handle_mapti(struct virt_its *its, uint64_t *cmdptr)
>> +{
>> +    uint32_t devid = its_cmd_get_deviceid(cmdptr);
>> +    uint32_t eventid = its_cmd_get_id(cmdptr);
>> +    uint32_t intid = its_cmd_get_physical_id(cmdptr);
>> +    int collid = its_cmd_get_collection(cmdptr);
> 
> This should be unsigned.

Fixed in v4.

>> +    struct vcpu *vcpu;
>> +
>> +    if ( its_cmd_get_command(cmdptr) == GITS_CMD_MAPI )
>> +        intid = eventid;
>> +
>> +    if ( !write_itte(its, devid, eventid, collid, intid, &vcpu) )
>> +        return -1;
>> +
>> +    gicv3_assign_guest_event(its->d, its->doorbell_address, devid,
>> eventid, vcpu, intid);
> 
> If you have a function returning an error, then you should check it and
> not ignoring it.

Fixed in v3.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-03-23 19:08   ` Julien Grall
@ 2017-04-03 19:30     ` Andre Przywara
  2017-04-03 20:13       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 19:30 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 23/03/17 19:08, Julien Grall wrote:
> Hi Andre,
> 
> On 16/03/17 11:20, Andre Przywara wrote:
>> The number of LPIs on a host can be potentially huge (millions),
>> although in practise will be mostly reasonable. So prematurely allocating
>> an array of struct irq_desc's for each LPI is not an option.
>> However Xen itself does not care about LPIs, as every LPI will be
>> injected
>> into a guest (Dom0 for now).
>> Create a dense data structure (8 Bytes) for each LPI which holds just
>> enough information to determine the virtual IRQ number and the VCPU into
>> which the LPI needs to be injected.
>> Also to not artificially limit the number of LPIs, we create a 2-level
>> table for holding those structures.
>> This patch introduces functions to initialize these tables and to
>> create, lookup and destroy entries for a given LPI.
>> By using the naturally atomic access guarantee the native uint64_t data
>> type gives us, we allocate and access LPI information in a way that does
>> not require a lock.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c        |  90 ++++++++++++++++++-
>>  xen/arch/arm/gic-v3-lpi.c        | 188
>> +++++++++++++++++++++++++++++++++++++++
>>  xen/include/asm-arm/gic.h        |   5 ++
>>  xen/include/asm-arm/gic_v3_its.h |  11 +++
>>  4 files changed, 292 insertions(+), 2 deletions(-)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 60b15b5..ed14d95 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -148,6 +148,20 @@ static int its_send_cmd_sync(struct host_its
>> *its, unsigned int cpu)
>>      return its_send_command(its, cmd);
>>  }
>>
>> +static int its_send_cmd_mapti(struct host_its *its,
>> +                              uint32_t deviceid, uint32_t eventid,
>> +                              uint32_t pintid, uint16_t icid)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    cmd[0] = GITS_CMD_MAPTI | ((uint64_t)deviceid << 32);
>> +    cmd[1] = eventid | ((uint64_t)pintid << 32);
>> +    cmd[2] = icid;
>> +    cmd[3] = 0x00;
>> +
>> +    return its_send_command(its, cmd);
>> +}
>> +
>>  static int its_send_cmd_mapc(struct host_its *its, uint32_t
>> collection_id,
>>                               unsigned int cpu)
>>  {
>> @@ -180,6 +194,19 @@ static int its_send_cmd_mapd(struct host_its
>> *its, uint32_t deviceid,
>>      return its_send_command(its, cmd);
>>  }
>>
>> +static int its_send_cmd_inv(struct host_its *its,
>> +                            uint32_t deviceid, uint32_t eventid)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    cmd[0] = GITS_CMD_INV | ((uint64_t)deviceid << 32);
>> +    cmd[1] = eventid;
>> +    cmd[2] = 0x00;
>> +    cmd[3] = 0x00;
>> +
>> +    return its_send_command(its, cmd);
>> +}
>> +
>>  /* Set up the (1:1) collection mapping for the given host CPU. */
>>  int gicv3_its_setup_collection(unsigned int cpu)
>>  {
>> @@ -462,7 +489,7 @@ int gicv3_its_init(void)
>>
>>  static int remove_mapped_guest_device(struct its_devices *dev)
>>  {
>> -    int ret;
>> +    int ret, i;
>>
>>      if ( dev->hw_its )
>>      {
>> @@ -471,11 +498,19 @@ static int remove_mapped_guest_device(struct
>> its_devices *dev)
>>              return ret;
>>      }
>>
>> +    /*
>> +     * The only error the function below would return is -ENOENT, in
>> which
>> +     * case there is nothing to free here. So we just ignore it.
>> +     */
> 
> The function gicv3_free_host_lpi_block will only be used here. And to be
> fair, if you try to free something that does not exist then it is not
> really an error...
> 
> So I would prefer to see the function to be void.

Fixed in v3.

>> +    for ( i = 0; i < DIV_ROUND_UP(dev->eventids, LPI_BLOCK); i++ )
>> +        gicv3_free_host_lpi_block(dev->hw_its, dev->host_lpis[i]);
> 
> Again, without looking at the implementation of
> gicv3_free_host_lpi_block, I think the usage of the function is very
> confusing. When I read host_lpis, I expect to see one LPI per event. But
> instead it be the first LPI of a block. The lack of documentation of the
> field in its_devices does not help to understand what's going on.
> 
> So please add some documentation and probably renaming some fields.

Fixed in v3.

> 
> Lastly, you have not answered to my question: should not we discard the
> LPIs before removing the device? Or does MAPD take care for you?"

Yes: "MAPD removes the mapping of the specified DeviceID. and interrupt
requests from that device are discarded."

So we don't need to issue individual DISCARDs if the device in unmapped.

>> +
>>      ret = gicv3_its_wait_commands(dev->hw_its);
>>      if ( ret )
>>          return ret;
> 
> I know I asked to wait the command, thank you for addressing it. But
> now, if the function fail you will end-up to leak memory. This is not
> better than failing to wait commands.

Fixed in v3.

>>
>>      xfree(dev->itt_addr);
>> +    xfree(dev->host_lpis);
>>      xfree(dev);
>>
>>      return 0;
>> @@ -513,6 +548,36 @@ static int compare_its_guest_devices(struct
>> its_devices *dev,
>>  }
>>
>>  /*
>> + * On the host ITS @its, map @nr_events consecutive LPIs.
>> + * The mapping connects a device @devid and event @eventid pair to
>> LPI @lpi,
>> + * increasing both @eventid and @lpi to cover the number of requested
>> LPIs.
>> + */
>> +int gicv3_its_map_host_events(struct host_its *its,
>> +                              uint32_t devid, uint32_t eventid,
>> uint32_t lpi,
>> +                              uint32_t nr_events)
>> +{
>> +    uint32_t i;
>> +    int ret;
>> +
>> +    for ( i = 0; i < nr_events; i++ )
>> +    {
>> +        /* For now we map every host LPI to host CPU 0 */
>> +        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
>> +        if ( ret )
>> +            return ret;
>> +        ret = its_send_cmd_inv(its, devid, eventid + i);
> 
> 
> So the spec allows up to 32KB event per device. As all the LPIs will be
> routed to CPU0 (e.g collection 0), it would be more efficient to do an
> INVALL. I would be happy to defer that to post Xen 4.9, but the rest
> needs to be fixed.

I tried INVALL and it didn't work, at least on the model. I can't see
why, so I kept the individual INVs in.
I have the patch still lying around, so we can revisit this later.

> Furthermore, what if the queue is full? AFAIU, you will return an error
> but it is not propagate. So Xen will think the device has been mapped
> even if it is not true.

In v3 the error is now checked in the caller and
gicv3_its_map_guest_device() cleans up and returns an error.
In the moment this is called from the MAPD emulation, so there is no
good way of communicating this back to the guest, but in the future this
can return the error to the hypercall and Dom0 can react accordingly.

> I think we need to have a plan here as this may likely happen if a
> device has many MSI and/or the queue is nearly full.
> 
>> +        if ( ret )
>> +            return ret;
>> +    }
>> +
>> +    ret = its_send_cmd_sync(its, 0);
>> +    if ( ret )
>> +        return ret;
>> +
>> +    return gicv3_its_wait_commands(its);
>> +}
>> +
>> +/*
>>   * Map a hardware device, identified by a certain host ITS and its
>> device ID
>>   * to domain d, a guest ITS (identified by its doorbell address) and
>> device ID.
>>   * Also provide the number of events (MSIs) needed for that device.
>> @@ -528,7 +593,7 @@ int gicv3_its_map_guest_device(struct domain *d,
>>      struct host_its *hw_its;
>>      struct its_devices *dev = NULL, *temp;
>>      struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent
>> = NULL;
>> -    int ret = -ENOENT;
>> +    int ret = -ENOENT, i;
> 
> Again, i should be unsigned.

Fixed in v4.

>>
>>      hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>>      if ( !hw_its )
>> @@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
>>      if ( !dev )
>>          goto out_unlock;
>>
>> +    dev->host_lpis = xzalloc_array(uint32_t,
>> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
> 
> Rather than having DIV_ROUND_UP spread everywhere. Would not be easier
> to round_up nr_events once for all?

I'd rather keep nr_events as the actual number around.
I think we might look into actually limiting the allocation later.

> 
>> +    if ( !dev->host_lpis )
>> +        goto out_unlock;
>> +
>>      ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
>>                              virt_to_maddr(itt_addr), true);
>>      if ( ret )
>> @@ -591,10 +661,26 @@ int gicv3_its_map_guest_device(struct domain *d,
>>
>>      spin_unlock(&d->arch.vgic.its_devices_lock);
>>
>> +    /*
>> +     * Map all host LPIs within this device already. We can't afford
>> to queue
>> +     * any host ITS commands later on during the guest's runtime.
>> +     */
>> +    for ( i = 0; i < DIV_ROUND_UP(nr_events, LPI_BLOCK); i++ )
>> +    {
>> +        ret = gicv3_allocate_host_lpi_block(hw_its, d, host_devid,
>> +                                            i * LPI_BLOCK);
>> +        if ( ret < 0 )
>> +            goto out;
> 
> This sounds quite wrong to me. If you fail to allocate a block, you will
> end-up with half of the device configured. However, you are freeing the
> memory but not reverting the ITS command. This is going to end up very
> badly.

I need to check again whether this is really a problem here.

> Please be really careful with the error path. Most of them are not
> correct and I don't want to end up dealing with security issue later on.
> 
>> +
>> +        dev->host_lpis[i] = ret;
> 
> Hmmm... as I said on the previous version you should not use the return
> for both error code and LPIs. LPIs are encoded on 32-bit, sooner or
> later there will be a clash between the error and the LPI.

Fixed in v3.

> 
>> +    }
>> +
>>      return 0;
>>
>>  out_unlock:
>>      spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +out:
>>      if ( dev )
>>          xfree(dev->host_lpis);
>>      xfree(itt_addr);
>> diff --git a/xen/arch/arm/gic-v3-lpi.c b/xen/arch/arm/gic-v3-lpi.c
>> index 51d7425..59d3ba4 100644
>> --- a/xen/arch/arm/gic-v3-lpi.c
>> +++ b/xen/arch/arm/gic-v3-lpi.c
>> @@ -20,24 +20,44 @@
>>
>>  #include <xen/lib.h>
>>  #include <xen/mm.h>
>> +#include <xen/sched.h>
>> +#include <asm/atomic.h>
>> +#include <asm/domain.h>
>>  #include <asm/gic.h>
>>  #include <asm/gic_v3_defs.h>
>>  #include <asm/gic_v3_its.h>
>>  #include <asm/io.h>
>>  #include <asm/page.h>
>>
>> +/* LPIs on the host always go to a guest, so no struct irq_desc for
>> them. */
>> +union host_lpi {
>> +    uint64_t data;
>> +    struct {
>> +        uint32_t virt_lpi;
>> +        uint16_t dom_id;
>> +        uint16_t vcpu_id;
>> +    };
>> +};
>> +
>>  #define LPI_PROPTABLE_NEEDS_FLUSHING    (1U << 0)
>>  /* Global state */
>>  static struct {
>>      /* The global LPI property table, shared by all redistributors. */
>>      uint8_t *lpi_property;
>>      /*
>> +     * A two-level table to lookup LPIs firing on the host and look
>> up the
>> +     * domain and virtual LPI number to inject.
>> +     */
>> +    union host_lpi **host_lpis;
>> +    /*
>>       * Number of physical LPIs the host supports. This is a property of
>>       * the GIC hardware. We depart from the habit of naming these things
>>       * "physical" in Xen, as the GICv3/4 spec uses the term "physical
>> LPI"
>>       * in a different context to differentiate them from "virtual LPIs".
>>       */
>>      unsigned long int nr_host_lpis;
>> +    /* Protects allocation and deallocation of host LPIs, but not the
>> access */
>> +    spinlock_t host_lpis_lock;
>>      unsigned int flags;
>>  } lpi_data;
>>
>> @@ -50,6 +70,19 @@ struct lpi_redist_data {
>>  static DEFINE_PER_CPU(struct lpi_redist_data, lpi_redist);
>>
>>  #define MAX_PHYS_LPIS   (lpi_data.nr_host_lpis - LPI_OFFSET)
>> +#define HOST_LPIS_PER_PAGE      (PAGE_SIZE / sizeof(union host_lpi))
>> +
>> +static union host_lpi *gic_get_host_lpi(uint32_t plpi)
>> +{
>> +    if ( !is_lpi(plpi) || plpi >= MAX_PHYS_LPIS + LPI_OFFSET )
>> +        return NULL;
>> +
>> +    plpi -= LPI_OFFSET;
>> +    if ( !lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE] )
>> +        return NULL;
>> +
>> +    return &lpi_data.host_lpis[plpi / HOST_LPIS_PER_PAGE][plpi %
>> HOST_LPIS_PER_PAGE];
>> +}
>>
>>  /* Stores this redistributor's physical address and ID in a per-CPU
>> variable */
>>  void gicv3_set_redist_address(paddr_t address, unsigned int redist_id)
>> @@ -204,15 +237,170 @@ int gicv3_lpi_init_rdist(void __iomem *
>> rdist_base)
>>  static unsigned int max_lpi_bits = CONFIG_MAX_PHYS_LPI_BITS;
>>  integer_param("max_lpi_bits", max_lpi_bits);
>>
>> +/*
>> + * Allocate the 2nd level array for host LPIs. This one holds pointers
>> + * to the page with the actual "union host_lpi" entries. Our LPI limit
>> + * avoids excessive memory usage.
>> + */
>>  int gicv3_lpi_init_host_lpis(unsigned int hw_lpi_bits)
>>  {
>> +    int nr_lpi_ptrs;
> 
> unsigned int please.

Fixed in v4.

>> +
>> +    /* We rely on the data structure being atomically accessible. */
>> +    BUILD_BUG_ON(sizeof(union host_lpi) > sizeof(unsigned long));
>> +
>>      lpi_data.nr_host_lpis = BIT_ULL(min(hw_lpi_bits, max_lpi_bits));
>>
>> +    spin_lock_init(&lpi_data.host_lpis_lock);
>> +
>> +    nr_lpi_ptrs = MAX_PHYS_LPIS / (PAGE_SIZE / sizeof(union host_lpi));
>> +    lpi_data.host_lpis = xzalloc_array(union host_lpi *, nr_lpi_ptrs);
>> +    if ( !lpi_data.host_lpis )
>> +        return -ENOMEM;
>> +
>>      printk("GICv3: using at most %ld LPIs on the host.\n",
>> MAX_PHYS_LPIS);
>>
>>      return 0;
>>  }
>>
>> +/* Must be called with host_lpis_lock held. */
> 
> Again, this is a call for adding an ASSERT in the function.

This comment is more like lock documentation, to give code writers a
guidance how the locking should be handled here.
I am not convinced that having ASSERTS in *static* functions is really
useful.

>> +static int find_unused_host_lpi(int start, uint32_t *index)
> 
> start should probably be unsigned.
> 
>> +{
>> +    int chunk;
> 
> Ditto.

Fixed in v3.

>> +    uint32_t i = *index;
>> +
>> +    for ( chunk = start; chunk < MAX_PHYS_LPIS / HOST_LPIS_PER_PAGE;
>> chunk++ )
>> +    {
>> +        /* If we hit an unallocated chunk, use entry 0 in that one. */
>> +        if ( !lpi_data.host_lpis[chunk] )
>> +        {
>> +            *index = 0;
>> +            return chunk;
>> +        }
>> +
>> +        /* Find an unallocated entry in this chunk. */
>> +        for ( ; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
>> +        {
>> +            if ( lpi_data.host_lpis[chunk][i].dom_id == INVALID_DOMID )
>> +            {
>> +                *index = i;
>> +                return chunk;
>> +            }
>> +        }
>> +        i = 0;
>> +    }
>> +
>> +    return -1;
>> +}
>> +
>> +/*
>> + * Allocate a block of 32 LPIs on the given host ITS for device "devid",
>> + * starting with "eventid". Put them into the respective ITT by
>> issuing a
>> + * MAPTI command for each of them.
>> + */
>> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain
>> *d,
>> +                                  uint32_t host_devid, uint32_t eventid)
> 
> Again, most of the parameters here is just for calling back ITS and the
> caller should all in hand. So I would much prefer to avoid a call chain
> ITS -> LPI -> ITS and make the LPI code ITS agnostic.
> 
> Furthermore, the return value is both used to return an error and the
> LPI. Given that an LPI is encoded on 32-bit, sooner or later there will
> be a clash with between the error and the LPI. So it would be wise to
> dissociate the error code and the LPIs.
> 
> The prototype of this function would like:
> 
> int gicv3_allocate_host_lpi_block(struct domain *domain, uint32_t
> *first_lpi);

Fixed in v3.

>> +{
>> +    static uint32_t next_lpi = 0;
>> +    uint32_t lpi, lpi_idx = next_lpi % HOST_LPIS_PER_PAGE;
>> +    int chunk;
>> +    int i;
>> +
>> +    spin_lock(&lpi_data.host_lpis_lock);
>> +    chunk = find_unused_host_lpi(next_lpi / HOST_LPIS_PER_PAGE,
>> &lpi_idx);
>> +
>> +    if ( chunk == - 1 )          /* rescan for a hole from the
>> beginning */
>> +    {
>> +        lpi_idx = 0;
>> +        chunk = find_unused_host_lpi(0, &lpi_idx);
>> +        if ( chunk == -1 )
>> +        {
>> +            spin_unlock(&lpi_data.host_lpis_lock);
>> +            return -ENOSPC;
>> +        }
> 
> The algo does not seem to have changed since the previous version.
> Looking at it again, my understanding is you will always try to allocate
> forward. So if the current chunk is full, you will allocate the next one
> rather than looking whether a previous chunk has space available. This
> will result to allocate more memory than necessary.

In v4 I amended the code slightly to move next_lpi outside of the
function. When we now free an LPI block, we check if the previous
next_lpi was pointing after this block and adjust it in this case.

> Similarly unused chunk could be freed to save memory.

No, we can't, because this breaks the lockless property. A user
(incoming LPI) would find the first level pointer and go into that
block. So we can't never replace this block pointer now. That smells
like a case for RCU, but I am not sure if Xen can properly handle this case.
But with the above optimization (adjusting next_lpi on freeing a block)
I am pretty sure this isn't a problem for now, especially for Dom0.

> To be clear, I am asking for confirmation, documentation, and listing as
> an improvement.
> 
>> +    }
>> +
>> +    /* If we hit an unallocated chunk, we initialize it and use entry
>> 0. */
>> +    if ( !lpi_data.host_lpis[chunk] )
>> +    {
>> +        union host_lpi *new_chunk;
>> +
>> +        new_chunk = alloc_xenheap_pages(0, 0);
> 
> As said on the previous version, please use alloc_xenheap_page as you
> only allocate one page.

Fixed in v3.

> Also, when NUMA support will be added we may want to take into account
> the node associated to the device saving us some time when reading the
> memory. You don't need to handle that now, but a TODO would be quite
> helpful.

TODO added in v3.

>> +        if ( !new_chunk )
>> +        {
>> +            spin_unlock(&lpi_data.host_lpis_lock);
>> +            return -ENOMEM;
>> +        }
>> +
>> +        for ( i = 0; i < HOST_LPIS_PER_PAGE; i += LPI_BLOCK )
>> +            new_chunk[i].dom_id = INVALID_DOMID;
>> +
>> +        lpi_data.host_lpis[chunk] = new_chunk;
>> +        lpi_idx = 0;
>> +    }
>> +
>> +    lpi = chunk * HOST_LPIS_PER_PAGE + lpi_idx;
>> +
>> +    for ( i = 0; i < LPI_BLOCK; i++ )
>> +    {
>> +        union host_lpi hlpi;
>> +
>> +        /*
>> +         * Mark this host LPI as belonging to the domain, but don't
>> assign
>> +         * any virtual LPI or a VCPU yet.
>> +         */
>> +        hlpi.virt_lpi = INVALID_LPI;
>> +        hlpi.dom_id = d->domain_id;
>> +        hlpi.vcpu_id = INVALID_DOMID;
> 
> Again, please don't mix vcpu and domain. If INVALID_VCPUID does not
> exist then it might be worth adding one.

Fixed in v4.

>> +        write_u64_atomic(&lpi_data.host_lpis[chunk][lpi_idx + i].data,
>> +                         hlpi.data);
>> +
>> +        /*
>> +         * Enable this host LPI, so we don't have to do this during the
>> +         * guest's runtime.
>> +         */
>> +        lpi_data.lpi_property[lpi + i] |= LPI_PROP_ENABLED;
>> +    }
>> +
>> +    /*
>> +     * We have allocated and initialized the host LPI entries, so
>> it's safe
>> +     * to drop the lock now. Access to the structures can be done
>> concurrently
>> +     * as it involves only an atomic uint64_t access.
>> +     */
>> +    spin_unlock(&lpi_data.host_lpis_lock);
>> +
>> +    if ( lpi_data.flags & LPI_PROPTABLE_NEEDS_FLUSHING )
>> +       
>> clean_and_invalidate_dcache_va_range(&lpi_data.lpi_property[lpi],
>> +                                             LPI_BLOCK);
>> +
>> +    gicv3_its_map_host_events(its, host_devid, eventid, lpi +
>> LPI_OFFSET,
>> +                              LPI_BLOCK);
> 
> See my comment at the top of the function.

Fixed in v3 (I believe).

>> +
>> +    next_lpi = lpi + LPI_BLOCK;
> 
> Newline here please.

Fixed in v3.

> 
>> +    return lpi + LPI_OFFSET;
>> +}
>> +
>> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi)
>> +{
>> +    union host_lpi *hlpi, empty_lpi = { .dom_id = INVALID_DOMID };
>> +    int i;
>> +
>> +    hlpi = gic_get_host_lpi(lpi);
>> +    if ( !hlpi )
>> +        return -ENOENT;
>> +
>> +    spin_lock(&lpi_data.host_lpis_lock);
>> +
>> +    for ( i = 0; i < LPI_BLOCK; i++ )
>> +        write_u64_atomic(&hlpi[i].data, empty_lpi.data);
>> +
>> +    /* TODO: Call a function in gic-v3-its.c to send DISCARDs */
> 
> It is quite worrying to see a TODO in a code that is supposed to be
> merged really soon.
> 
> I think this should be done by the caller and not here. You also need to
> disable the interrupts which has not been done here.

I removed the TODO in v3, because it was wrong. We unmap the device
before, so no need for individual DISCARDs (as mentioned above).

>> +
>> +    spin_unlock(&lpi_data.host_lpis_lock);
>> +
>> +    return 0;
>> +}
>> +
>>  /*
>>   * Local variables:
>>   * mode: C
>> diff --git a/xen/include/asm-arm/gic.h b/xen/include/asm-arm/gic.h
>> index 12bd155..7825575 100644
>> --- a/xen/include/asm-arm/gic.h
>> +++ b/xen/include/asm-arm/gic.h
>> @@ -220,7 +220,12 @@ enum gic_version {
>>      GIC_V3,
>>  };
>>
>> +#define INVALID_LPI     0
>>  #define LPI_OFFSET      8192
> 
> Again, newline here please.

Fixed in v3.

>> +static inline bool is_lpi(unsigned int irq)
>> +{
>> +    return irq >= LPI_OFFSET;
>> +}
> 
> Again, I think both INVALID_LPI and is_lpi should be moved in irq.h.

Fixed in v3.

>>
>>  extern enum gic_version gic_hw_version(void);
>>
>> diff --git a/xen/include/asm-arm/gic_v3_its.h
>> b/xen/include/asm-arm/gic_v3_its.h
>> index 3421ea0..2387972 100644
>> --- a/xen/include/asm-arm/gic_v3_its.h
>> +++ b/xen/include/asm-arm/gic_v3_its.h
>> @@ -106,6 +106,11 @@
>>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>>  #define HOST_ITS_USES_PTA               (1U << 1)
>>
>> +#define INVALID_DOMID ((uint16_t)~0)
> 
> Again, rather than defining your own invalid domid, it would be better
> to use the one defined in xen.h (see DOMID_INVALID).

Fixed in v3.

Cheers,
Andre.


>> +
>> +/* We allocate LPIs on the hosts in chunks of 32 to reduce handling
>> overhead. */
>> +#define LPI_BLOCK                       32
>> +
>>  /* data structure for each hardware ITS */
>>  struct host_its {
>>      struct list_head entry;
>> @@ -151,6 +156,12 @@ int gicv3_its_map_guest_device(struct domain *d,
>>                                 paddr_t host_doorbell, uint32_t
>> host_devid,
>>                                 paddr_t guest_doorbell, uint32_t
>> guest_devid,
>>                                 uint32_t nr_events, bool valid);
>> +int gicv3_its_map_host_events(struct host_its *its,
>> +                              uint32_t host_devid, uint32_t eventid,
>> +                              uint32_t lpi, uint32_t nrevents);
>> +int gicv3_allocate_host_lpi_block(struct host_its *its, struct domain
>> *d,
>> +                                  uint32_t host_devid, uint32_t
>> eventid);
>> +int gicv3_free_host_lpi_block(struct host_its *its, uint32_t lpi);
>>
>>  #else
>>
>>
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-03-22 22:45   ` Stefano Stabellini
@ 2017-04-03 19:45     ` Andre Przywara
  0 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 19:45 UTC (permalink / raw)
  To: Stefano Stabellini
  Cc: xen-devel, Julien Grall, Shanker Donthineni, Vijay Kilari

Hi,

On 22/03/17 22:45, Stefano Stabellini wrote:
> On Thu, 16 Mar 2017, Andre Przywara wrote:
>> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
>> those IDs, which we directly pass on to the host.
>> For this we have to map each device that Dom0 may request to a host
>> ITS device with the same identifier.
>> Allocate the respective memory and enter each device into an rbtree to
>> later be able to iterate over it or to easily teardown guests.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c        | 207 +++++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/vgic-v3.c           |   3 +
>>  xen/include/asm-arm/domain.h     |   3 +
>>  xen/include/asm-arm/gic_v3_its.h |  18 ++++
>>  4 files changed, 231 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 5c11b0d..60b15b5 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -21,6 +21,8 @@
>>  #include <xen/lib.h>
>>  #include <xen/delay.h>
>>  #include <xen/mm.h>
>> +#include <xen/rbtree.h>
>> +#include <xen/sched.h>
>>  #include <xen/sizes.h>
>>  #include <asm/gic.h>
>>  #include <asm/gic_v3_defs.h>
>> @@ -32,6 +34,17 @@
>>  
>>  LIST_HEAD(host_its_list);
>>  
>> +struct its_devices {
>> +    struct rb_node rbnode;
>> +    struct host_its *hw_its;
>> +    void *itt_addr;
>> +    paddr_t guest_doorbell;
>> +    uint32_t host_devid;
>> +    uint32_t guest_devid;
>> +    uint32_t eventids;
>> +    uint32_t *host_lpis;
>> +};
>> +
>>  bool gicv3_its_host_has_its(void)
>>  {
>>      return !list_empty(&host_its_list);
>> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its *its, uint32_t collection_id,
>>      return its_send_command(its, cmd);
>>  }
>>  
>> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>> +                             uint8_t size_bits, paddr_t itt_addr, bool valid)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    if ( valid )
>> +    {
>> +        ASSERT(size_bits < 32);
>> +        ASSERT(!(itt_addr & ~GENMASK(51, 8)));
>> +    }
>> +    cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
>> +    cmd[1] = valid ? size_bits : 0x00;
>> +    cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
>> +    cmd[3] = 0x00;
>> +
>> +    return its_send_command(its, cmd);
>> +}
>> +
>>  /* Set up the (1:1) collection mapping for the given host CPU. */
>>  int gicv3_its_setup_collection(unsigned int cpu)
>>  {
>> @@ -379,6 +410,7 @@ static int gicv3_its_init_single_its(struct host_its *hw_its)
>>      devid_bits = min(devid_bits, max_its_device_bits);
>>      if ( reg & GITS_TYPER_PTA )
>>          hw_its->flags |= HOST_ITS_USES_PTA;
>> +    hw_its->itte_size = GITS_TYPER_ITT_SIZE(reg);
>>  
>>      for ( i = 0; i < GITS_BASER_NR_REGS; i++ )
>>      {
>> @@ -428,6 +460,180 @@ int gicv3_its_init(void)
>>      return 0;
>>  }
>>  
>> +static int remove_mapped_guest_device(struct its_devices *dev)
>> +{
>> +    int ret;
>> +
>> +    if ( dev->hw_its )
>> +    {
>> +        int ret = its_send_cmd_mapd(dev->hw_its, dev->host_devid, 0, 0, false);
>> +        if ( ret )
>> +            return ret;
>> +    }
>> +
>> +    ret = gicv3_its_wait_commands(dev->hw_its);
>> +    if ( ret )
>> +        return ret;
>> +
>> +    xfree(dev->itt_addr);
>> +    xfree(dev);
>> +
>> +    return 0;
>> +}
>> +
>> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t doorbell_address)
>> +{
>> +    struct host_its *hw_its;
>> +
>> +    list_for_each_entry(hw_its, &host_its_list, entry)
> 
> Does this need to take a spinlock to protect host_its_list? I guess not
> because the list is not modified after boot?

Exactly, I added a comment in v4 explaining this.

>> +    {
>> +        if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
>> +            return hw_its;
>> +    }
>> +
>> +    return NULL;
>> +}
>> +
>> +static int compare_its_guest_devices(struct its_devices *dev,
>> +                                     paddr_t doorbell, uint32_t devid)
>> +{
>> +    if ( dev->guest_doorbell < doorbell )
>> +        return -1;
>> +
>> +    if ( dev->guest_doorbell > doorbell )
>> +        return 1;
>> +
>> +    if ( dev->guest_devid < devid )
>> +        return -1;
>> +
>> +    if ( dev->guest_devid > devid )
>> +        return 1;
>> +
>> +    return 0;
>> +}
>> +
>> +/*
>> + * Map a hardware device, identified by a certain host ITS and its device ID
>> + * to domain d, a guest ITS (identified by its doorbell address) and device ID.
>> + * Also provide the number of events (MSIs) needed for that device.
>> + * This does not check if this particular hardware device is already mapped
>> + * at another domain, it is expected that this would be done by the caller.
>> + */
>> +int gicv3_its_map_guest_device(struct domain *d,
>> +                               paddr_t host_doorbell, uint32_t host_devid,
>> +                               paddr_t guest_doorbell, uint32_t guest_devid,
>> +                               uint32_t nr_events, bool valid)
>> +{
>> +    void *itt_addr = NULL;
>> +    struct host_its *hw_its;
>> +    struct its_devices *dev = NULL, *temp;
>> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent = NULL;
>> +    int ret = -ENOENT;
>> +
>> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>> +    if ( !hw_its )
>> +        return ret;
>> +
>> +    /* check for already existing mappings */
>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>> +    while ( *new )
>> +    {
>> +        temp = rb_entry(*new, struct its_devices, rbnode);
>> +
>> +        parent = *new;
>> +        if ( !compare_its_guest_devices(temp, guest_doorbell, guest_devid) )
>> +        {
>> +            if ( !valid )
>> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
>> +
>> +            spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +            if ( valid )
>> +                return -EBUSY;
>> +
>> +            return remove_mapped_guest_device(temp);
>> +        }
>> +
> 
> Plese don't run compare_its_guest_devices twice with the same arguments,
> just store the return value.

Fixed in v3.

>> +        if ( compare_its_guest_devices(temp, guest_doorbell, guest_devid) > 0 )
>> +            new = &((*new)->rb_left);
>> +        else
>> +            new = &((*new)->rb_right);
>> +    }
>> +
>> +    if ( !valid )
>> +        goto out_unlock;
>> +
>> +    ret = -ENOMEM;
>> +
>> +    /* An Interrupt Translation Table needs to be 256-byte aligned. */
>> +    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
>> +    if ( !itt_addr )
>> +        goto out_unlock;
>> +
>> +    dev = xzalloc(struct its_devices);
>> +    if ( !dev )
>> +        goto out_unlock;
>> +
>> +    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
>> +                            virt_to_maddr(itt_addr), true);
>> +    if ( ret )
>> +        goto out_unlock;
>> +
>> +    dev->itt_addr = itt_addr;
>> +    dev->hw_its = hw_its;
>> +    dev->guest_doorbell = guest_doorbell;
>> +    dev->guest_devid = guest_devid;
>> +    dev->host_devid = host_devid;
>> +    dev->eventids = nr_events;
>> +
>> +    rb_link_node(&dev->rbnode, parent, new);
>> +    rb_insert_color(&dev->rbnode, &d->arch.vgic.its_devices);
>> +
>> +    spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +    return 0;
>> +
>> +out_unlock:
>> +    spin_unlock(&d->arch.vgic.its_devices_lock);
>> +    if ( dev )
>> +        xfree(dev->host_lpis);
> 
> Where is host_lpis allocated? Why is it freed here?

So I swapped this patch with the next one in v4, because this solves
this issue.

>> +    xfree(itt_addr);
>> +    xfree(dev);
>> +    return ret;
>> +}
>> +
>> +/* Removing any connections a domain had to any ITS in the system. */
>> +void gicv3_its_unmap_all_devices(struct domain *d)
>> +{
>> +    struct rb_node *victim;
>> +    struct its_devices *dev;
>> +
>> +    /*
>> +     * This is an easily readable, yet inefficient implementation.
>> +     * It uses the provided iteration wrapper and erases each node, which
>> +     * possibly triggers rebalancing.
>> +     * This seems overkill since we are going to abolish the whole tree, but
>> +     * avoids an open-coded re-implementation of the traversal functions with
>> +     * some recursive function calls.
>> +     * Performance does not matter here, since we are destroying a domain.
>> +     */
>> +restart:
>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>> +    if ( (victim = rb_first(&d->arch.vgic.its_devices)) )
>> +    {
>> +        dev = rb_entry(victim, struct its_devices, rbnode);
>> +        rb_erase(victim, &d->arch.vgic.its_devices);
>> +
>> +        spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +        remove_mapped_guest_device(dev);
>> +
>> +        goto restart;
>> +    }
> 
> There is no need to use goto here, a simple while loop should suffice.

Fixed in v4.

>> +    spin_unlock(&d->arch.vgic.its_devices_lock);
>> +}
>> +
>>  /* Scan the DT for any ITS nodes and create a list of host ITSes out of it. */
>>  void gicv3_its_dt_init(const struct dt_device_node *node)
>>  {
>> @@ -455,6 +661,7 @@ void gicv3_its_dt_init(const struct dt_device_node *node)
>>          its_data->addr = addr;
>>          its_data->size = size;
>>          its_data->dt_node = its;
>> +        spin_lock_init(&its_data->cmd_lock);
> 
> This change should be in the previous patch

Fixed in v4.

>>          printk("GICv3: Found ITS @0x%lx\n", addr);
>>  
>> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
>> index d61479d..1fadb00 100644
>> --- a/xen/arch/arm/vgic-v3.c
>> +++ b/xen/arch/arm/vgic-v3.c
>> @@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
>>      d->arch.vgic.nr_regions = rdist_count;
>>      d->arch.vgic.rdist_regions = rdist_regions;
>>  
>> +    spin_lock_init(&d->arch.vgic.its_devices_lock);
>> +    d->arch.vgic.its_devices = RB_ROOT;
>> +
>>      /*
>>       * Domain 0 gets the hardware address.
>>       * Guests get the virtual platform layout.
>> diff --git a/xen/include/asm-arm/domain.h b/xen/include/asm-arm/domain.h
>> index 2d6fbb1..00b9c1a 100644
>> --- a/xen/include/asm-arm/domain.h
>> +++ b/xen/include/asm-arm/domain.h
>> @@ -11,6 +11,7 @@
>>  #include <asm/gic.h>
>>  #include <public/hvm/params.h>
>>  #include <xen/serial.h>
>> +#include <xen/rbtree.h>
>>  
>>  struct hvm_domain
>>  {
>> @@ -109,6 +110,8 @@ struct arch_domain
>>          } *rdist_regions;
>>          int nr_regions;                     /* Number of rdist regions */
>>          uint32_t rdist_stride;              /* Re-Distributor stride */
>> +        struct rb_root its_devices;         /* devices mapped to an ITS */
>> +        spinlock_t its_devices_lock;        /* protects the its_devices tree */
>>  #endif
>>      } vgic;
>>  
>> diff --git a/xen/include/asm-arm/gic_v3_its.h b/xen/include/asm-arm/gic_v3_its.h
>> index 8b493fb..3421ea0 100644
>> --- a/xen/include/asm-arm/gic_v3_its.h
>> +++ b/xen/include/asm-arm/gic_v3_its.h
>> @@ -48,6 +48,10 @@
>>  #define GITS_TYPER_DEVICE_ID_BITS(r)    (((r & GITS_TYPER_DEVIDS_MASK) >> \
>>                                                 GITS_TYPER_DEVIDS_SHIFT) + 1)
>>  #define GITS_TYPER_IDBITS_SHIFT         8
>> +#define GITS_TYPER_ITT_SIZE_SHIFT       4
>> +#define GITS_TYPER_ITT_SIZE_MASK        (0xfUL << GITS_TYPER_ITT_SIZE_SHIFT)
>> +#define GITS_TYPER_ITT_SIZE(r)          ((((r) & GITS_TYPER_ITT_SIZE_MASK) >> \
>> +                                                GITS_TYPER_ITT_SIZE_SHIFT) + 1)
>>  
>>  #define GITS_IIDR_VALUE                 0x34c
>>  
>> @@ -94,7 +98,10 @@
>>  #define GITS_CMD_MOVALL                 0x0e
>>  #define GITS_CMD_DISCARD                0x0f
>>  
>> +#define ITS_DOORBELL_OFFSET             0x10040
>> +
>>  #include <xen/device_tree.h>
>> +#include <xen/rbtree.h>
>>  
>>  #define HOST_ITS_FLUSH_CMD_QUEUE        (1U << 0)
>>  #define HOST_ITS_USES_PTA               (1U << 1)
>> @@ -108,6 +115,7 @@ struct host_its {
>>      void __iomem *its_base;
>>      spinlock_t cmd_lock;
>>      void *cmd_buf;
>> +    unsigned int itte_size;
>>      unsigned int flags;
>>  };
> 
> I would move itte_size and its initialization to the patch that
> introduced struct host_its.

Puh, OK. This wasn't as innocent as it looks like on the first glance,
so I created a new patch to introduce the host ITS init early and
separate this from the ITS table init.

Cheers,
Andre.


>> @@ -134,6 +142,16 @@ uint64_t gicv3_get_redist_address(unsigned int cpu, bool use_pta);
>>  /* Map a collection for this host CPU to each host ITS. */
>>  int gicv3_its_setup_collection(unsigned int cpu);
>>  
>> +/*
>> + * Map a device on the host by allocating an ITT on the host (ITS).
>> + * "nr_event" specifies how many events (interrupts) this device will need.
>> + * Setting "valid" to false deallocates the device.
>> + */
>> +int gicv3_its_map_guest_device(struct domain *d,
>> +                               paddr_t host_doorbell, uint32_t host_devid,
>> +                               paddr_t guest_doorbell, uint32_t guest_devid,
>> +                               uint32_t nr_events, bool valid);
>> +
>>  #else
>>  
>>  static LIST_HEAD(host_its_list);
>> -- 
>> 2.9.0
>>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-03-22 17:29   ` Julien Grall
@ 2017-04-03 20:08     ` Andre Przywara
  2017-04-03 20:41       ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-04-03 20:08 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 22/03/17 17:29, Julien Grall wrote:
> Hi Andre,
> 
> On 16/03/17 11:20, Andre Przywara wrote:
>> The ITS uses device IDs to map LPIs to a device. Dom0 will later use
>> those IDs, which we directly pass on to the host.
>> For this we have to map each device that Dom0 may request to a host
>> ITS device with the same identifier.
>> Allocate the respective memory and enter each device into an rbtree to
>> later be able to iterate over it or to easily teardown guests.
>>
>> Signed-off-by: Andre Przywara <andre.przywara@arm.com>
>> ---
>>  xen/arch/arm/gic-v3-its.c        | 207
>> +++++++++++++++++++++++++++++++++++++++
>>  xen/arch/arm/vgic-v3.c           |   3 +
>>  xen/include/asm-arm/domain.h     |   3 +
>>  xen/include/asm-arm/gic_v3_its.h |  18 ++++
>>  4 files changed, 231 insertions(+)
>>
>> diff --git a/xen/arch/arm/gic-v3-its.c b/xen/arch/arm/gic-v3-its.c
>> index 5c11b0d..60b15b5 100644
>> --- a/xen/arch/arm/gic-v3-its.c
>> +++ b/xen/arch/arm/gic-v3-its.c
>> @@ -21,6 +21,8 @@
>>  #include <xen/lib.h>
>>  #include <xen/delay.h>
>>  #include <xen/mm.h>
>> +#include <xen/rbtree.h>
>> +#include <xen/sched.h>
>>  #include <xen/sizes.h>
>>  #include <asm/gic.h>
>>  #include <asm/gic_v3_defs.h>
>> @@ -32,6 +34,17 @@
>>
>>  LIST_HEAD(host_its_list);
>>
>> +struct its_devices {
>> +    struct rb_node rbnode;
>> +    struct host_its *hw_its;
>> +    void *itt_addr;
>> +    paddr_t guest_doorbell;
> 
> I think it would be worth to explain in the commit message why you need
> the guest_doorbell in the struct its_devices and how you plan to use it.

In v4 I now also elaborated on the reason in a comment (before the
struct), which I deem more useful than something in the commit message.

>> +    uint32_t host_devid;
>> +    uint32_t guest_devid;
>> +    uint32_t eventids;
>> +    uint32_t *host_lpis;
>> +};
>> +
>>  bool gicv3_its_host_has_its(void)
>>  {
>>      return !list_empty(&host_its_list);
>> @@ -149,6 +162,24 @@ static int its_send_cmd_mapc(struct host_its
>> *its, uint32_t collection_id,
>>      return its_send_command(its, cmd);
>>  }
>>
>> +static int its_send_cmd_mapd(struct host_its *its, uint32_t deviceid,
>> +                             uint8_t size_bits, paddr_t itt_addr,
>> bool valid)
>> +{
>> +    uint64_t cmd[4];
>> +
>> +    if ( valid )
>> +    {
>> +        ASSERT(size_bits < 32);
> 
> It would be better if you do the check against the real number in
> hardware (i.e GITS_TYPER.ID_bits).

Added in v4.

> 
>> +        ASSERT(!(itt_addr & ~GENMASK(51, 8)));
>> +    }
>> +    cmd[0] = GITS_CMD_MAPD | ((uint64_t)deviceid << 32);
>> +    cmd[1] = valid ? size_bits : 0x00;
> 
> This is really confusing. The check was not on the previous version. So
> why do you need that?

Admittedly I was taken away be some intention to check this here
properly. But since itt_addr and size are only valid with V=1, I removed
this in v3.

> Also, it would have been better to hide the "size - 1" in the helper
> avoiding to really on the caller to do the right thing.

I tend to agree, but then we have the awkward case where an unmap passes
0 in size, which then gets decremented by one. But you are right that
it's still saner this way, so I pass 1 now in the unmap call and do the
"-1" encoding in here.

>> +    cmd[2] = valid ? (itt_addr | GITS_VALID_BIT) : 0x00;
> 
> Ditto about "valid? ...".

Removed in v3.

> [...]
> 
>> +static struct host_its *gicv3_its_find_by_doorbell(paddr_t
>> doorbell_address)
>> +{
>> +    struct host_its *hw_its;
>> +
>> +    list_for_each_entry(hw_its, &host_its_list, entry)
>> +    {
>> +        if ( hw_its->addr + ITS_DOORBELL_OFFSET == doorbell_address )
> 
> Why not storing the ITS address rather than the doorbell to avoid this
> check?

Because the doorbell address is a nice architectural property of MSIs in
general. And we need this check anyway, it's just the addition of the
doorbell offset that is different.

> [...]
> 
>> +int gicv3_its_map_guest_device(struct domain *d,
>> +                               paddr_t host_doorbell, uint32_t
>> host_devid,
>> +                               paddr_t guest_doorbell, uint32_t
>> guest_devid,
>> +                               uint32_t nr_events, bool valid)
>> +{
>> +    void *itt_addr = NULL;
>> +    struct host_its *hw_its;
>> +    struct its_devices *dev = NULL, *temp;
>> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent
>> = NULL;
>> +    int ret = -ENOENT;
>> +
>> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>> +    if ( !hw_its )
>> +        return ret;
>> +
>> +    /* check for already existing mappings */
>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>> +    while ( *new )
>> +    {
>> +        temp = rb_entry(*new, struct its_devices, rbnode);
>> +
>> +        parent = *new;
>> +        if ( !compare_its_guest_devices(temp, guest_doorbell,
>> guest_devid) )
>> +        {
>> +            if ( !valid )
>> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
>> +
>> +            spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +            if ( valid )
> 
> Again, a printk(XENLOG_GUEST...) here would be useful to know which host
> DeviceID was associated to the guest DeviceID.

I added a gprintk(XENLOG_DEBUG, ), which I think is more appropriate (as
it may spam the console when some stupid guest is running). Let me know
if you want to have a different loglevel.

>> +                return -EBUSY;
>> +
>> +            return remove_mapped_guest_device(temp);
> 
> Just above you removed the device from the RB-tree but this function may
> fail and never free the memory. This means that memory will be leaked
> leading to a potential denial of service.

So I fixed this case in v4, though there is still a tiny chance of a
memleak: if the MAPD(V=0) command fails. We can't free the ITT table
then, really, because it still belongs to the ITS. I don't think we can
do much about it, though.
I free the other allocations of the memory now, anyway.

>> +        }
>> +
>> +        if ( compare_its_guest_devices(temp, guest_doorbell,
>> guest_devid) > 0 )
>> +            new = &((*new)->rb_left);
>> +        else
>> +            new = &((*new)->rb_right);
>> +    }
>> +
>> +    if ( !valid )
>> +        goto out_unlock;
>> +
>> +    ret = -ENOMEM;
>> +
>> +    /* An Interrupt Translation Table needs to be 256-byte aligned. */
>> +    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
>> +    if ( !itt_addr )
>> +        goto out_unlock;
>> +
>> +    dev = xzalloc(struct its_devices);
>> +    if ( !dev )
>> +        goto out_unlock;
>> +
>> +    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
> 
> I don't understand why nr_events - 1. Can you explain?

Xen lacks an ilog2, so "fls" is the closest I could find. "fls" has this
slightly weird semantic (from the Linux source):
"Note fls(0) = 0, fls(1) = 1, fls(0x80000000) = 32."
I think this translates into: "How many bits do I need to express this
number?". For our case the highest event number we need to encode is
"nr_events - 1", hence the subtraction.
So is it worth to introduce a:
	static inline int ilog2_64(uint64_t n) ...
in xen/include/asm-arm/bitops.h to document this?

> 
> [...]
> 
>> +/* Removing any connections a domain had to any ITS in the system. */
>> +void gicv3_its_unmap_all_devices(struct domain *d)
>> +{
>> +    struct rb_node *victim;
>> +    struct its_devices *dev;
>> +
>> +    /*
>> +     * This is an easily readable, yet inefficient implementation.
>> +     * It uses the provided iteration wrapper and erases each node,
>> which
>> +     * possibly triggers rebalancing.
>> +     * This seems overkill since we are going to abolish the whole
>> tree, but
>> +     * avoids an open-coded re-implementation of the traversal
>> functions with
>> +     * some recursive function calls.
>> +     * Performance does not matter here, since we are destroying a
>> domain.
> 
> Again, this is slightly untrue. Performance matter when destroying a
> domain as Xen cannot be preempted. So if it takes too long, you will
> have an impact on the overall system.

I reworded this sentence in v3, since you apparently misunderstood me.
By inefficient I meant sub-optimal, but this is not a _critical_ path,
so we don't care too much. The execution time is clearly bounded by the
number of devices. We simply shouldn't allow gazillion of devices on a
DomU if we care about those things.

> However, I think it would be fair to assume that all device will be
> deassigned before the ITS is destroyed. So I would just drop this
> function. Note that we have the same assumption in the SMMU driver.

Well, why not keep it as a safeguard, then? If a domain gets destroyed,
aren't we supposed to clean up?

>> +     */
>> +restart:
>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>> +    if ( (victim = rb_first(&d->arch.vgic.its_devices)) )
>> +    {
>> +        dev = rb_entry(victim, struct its_devices, rbnode);
>> +        rb_erase(victim, &d->arch.vgic.its_devices);
>> +
>> +        spin_unlock(&d->arch.vgic.its_devices_lock);
>> +
>> +        remove_mapped_guest_device(dev);
>> +
>> +        goto restart;
>> +    }
>> +
>> +    spin_unlock(&d->arch.vgic.its_devices_lock);
>> +}
>> +
>>  /* Scan the DT for any ITS nodes and create a list of host ITSes out
>> of it. */
>>  void gicv3_its_dt_init(const struct dt_device_node *node)
>>  {
>> @@ -455,6 +661,7 @@ void gicv3_its_dt_init(const struct dt_device_node
>> *node)
>>          its_data->addr = addr;
>>          its_data->size = size;
>>          its_data->dt_node = its;
>> +        spin_lock_init(&its_data->cmd_lock);
> 
> This should be in patch #5.

Done in v4.

>>
>>          printk("GICv3: Found ITS @0x%lx\n", addr);
>>
>> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
>> index d61479d..1fadb00 100644
>> --- a/xen/arch/arm/vgic-v3.c
>> +++ b/xen/arch/arm/vgic-v3.c
>> @@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
>>      d->arch.vgic.nr_regions = rdist_count;
>>      d->arch.vgic.rdist_regions = rdist_regions;
>>
>> +    spin_lock_init(&d->arch.vgic.its_devices_lock);
>> +    d->arch.vgic.its_devices = RB_ROOT;
> 
> Again, the placement of those 2 lines are likely wrong. This should
> belong to the vITS and not the vgic-v3.

Well, it's a "domain which has an ITS" thing, as this is not per ITS. So
far we only have an per-ITS init function, as we don't iterate over the
host ITSes there anymore.
So I refrained from adding a separate function to initialize these
simple and generic data structures. Please let me know if you insist on
some ITS-per-domain init function.

Cheers,
Andre.

> I think it would make sense to get a patch that introduces a skeleton
> for the vITS before this patch and start plumbing through.
> 
>> +
>>      /*
>>       * Domain 0 gets the hardware address.
>>       * Guests get the virtual platform layout.
> 
> Cheers,
> 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array
  2017-04-03 19:30     ` Andre Przywara
@ 2017-04-03 20:13       ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-04-03 20:13 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

On 04/03/2017 08:30 PM, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 23/03/17 19:08, Julien Grall wrote:
>>>  /*
>>> + * On the host ITS @its, map @nr_events consecutive LPIs.
>>> + * The mapping connects a device @devid and event @eventid pair to
>>> LPI @lpi,
>>> + * increasing both @eventid and @lpi to cover the number of requested
>>> LPIs.
>>> + */
>>> +int gicv3_its_map_host_events(struct host_its *its,
>>> +                              uint32_t devid, uint32_t eventid,
>>> uint32_t lpi,
>>> +                              uint32_t nr_events)
>>> +{
>>> +    uint32_t i;
>>> +    int ret;
>>> +
>>> +    for ( i = 0; i < nr_events; i++ )
>>> +    {
>>> +        /* For now we map every host LPI to host CPU 0 */
>>> +        ret = its_send_cmd_mapti(its, devid, eventid + i, lpi + i, 0);
>>> +        if ( ret )
>>> +            return ret;
>>> +        ret = its_send_cmd_inv(its, devid, eventid + i);
>>
>>
>> So the spec allows up to 32KB event per device. As all the LPIs will be
>> routed to CPU0 (e.g collection 0), it would be more efficient to do an
>> INVALL. I would be happy to defer that to post Xen 4.9, but the rest
>> needs to be fixed.
>
> I tried INVALL and it didn't work, at least on the model. I can't see
> why, so I kept the individual INVs in.
> I have the patch still lying around, so we can revisit this later.

Can you add a TODO comment please?

>>>
>>>      hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>>>      if ( !hw_its )
>>> @@ -574,6 +639,11 @@ int gicv3_its_map_guest_device(struct domain *d,
>>>      if ( !dev )
>>>          goto out_unlock;
>>>
>>> +    dev->host_lpis = xzalloc_array(uint32_t,
>>> +                                   DIV_ROUND_UP(nr_events, LPI_BLOCK));
>>
>> Rather than having DIV_ROUND_UP spread everywhere. Would not be easier
>> to round_up nr_events once for all?
>
> I'd rather keep nr_events as the actual number around.
> I think we might look into actually limiting the allocation later.

Why? This number will likely be a multiple of bit because of the ITS 
works. You would also have to keep around multiple different value that 
will make the code more complicate to read...


>>> +/* Must be called with host_lpis_lock held. */
>>
>> Again, this is a call for adding an ASSERT in the function.
>
> This comment is more like lock documentation, to give code writers a
> guidance how the locking should be handled here.
> I am not convinced that having ASSERTS in *static* functions is really
> useful.

Well, you seem to assume that you will be the only one to modify this 
code and it is very easy to skip reading a comment by mistake.

So the ASSERT will catch such error. Give me a reason that the ASSERT is 
a bad idea and I will re-think my position.

[...]

>> The algo does not seem to have changed since the previous version.
>> Looking at it again, my understanding is you will always try to allocate
>> forward. So if the current chunk is full, you will allocate the next one
>> rather than looking whether a previous chunk has space available. This
>> will result to allocate more memory than necessary.
>
> In v4 I amended the code slightly to move next_lpi outside of the
> function. When we now free an LPI block, we check if the previous
> next_lpi was pointing after this block and adjust it in this case.
>
>> Similarly unused chunk could be freed to save memory.
>
> No, we can't, because this breaks the lockless property. A user
> (incoming LPI) would find the first level pointer and go into that
> block. So we can't never replace this block pointer now. That smells
> like a case for RCU, but I am not sure if Xen can properly handle this case.
> But with the above optimization (adjusting next_lpi on freeing a block)
> I am pretty sure this isn't a problem for now, especially for Dom0.

Then document it, because this is a call to forget to revisit that in 
the future.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-04-03 20:08     ` Andre Przywara
@ 2017-04-03 20:41       ` Julien Grall
  2017-04-04  9:57         ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-04-03 20:41 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari



On 04/03/2017 09:08 PM, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 22/03/17 17:29, Julien Grall wrote:
>>> +int gicv3_its_map_guest_device(struct domain *d,
>>> +                               paddr_t host_doorbell, uint32_t
>>> host_devid,
>>> +                               paddr_t guest_doorbell, uint32_t
>>> guest_devid,
>>> +                               uint32_t nr_events, bool valid)
>>> +{
>>> +    void *itt_addr = NULL;
>>> +    struct host_its *hw_its;
>>> +    struct its_devices *dev = NULL, *temp;
>>> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent
>>> = NULL;
>>> +    int ret = -ENOENT;
>>> +
>>> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>>> +    if ( !hw_its )
>>> +        return ret;
>>> +
>>> +    /* check for already existing mappings */
>>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>>> +    while ( *new )
>>> +    {
>>> +        temp = rb_entry(*new, struct its_devices, rbnode);
>>> +
>>> +        parent = *new;
>>> +        if ( !compare_its_guest_devices(temp, guest_doorbell,
>>> guest_devid) )
>>> +        {
>>> +            if ( !valid )
>>> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
>>> +
>>> +            spin_unlock(&d->arch.vgic.its_devices_lock);
>>> +
>>> +            if ( valid )
>>
>> Again, a printk(XENLOG_GUEST...) here would be useful to know which host
>> DeviceID was associated to the guest DeviceID.
>
> I added a gprintk(XENLOG_DEBUG, ), which I think is more appropriate (as
> it may spam the console when some stupid guest is running). Let me know
> if you want to have a different loglevel.

I don't think this is more appropriate. gprintk will print the domain ID 
of the current domain, whilst this function will be called by the 
toolstack in the future.

Furthemore, if you look at the implementation of gprintk you will notice 
that it is basically turning into printk(XENLOG_GUEST...) and adding 
information of the current vCPU.

What matters for ratelimiting is XENLOG_GUEST.

>
>>> +                return -EBUSY;
>>> +
>>> +            return remove_mapped_guest_device(temp);
>>
>> Just above you removed the device from the RB-tree but this function may
>> fail and never free the memory. This means that memory will be leaked
>> leading to a potential denial of service.
>
> So I fixed this case in v4, though there is still a tiny chance of a
> memleak: if the MAPD(V=0) command fails. We can't free the ITT table
> then, really, because it still belongs to the ITS. I don't think we can
> do much about it, though.

This is a leak and even tiny is quite worrying. How do you plan to 
address this in the future? What is the best thing to do?

> I free the other allocations of the memory now, anyway.
>
>>> +        }
>>> +
>>> +        if ( compare_its_guest_devices(temp, guest_doorbell,
>>> guest_devid) > 0 )
>>> +            new = &((*new)->rb_left);
>>> +        else
>>> +            new = &((*new)->rb_right);
>>> +    }
>>> +
>>> +    if ( !valid )
>>> +        goto out_unlock;
>>> +
>>> +    ret = -ENOMEM;
>>> +
>>> +    /* An Interrupt Translation Table needs to be 256-byte aligned. */
>>> +    itt_addr = _xzalloc(nr_events * hw_its->itte_size, 256);
>>> +    if ( !itt_addr )
>>> +        goto out_unlock;
>>> +
>>> +    dev = xzalloc(struct its_devices);
>>> +    if ( !dev )
>>> +        goto out_unlock;
>>> +
>>> +    ret = its_send_cmd_mapd(hw_its, host_devid, fls(nr_events - 1) - 1,
>>
>> I don't understand why nr_events - 1. Can you explain?
>
> Xen lacks an ilog2, so "fls" is the closest I could find. "fls" has this
> slightly weird semantic (from the Linux source):
> "Note fls(0) = 0, fls(1) = 1, fls(0x80000000) = 32."
> I think this translates into: "How many bits do I need to express this
> number?". For our case the highest event number we need to encode is
> "nr_events - 1", hence the subtraction.
> So is it worth to introduce a:
> 	static inline int ilog2_64(uint64_t n) ...
> in xen/include/asm-arm/bitops.h to document this?

This might make easier to read the code.

>>
>> [...]
>>
>>> +/* Removing any connections a domain had to any ITS in the system. */
>>> +void gicv3_its_unmap_all_devices(struct domain *d)
>>> +{
>>> +    struct rb_node *victim;
>>> +    struct its_devices *dev;
>>> +
>>> +    /*
>>> +     * This is an easily readable, yet inefficient implementation.
>>> +     * It uses the provided iteration wrapper and erases each node,
>>> which
>>> +     * possibly triggers rebalancing.
>>> +     * This seems overkill since we are going to abolish the whole
>>> tree, but
>>> +     * avoids an open-coded re-implementation of the traversal
>>> functions with
>>> +     * some recursive function calls.
>>> +     * Performance does not matter here, since we are destroying a
>>> domain.
>>
>> Again, this is slightly untrue. Performance matter when destroying a
>> domain as Xen cannot be preempted. So if it takes too long, you will
>> have an impact on the overall system.
>
> I reworded this sentence in v3, since you apparently misunderstood me.
> By inefficient I meant sub-optimal, but this is not a _critical_ path,
> so we don't care too much. The execution time is clearly bounded by the
> number of devices. We simply shouldn't allow gazillion of devices on a
> DomU if we care about those things.

This is a very naive way of thinking how domain destruction is working 
on Xen. Domain destruction is likely to get time, you have to release 
all the resources (memory, devices...). So we usually split in smaller 
pieces to be able to handle interrupts or allowing preemption.

>
>> However, I think it would be fair to assume that all device will be
>> deassigned before the ITS is destroyed. So I would just drop this
>> function. Note that we have the same assumption in the SMMU driver.
>
> Well, why not keep it as a safeguard, then? If a domain gets destroyed,
> aren't we supposed to clean up?

See above. We use ASSERT(....) in the SMMU code to check this assumption.

>>>
>>>          printk("GICv3: Found ITS @0x%lx\n", addr);
>>>
>>> diff --git a/xen/arch/arm/vgic-v3.c b/xen/arch/arm/vgic-v3.c
>>> index d61479d..1fadb00 100644
>>> --- a/xen/arch/arm/vgic-v3.c
>>> +++ b/xen/arch/arm/vgic-v3.c
>>> @@ -1450,6 +1450,9 @@ static int vgic_v3_domain_init(struct domain *d)
>>>      d->arch.vgic.nr_regions = rdist_count;
>>>      d->arch.vgic.rdist_regions = rdist_regions;
>>>
>>> +    spin_lock_init(&d->arch.vgic.its_devices_lock);
>>> +    d->arch.vgic.its_devices = RB_ROOT;
>>
>> Again, the placement of those 2 lines are likely wrong. This should
>> belong to the vITS and not the vgic-v3.
>
> Well, it's a "domain which has an ITS" thing, as this is not per ITS. So
> far we only have an per-ITS init function, as we don't iterate over the
> host ITSes there anymore.
> So I refrained from adding a separate function to initialize these
> simple and generic data structures. Please let me know if you insist on
> some ITS-per-domain init function.

Where is the limit to refrain from creating function then? What matter 
is logic...

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping
  2017-04-03 20:41       ` Julien Grall
@ 2017-04-04  9:57         ` Andre Przywara
  0 siblings, 0 replies; 119+ messages in thread
From: Andre Przywara @ 2017-04-04  9:57 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 03/04/17 21:41, Julien Grall wrote:
> 
> 
> On 04/03/2017 09:08 PM, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 22/03/17 17:29, Julien Grall wrote:
>>>> +int gicv3_its_map_guest_device(struct domain *d,
>>>> +                               paddr_t host_doorbell, uint32_t
>>>> host_devid,
>>>> +                               paddr_t guest_doorbell, uint32_t
>>>> guest_devid,
>>>> +                               uint32_t nr_events, bool valid)
>>>> +{
>>>> +    void *itt_addr = NULL;
>>>> +    struct host_its *hw_its;
>>>> +    struct its_devices *dev = NULL, *temp;
>>>> +    struct rb_node **new = &d->arch.vgic.its_devices.rb_node, *parent
>>>> = NULL;
>>>> +    int ret = -ENOENT;
>>>> +
>>>> +    hw_its = gicv3_its_find_by_doorbell(host_doorbell);
>>>> +    if ( !hw_its )
>>>> +        return ret;
>>>> +
>>>> +    /* check for already existing mappings */
>>>> +    spin_lock(&d->arch.vgic.its_devices_lock);
>>>> +    while ( *new )
>>>> +    {
>>>> +        temp = rb_entry(*new, struct its_devices, rbnode);
>>>> +
>>>> +        parent = *new;
>>>> +        if ( !compare_its_guest_devices(temp, guest_doorbell,
>>>> guest_devid) )
>>>> +        {
>>>> +            if ( !valid )
>>>> +                rb_erase(&temp->rbnode, &d->arch.vgic.its_devices);
>>>> +
>>>> +            spin_unlock(&d->arch.vgic.its_devices_lock);
>>>> +
>>>> +            if ( valid )
>>>
>>> Again, a printk(XENLOG_GUEST...) here would be useful to know which host
>>> DeviceID was associated to the guest DeviceID.
>>
>> I added a gprintk(XENLOG_DEBUG, ), which I think is more appropriate (as
>> it may spam the console when some stupid guest is running). Let me know
>> if you want to have a different loglevel.
> 
> I don't think this is more appropriate. gprintk will print the domain ID
> of the current domain, whilst this function will be called by the
> toolstack in the future.
> 
> Furthemore, if you look at the implementation of gprintk you will notice
> that it is basically turning into printk(XENLOG_GUEST...) and adding
> information of the current vCPU.
> 
> What matters for ratelimiting is XENLOG_GUEST.
> 
>>
>>>> +                return -EBUSY;
>>>> +
>>>> +            return remove_mapped_guest_device(temp);
>>>
>>> Just above you removed the device from the RB-tree but this function may
>>> fail and never free the memory. This means that memory will be leaked
>>> leading to a potential denial of service.
>>
>> So I fixed this case in v4, though there is still a tiny chance of a
>> memleak: if the MAPD(V=0) command fails. We can't free the ITT table
>> then, really, because it still belongs to the ITS. I don't think we can
>> do much about it, though.
> 
> This is a leak and even tiny is quite worrying. How do you plan to
> address this in the future? What is the best thing to do?

In an ideal world we would probably have a spillover queue for ITS
commands. Whenever a command can't be queued to the hardware ITS queue
immediately, the guest should be put to wait somehow and the commands
recorded, to be executed whenever the hardware command queue gets free
slots again.
However I think we can't really do this today, because we don't have a
good way of putting a guest to sleep when it trapped on an MMIO access.
When in the future device assignment / de-assignment is handled via a
hypercall, we can probably address this more easily.

As for the likeliness: I think this is extremely rare. With our large
command queue and the ITS running at multiple hundred MHz I don't think
we will ever run into this situation, really, especially not with just Dom0.

So for now are we OK with this comment noting the corner case? Maybe a TODO?

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests
  2017-04-03 14:18     ` Andre Przywara
@ 2017-04-04 11:49       ` Julien Grall
  2017-04-04 12:51         ` Andre Przywara
  0 siblings, 1 reply; 119+ messages in thread
From: Julien Grall @ 2017-04-04 11:49 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari



On 03/04/17 15:18, Andre Przywara wrote:
> Hi,

Hi Andre,

> On 24/03/17 12:03, Julien Grall wrote:
>>> +    /* We may have mapped more host LPIs than the guest actually
>>> asked for. */
>>
>> Another way, is the interrupt has been received at the same time the
>> guest is configuring it. What will happen if the interrupt is lost?
>
> I don't see how this would be our problem. If the guest is still about
> to configure an LPI, then any incoming one is pretty clearly a spurious
> interrupt.
> I take it that you mean "mapping an LPI using ITS commands" when you say
> "configuring" here. If this is not what you mean, please correct me.
>
> I'd expect a guest to first map the LPI, then enable it and send an INV
> command. Doing that differently will not result in any sane behavior (as
> in: a guest cannot expect an LPI to come through), so nothing we need to
> worry about (from a lost IRQ perspective).
>
> Or what do I miss here?

All host LPIs will be enabled when a device is assigned to a guest. So 
technically an LPI can come at anytime before the guest is configuring 
the virtual mapping.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests
  2017-04-04 12:51         ` Andre Przywara
@ 2017-04-04 12:50           ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-04-04 12:50 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

On 04/04/17 13:51, Andre Przywara wrote:
> On 04/04/17 12:49, Julien Grall wrote:
>>
>>
>> On 03/04/17 15:18, Andre Przywara wrote:
>>> Hi,
>>
>> Hi Andre,
>>
>>> On 24/03/17 12:03, Julien Grall wrote:
>>>>> +    /* We may have mapped more host LPIs than the guest actually
>>>>> asked for. */
>>>>
>>>> Another way, is the interrupt has been received at the same time the
>>>> guest is configuring it. What will happen if the interrupt is lost?
>>>
>>> I don't see how this would be our problem. If the guest is still about
>>> to configure an LPI, then any incoming one is pretty clearly a spurious
>>> interrupt.
>>> I take it that you mean "mapping an LPI using ITS commands" when you say
>>> "configuring" here. If this is not what you mean, please correct me.
>>>
>>> I'd expect a guest to first map the LPI, then enable it and send an INV
>>> command. Doing that differently will not result in any sane behavior (as
>>> in: a guest cannot expect an LPI to come through), so nothing we need to
>>> worry about (from a lost IRQ perspective).
>>>
>>> Or what do I miss here?
>>
>> All host LPIs will be enabled when a device is assigned to a guest. So
>> technically an LPI can come at anytime before the guest is configuring
>> the virtual mapping.
>
> As I said: what is the problem? Any LPI coming in without being mapped
> by a guest would be spurious from a guest point of view, as in not
> expected and ignored. But a guest will never receive it, since we
> explicitly check for that and on the host side and bail out early. From
> do_LPI():
>     /* Unmapped events are marked with an invalid LPI ID. */
>     if ( hlpi.virt_lpi == INVALID_LPI )
>         return;
>
> And we (atomically) update the entry once the guest has configured it,
> so either it's ignored or delivered. And this is a benign race you have
> on real hardware too.

The document it... if I ask a questions multiple time it is likely the 
code is not clear enough.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests
  2017-04-04 11:49       ` Julien Grall
@ 2017-04-04 12:51         ` Andre Przywara
  2017-04-04 12:50           ` Julien Grall
  0 siblings, 1 reply; 119+ messages in thread
From: Andre Przywara @ 2017-04-04 12:51 UTC (permalink / raw)
  To: Julien Grall, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi,

On 04/04/17 12:49, Julien Grall wrote:
> 
> 
> On 03/04/17 15:18, Andre Przywara wrote:
>> Hi,
> 
> Hi Andre,
> 
>> On 24/03/17 12:03, Julien Grall wrote:
>>>> +    /* We may have mapped more host LPIs than the guest actually
>>>> asked for. */
>>>
>>> Another way, is the interrupt has been received at the same time the
>>> guest is configuring it. What will happen if the interrupt is lost?
>>
>> I don't see how this would be our problem. If the guest is still about
>> to configure an LPI, then any incoming one is pretty clearly a spurious
>> interrupt.
>> I take it that you mean "mapping an LPI using ITS commands" when you say
>> "configuring" here. If this is not what you mean, please correct me.
>>
>> I'd expect a guest to first map the LPI, then enable it and send an INV
>> command. Doing that differently will not result in any sane behavior (as
>> in: a guest cannot expect an LPI to come through), so nothing we need to
>> worry about (from a lost IRQ perspective).
>>
>> Or what do I miss here?
> 
> All host LPIs will be enabled when a device is assigned to a guest. So
> technically an LPI can come at anytime before the guest is configuring
> the virtual mapping.

As I said: what is the problem? Any LPI coming in without being mapped
by a guest would be spurious from a guest point of view, as in not
expected and ignored. But a guest will never receive it, since we
explicitly check for that and on the host side and bail out early. From
do_LPI():
    /* Unmapped events are marked with an invalid LPI ID. */
    if ( hlpi.virt_lpi == INVALID_LPI )
        return;

And we (atomically) update the entry once the guest has configured it,
so either it's ignored or delivered. And this is a benign race you have
on real hardware too.

Cheers,
Andre.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

* Re: [PATCH v2 15/27] ARM: vITS: introduce translation table walks
  2017-04-03 18:25     ` Andre Przywara
@ 2017-04-04 15:59       ` Julien Grall
  0 siblings, 0 replies; 119+ messages in thread
From: Julien Grall @ 2017-04-04 15:59 UTC (permalink / raw)
  To: Andre Przywara, Stefano Stabellini
  Cc: xen-devel, Shanker Donthineni, Vijay Kilari

Hi Andre,

On 03/04/17 19:25, Andre Przywara wrote:
> On 24/03/17 13:00, Julien Grall wrote:
>> On 03/16/2017 11:20 AM, Andre Przywara wrote:
>>> + */
>>> +bool read_itte(struct virt_its *its, uint32_t devid, uint32_t evid,
>>> +               struct vcpu **vcpu, uint32_t *vlpi)
>>> +{
>>> +    struct vits_itte *itte;
>>> +    int collid;
>>> +    uint32_t _vlpi;
>>> +    struct vcpu *_vcpu;
>>> +
>>> +    spin_lock(&its->its_lock);
>>
>> Do we expect multiple vCPU calling read_itte at the same time? This
>> needs to be clarify in the comments because this current function is not
>> scalable.
>
> We need this lock here because this protects our data structures.
> A guest locks accesses to the ITS command queue anyway (otherwrite
> multiple VCPUs would race for CWRITER and CREADR), so I don't see a real
> problem. And this lock is pre ITS, not per guest.

Fair point, please ignore my request then.

Cheers,

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel

^ permalink raw reply	[flat|nested] 119+ messages in thread

end of thread, other threads:[~2017-04-04 15:59 UTC | newest]

Thread overview: 119+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-03-16 11:20 [PATCH v2 00/27] arm64: Dom0 ITS emulation Andre Przywara
2017-03-16 11:20 ` [PATCH v2 01/27] ARM: GICv3 ITS: parse and store ITS subnodes from hardware DT Andre Przywara
2017-03-21 20:17   ` Julien Grall
2017-03-23 10:57     ` Andre Przywara
2017-03-23 17:32       ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 02/27] ARM: GICv3: allocate LPI pending and property table Andre Przywara
2017-03-21 21:23   ` Julien Grall
2017-03-23 14:40     ` Andre Przywara
2017-03-23 17:42       ` Julien Grall
2017-03-23 17:45         ` Stefano Stabellini
2017-03-23 17:49           ` Julien Grall
2017-03-23 18:01             ` Stefano Stabellini
2017-03-23 18:21               ` Andre Przywara
2017-03-24 11:45                 ` Julien Grall
2017-03-24 17:22                   ` Stefano Stabellini
2017-03-21 22:57   ` Stefano Stabellini
2017-03-21 23:08     ` André Przywara
2017-03-21 23:27       ` Stefano Stabellini
2017-03-23 10:50         ` Andre Przywara
2017-03-23 17:47           ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 03/27] ARM: GICv3 ITS: allocate device and collection table Andre Przywara
2017-03-21 23:29   ` Stefano Stabellini
2017-03-22 13:52   ` Julien Grall
2017-03-22 16:08     ` André Przywara
2017-03-22 16:33       ` Julien Grall
2017-03-29 13:58         ` Andre Przywara
2017-03-16 11:20 ` [PATCH v2 04/27] ARM: GICv3 ITS: map ITS command buffer Andre Przywara
2017-03-21 23:48   ` Stefano Stabellini
2017-03-22 15:23   ` Julien Grall
2017-03-22 16:31     ` André Przywara
2017-03-22 16:41       ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 05/27] ARM: GICv3 ITS: introduce ITS command handling Andre Przywara
2017-03-16 15:05   ` Shanker Donthineni
2017-03-16 15:18     ` Andre Przywara
2017-03-22  0:02   ` Stefano Stabellini
2017-03-22 15:59   ` Julien Grall
2017-04-03 10:58     ` Andre Przywara
2017-04-03 11:23       ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 06/27] ARM: GICv3 ITS: introduce device mapping Andre Przywara
2017-03-22 17:29   ` Julien Grall
2017-04-03 20:08     ` Andre Przywara
2017-04-03 20:41       ` Julien Grall
2017-04-04  9:57         ` Andre Przywara
2017-03-22 22:45   ` Stefano Stabellini
2017-04-03 19:45     ` Andre Przywara
2017-03-30 11:17   ` Vijay Kilari
2017-03-16 11:20 ` [PATCH v2 07/27] ARM: arm64: activate atomic 64-bit accessors Andre Przywara
2017-03-22 17:30   ` Julien Grall
2017-03-22 22:49     ` Stefano Stabellini
2017-03-16 11:20 ` [PATCH v2 08/27] ARM: GICv3 ITS: introduce host LPI array Andre Przywara
2017-03-22 23:38   ` Stefano Stabellini
2017-03-23  8:48     ` Julien Grall
2017-03-23 10:21     ` Andre Przywara
2017-03-23 17:52       ` Stefano Stabellini
2017-03-24 11:54         ` Julien Grall
2017-03-23 19:08   ` Julien Grall
2017-04-03 19:30     ` Andre Przywara
2017-04-03 20:13       ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 09/27] ARM: GICv3: introduce separate pending_irq structs for LPIs Andre Przywara
2017-03-22 23:44   ` Stefano Stabellini
2017-03-23 20:08     ` André Przywara
2017-03-24 10:59       ` Julien Grall
2017-03-24 11:40   ` Julien Grall
2017-03-24 15:50     ` Andre Przywara
2017-03-24 16:19       ` Julien Grall
2017-03-24 17:26       ` Stefano Stabellini
2017-03-27  9:02         ` Andre Przywara
2017-03-27 14:01           ` Julien Grall
2017-03-27 17:44             ` Stefano Stabellini
2017-03-27 17:49               ` Julien Grall
2017-03-27 18:39                 ` Stefano Stabellini
2017-03-27 21:24                   ` Julien Grall
2017-03-28  7:58                   ` Jan Beulich
2017-03-28 13:12                     ` Julien Grall
2017-03-28 13:34                       ` Jan Beulich
2017-03-16 11:20 ` [PATCH v2 10/27] ARM: GICv3: forward pending LPIs to guests Andre Przywara
2017-03-24 12:03   ` Julien Grall
2017-04-03 14:18     ` Andre Przywara
2017-04-04 11:49       ` Julien Grall
2017-04-04 12:51         ` Andre Przywara
2017-04-04 12:50           ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 11/27] ARM: GICv3: enable ITS and LPIs on the host Andre Przywara
2017-03-16 11:20 ` [PATCH v2 12/27] ARM: vGICv3: handle virtual LPI pending and property tables Andre Przywara
2017-03-24 12:09   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 13/27] ARM: vGICv3: Handle disabled LPIs Andre Przywara
2017-03-24 12:20   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 14/27] ARM: vGICv3: introduce basic ITS emulation bits Andre Przywara
2017-03-16 16:25   ` Shanker Donthineni
2017-03-20 12:17     ` Vijay Kilari
2017-03-24 12:41   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 15/27] ARM: vITS: introduce translation table walks Andre Przywara
2017-03-24 13:00   ` Julien Grall
2017-04-03 18:25     ` Andre Przywara
2017-04-04 15:59       ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 16/27] ARM: vITS: handle CLEAR command Andre Przywara
2017-03-24 14:27   ` Julien Grall
2017-03-24 15:53     ` Andre Przywara
2017-03-24 17:17       ` Stefano Stabellini
2017-03-27  8:44         ` Andre Przywara
2017-03-27 14:12           ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 17/27] ARM: vITS: handle INT command Andre Przywara
2017-03-24 14:38   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 18/27] ARM: vITS: handle MAPC command Andre Przywara
2017-03-16 11:20 ` [PATCH v2 19/27] ARM: vITS: handle MAPD command Andre Przywara
2017-03-24 14:41   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 20/27] ARM: vITS: handle MAPTI command Andre Przywara
2017-03-24 14:54   ` Julien Grall
2017-04-03 18:47     ` Andre Przywara
2017-03-16 11:20 ` [PATCH v2 21/27] ARM: vITS: handle MOVI command Andre Przywara
2017-03-24 15:00   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 22/27] ARM: vITS: handle DISCARD command Andre Przywara
2017-03-16 11:20 ` [PATCH v2 23/27] ARM: vITS: handle INV command Andre Przywara
2017-03-16 11:20 ` [PATCH v2 24/27] ARM: vITS: handle INVALL command Andre Przywara
2017-03-24 15:12   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 25/27] ARM: vITS: create and initialize virtual ITSes for Dom0 Andre Przywara
2017-03-24 15:18   ` Julien Grall
2017-03-16 11:20 ` [PATCH v2 26/27] ARM: vITS: create ITS subnodes for Dom0 DT Andre Przywara
2017-03-16 11:20 ` [PATCH v2 27/27] ARM: vGIC: advertise LPI support Andre Przywara
2017-03-24 15:25   ` Julien Grall

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.