[Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Brandon Perez]

[-- Attachment #1: Type: text/plain, Size: 2388 bytes --]

Hello All,

     You can find the relevant thread that outlines the issue at [1].

     In short, the issue is as follows. On the TI DRA72 chip, there is a 
piece of hardware called the IRQ crossbar. Due to the large number of 
peripheral devices, not all devices can fit onto the SPI lines on the 
GIC. The crossbar maps peripheral devices to interrupt lines on the GIC.

     The Linux kernel handles this by performing an on-demand mapping. 
Whenever a driver requires an interrupt, it requests it from the kernel. 
The Kernel keeps an internal map of the SPI lines, and checks for the 
next free line. It then sets up the appropriate crossbar register to map 
the given device to the SPI line, and marks that line as no longer free.

     In the device tree, the "interrupts" property no longer contains 
the IRQ line corresponding to the device, but rather the crossbar input 
line corresponding to the device. This causes issues in Xen since it is 
expecting an IRQ line.

     Ideally (in my opinion), the solution would be to mark these device 
tree entries (possibly with a different "interrupt-parent" property), so 
that Xen only maps the memory-mapped registers, and passes the interrupt 
on to Dom0. Dom0 would then handle the on-demand mapping of the devices. 
That way, Xen wouldn't need to repeat some fairly hardware-specific code 
that's already in the Kernel.

     Below I've attached the patches of the changes I have made to Xen 
and the Linux kernel so far. For reference, here is my setup:
     - Hardware: TI DRA72 Chip, Arm Cortex A15
     - Xen:
         - Version: 4.6-unstable
         - Compiled from source, with some local changes
         - Branch: master
         - Commit: ecdae1cfaa7f6123decaa1b9d7205c3ff726b941
         - Repo URL: git://xenbits.xen.org/xen.git
     - Linux Kernel:
         - Version: 3.14
         - Compiled From source, with some local changes
         - Branch: android-3.14-6AL.1.0
         - Commit: 7b2f1133857414b96927c06f08ed6c440f5472e7
         - Repo URL: git://git.omapzoom.org/kernel/omap.git

     I have additional patches for U-Boot which I can provide if needed, 
but they aren't directly relevant to this issue. I also have patches for 
a static crossbar mapping with Xen, which is the temporary work-around 
I'm currently working with.

[1] http://www.gossamer-threads.com/lists/xen/users/389509

Brandon Perez


[-- Attachment #2: xen_0001_io_virt_remap.patch --]
[-- Type: text/x-patch, Size: 3291 bytes --]

>From f2bf190255c8f872d15063d7f8a6382c279e312d Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 17:56:49 -0400
Subject: [PATCH 1/3] This patch adds in IO mappings for several devices which
 are not explicitly laid out in the device tree, and
 mappings for DRA72x specific regions that are not
 devices.

In the ARM virtualization setup, the virtual memory uses a 2-stage
translation. The guest OS VM performs a virtual address translation to
an intermediate physical address, which is still not a true physical
memory address. The Xen hypervisor VM then translates this IPA to an
actual physical address.

Thus, in order for a guest OS (even Dom0) to access a physical address,
Xen must explicitly setup a mapping in its VM for the guest OS to access
this location. Several devices were missing from the device tree, so Xen
was unware of the need to map their MMIO registers. Thus, the platform
has specific_mapping() function, which patches up these holes. The OMAP5
specific mapping function is missing a few items that DRA72x chips need,
so these were added in.
---
 xen/arch/arm/platforms/omap5.c        |   27 +++++++++++++++++++++++++++
 xen/include/asm-arm/platforms/omap5.h |    3 +++
 2 files changed, 30 insertions(+)

diff --git a/xen/arch/arm/platforms/omap5.c b/xen/arch/arm/platforms/omap5.c
index e7bf30d..3c6495a 100644
--- a/xen/arch/arm/platforms/omap5.c
+++ b/xen/arch/arm/platforms/omap5.c
@@ -120,6 +120,32 @@ static int omap5_specific_mapping(struct domain *d)
     return 0;
 }
 
+/* Additional mappings for dom0 (not in the DTS) */
+static int dra7_specific_mapping(struct domain *d)
+{
+    /* Map the PRM module */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_PRM_BASE), 2,
+                     paddr_to_pfn(OMAP5_PRM_BASE));
+
+    /* Map the PRM_MPU */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_PRCM_MPU_BASE), 1,
+                     paddr_to_pfn(OMAP5_PRCM_MPU_BASE));
+
+    /* Map the Wakeup Gen */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_WKUPGEN_BASE), 1,
+                     paddr_to_pfn(OMAP5_WKUPGEN_BASE));
+
+    /* Map the on-chip SRAM */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_SRAM_PA), 32,
+                     paddr_to_pfn(OMAP5_SRAM_PA));
+
+    /* Map GPMC address space for NAND flash. */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_GPMC_PA), 65536,
+                     paddr_to_pfn(OMAP5_GPMC_PA));
+
+    return 0;
+}
+
 static int __init omap5_smp_init(void)
 {
     void __iomem *wugen_base;
@@ -171,6 +197,7 @@ PLATFORM_START(dra7, "TI DRA7")
     .init_time = omap5_init_time,
     .cpu_up = cpu_up_send_sgi,
     .smp_init = omap5_smp_init,
+    .specific_mapping = dra7_specific_mapping,
 
     .dom0_gnttab_start = 0x4b000000,
     .dom0_gnttab_size = 0x20000,
diff --git a/xen/include/asm-arm/platforms/omap5.h b/xen/include/asm-arm/platforms/omap5.h
index c559c84..d87e7d2 100644
--- a/xen/include/asm-arm/platforms/omap5.h
+++ b/xen/include/asm-arm/platforms/omap5.h
@@ -20,6 +20,9 @@
 #define OMAP_AUX_CORE_BOOT_0_OFFSET             0x800
 #define OMAP_AUX_CORE_BOOT_1_OFFSET             0x804
 
+#define OMAP5_GPMC_PA                           0x01000000
+#define OMAP5_TILER_PA                          0x60000000
+
 #endif /* __ASM_ARM_PLATFORMS_OMAP5_H */
 
 /*
-- 
1.7.9.5


[-- Attachment #3: xen_0002_smc_call_passthrough.patch --]
[-- Type: text/x-patch, Size: 2125 bytes --]

>From e53fdc1ea750dd3143e2d7cd62a5d38eb446afde Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 17:58:24 -0400
Subject: [PATCH 2/3] This patch enables pass-through SMC call support for
 guest OS's. By default, Xen does not allow for guests
 to make SMC calls. However, on the DRA72x chips, the
 kernel needs to make several SMC calls to interact with
 the secure ROM code.

This patch contains two options for solving this. The selected method is
to simply allow the guest to make the SMC call, so it passes through the
hypervisor without being trapped. The other method, commented out, it to
trap and emulate the call. For now, both are committed until it is
decided which is more appropiate.
---
 xen/arch/arm/traps.c |   15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 258d4c5..9b9de7b 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -123,8 +123,9 @@ void __cpuinit init_traps(void)
                  CPTR_EL2);
 
     /* Setup hypervisor traps */
+    // TODO: Choose method
     WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
-                 HCR_TWE|HCR_TWI|HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP, HCR_EL2);
+                 HCR_TWE|HCR_TWI/*|HCR_TSC*/|HCR_TAC|HCR_SWIO|HCR_TIDCP, HCR_EL2);
     isb();
 }
 
@@ -2494,6 +2495,18 @@ asmlinkage void do_trap_hypervisor(struct cpu_user_regs *regs)
         GUEST_BUG_ON(!psr_mode_is_32bit(regs->cpsr));
         perfc_incr(trap_smc32);
         inject_undef32_exception(regs);
+// TODO: Choose method
+/*#define omap5_smc(func_id, arg1) \
+        asm volatile ("push {r1-r12, lr}\n\t" \
+                      "mov r12,%0\n\t" \
+                      "mov r0,%1\n\t" \
+                      "smc #1\n\t" \
+                      "pop {r1-r12, lr}" \
+                      : \
+                      : "r" (func_id), "r" (arg1))
+
+        omap5_smc(regs->r12, regs->r0);
+        advance_pc(regs, hsr); */
         break;
     case HSR_EC_HVC32:
         GUEST_BUG_ON(!psr_mode_is_32bit(regs->cpsr));
-- 
1.7.9.5


[-- Attachment #4: xen_0003_static_xbar_mapping_aware.patch --]
[-- Type: text/x-patch, Size: 2285 bytes --]

>From 03f444d9042b1730bf3d8ff9ab3369e744e5e69b Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 18:04:36 -0400
Subject: [PATCH 3/3] This commit adds in support for Xen to be aware of
 devices being statically mapped in the crossbar.

In general, Xen needs very few interrupts to run. Most of the interrupts
it directly uses (and therefore owns) are timer interrupts, which are
internal to the processor (PPIs), and therefore do not involve the
interrupts crossbar. The rest of the interrupts (SPIs) are simply passed
onto the Dom0 kernel, who handles them appropiately. Xen simply
internally tracks which interrupts belong to which domain.

There is one notable exception to this: the serial device, which is a
peripheral, and therefore must be routed through the crossbar. In a
setup where the kernel does pseudo-dynamic crossbar mapping, Xen must
still own this serial interrupt, and it therefore must be statically
mapped.

The interrupt property will contain the crossbar input number of the
device, so a new property is added to a device if it is needed by Xen:
"default-mapping", which contains the interrupt number that the device
will correspond to in the mapping setup by the bootloader. This will not
be changed by the Dom0 kernel.
---
 xen/common/device_tree.c |   11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index 31f169b..fc82eb4 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -1036,10 +1036,13 @@ int dt_device_get_raw_irq(const struct dt_device_node *device,
     dt_dprintk("dt_device_get_raw_irq: dev=%s, index=%u\n",
                device->full_name, index);
 
-    /* Get the interrupts property */
-    intspec = dt_get_property(device, "interrupts", &intlen);
-    if ( intspec == NULL )
-        return -EINVAL;
+    /* Get the appropiate interrupts property */
+    intspec = dt_get_property(device, "default-mapping", &intlen);
+    if (intspec == NULL) {
+        intspec = dt_get_property(device, "interrupts", &intlen);
+        if (intspec == NULL)
+            return -EINVAL;
+    }
     intlen /= sizeof(*intspec);
 
     dt_dprintk(" intspec=%d intlen=%d\n", be32_to_cpup(intspec), intlen);
-- 
1.7.9.5


[-- Attachment #5: linux_0001_dt_chosen_node.patch --]
[-- Type: text/x-patch, Size: 1332 bytes --]

>From 9b01f0062cd11c9c676e279182ba61524673adb8 Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Tue, 14 Jul 2015 10:56:38 -0400
Subject: [PATCH 1/6] This commit adds in parts of the "chosen" node into the
 device tree, which are not dynamic.

The "chosen" node in a device tree is used to convey dynamic information
to the kernel that is booting (Xen in our case). Normally, this can be
set from the UBoot command line, but for some reason, the UBoot command
parser freezes when it sees an argument like "<1>". Thus, we simply set
these properties in the device tree.
---
 arch/arm/boot/dts/dra72-evm.dts |   11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/arch/arm/boot/dts/dra72-evm.dts b/arch/arm/boot/dts/dra72-evm.dts
index c747dd6..e2d6170 100644
--- a/arch/arm/boot/dts/dra72-evm.dts
+++ b/arch/arm/boot/dts/dra72-evm.dts
@@ -15,6 +15,17 @@
 	model = "TI DRA722";
 	compatible = "ti,dra72-evm", "ti,dra722", "ti,dra72", "ti,dra7";
 
+    chosen {
+        modules {
+            #address-cells = <1>;
+            #size-cells = <1>;
+            module@0 {
+                compatible = "xen,linux-zimage", "xen,multiboot-module";
+                reg = <0xa0000000 0xa00000>;
+            };
+        };
+    };
+
 	aliases {
 		display0 = &hdmi0;
 		sound0 = &primary_sound;
-- 
1.7.9.5


[-- Attachment #6: linux_0002_static_crossbar_support.patch --]
[-- Type: text/x-patch, Size: 3863 bytes --]

>From b81137bc437c969465a070dfbced506d981efd12 Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Tue, 14 Jul 2015 11:07:24 -0400
Subject: [PATCH 2/6] This commit enables the kernel to accept a static
 crossbar mapping from the bootloader, where the mapping
 of peripherals to IRQ lines is handled by the
 bootloader.

Due to the large number of peripheral interrupts on the J6/DRA72x
boards, there is a crossbar to route peripheral devices to SPI lines on
the GIC. Normally, the mapping and configuration of the crossbar
registers (which perform the mapping) is handled in a pseudo-dynamic
manner by the kernel. The kernel maps interrupts from devices as drivers
request the interrupts from the kernel. Naturally, with a given
configuration, the mapping is always the same (and thus static), but
this allows for flexibility with adding and removing devices.

Due to the difficultly of performing this pseudo-dynamic mapping while
the kernel is virtualized by Xen, a static mapping is used. In this case,
the kernel does not attempt to map IRQ's, and simply discovers the crossbar
configuration from the registers. This feature is enabled by
CONFIG_DRA7_CROSSBAR_STATIC_MAPPING. For now, this is simply present in the
irqchip.c file, but will eventually be integrated as a full config option.
---
 arch/arm/boot/dts/dra7.dtsi    |    1 +
 drivers/irqchip/irq-crossbar.c |   36 ++++++++++++++++++++++++------------
 2 files changed, 25 insertions(+), 12 deletions(-)

diff --git a/arch/arm/boot/dts/dra7.dtsi b/arch/arm/boot/dts/dra7.dtsi
index 6b69496..0f99bfa 100644
--- a/arch/arm/boot/dts/dra7.dtsi
+++ b/arch/arm/boot/dts/dra7.dtsi
@@ -384,6 +384,7 @@
 		uart1: serial@4806a000 {
 			compatible = "ti,omap4-uart";
 			reg = <0x4806a000 0x100>;
+			interrupts = <GIC_SPI 67 IRQ_TYPE_LEVEL_HIGH>;
 			interrupts-extended = <&gic GIC_SPI 67 IRQ_TYPE_LEVEL_HIGH>;
 			ti,hwmods = "uart1";
 			clock-frequency = <48000000>;
diff --git a/drivers/irqchip/irq-crossbar.c b/drivers/irqchip/irq-crossbar.c
index 85c2985..4d58650 100644
--- a/drivers/irqchip/irq-crossbar.c
+++ b/drivers/irqchip/irq-crossbar.c
@@ -22,6 +22,9 @@
 #define IRQ_SKIP	-3
 #define GIC_IRQ_START	32
 
+// FIXME:
+#define CONFIG_DRA7_CROSSBAR_STATIC_MAPPING
+
 /**
  * struct crossbar_device - crossbar device description
  * @int_max: maximum number of supported interrupts
@@ -100,10 +103,11 @@ static inline bool needs_crossbar_write(irq_hw_number_t hw)
 static int crossbar_domain_map(struct irq_domain *d, unsigned int irq,
 			       irq_hw_number_t hw)
 {
-	if (needs_crossbar_write(hw))
-		cb->write(hw - GIC_IRQ_START, cb->irq_map[hw - GIC_IRQ_START]);
-
-	return 0;
+#ifndef CONFIG_DRA7_CROSSBAR_STATIC_MAPPING
+    if (needs_crossbar_write(hw))
+        cb->write(hw - GIC_IRQ_START, cb->irq_map[hw - GIC_IRQ_START]);
+#endif
+    return 0;
 }
 
 /**
@@ -284,14 +288,22 @@ static int __init crossbar_of_init(struct device_node *node)
 	}
 
 	of_property_read_u32(node, "ti,irqs-safe-map", &cb->safe_map);
-	/* Initialize the crossbar with safe map to start with */
-	for (i = 0; i < max; i++) {
-		if (cb->irq_map[i] == IRQ_RESERVED ||
-		    cb->irq_map[i] == IRQ_SKIP)
-			continue;
-
-		cb->write(i, cb->safe_map);
-	}
+#ifndef CONFIG_DRA7_CROSSBAR_STATIC_MAPPING
+    /* Initialize the crossbar with safe map to start with */
+    for (i = 0; i < max; i++) {
+        if (cb->irq_map[i] == IRQ_RESERVED || cb->irq_map[i] == IRQ_SKIP)
+            continue;
+
+        cb->write(i, cb->safe_map);
+    }
+#else
+    // Override the mapping with the default mapping given by the bootloader
+    for (i = 0; i < max; i++) {
+        if (cb->irq_map[i] != IRQ_RESERVED) {
+            cb->irq_map[i] = readw(cb->crossbar_base + cb->register_offsets[i]);
+        }
+    }
+#endif
 
 	register_routable_domain_ops(&routable_irq_domain_ops);
 	return 0;
-- 
1.7.9.5


[-- Attachment #7: linux_0003_dt_io_remap_nodes.patch --]
[-- Type: text/x-patch, Size: 2519 bytes --]

>From e6093de127e7efb5335608fc648882adc70ea797 Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Tue, 14 Jul 2015 11:17:06 -0400
Subject: [PATCH 3/6] This commit adds in several devices into the device tree
 which are implicitly used by the DRA72x chip. That is,
 the kernel simply knows about these devices, without
 them being listed in the device tree.

This is a problem, however, when the kernel is being virutalized.
In the ARM virtualization setup, the virtual memory uses a 2-stage
translation. The guest OS VM performs a virtual address translation
to an intermediate physical address, which is still not a true physical
memory address. The Xen hypervisor VM then translates this IPA to an
actual physical address.

Thus, for the kernel to access a physical address Xen (or some other
hypervisor), must map the physical address in its VM map. So, Xen must
know about all the devices that the kernel is going to use so that it
can map its MMIO registers.
---
 arch/arm/boot/dts/dra7.dtsi     |   18 ++++++++++++++++++
 arch/arm/boot/dts/dra72-evm.dts |    8 ++++++++
 2 files changed, 26 insertions(+)

diff --git a/arch/arm/boot/dts/dra7.dtsi b/arch/arm/boot/dts/dra7.dtsi
index 0f99bfa..22c720d 100644
--- a/arch/arm/boot/dts/dra7.dtsi
+++ b/arch/arm/boot/dts/dra7.dtsi
@@ -1979,6 +1979,24 @@
 			#size-cells = <0>;
 			status = "disabled";
 		};
+
+        hdq1w: hdq1w@0x480B2000 {
+            compatible = "ti,hdq1w";
+            ti,hwmods = "hdq1w";
+            reg = <0x480B2000 0x1000>;
+        };
+
+        smartreflex_core: smartreflex_core@0x4A0DD000  {
+            compatible = "ti,smartreflex_core";
+            ti,hwmods = "ti,smartreflex_core";
+            reg = <0x4A0DD000 0x1000>;
+        };
+
+        smartreflex_mpu: smartreflex_mpu@0x4A0D9000 {
+            compatible = "ti,smartreflex_mpu";
+            ti,hwmods = "ti,smartreflex_mpu";
+            reg = <0x4A0D9000 0x1000>;
+        };
 	};
 };
 
diff --git a/arch/arm/boot/dts/dra72-evm.dts b/arch/arm/boot/dts/dra72-evm.dts
index e2d6170..729e176 100644
--- a/arch/arm/boot/dts/dra72-evm.dts
+++ b/arch/arm/boot/dts/dra72-evm.dts
@@ -87,6 +87,14 @@
 		#size-cells = <1>;
 		ranges;
 
+        default_cma: default_cma@0xae000000 {
+			compatible = "shared-dma-pool";
+			reg = <0xae000000 0x1800000>;
+			reusable;
+			status = "okay";
+            linux,cma-default;
+        };
+
 		ipu2_cma_pool: ipu2_cma@95800000 {
 			compatible = "shared-dma-pool";
 			reg = <0x95800000 0x3800000>;
-- 
1.7.9.5


[-- Attachment #8: linux_0004_static_crossbar_kconfig.patch --]
[-- Type: text/x-patch, Size: 2629 bytes --]

>From 49921a37843518435f103c1239143a32302d4cf1 Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Thu, 16 Jul 2015 10:52:31 -0400
Subject: [PATCH 4/6] This commit makes the static crossbar mapping support a
 Kernel configuration option, able to be turned off and
 on. The option can be manually added to the .config
 file, and build with "make olddefconfig", or it can be
 added using the menu configuration.

---
 drivers/irqchip/Kconfig        |    9 +++++++++
 drivers/irqchip/irq-crossbar.c |    7 ++-----
 2 files changed, 11 insertions(+), 5 deletions(-)

diff --git a/drivers/irqchip/Kconfig b/drivers/irqchip/Kconfig
index 1110687..6824853 100644
--- a/drivers/irqchip/Kconfig
+++ b/drivers/irqchip/Kconfig
@@ -77,3 +77,12 @@ config IRQ_CROSSBAR
 	  The primary irqchip invokes the crossbar's callback which inturn allocates
 	  a free irq and configures the IP. Thus the peripheral interrupts are
 	  routed to one of the free irqchip interrupt lines.
+
+config IRQ_CROSSBAR_STATIC_MAPPING
+	bool "IRQ crossbar static mapping support"
+	depends on IRQ_CROSSBAR
+	help
+	  Support for a static CROSSBAR mapping. This means that the kernel accepts
+	  the mapping of peripherals on the crossbar to IRQ lines on the chip, and
+	  not attempt to map peripherals to IRQ lines at runtime. Either the system
+	  default is used, or the mapping is setup by the bootloader.
diff --git a/drivers/irqchip/irq-crossbar.c b/drivers/irqchip/irq-crossbar.c
index 4d58650..cf48296 100644
--- a/drivers/irqchip/irq-crossbar.c
+++ b/drivers/irqchip/irq-crossbar.c
@@ -22,9 +22,6 @@
 #define IRQ_SKIP	-3
 #define GIC_IRQ_START	32
 
-// FIXME:
-#define CONFIG_DRA7_CROSSBAR_STATIC_MAPPING
-
 /**
  * struct crossbar_device - crossbar device description
  * @int_max: maximum number of supported interrupts
@@ -103,7 +100,7 @@ static inline bool needs_crossbar_write(irq_hw_number_t hw)
 static int crossbar_domain_map(struct irq_domain *d, unsigned int irq,
 			       irq_hw_number_t hw)
 {
-#ifndef CONFIG_DRA7_CROSSBAR_STATIC_MAPPING
+#ifndef CONFIG_IRQ_CROSSBAR_STATIC_MAPPING
     if (needs_crossbar_write(hw))
         cb->write(hw - GIC_IRQ_START, cb->irq_map[hw - GIC_IRQ_START]);
 #endif
@@ -288,7 +285,7 @@ static int __init crossbar_of_init(struct device_node *node)
 	}
 
 	of_property_read_u32(node, "ti,irqs-safe-map", &cb->safe_map);
-#ifndef CONFIG_DRA7_CROSSBAR_STATIC_MAPPING
+#ifndef CONFIG_IRQ_CROSSBAR_STATIC_MAPPING
     /* Initialize the crossbar with safe map to start with */
     for (i = 0; i < max; i++) {
         if (cb->irq_map[i] == IRQ_RESERVED || cb->irq_map[i] == IRQ_SKIP)
-- 
1.7.9.5


[-- Attachment #9: linux_0005_uart_static_crossbar_mapping.patch --]
[-- Type: text/x-patch, Size: 1829 bytes --]

>From 683334b1c319590b7815c56520f47b6d4fdfbccd Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 18:58:28 -0400
Subject: [PATCH 5/6] This commit modifies the UART device tree entry to allow
 the serial device to be statically mapped, even in a
 configuration where the kernel pseudo-dynmically maps
 IRQs.

In general, the hypervisor will need exclusive ownership of the serial
interrupt. Since the hypervisor will generally be unaware of the
crossbar, the mapping of the serial device should be static. This is
reflected in the "default-mapping" property of the serial device, which
tells the hypervisor which IRQ line it will be mapped to by the
bootloader.

Also, the kernel must take care not to override this mapping, or else
the serial communication will stop working. Thus, the serial device's
IRQ is also added to the "irq-skip" list in the "crossbar_mpu" device,
so that the pseudo-dynamic mapping of interrupts skips the serial
device's interrupt line.
---
 arch/arm/boot/dts/dra7.dtsi |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/arch/arm/boot/dts/dra7.dtsi b/arch/arm/boot/dts/dra7.dtsi
index 22c720d..4692bb5 100644
--- a/arch/arm/boot/dts/dra7.dtsi
+++ b/arch/arm/boot/dts/dra7.dtsi
@@ -386,6 +386,7 @@
 			reg = <0x4806a000 0x100>;
 			interrupts = <GIC_SPI 67 IRQ_TYPE_LEVEL_HIGH>;
 			interrupts-extended = <&gic GIC_SPI 67 IRQ_TYPE_LEVEL_HIGH>;
+			default-mapping = <GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>;
 			ti,hwmods = "uart1";
 			clock-frequency = <48000000>;
 			status = "disabled";
@@ -1631,7 +1632,7 @@
 			ti,max-crossbar-sources = <MAX_SOURCES>;
 			ti,reg-size = <2>;
 			ti,irqs-reserved = <0 1 2 3 5 6 131 132>;
-			ti,irqs-skip = <10 133 139 140>;
+			ti,irqs-skip = <10 72 133 139 140>;
 			ti,irqs-safe-map = <0>;
 		};
 
-- 
1.7.9.5


[-- Attachment #10: linux_0006_new_interrupt_parent.patch --]
[-- Type: text/x-patch, Size: 1718 bytes --]

>From e1c4bac4a1a72da465584b001c03374106f9f3df Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 19:08:16 -0400
Subject: [PATCH 6/6] This commit fixes a conflict in Xen with the IRQ
 numbers.

Since the UART device defaults to SPI line 72, we added in the
default-mapping property into its device tree entry to inform Xen of its
true interrupt number. However, another device, omap3_dwc_1, has one of
its interrupts on crossbar line 72. Thus, Xen interrupts this as a
conflict between two IRQs, leading to an aborted boot.

To fix this issue, we changed the interrupt parent of omap3_dwc_1 to the
crossbar_mpu device, and updated that device tree node to be a proper
interrupt controller.
---
 arch/arm/boot/dts/dra7.dtsi |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/arm/boot/dts/dra7.dtsi b/arch/arm/boot/dts/dra7.dtsi
index 4692bb5..e0fd64e 100644
--- a/arch/arm/boot/dts/dra7.dtsi
+++ b/arch/arm/boot/dts/dra7.dtsi
@@ -1516,6 +1516,7 @@
 			ti,hwmods = "usb_otg_ss1";
 			reg = <0x48880000 0x10000>;
 			interrupts = <GIC_SPI 72 IRQ_TYPE_LEVEL_HIGH>;
+			interrupt-parent = <&crossbar_mpu>;
 			#address-cells = <1>;
 			#size-cells = <1>;
 			utmi-mode = <2>;
@@ -1529,6 +1530,7 @@
 				interrupt-names = "peripheral",
 						  "host",
 						  "otg";
+				interrupt-parent = <&crossbar_mpu>;
 				phys = <&usb2_phy1>, <&usb3_phy1>;
 				phy-names = "usb2-phy", "usb3-phy";
 				tx-fifo-resize;
@@ -1627,6 +1629,8 @@
 
 		crossbar_mpu: crossbar@4a020000 {
 			compatible = "ti,irq-crossbar";
+			interrupt-controller;
+			#interrupt-cells = <3>;
 			reg = <0x4a002a48 0x130>;
 			ti,max-irqs = <160>;
 			ti,max-crossbar-sources = <MAX_SOURCES>;
-- 
1.7.9.5


[-- Attachment #11: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Julien Grall]

On 21/07/2015 00:17, Brandon Perez wrote:
> Hello All,

Hi Brandon,

We use to send one mail by patch rather than sending them as an 
attachment of a single email. It's easier for reviewing the patches.
You also need to add you Signed-off-by on each patch and CC all the 
relevant maintainers. Please see [1] for all the guidelines to submit a 
patch to Xen.

A couple of comments I about this series:
	- Patch #2: You are allowing any guest to do smc which, unless you 
trust all the guest, is unsecure. There was some discussion about 
different solution to handle SMC back in 2013 [2]. So far I didn't see 
any more update on it. It may be worth to send a separate thread about 
how to handle SMC.
	- Patch #3, I can't find any documentation or implementation of the 
property "default-mapping" in Linux. Can you provide a link about it?

I will comment more when you will resend the patches inline.

Regards,

[1] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches
[2] http://lists.xen.org/archives/html/xen-devel/2013-07/msg02779.html

-- 
Julien Grall

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Brandon Perez]

On 07/21/2015 05:41 AM, Julien Grall wrote:
>
>
> On 21/07/2015 00:17, Brandon Perez wrote:
>> Hello All,
>
> Hi Brandon,
>
> We use to send one mail by patch rather than sending them as an
> attachment of a single email. It's easier for reviewing the patches.
> You also need to add you Signed-off-by on each patch and CC all the
> relevant maintainers. Please see [1] for all the guidelines to submit a
> patch to Xen.
>
> A couple of comments I about this series:
>      - Patch #2: You are allowing any guest to do smc which, unless you
> trust all the guest, is unsecure. There was some discussion about
> different solution to handle SMC back in 2013 [2]. So far I didn't see
> any more update on it. It may be worth to send a separate thread about
> how to handle SMC.
>      - Patch #3, I can't find any documentation or implementation of the
> property "default-mapping" in Linux. Can you provide a link about it?
>
> I will comment more when you will resend the patches inline.
>
> Regards,
>
> [1] http://wiki.xenproject.org/wiki/Submitting_Xen_Project_Patches
> [2] http://lists.xen.org/archives/html/xen-devel/2013-07/msg02779.html
>

Hi Julien,

     I'm not sure that these patches are quite ready yet to be put into 
the Xen repo. For one, these patches don't solve the problem of the 
crossbar as outlined in the thread this came from [1]. Also, I haven't 
had a chance to clean up these patches yet. I've provided these patches 
at the request of Ian, so that you guys could see the changes I have 
made so far, and we could discuss what changes would be needed for the 
crossbar.

     I agree about Patch #2, that makes sense, this was a workaround 
I've been using for now. Perhaps a check could be added to see if the 
domain is the privileged domain?

     That's correct, the "default-mapping" property is not standard. 
It's another workaround that I'm working with for now. The interrupts 
property is going to contain the crossbar input number, not the actual 
SPI GIC line, so I needed a way to convey this to Xen.

     The patches are inlined below:

Patch #1:

 From f2bf190255c8f872d15063d7f8a6382c279e312d Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 17:56:49 -0400
Subject: DRA7: Add specific mappings for devices/regions not in the 
device tree.

The DRA7 chip, which is similar to the OMAP5 chip, also requires 
specific mappings. These are MMIO mappings which are not explicitly 
stated in the device tree, so Xen does not know to map them. This patch 
adds these regions required by the DRA7 to be mapped.

Signed-off-by: Brandon Perez <bperez-1@ti.com>

---
  xen/arch/arm/platforms/omap5.c        |   27 +++++++++++++++++++++++++++
  xen/include/asm-arm/platforms/omap5.h |    3 +++
  2 files changed, 30 insertions(+)

diff --git a/xen/arch/arm/platforms/omap5.c b/xen/arch/arm/platforms/omap5.c
index e7bf30d..3c6495a 100644
--- a/xen/arch/arm/platforms/omap5.c
+++ b/xen/arch/arm/platforms/omap5.c
@@ -120,6 +120,32 @@ static int omap5_specific_mapping(struct domain *d)
      return 0;
  }

+/* Additional mappings for dom0 (not in the DTS) */
+static int dra7_specific_mapping(struct domain *d)
+{
+    /* Map the PRM module */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_PRM_BASE), 2,
+                     paddr_to_pfn(OMAP5_PRM_BASE));
+
+    /* Map the PRM_MPU */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_PRCM_MPU_BASE), 1,
+                     paddr_to_pfn(OMAP5_PRCM_MPU_BASE));
+
+    /* Map the Wakeup Gen */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_WKUPGEN_BASE), 1,
+                     paddr_to_pfn(OMAP5_WKUPGEN_BASE));
+
+    /* Map the on-chip SRAM */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_SRAM_PA), 32,
+                     paddr_to_pfn(OMAP5_SRAM_PA));
+
+    /* Map GPMC address space for NAND flash. */
+    map_mmio_regions(d, paddr_to_pfn(OMAP5_GPMC_PA), 65536,
+                     paddr_to_pfn(OMAP5_GPMC_PA));
+
+    return 0;
+}
+
  static int __init omap5_smp_init(void)
  {
      void __iomem *wugen_base;
@@ -171,6 +197,7 @@ PLATFORM_START(dra7, "TI DRA7")
      .init_time = omap5_init_time,
      .cpu_up = cpu_up_send_sgi,
      .smp_init = omap5_smp_init,
+    .specific_mapping = dra7_specific_mapping,

      .dom0_gnttab_start = 0x4b000000,
      .dom0_gnttab_size = 0x20000,
diff --git a/xen/include/asm-arm/platforms/omap5.h 
b/xen/include/asm-arm/platforms/omap5.h
index c559c84..d87e7d2 100644
--- a/xen/include/asm-arm/platforms/omap5.h
+++ b/xen/include/asm-arm/platforms/omap5.h
@@ -20,6 +20,9 @@
  #define OMAP_AUX_CORE_BOOT_0_OFFSET             0x800
  #define OMAP_AUX_CORE_BOOT_1_OFFSET             0x804

+#define OMAP5_GPMC_PA                           0x01000000
+#define OMAP5_TILER_PA                          0x60000000
+
  #endif /* __ASM_ARM_PLATFORMS_OMAP5_H */

  /*
-- 
1.7.9.5

Patch #2:

 From e53fdc1ea750dd3143e2d7cd62a5d38eb446afde Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 17:58:24 -0400
Subject: Traps: Enable pass-through SMC call support for guest OS's.

Originally, Xen did not allow for guests to make SMC calls. However, on 
the DRA7 chips, the kernel needs to make several SMC calls to interact 
with the secure ROM code.

There are two solutions for solving this in the patch. The selected 
method is to simply allow the kernel to make the call, without going 
through the hypervisor. The other is to trap and emulate the call in the 
hypervisor.

Signed-off-by: Brandon Perez <bperez-1@ti.com>

---
  xen/arch/arm/traps.c |   15 ++++++++++++++-
  1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/traps.c b/xen/arch/arm/traps.c
index 258d4c5..9b9de7b 100644
--- a/xen/arch/arm/traps.c
+++ b/xen/arch/arm/traps.c
@@ -123,8 +123,9 @@ void __cpuinit init_traps(void)
                   CPTR_EL2);

      /* Setup hypervisor traps */
+    // TODO: Choose method
      WRITE_SYSREG(HCR_PTW|HCR_BSU_INNER|HCR_AMO|HCR_IMO|HCR_FMO|HCR_VM|
-                 HCR_TWE|HCR_TWI|HCR_TSC|HCR_TAC|HCR_SWIO|HCR_TIDCP, 
HCR_EL2);
+ 
HCR_TWE|HCR_TWI/*|HCR_TSC*/|HCR_TAC|HCR_SWIO|HCR_TIDCP, HCR_EL2);
      isb();
  }

@@ -2494,6 +2495,18 @@ asmlinkage void do_trap_hypervisor(struct 
cpu_user_regs *regs)
          GUEST_BUG_ON(!psr_mode_is_32bit(regs->cpsr));
          perfc_incr(trap_smc32);
          inject_undef32_exception(regs);
+// TODO: Choose method
+/*#define omap5_smc(func_id, arg1) \
+        asm volatile ("push {r1-r12, lr}\n\t" \
+                      "mov r12,%0\n\t" \
+                      "mov r0,%1\n\t" \
+                      "smc #1\n\t" \
+                      "pop {r1-r12, lr}" \
+                      : \
+                      : "r" (func_id), "r" (arg1))
+
+        omap5_smc(regs->r12, regs->r0);
+        advance_pc(regs, hsr); */
          break;
      case HSR_EC_HVC32:
          GUEST_BUG_ON(!psr_mode_is_32bit(regs->cpsr));
-- 
1.7.9.5

Patch #3:

 From 03f444d9042b1730bf3d8ff9ab3369e744e5e69b Mon Sep 17 00:00:00 2001
From: Brandon Perez <bperez-1@ti.com>
Date: Mon, 20 Jul 2015 18:04:36 -0400
Subject: ARM/IRQ-Crossbar: Make Xen aware of devices being statically 
mapped in the IRQ crossbar.

In general, Xen needs to own very few interrupts to run. Most of these 
are timer PPIs, which do not involve the crossbar. The notable exception 
to this is the serial device, which is an SPI. On the DRA7 chips, this 
involves going through the interrupt crossbar.

As the device tree entry will contain the crossbar input number, and not 
the IRQ line, Xen must be given the SPI input line number. This is 
achieved with the "default-mapping" property, which contains the SPI 
number that the device will correspond to in the mapping setup by the 
bootloader.

Signed-off-by: Brandon Perez <bperez-1@ti.com>

---
  xen/common/device_tree.c |   11 +++++++----
  1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/xen/common/device_tree.c b/xen/common/device_tree.c
index 31f169b..fc82eb4 100644
--- a/xen/common/device_tree.c
+++ b/xen/common/device_tree.c
@@ -1036,10 +1036,13 @@ int dt_device_get_raw_irq(const struct 
dt_device_node *device,
      dt_dprintk("dt_device_get_raw_irq: dev=%s, index=%u\n",
                 device->full_name, index);

-    /* Get the interrupts property */
-    intspec = dt_get_property(device, "interrupts", &intlen);
-    if ( intspec == NULL )
-        return -EINVAL;
+    /* Get the appropiate interrupts property */
+    intspec = dt_get_property(device, "default-mapping", &intlen);
+    if (intspec == NULL) {
+        intspec = dt_get_property(device, "interrupts", &intlen);
+        if (intspec == NULL)
+            return -EINVAL;
+    }
      intlen /= sizeof(*intspec);

      dt_dprintk(" intspec=%d intlen=%d\n", be32_to_cpup(intspec), intlen);
-- 
1.7.9.5


Brandon Perez

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Ian Campbell]

On Tue, 2015-07-21 at 10:07 -0400, Brandon Perez wrote:
> 
>      I'm not sure that these patches are quite ready yet to be put 
> into 
> the Xen repo.

That's ok, but even for an RFC (Request For Comments) please post them
one patch per email in the manner of git send-email. You can use -
-subject-prefix='PATCH RFC' to tag them as such.

Thanks,
Ian.

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Brandon Perez]

On 07/21/2015 11:12 AM, Ian Campbell wrote:
> On Tue, 2015-07-21 at 10:07 -0400, Brandon Perez wrote:
>>
>>       I'm not sure that these patches are quite ready yet to be put
>> into
>> the Xen repo.
>
> That's ok, but even for an RFC (Request For Comments) please post them
> one patch per email in the manner of git send-email. You can use -
> -subject-prefix='PATCH RFC' to tag them as such.
>
> Thanks,
> Ian.
>

All,

     Apologies for the delay, I've been pretty busy the last week. Also, 
sorry for the confusion with sending the patches, this is my first time 
working with an open-source dev group.

     I've sent the patches, one per email, via git.

Brandon

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Ian Campbell]

On Wed, 2015-07-29 at 16:36 -0400, Brandon Perez wrote:
> On 07/21/2015 11:12 AM, Ian Campbell wrote:
> > On Tue, 2015-07-21 at 10:07 -0400, Brandon Perez wrote:
> > > 
> > >       I'm not sure that these patches are quite ready yet to be put
> > > into
> > > the Xen repo.
> > 
> > That's ok, but even for an RFC (Request For Comments) please post them
> > one patch per email in the manner of git send-email. You can use -
> > -subject-prefix='PATCH RFC' to tag them as such.
> > 
> > Thanks,
> > Ian.
> > 
> 
> All,
> 
>      Apologies for the delay, I've been pretty busy the last week. Also, 
> sorry for the confusion with sending the patches, this is my first time 
> working with an open-source dev group.
> 
>      I've sent the patches, one per email, via git.

Thanks. For next time please can you send a batch of related patches all
with the same git send-email invocation such that they arrive as a set of
threaded mails (i.e. with a References header on the previous). That keeps
the related patches together in peoples in box, and also labels them 1/N,
2/N etc so we know which order they are applicable in.

Invoking send-email with --dry-run will print out all the headers it would
send, so you can check it will do as expected.

http://wiki.xen.org/wiki/Submitting_Xen_Patches#Git_send-email talks about
this stuff a bit.

Thanks,

Ian.

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Ian Campbell]

On Wed, 2015-07-29 at 16:36 -0400, Brandon Perez wrote:
> On 07/21/2015 11:12 AM, Ian Campbell wrote:
> > On Tue, 2015-07-21 at 10:07 -0400, Brandon Perez wrote:
> > > 
> > >       I'm not sure that these patches are quite ready yet to be put
> > > into
> > > the Xen repo.
> > 
> > That's ok, but even for an RFC (Request For Comments) please post them
> > one patch per email in the manner of git send-email. You can use -
> > -subject-prefix='PATCH RFC' to tag them as such.
> > 
> > Thanks,
> > Ian.
> > 
> 
> All,
> 
>      Apologies for the delay, I've been pretty busy the last week. Also, 
> sorry for the confusion with sending the patches, this is my first time 
> working with an open-source dev group.
> 
>      I've sent the patches, one per email, via git.

They were "From: bperez-1@ti.com", which bounces my replies. I can resend
if you are not subscribed to the list or you can fish them out of the
archives I guess.

Ian.

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Brandon Perez]

On 07/30/2015 07:23 AM, Ian Campbell wrote:
> On Wed, 2015-07-29 at 16:36 -0400, Brandon Perez wrote:
>> On 07/21/2015 11:12 AM, Ian Campbell wrote:
>>> On Tue, 2015-07-21 at 10:07 -0400, Brandon Perez wrote:
>>>>
>>>>        I'm not sure that these patches are quite ready yet to be put
>>>> into
>>>> the Xen repo.
>>>
>>> That's ok, but even for an RFC (Request For Comments) please post them
>>> one patch per email in the manner of git send-email. You can use -
>>> -subject-prefix='PATCH RFC' to tag them as such.
>>>
>>> Thanks,
>>> Ian.
>>>
>>
>> All,
>>
>>       Apologies for the delay, I've been pretty busy the last week. Also,
>> sorry for the confusion with sending the patches, this is my first time
>> working with an open-source dev group.
>>
>>       I've sent the patches, one per email, via git.
>
> They were "From: bperez-1@ti.com", which bounces my replies. I can resend
> if you are not subscribed to the list or you can fish them out of the
> archives I guess.
>
> Ian.
>

Apologies, I mistyped my email address. It should be "b-perez1@ti.com".

Brandon

Re: [Xen-Devel] Enabling IRQ Crossbar (Secondary Interrupt Controller) Support

[Brandon Perez]

On 07/30/2015 06:57 AM, Ian Campbell wrote:
> On Wed, 2015-07-29 at 16:36 -0400, Brandon Perez wrote:
>> On 07/21/2015 11:12 AM, Ian Campbell wrote:
>>> On Tue, 2015-07-21 at 10:07 -0400, Brandon Perez wrote:
>>>>
>>>>        I'm not sure that these patches are quite ready yet to be put
>>>> into
>>>> the Xen repo.
>>>
>>> That's ok, but even for an RFC (Request For Comments) please post them
>>> one patch per email in the manner of git send-email. You can use -
>>> -subject-prefix='PATCH RFC' to tag them as such.
>>>
>>> Thanks,
>>> Ian.
>>>
>>
>> All,
>>
>>       Apologies for the delay, I've been pretty busy the last week. Also,
>> sorry for the confusion with sending the patches, this is my first time
>> working with an open-source dev group.
>>
>>       I've sent the patches, one per email, via git.
>
> Thanks. For next time please can you send a batch of related patches all
> with the same git send-email invocation such that they arrive as a set of
> threaded mails (i.e. with a References header on the previous). That keeps
> the related patches together in peoples in box, and also labels them 1/N,
> 2/N etc so we know which order they are applicable in.
>
> Invoking send-email with --dry-run will print out all the headers it would
> send, so you can check it will do as expected.
>
> http://wiki.xen.org/wiki/Submitting_Xen_Patches#Git_send-email talks about
> this stuff a bit.
>
> Thanks,
>
> Ian.
>

Hi Ian,

     I see that makes sense, I'll make sure to batch the patches next time.

Brandon