[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Andre Przywara]

(for GIT URL and Changelog see below)

ARM CPUs with the virtualization extension have a new mode called
HYP mode, which allows hypervisors to safely control and monitor
guests. The current hypervisor implementations (KVM and Xen)
require the kernel to be entered in that HYP mode.

This patch series introduces a configuration variable
CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP
mode. This is done automatically during execution of the bootm
command.

The process of switching into HYP mode requires the CPU to be in
secure state initially when entering u-boot, it will then setup some
register and switch to non-secure state. This requires the GIC to be
programmed properly first. Explanations about the details are in the
commit messages of the respective patches.

The patches are structured like this:
1/8: prepare header file
2/8: add monitor handler (assembly)
3/8: add per CPU non-secure switch routine (assembly)
4/8: add system wide non-secure setup (C)
5/8: trigger non-secure switch during bootm command
6/8: add generic SMP functionality
7/8: add HYP mode switching
8/8: board specific code for ARM Versatile Express TC2

Since up to patch 6/8 this code works on non-virtualization capable
CPUs also and there has been a request, there is now a second
configuration variable CONFIG_ARMV7_NONSEC, which omits the final
HYP mode switch and just goes into non-secure SVC state.
You can specify either (or none) of them, the code cares about
the dependency.

The code aims to be as generic as possible, though currently it has
only been tested on the Versatile Express TC-2 board. The last patch
thus enables the feature for that board and should serve as an
example for supporting other boards.

For convenience there is a GIT tree which you can pull these patches
from ("hypmode_v4" branch):
git://git.linaro.org/people/aprzywara/u-boot.git

Changes RFC..v1
* not a dedicated command anymore, code run by bootm & friends
* protecting code with #ifdefs to avoid unnecessary inclusion and
  accidental crashing (when accessing restricted registers)
* moving prototypes to header file to meet checkpatch recommendation
* adding comment as proposed by Christoffer

Changes v1..v2
mostly style and code layout changes 
* restructure assembly code to live in a new file and not start.S
* split smp, nonsec_init and hyp_init to be separate functions
* used named constants from common header files
* split C function to be more readable
* extend comments to be more precise and elaborate
* add provision to override GIC base address (needed for Arndale?)
* add configuration variable to enable VExpress specific SMP code
* use writel/readl for MMIO GIC accesses
* remove superfluous isb instructions
* more minor fixes

Changes v2..v3
* fix clobbering of GICC address actually spoiling the stack
* do CNTFRQ setup in assembly per core (and not only once per SoC)
* moving the new code files into arch/arm/cpu/armv7
* add config variable for doing non-secure switch only
* use actual HYP and secure instructions mnemonics instead of
  the encoded byte sequence. This requires more recent compilers.
* make the identification of the CPU core more robust and saner
* use enum for error codes and rename them
* lots of smaller layout and style fixes

Changes v3..v4
* mask reserved bits in CBAR register
* move the VExpress board specific SMP code into the board directory
* embed error reporting in the respective functions and getting
  rid of the error code enum at all (by popular demand ;-)
* minor style fixes

Please review and comment!

Contributions and comments to support other boards are welcome.

Andre Przywara (8):
  ARM: prepare armv7.h to be included from assembly source
  ARM: add secure monitor handler to switch to non-secure state
  ARM: add assembly routine to switch to non-secure state
  ARM: add C function to switch to non-secure state
  ARM: trigger non-secure state switch during bootm execution
  ARM: add SMP support for non-secure switch
  ARM: extend non-secure switch to also go into HYP mode
  ARM: VExpress: enable ARMv7 virt support for VExpress A15

 arch/arm/cpu/armv7/Makefile             |   5 +
 arch/arm/cpu/armv7/nonsec_virt.S        | 196 ++++++++++++++++++++++++++++++++
 arch/arm/cpu/armv7/virt-v7.c            | 172 ++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h            |  33 +++++-
 arch/arm/include/asm/gic.h              |  19 ++++
 arch/arm/lib/bootm.c                    |  18 +++
 board/armltd/vexpress/Makefile          |   7 +-
 board/armltd/vexpress/vexpress_common.c |  13 +++
 board/armltd/vexpress/vexpress_smp.S    |  36 ++++++
 include/common.h                        |   2 +
 include/configs/vexpress_ca15_tc2.h     |   4 +-
 11 files changed, 501 insertions(+), 4 deletions(-)
 create mode 100644 arch/arm/cpu/armv7/nonsec_virt.S
 create mode 100644 arch/arm/cpu/armv7/virt-v7.c
 create mode 100644 arch/arm/include/asm/gic.h
 create mode 100644 board/armltd/vexpress/vexpress_smp.S

-- 
1.7.12.1

[U-Boot] [PATCH v4 1/8] ARM: prepare armv7.h to be included from assembly source

[Andre Przywara]

armv7.h contains some useful constants, but also C prototypes.
To include it also in assembly files, protect the non-assembly
part appropriately.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/include/asm/armv7.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index a73630b..20caa7c 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -23,7 +23,6 @@
  */
 #ifndef ARMV7_H
 #define ARMV7_H
-#include <linux/types.h>
 
 /* Cortex-A9 revisions */
 #define MIDR_CORTEX_A9_R0P1	0x410FC091
@@ -57,6 +56,9 @@
 #define ARMV7_CLIDR_CTYPE_INSTRUCTION_DATA	3
 #define ARMV7_CLIDR_CTYPE_UNIFIED		4
 
+#ifndef __ASSEMBLY__
+#include <linux/types.h>
+
 /*
  * CP15 Barrier instructions
  * Please note that we have separate barrier instructions in ARMv7
@@ -74,4 +76,6 @@ void v7_outer_cache_inval_all(void);
 void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
+#endif /* ! __ASSEMBLY__ */
+
 #endif
-- 
1.7.12.1

[U-Boot] [PATCH v4 2/8] ARM: add secure monitor handler to switch to non-secure state

[Andre Przywara]

A prerequisite for using virtualization is to be in HYP mode, which
requires the CPU to be in non-secure state first.
Add a new file in arch/arm/cpu/armv7 to hold a monitor handler routine
which switches the CPU to non-secure state by setting the NS and
associated bits.
According to the ARM architecture reference manual this should not be
done in SVC mode, so we have to setup a SMC handler for this.
We create a new vector table to avoid interference with other boards.
The MVBAR register will be programmed later just before the smc call.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/Makefile      |  4 +++
 arch/arm/cpu/armv7/nonsec_virt.S | 55 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 59 insertions(+)
 create mode 100644 arch/arm/cpu/armv7/nonsec_virt.S

diff --git a/arch/arm/cpu/armv7/Makefile b/arch/arm/cpu/armv7/Makefile
index 7a8c2d0..11a8ad5 100644
--- a/arch/arm/cpu/armv7/Makefile
+++ b/arch/arm/cpu/armv7/Makefile
@@ -36,6 +36,10 @@ ifneq ($(CONFIG_AM33XX)$(CONFIG_OMAP44XX)$(CONFIG_OMAP54XX)$(CONFIG_TEGRA)$(CONF
 SOBJS	+= lowlevel_init.o
 endif
 
+ifneq ($(CONFIG_ARMV7_NONSEC),)
+SOBJS	+= nonsec_virt.o
+endif
+
 SRCS	:= $(START:.o=.S) $(COBJS:.o=.c)
 OBJS	:= $(addprefix $(obj),$(COBJS) $(SOBJS))
 START	:= $(addprefix $(obj),$(START))
diff --git a/arch/arm/cpu/armv7/nonsec_virt.S b/arch/arm/cpu/armv7/nonsec_virt.S
new file mode 100644
index 0000000..dbe5c0f
--- /dev/null
+++ b/arch/arm/cpu/armv7/nonsec_virt.S
@@ -0,0 +1,55 @@
+/*
+ * code for switching cores into non-secure state
+ *
+ * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <config.h>
+
+/* the vector table for secure state */
+_monitor_vectors:
+	.word 0	/* reset */
+	.word 0 /* undef */
+	adr pc, _secure_monitor
+	.word 0
+	.word 0
+	.word 0
+	.word 0
+	.word 0
+	.word 0	/* pad */
+
+/*
+ * secure monitor handler
+ * U-boot calls this "software interrupt" in start.S
+ * This is executed on a "smc" instruction, we use a "smc #0" to switch
+ * to non-secure state.
+ * We use only r0 and r1 here, due to constraints in the caller.
+ */
+	.align	5
+_secure_monitor:
+	mrc	p15, 0, r1, c1, c1, 0		@ read SCR
+	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits
+	orr	r1, r1, #0x31			@ enable NS, AW, FW bits
+
+	mcr	p15, 0, r1, c1, c1, 0		@ write SCR (with NS bit set)
+
+	movs	pc, lr				@ return to non-secure SVC
+
-- 
1.7.12.1

[U-Boot] [PATCH v4 3/8] ARM: add assembly routine to switch to non-secure state

[Andre Przywara]

While actually switching to non-secure state is one thing, another
part of this process is to make sure that we still have full access
to the interrupt controller (GIC).
The GIC is fully aware of secure vs. non-secure state, some
registers are banked, others may be configured to be accessible from
secure state only.
To be as generic as possible, we get the GIC memory mapped address
based on the PERIPHBASE value in the CBAR register. Since this
register is not architecturally defined, we check the MIDR before to
be from an A15 or A7.
For CPUs not having the CBAR or boards with wrong information herein
we allow providing the base address as a configuration variable.

Now that we know the GIC address, we:
a) allow private interrupts to be delivered to the core
   (GICD_IGROUPR0 = 0xFFFFFFFF)
b) enable the CPU interface (GICC_CTLR[0] = 1)
c) set the priority filter to allow non-secure interrupts
   (GICC_PMR = 0xFF)

Also we allow access to all coprocessor interfaces from non-secure
state by writing the appropriate bits in the NSACR register.

The generic timer base frequency register is only accessible from
secure state, so we have to program it now. Actually this should be
done from primary firmware before, but some boards seems to omit
this, so if needed we do this here with a board specific value.
The Versatile Express board does not need this, so we remove the
frequency from the configuration file here.

After having switched to non-secure state, we also enable the
non-secure GIC CPU interface, since this register is banked.

Since we need to call this routine also directly from the smp_pen
later (where we don't have any stack), we can only use caller saved
registers r0-r3 and r12 to not mess with the compiler.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/nonsec_virt.S    | 86 +++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h        | 21 +++++++++
 arch/arm/include/asm/gic.h          | 17 ++++++++
 include/configs/vexpress_ca15_tc2.h |  2 -
 4 files changed, 124 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/include/asm/gic.h

diff --git a/arch/arm/cpu/armv7/nonsec_virt.S b/arch/arm/cpu/armv7/nonsec_virt.S
index dbe5c0f..81384df 100644
--- a/arch/arm/cpu/armv7/nonsec_virt.S
+++ b/arch/arm/cpu/armv7/nonsec_virt.S
@@ -23,6 +23,11 @@
  */
 
 #include <config.h>
+#include <linux/linkage.h>
+#include <asm/gic.h>
+#include <asm/armv7.h>
+
+.arch_extension sec
 
 /* the vector table for secure state */
 _monitor_vectors:
@@ -53,3 +58,84 @@ _secure_monitor:
 
 	movs	pc, lr				@ return to non-secure SVC
 
+/*
+ * Switch a core to non-secure state.
+ *
+ *  1. initialize the GIC per-core interface
+ *  2. allow coprocessor access in non-secure modes
+ *  3. switch the cpu mode (by calling "smc #0")
+ *
+ * Called from smp_pen by secondary cores and directly by the BSP.
+ * Do not assume that the stack is available and only use registers
+ * r0-r3 and r12.
+ *
+ * PERIPHBASE is used to get the GIC address. This could be 40 bits long,
+ * though, but we check this in C before calling this function.
+ */
+ENTRY(_nonsec_init)
+#ifdef CONFIG_ARM_GIC_BASE_ADDRESS
+	ldr	r2, =CONFIG_ARM_GIC_BASE_ADDRESS
+#else
+	mrc	p15, 4, r2, c15, c0, 0		@ read CBAR
+	bfc	r2, #0, #15			@ clear reserved bits
+#endif
+	add	r3, r2, #GIC_DIST_OFFSET	@ GIC dist i/f offset
+	mvn	r1, #0				@ all bits to 1
+	str	r1, [r3, #GICD_IGROUPRn]	@ allow private interrupts
+
+	mrc	p15, 0, r0, c0, c0, 0		@ read MIDR
+	ldr	r1, =MIDR_PRIMARY_PART_MASK
+	and	r0, r0, r1			@ mask out variant and revision
+
+	ldr	r1, =MIDR_CORTEX_A7_R0P0 & MIDR_PRIMARY_PART_MASK
+	cmp	r0, r1				@ check for Cortex-A7
+
+	ldr	r1, =MIDR_CORTEX_A15_R0P0 & MIDR_PRIMARY_PART_MASK
+	cmpne	r0, r1				@ check for Cortex-A15
+
+	movne	r1, #GIC_CPU_OFFSET_A9		@ GIC CPU offset for A9
+	moveq	r1, #GIC_CPU_OFFSET_A15		@ GIC CPU offset for A15/A7
+	add	r3, r2, r1			@ r3 = GIC CPU i/f addr
+
+	mov	r1, #1				@ set GICC_CTLR[enable]
+	str	r1, [r3, #GICC_CTLR]		@ and clear all other bits
+	mov	r1, #0xff
+	str	r1, [r3, #GICC_PMR]		@ set priority mask register
+
+	movw	r1, #0x3fff
+	movt	r1, #0x0006
+	mcr	p15, 0, r1, c1, c1, 2		@ NSACR = all copros to non-sec
+
+/* The CNTFRQ register of the generic timer needs to be
+ * programmed in secure state. Some primary bootloaders / firmware
+ * omit this, so if the frequency is provided in the configuration,
+ * we do this here instead.
+ * But first check if we have the generic timer.
+ */
+#ifdef CONFIG_SYS_CLK_FREQ
+	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
+	and	r0, r0, #CPUID_ARM_GENTIMER_MASK	@ mask arch timer bits
+	cmp	r0, #(1 << CPUID_ARM_GENTIMER_SHIFT)
+	ldreq	r1, =CONFIG_SYS_CLK_FREQ
+	mcreq	p15, 0, r1, c14, c0, 0		@ write CNTFRQ
+#endif
+
+	adr	r1, _monitor_vectors
+	mcr	p15, 0, r1, c12, c0, 1		@ set MVBAR to secure vectors
+
+	mrc	p15, 0, ip, c12, c0, 0		@ save secure copy of VBAR
+
+	isb
+	smc	#0				@ call into MONITOR mode
+
+	mcr	p15, 0, ip, c12, c0, 0		@ write non-secure copy of VBAR
+
+	mov	r1, #1
+	str	r1, [r3, #GICC_CTLR]		@ enable non-secure CPU i/f
+	add	r2, r2, #GIC_DIST_OFFSET
+	str	r1, [r2, #GICD_CTLR]		@ allow private interrupts
+
+	mov	r0, r3				@ return GICC address
+
+	bx	lr
+ENDPROC(_nonsec_init)
diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index 20caa7c..3e4b743 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -34,6 +34,22 @@
 #define MIDR_CORTEX_A15_R0P0	0x410FC0F0
 #define MIDR_CORTEX_A15_R2P2	0x412FC0F2
 
+/* Cortex-A7 revisions */
+#define MIDR_CORTEX_A7_R0P0	0x410FC070
+
+#define MIDR_PRIMARY_PART_MASK	0xFF0FFFF0
+
+/* ID_PFR1 feature fields */
+#define CPUID_ARM_SEC_SHIFT		4
+#define CPUID_ARM_SEC_MASK		(0xF << CPUID_ARM_SEC_SHIFT)
+#define CPUID_ARM_VIRT_SHIFT		12
+#define CPUID_ARM_VIRT_MASK		(0xF << CPUID_ARM_VIRT_SHIFT)
+#define CPUID_ARM_GENTIMER_SHIFT	16
+#define CPUID_ARM_GENTIMER_MASK		(0xF << CPUID_ARM_GENTIMER_SHIFT)
+
+/* valid bits in CBAR register / PERIPHBASE value */
+#define CBAR_MASK			0xFFFF8000
+
 /* CCSIDR */
 #define CCSIDR_LINE_SIZE_OFFSET		0
 #define CCSIDR_LINE_SIZE_MASK		0x7
@@ -76,6 +92,11 @@ void v7_outer_cache_inval_all(void);
 void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
+#ifdef CONFIG_ARMV7_NONSEC
+/* defined in assembly file */
+unsigned int _nonsec_init(void);
+#endif /* CONFIG_ARMV7_NONSEC */
+
 #endif /* ! __ASSEMBLY__ */
 
 #endif
diff --git a/arch/arm/include/asm/gic.h b/arch/arm/include/asm/gic.h
new file mode 100644
index 0000000..c2b1e28
--- /dev/null
+++ b/arch/arm/include/asm/gic.h
@@ -0,0 +1,17 @@
+#ifndef __GIC_V2_H__
+#define __GIC_V2_H__
+
+/* register offsets for the ARM generic interrupt controller (GIC) */
+
+#define GIC_DIST_OFFSET		0x1000
+#define GICD_CTLR		0x0000
+#define GICD_TYPER		0x0004
+#define GICD_IGROUPRn		0x0080
+#define GICD_SGIR		0x0F00
+
+#define GIC_CPU_OFFSET_A9	0x0100
+#define GIC_CPU_OFFSET_A15	0x2000
+#define GICC_CTLR		0x0000
+#define GICC_PMR		0x0004
+
+#endif
diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
index 9e230ad..4f425ac 100644
--- a/include/configs/vexpress_ca15_tc2.h
+++ b/include/configs/vexpress_ca15_tc2.h
@@ -31,6 +31,4 @@
 #include "vexpress_common.h"
 #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
 
-#define CONFIG_SYS_CLK_FREQ 24000000
-
 #endif
-- 
1.7.12.1

[U-Boot] [PATCH v4 4/8] ARM: add C function to switch to non-secure state

[Andre Przywara]

The core specific part of the work is done in the assembly routine
in nonsec_virt.S, introduced with the previous patch, but for the full
glory we need to setup the GIC distributor interface once for the
whole system, which is done in C here.
The routine is placed in arch/arm/cpu/armv7 to allow easy access from
other ARMv7 boards.

We check the availability of the security extensions first.

Since we need a safe way to access the GIC, we use the PERIPHBASE
registers on Cortex-A15 and A7 CPUs and do some sanity checks.
Boards not implementing the CBAR can override this value via a
configuration file variable.

Then we actually do the GIC enablement:
a) enable the GIC distributor, both for non-secure and secure state
   (GICD_CTLR[1:0] = 11b)
b) allow all interrupts to be handled from non-secure state
   (GICD_IGROUPRn = 0xFFFFFFFF)

The core specific GIC setup is then done in the assembly routine.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/Makefile  |   1 +
 arch/arm/cpu/armv7/virt-v7.c | 121 +++++++++++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h |   3 ++
 3 files changed, 125 insertions(+)
 create mode 100644 arch/arm/cpu/armv7/virt-v7.c

diff --git a/arch/arm/cpu/armv7/Makefile b/arch/arm/cpu/armv7/Makefile
index 11a8ad5..5813e87 100644
--- a/arch/arm/cpu/armv7/Makefile
+++ b/arch/arm/cpu/armv7/Makefile
@@ -38,6 +38,7 @@ endif
 
 ifneq ($(CONFIG_ARMV7_NONSEC),)
 SOBJS	+= nonsec_virt.o
+COBJS	+= virt-v7.o
 endif
 
 SRCS	:= $(START:.o=.S) $(COBJS:.o=.c)
diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
new file mode 100644
index 0000000..689023f
--- /dev/null
+++ b/arch/arm/cpu/armv7/virt-v7.c
@@ -0,0 +1,121 @@
+/*
+ * (C) Copyright 2013
+ * Andre Przywara, Linaro
+ *
+ * Routines to transition ARMv7 processors from secure into non-secure state
+ * needed to enable ARMv7 virtualization for current hypervisors
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <common.h>
+#include <asm/armv7.h>
+#include <asm/gic.h>
+#include <asm/io.h>
+
+unsigned long gic_dist_addr;
+
+static unsigned int read_id_pfr1(void)
+{
+	unsigned int reg;
+
+	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
+	return reg;
+}
+
+static unsigned long get_gicd_base_address(void)
+{
+#ifdef CONFIG_ARM_GIC_BASE_ADDRESS
+	return CONFIG_ARM_GIC_BASE_ADDRESS + GIC_DIST_OFFSET;
+#else
+	unsigned midr;
+	unsigned periphbase;
+
+	/* check whether we are an Cortex-A15 or A7.
+	 * The actual HYP switch should work with all CPUs supporting
+	 * the virtualization extension, but we need the GIC address,
+	 * which we know only for sure for those two CPUs.
+	 */
+	asm("mrc p15, 0, %0, c0, c0, 0\n" : "=r"(midr));
+	switch (midr & MIDR_PRIMARY_PART_MASK) {
+	case MIDR_CORTEX_A9_R0P1:
+	case MIDR_CORTEX_A15_R0P0:
+	case MIDR_CORTEX_A7_R0P0:
+		break;
+	default:
+		printf("nonsec: could not determine GIC address.\n");
+		return -1;
+	}
+
+	/* get the GIC base address from the CBAR register */
+	asm("mrc p15, 4, %0, c15, c0, 0\n" : "=r" (periphbase));
+
+	/* the PERIPHBASE can be mapped above 4 GB (lower 8 bits used to
+	 * encode this). Bail out here since we cannot access this without
+	 * enabling paging.
+	 */
+	if ((periphbase & 0xff) != 0) {
+		printf("nonsec: PERIPHBASE is above 4 GB, no access.\n");
+		return -1;
+	}
+
+	return (periphbase & CBAR_MASK) + GIC_DIST_OFFSET;
+#endif
+}
+
+int armv7_switch_nonsec(void)
+{
+	unsigned int reg;
+	unsigned itlinesnr, i;
+
+	/* check whether the CPU supports the security extensions */
+	reg = read_id_pfr1();
+	if ((reg & 0xF0) == 0) {
+		printf("nonsec: Security extensions not implemented.\n");
+		return -1;
+	}
+
+	/* the SCR register will be set directly in the monitor mode handler,
+	 * according to the spec one should not tinker with it in secure state
+	 * in SVC mode. Do not try to read it once in non-secure state,
+	 * any access to it will trap.
+	 */
+
+	gic_dist_addr = get_gicd_base_address();
+	if (gic_dist_addr == -1)
+		return -1;
+
+	/* enable the GIC distributor */
+	writel(readl(gic_dist_addr + GICD_CTLR) | 0x03,
+	       gic_dist_addr + GICD_CTLR);
+
+	/* TYPER[4:0] contains an encoded number of available interrupts */
+	itlinesnr = readl(gic_dist_addr + GICD_TYPER) & 0x1f;
+
+	/* set all bits in the GIC group registers to one to allow access
+	 * from non-secure state
+	 */
+	for (i = 0; i <= itlinesnr; i++)
+		writel((unsigned)-1, gic_dist_addr + GICD_IGROUPRn + 4 * i);
+
+	/* call the non-sec switching code on this CPU */
+	_nonsec_init();
+
+	return 0;
+}
diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index 3e4b743..10ced11 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -93,6 +93,9 @@ void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
 #ifdef CONFIG_ARMV7_NONSEC
+
+int armv7_switch_nonsec(void);
+
 /* defined in assembly file */
 unsigned int _nonsec_init(void);
 #endif /* CONFIG_ARMV7_NONSEC */
-- 
1.7.12.1

[U-Boot] [PATCH v4 5/8] ARM: trigger non-secure state switch during bootm execution

[Andre Przywara]

To actually trigger the non-secure switch we just implemented, call
the switching routine from within the bootm command implementation.
This way we automatically enable this feature without further user
intervention.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/lib/bootm.c | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm/lib/bootm.c b/arch/arm/lib/bootm.c
index 0325d08..6470eac 100644
--- a/arch/arm/lib/bootm.c
+++ b/arch/arm/lib/bootm.c
@@ -34,6 +34,10 @@
 #include <asm/bootm.h>
 #include <linux/compiler.h>
 
+#ifdef CONFIG_ARMV7_NONSEC
+#include <asm/armv7.h>
+#endif
+
 DECLARE_GLOBAL_DATA_PTR;
 
 static struct tag *params;
@@ -193,6 +197,14 @@ static void setup_end_tag(bd_t *bd)
 
 __weak void setup_board_tags(struct tag **in_params) {}
 
+static void do_nonsec_virt_switch(void)
+{
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
+	if (armv7_switch_nonsec() == 0)
+		debug("entered non-secure state\n");
+#endif
+}
+
 /* Subcommand: PREP */
 static void boot_prep_linux(bootm_headers_t *images)
 {
@@ -229,6 +241,7 @@ static void boot_prep_linux(bootm_headers_t *images)
 		printf("FDT and ATAGS support not compiled in - hanging\n");
 		hang();
 	}
+	do_nonsec_virt_switch();
 }
 
 /* Subcommand: GO */
-- 
1.7.12.1

[U-Boot] [PATCH v4 6/8] ARM: add SMP support for non-secure switch

[Andre Przywara]

Currently the non-secure switch is only done for the boot processor.
To enable full SMP support, we have to switch all secondary cores
into non-secure state also.

So we add an entry point for secondary CPUs coming out of low-power
state and make sure we put them into WFI again after having switched
to non-secure state.
For this we acknowledge and EOI the wake-up IPI, then go into WFI.
Once being kicked out of it later, we sanity check that the start
address has actually been changed (since another attempt to switch
to non-secure would block the core) and jump to the new address.

The actual CPU kick is done by sending an inter-processor interrupt
via the GIC to all CPU interfaces except the requesting processor.
The secondary cores will then setup their respective GIC CPU
interface.
While this approach is pretty universal across several ARMv7 boards,
we make this function weak in case someone needs to tweak this for
a specific board.

The way of setting the secondary's start address is board specific,
so each supported board should implement smp_set_boot_cpu_addr().

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/nonsec_virt.S | 22 ++++++++++++++++++++++
 arch/arm/cpu/armv7/virt-v7.c     | 16 +++++++++++++++-
 arch/arm/include/asm/armv7.h     |  1 +
 arch/arm/include/asm/gic.h       |  2 ++
 include/common.h                 |  2 ++
 5 files changed, 42 insertions(+), 1 deletion(-)

diff --git a/arch/arm/cpu/armv7/nonsec_virt.S b/arch/arm/cpu/armv7/nonsec_virt.S
index 81384df..a88fa6b 100644
--- a/arch/arm/cpu/armv7/nonsec_virt.S
+++ b/arch/arm/cpu/armv7/nonsec_virt.S
@@ -59,6 +59,28 @@ _secure_monitor:
 	movs	pc, lr				@ return to non-secure SVC
 
 /*
+ * Secondary CPUs start here and call the code for the core specific parts
+ * of the non-secure and HYP mode transition. The GIC distributor specific
+ * code has already been executed by a C function before.
+ * Then they go back to wfi and wait to be woken up by the kernel again.
+ */
+ENTRY(_smp_pen)
+	mrs	r0, cpsr
+	orr	r0, r0, #0xc0
+	msr	cpsr, r0			@ disable interrupts
+	ldr	r1, =_start
+	mcr	p15, 0, r1, c12, c0, 0		@ set VBAR
+
+	bl	_nonsec_init
+
+	ldr	r1, [r0, #GICC_IAR]		@ acknowledge IPI
+	str	r1, [r0, #GICC_EOIR]		@ signal end of interrupt
+
+	adr	r0, _smp_pen			@ do not use this address again
+	b	smp_waitloop			@ wait for IPIs, board specific
+ENDPROC(_smp_pen)
+
+/*
  * Switch a core to non-secure state.
  *
  *  1. initialize the GIC per-core interface
diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
index 689023f..50f0a3a 100644
--- a/arch/arm/cpu/armv7/virt-v7.c
+++ b/arch/arm/cpu/armv7/virt-v7.c
@@ -79,6 +79,17 @@ static unsigned long get_gicd_base_address(void)
 #endif
 }
 
+static void kick_secondary_cpus_gic(unsigned long gicdaddr)
+{
+	/* kick all CPUs (except this one) by writing to GICD_SGIR */
+	writel(1U << 24, gicdaddr + GICD_SGIR);
+}
+
+void __weak smp_kick_all_cpus(void)
+{
+	kick_secondary_cpus_gic(gic_dist_addr);
+}
+
 int armv7_switch_nonsec(void)
 {
 	unsigned int reg;
@@ -114,7 +125,10 @@ int armv7_switch_nonsec(void)
 	for (i = 0; i <= itlinesnr; i++)
 		writel((unsigned)-1, gic_dist_addr + GICD_IGROUPRn + 4 * i);
 
-	/* call the non-sec switching code on this CPU */
+	smp_set_core_boot_addr((unsigned long)_smp_pen, -1);
+	smp_kick_all_cpus();
+
+	/* call the non-sec switching code on this CPU also */
 	_nonsec_init();
 
 	return 0;
diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index 10ced11..06657fe 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -98,6 +98,7 @@ int armv7_switch_nonsec(void);
 
 /* defined in assembly file */
 unsigned int _nonsec_init(void);
+void _smp_pen(void);
 #endif /* CONFIG_ARMV7_NONSEC */
 
 #endif /* ! __ASSEMBLY__ */
diff --git a/arch/arm/include/asm/gic.h b/arch/arm/include/asm/gic.h
index c2b1e28..a0891cc 100644
--- a/arch/arm/include/asm/gic.h
+++ b/arch/arm/include/asm/gic.h
@@ -13,5 +13,7 @@
 #define GIC_CPU_OFFSET_A15	0x2000
 #define GICC_CTLR		0x0000
 #define GICC_PMR		0x0004
+#define GICC_IAR		0x000C
+#define GICC_EOIR		0x0010
 
 #endif
diff --git a/include/common.h b/include/common.h
index e5220cf..05b30d7 100644
--- a/include/common.h
+++ b/include/common.h
@@ -642,6 +642,8 @@ void ft_pci_setup(void *blob, bd_t *bd);
 #endif
 #endif
 
+void smp_set_core_boot_addr(unsigned long addr, int corenr);
+void smp_kick_all_cpus(void);
 
 /* $(CPU)/serial.c */
 int	serial_init   (void);
-- 
1.7.12.1

[U-Boot] [PATCH v4 7/8] ARM: extend non-secure switch to also go into HYP mode

[Andre Przywara]

For the KVM and XEN hypervisors to be usable, we need to enter the
kernel in HYP mode. Now that we already are in non-secure state,
HYP mode switching is within short reach.

While doing the non-secure switch, we have to enable the HVC
instruction and setup the HYP mode HVBAR (while still secure).

The actual switch is done by dropping back from a HYP mode handler
without actually leaving HYP mode, so we introduce a new handler
routine in our new secure exception vector table.

In the assembly switching routine we save and restore the banked LR
and SP registers around the hypercall to do the actual HYP mode
switch.

The C routine first checks whether we are in HYP mode already and
also whether the virtualization extensions are available. It also
checks whether the HYP mode switch was finally successful.
The bootm command part only adds and adjusts some error reporting.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 arch/arm/cpu/armv7/Makefile      |  2 +-
 arch/arm/cpu/armv7/nonsec_virt.S | 43 +++++++++++++++++++++++++++++++++++-----
 arch/arm/cpu/armv7/virt-v7.c     | 37 ++++++++++++++++++++++++++++++++++
 arch/arm/include/asm/armv7.h     |  6 ++++--
 arch/arm/lib/bootm.c             |  7 ++++++-
 5 files changed, 86 insertions(+), 9 deletions(-)

diff --git a/arch/arm/cpu/armv7/Makefile b/arch/arm/cpu/armv7/Makefile
index 5813e87..c20df3d 100644
--- a/arch/arm/cpu/armv7/Makefile
+++ b/arch/arm/cpu/armv7/Makefile
@@ -36,7 +36,7 @@ ifneq ($(CONFIG_AM33XX)$(CONFIG_OMAP44XX)$(CONFIG_OMAP54XX)$(CONFIG_TEGRA)$(CONF
 SOBJS	+= lowlevel_init.o
 endif
 
-ifneq ($(CONFIG_ARMV7_NONSEC),)
+ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
 SOBJS	+= nonsec_virt.o
 COBJS	+= virt-v7.o
 endif
diff --git a/arch/arm/cpu/armv7/nonsec_virt.S b/arch/arm/cpu/armv7/nonsec_virt.S
index a88fa6b..fb1651d 100644
--- a/arch/arm/cpu/armv7/nonsec_virt.S
+++ b/arch/arm/cpu/armv7/nonsec_virt.S
@@ -1,5 +1,5 @@
 /*
- * code for switching cores into non-secure state
+ * code for switching cores into non-secure state and into HYP mode
  *
  * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
  *
@@ -28,15 +28,16 @@
 #include <asm/armv7.h>
 
 .arch_extension sec
+.arch_extension virt
 
-/* the vector table for secure state */
+/* the vector table for secure state and HYP mode */
 _monitor_vectors:
 	.word 0	/* reset */
 	.word 0 /* undef */
 	adr pc, _secure_monitor
 	.word 0
 	.word 0
-	.word 0
+	adr pc, _hyp_trap
 	.word 0
 	.word 0
 	.word 0	/* pad */
@@ -54,10 +55,27 @@ _secure_monitor:
 	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits
 	orr	r1, r1, #0x31			@ enable NS, AW, FW bits
 
+#ifdef CONFIG_ARMV7_VIRT
+	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
+	and	r0, r0, #CPUID_ARM_VIRT_MASK	@ mask virtualization bits
+	cmp	r0, #(1 << CPUID_ARM_VIRT_SHIFT)
+	orreq	r1, r1, #0x100			@ allow HVC instruction
+#endif
+
 	mcr	p15, 0, r1, c1, c1, 0		@ write SCR (with NS bit set)
 
+#ifdef CONFIG_ARMV7_VIRT
+	mrceq	p15, 0, r0, c12, c0, 1		@ get MVBAR value
+	mcreq	p15, 4, r0, c12, c0, 0		@ write HVBAR
+#endif
+
 	movs	pc, lr				@ return to non-secure SVC
 
+_hyp_trap:
+	mrs	lr, elr_hyp	@ for older asm: .byte 0x00, 0xe3, 0x0e, 0xe1
+	mov pc, lr				@ do no switch modes, but
+						@ return to caller
+
 /*
  * Secondary CPUs start here and call the code for the core specific parts
  * of the non-secure and HYP mode transition. The GIC distributor specific
@@ -72,9 +90,13 @@ ENTRY(_smp_pen)
 	mcr	p15, 0, r1, c12, c0, 0		@ set VBAR
 
 	bl	_nonsec_init
+	mov	r12, r0				@ save GICC address
+#ifdef CONFIG_ARMV7_VIRT
+	bl	_switch_to_hyp
+#endif
 
-	ldr	r1, [r0, #GICC_IAR]		@ acknowledge IPI
-	str	r1, [r0, #GICC_EOIR]		@ signal end of interrupt
+	ldr	r1, [r12, #GICC_IAR]		@ acknowledge IPI
+	str	r1, [r12, #GICC_EOIR]		@ signal end of interrupt
 
 	adr	r0, _smp_pen			@ do not use this address again
 	b	smp_waitloop			@ wait for IPIs, board specific
@@ -161,3 +183,14 @@ ENTRY(_nonsec_init)
 
 	bx	lr
 ENDPROC(_nonsec_init)
+
+ENTRY(_switch_to_hyp)
+	mov	r0, lr
+	mov	r1, sp				@ save SVC copy of LR and SP
+	isb
+	hvc #0			 @ for older asm: .byte 0x70, 0x00, 0x40, 0xe1
+	mov	sp, r1
+	mov	lr, r0				@ restore SVC copy of LR and SP
+
+	bx	lr
+ENDPROC(_switch_to_hyp)
diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
index 50f0a3a..4dc8c45 100644
--- a/arch/arm/cpu/armv7/virt-v7.c
+++ b/arch/arm/cpu/armv7/virt-v7.c
@@ -3,6 +3,7 @@
  * Andre Przywara, Linaro
  *
  * Routines to transition ARMv7 processors from secure into non-secure state
+ * and from non-secure SVC into HYP mode
  * needed to enable ARMv7 virtualization for current hypervisors
  *
  * See file CREDITS for list of people who contributed to this
@@ -31,6 +32,14 @@
 
 unsigned long gic_dist_addr;
 
+static unsigned int read_cpsr(void)
+{
+	unsigned int reg;
+
+	asm volatile ("mrs %0, cpsr\n" : "=r" (reg));
+	return reg;
+}
+
 static unsigned int read_id_pfr1(void)
 {
 	unsigned int reg;
@@ -90,6 +99,34 @@ void __weak smp_kick_all_cpus(void)
 	kick_secondary_cpus_gic(gic_dist_addr);
 }
 
+int armv7_switch_hyp(void)
+{
+	unsigned int reg;
+
+	/* check whether we are in HYP mode already */
+	if ((read_cpsr() & 0x1f) == 0x1a) {
+		debug("CPU already in HYP mode\n");
+		return 0;
+	}
+
+	/* check whether the CPU supports the virtualization extensions */
+	reg = read_id_pfr1();
+	if ((reg & CPUID_ARM_VIRT_MASK) != 1 << CPUID_ARM_VIRT_SHIFT) {
+		printf("HYP mode: Virtualization extensions not implemented.\n");
+		return -1;
+	}
+
+	/* call the HYP switching code on this CPU also */
+	_switch_to_hyp();
+
+	if ((read_cpsr() & 0x1F) != 0x1a) {
+		printf("HYP mode: switch not successful.\n");
+		return -1;
+	}
+
+	return 0;
+}
+
 int armv7_switch_nonsec(void)
 {
 	unsigned int reg;
diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
index 06657fe..1a7b150 100644
--- a/arch/arm/include/asm/armv7.h
+++ b/arch/arm/include/asm/armv7.h
@@ -92,14 +92,16 @@ void v7_outer_cache_inval_all(void);
 void v7_outer_cache_flush_range(u32 start, u32 end);
 void v7_outer_cache_inval_range(u32 start, u32 end);
 
-#ifdef CONFIG_ARMV7_NONSEC
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 
 int armv7_switch_nonsec(void);
+int armv7_switch_hyp(void);
 
 /* defined in assembly file */
 unsigned int _nonsec_init(void);
 void _smp_pen(void);
-#endif /* CONFIG_ARMV7_NONSEC */
+void _switch_to_hyp(void);
+#endif /* CONFIG_ARMV7_NONSEC || CONFIG_ARMV7_VIRT */
 
 #endif /* ! __ASSEMBLY__ */
 
diff --git a/arch/arm/lib/bootm.c b/arch/arm/lib/bootm.c
index 6470eac..ff36319 100644
--- a/arch/arm/lib/bootm.c
+++ b/arch/arm/lib/bootm.c
@@ -34,7 +34,7 @@
 #include <asm/bootm.h>
 #include <linux/compiler.h>
 
-#ifdef CONFIG_ARMV7_NONSEC
+#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 #include <asm/armv7.h>
 #endif
 
@@ -201,8 +201,13 @@ static void do_nonsec_virt_switch(void)
 {
 #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
 	if (armv7_switch_nonsec() == 0)
+#ifdef CONFIG_ARMV7_VIRT
+		if (armv7_switch_hyp() == 0)
+			debug("entered HYP mode\n");
+#else
 		debug("entered non-secure state\n");
 #endif
+#endif
 }
 
 /* Subcommand: PREP */
-- 
1.7.12.1

[U-Boot] [PATCH v4 8/8] ARM: VExpress: enable ARMv7 virt support for VExpress A15

[Andre Przywara]

To enable hypervisors utilizing the ARMv7 virtualization extension
on the Versatile Express board with the A15 core tile, we add the
required configuration variable.
Also we define the board specific functions to do the SMP bringup:
smp_set_cpu_boot_addr() to set the start address for secondary cores
and smp_waitloop() to wait for IPIs and jump to the start address.

This also serves as an example for what to do when adding support for
new boards.

Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
---
 board/armltd/vexpress/Makefile          |  7 +++++--
 board/armltd/vexpress/vexpress_common.c | 13 ++++++++++++
 board/armltd/vexpress/vexpress_smp.S    | 36 +++++++++++++++++++++++++++++++++
 include/configs/vexpress_ca15_tc2.h     |  4 ++++
 4 files changed, 58 insertions(+), 2 deletions(-)
 create mode 100644 board/armltd/vexpress/vexpress_smp.S

diff --git a/board/armltd/vexpress/Makefile b/board/armltd/vexpress/Makefile
index 6719f3d..282ef6d 100644
--- a/board/armltd/vexpress/Makefile
+++ b/board/armltd/vexpress/Makefile
@@ -26,9 +26,12 @@ include $(TOPDIR)/config.mk
 LIB	= $(obj)lib$(BOARD).o
 
 COBJS	:= vexpress_common.o
+ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
+SOBJS	:= vexpress_smp.o
+endif
 
-SRCS	:= $(COBJS:.o=.c)
-OBJS	:= $(addprefix $(obj),$(COBJS))
+SRCS	:= $(COBJS:.o=.c) $(SOBJS:.o=.S)
+OBJS	:= $(addprefix $(obj),$(COBJS) $(SOBJS))
 
 $(LIB):	$(obj).depend $(OBJS)
 	$(call cmd_link_o_target, $(OBJS))
diff --git a/board/armltd/vexpress/vexpress_common.c b/board/armltd/vexpress/vexpress_common.c
index 2c54869..66b810d 100644
--- a/board/armltd/vexpress/vexpress_common.c
+++ b/board/armltd/vexpress/vexpress_common.c
@@ -272,3 +272,16 @@ ulong get_tbclk(void)
 {
 	return (ulong)CONFIG_SYS_HZ;
 }
+
+/* Setting the address at which secondary cores start from.
+ * Versatile Express uses one address for all cores, so ignore corenr
+ */
+void smp_set_core_boot_addr(unsigned long addr, int corenr)
+{
+	/* The SYSFLAGS register on VExpress needs to be cleared first
+	 * by writing to the next address, since any writes to the address
+	 * at offset 0 will only be ORed in
+	 */
+	writel(~0, CONFIG_SYSFLAGS_ADDR + 4);
+	writel(addr, CONFIG_SYSFLAGS_ADDR);
+}
diff --git a/board/armltd/vexpress/vexpress_smp.S b/board/armltd/vexpress/vexpress_smp.S
new file mode 100644
index 0000000..41be2e7
--- /dev/null
+++ b/board/armltd/vexpress/vexpress_smp.S
@@ -0,0 +1,36 @@
+/*
+ * code for redirecting secondary cores to their start address
+ *
+ * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
+ *
+ * See file CREDITS for list of people who contributed to this
+ * project.
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License as
+ * published by the Free Software Foundation; either version 2 of
+ * the License, or (at your option) any later version.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program; if not, write to the Free Software
+ * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
+ * MA 02111-1307 USA
+ */
+
+#include <config.h>
+#include <linux/linkage.h>
+
+/* void _smp_waitloop(unsigned previous_address); */
+ENTRY(smp_waitloop)
+	wfi
+	ldr	r1, =CONFIG_SYSFLAGS_ADDR	@ load start address
+	ldr	r1, [r1]
+	cmp	r0, r1			@ make sure we dont execute this code
+	beq	smp_waitloop		@ again (due to a spurious wakeup)
+	mov	pc, r1
+ENDPROC(smp_waitloop)
diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
index 4f425ac..14aa78e 100644
--- a/include/configs/vexpress_ca15_tc2.h
+++ b/include/configs/vexpress_ca15_tc2.h
@@ -31,4 +31,8 @@
 #include "vexpress_common.h"
 #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
 
+#define CONFIG_SYSFLAGS_ADDR 0x1c010030
+
+#define CONFIG_ARMV7_VIRT
+
 #endif
-- 
1.7.12.1

[U-Boot] [PATCH v4 4/8] ARM: add C function to switch to non-secure state

[Christoffer Dall]

On Fri, Aug 09, 2013 at 05:03:08PM +0200, Andre Przywara wrote:
> The core specific part of the work is done in the assembly routine
> in nonsec_virt.S, introduced with the previous patch, but for the full
> glory we need to setup the GIC distributor interface once for the
> whole system, which is done in C here.
> The routine is placed in arch/arm/cpu/armv7 to allow easy access from
> other ARMv7 boards.
> 
> We check the availability of the security extensions first.
> 
> Since we need a safe way to access the GIC, we use the PERIPHBASE
> registers on Cortex-A15 and A7 CPUs and do some sanity checks.
> Boards not implementing the CBAR can override this value via a
> configuration file variable.
> 
> Then we actually do the GIC enablement:
> a) enable the GIC distributor, both for non-secure and secure state
>    (GICD_CTLR[1:0] = 11b)
> b) allow all interrupts to be handled from non-secure state
>    (GICD_IGROUPRn = 0xFFFFFFFF)
> 
> The core specific GIC setup is then done in the assembly routine.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>  arch/arm/cpu/armv7/Makefile  |   1 +
>  arch/arm/cpu/armv7/virt-v7.c | 121 +++++++++++++++++++++++++++++++++++++++++++
>  arch/arm/include/asm/armv7.h |   3 ++
>  3 files changed, 125 insertions(+)
>  create mode 100644 arch/arm/cpu/armv7/virt-v7.c
> 
> diff --git a/arch/arm/cpu/armv7/Makefile b/arch/arm/cpu/armv7/Makefile
> index 11a8ad5..5813e87 100644
> --- a/arch/arm/cpu/armv7/Makefile
> +++ b/arch/arm/cpu/armv7/Makefile
> @@ -38,6 +38,7 @@ endif
>  
>  ifneq ($(CONFIG_ARMV7_NONSEC),)
>  SOBJS	+= nonsec_virt.o
> +COBJS	+= virt-v7.o
>  endif
>  
>  SRCS	:= $(START:.o=.S) $(COBJS:.o=.c)
> diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
> new file mode 100644
> index 0000000..689023f
> --- /dev/null
> +++ b/arch/arm/cpu/armv7/virt-v7.c
> @@ -0,0 +1,121 @@
> +/*
> + * (C) Copyright 2013
> + * Andre Przywara, Linaro
> + *
> + * Routines to transition ARMv7 processors from secure into non-secure state
> + * needed to enable ARMv7 virtualization for current hypervisors
> + *
> + * See file CREDITS for list of people who contributed to this
> + * project.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License as
> + * published by the Free Software Foundation; either version 2 of
> + * the License, or (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
> + * MA 02111-1307 USA
> + */
> +
> +#include <common.h>
> +#include <asm/armv7.h>
> +#include <asm/gic.h>
> +#include <asm/io.h>
> +
> +unsigned long gic_dist_addr;
> +
> +static unsigned int read_id_pfr1(void)
> +{
> +	unsigned int reg;
> +
> +	asm("mrc p15, 0, %0, c0, c1, 1\n" : "=r"(reg));
> +	return reg;
> +}
> +
> +static unsigned long get_gicd_base_address(void)
> +{
> +#ifdef CONFIG_ARM_GIC_BASE_ADDRESS
> +	return CONFIG_ARM_GIC_BASE_ADDRESS + GIC_DIST_OFFSET;
> +#else
> +	unsigned midr;
> +	unsigned periphbase;
> +
> +	/* check whether we are an Cortex-A15 or A7.
> +	 * The actual HYP switch should work with all CPUs supporting
> +	 * the virtualization extension, but we need the GIC address,
> +	 * which we know only for sure for those two CPUs.
> +	 */
> +	asm("mrc p15, 0, %0, c0, c0, 0\n" : "=r"(midr));
> +	switch (midr & MIDR_PRIMARY_PART_MASK) {
> +	case MIDR_CORTEX_A9_R0P1:
> +	case MIDR_CORTEX_A15_R0P0:
> +	case MIDR_CORTEX_A7_R0P0:
> +		break;
> +	default:
> +		printf("nonsec: could not determine GIC address.\n");
> +		return -1;
> +	}
> +
> +	/* get the GIC base address from the CBAR register */
> +	asm("mrc p15, 4, %0, c15, c0, 0\n" : "=r" (periphbase));
> +
> +	/* the PERIPHBASE can be mapped above 4 GB (lower 8 bits used to
> +	 * encode this). Bail out here since we cannot access this without
> +	 * enabling paging.
> +	 */
> +	if ((periphbase & 0xff) != 0) {
> +		printf("nonsec: PERIPHBASE is above 4 GB, no access.\n");
> +		return -1;
> +	}
> +
> +	return (periphbase & CBAR_MASK) + GIC_DIST_OFFSET;
> +#endif
> +}
> +
> +int armv7_switch_nonsec(void)
> +{
> +	unsigned int reg;
> +	unsigned itlinesnr, i;
> +
> +	/* check whether the CPU supports the security extensions */
> +	reg = read_id_pfr1();
> +	if ((reg & 0xF0) == 0) {
> +		printf("nonsec: Security extensions not implemented.\n");
> +		return -1;
> +	}
> +
> +	/* the SCR register will be set directly in the monitor mode handler,
> +	 * according to the spec one should not tinker with it in secure state
> +	 * in SVC mode. Do not try to read it once in non-secure state,
> +	 * any access to it will trap.
> +	 */
> +
> +	gic_dist_addr = get_gicd_base_address();
> +	if (gic_dist_addr == -1)
> +		return -1;
> +
> +	/* enable the GIC distributor */
> +	writel(readl(gic_dist_addr + GICD_CTLR) | 0x03,
> +	       gic_dist_addr + GICD_CTLR);
> +
> +	/* TYPER[4:0] contains an encoded number of available interrupts */
> +	itlinesnr = readl(gic_dist_addr + GICD_TYPER) & 0x1f;
> +
> +	/* set all bits in the GIC group registers to one to allow access
> +	 * from non-secure state
> +	 */
> +	for (i = 0; i <= itlinesnr; i++)
> +		writel((unsigned)-1, gic_dist_addr + GICD_IGROUPRn + 4 * i);

I think Nikolay pointed out that you could start from 1 in this for
loop.

-Christoffer

> +
> +	/* call the non-sec switching code on this CPU */
> +	_nonsec_init();
> +
> +	return 0;
> +}
> diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
> index 3e4b743..10ced11 100644
> --- a/arch/arm/include/asm/armv7.h
> +++ b/arch/arm/include/asm/armv7.h
> @@ -93,6 +93,9 @@ void v7_outer_cache_flush_range(u32 start, u32 end);
>  void v7_outer_cache_inval_range(u32 start, u32 end);
>  
>  #ifdef CONFIG_ARMV7_NONSEC
> +
> +int armv7_switch_nonsec(void);
> +
>  /* defined in assembly file */
>  unsigned int _nonsec_init(void);
>  #endif /* CONFIG_ARMV7_NONSEC */
> -- 
> 1.7.12.1
> 

[U-Boot] [PATCH v4 7/8] ARM: extend non-secure switch to also go into HYP mode

[Christoffer Dall]

On Fri, Aug 09, 2013 at 05:03:11PM +0200, Andre Przywara wrote:
> For the KVM and XEN hypervisors to be usable, we need to enter the
> kernel in HYP mode. Now that we already are in non-secure state,
> HYP mode switching is within short reach.
> 
> While doing the non-secure switch, we have to enable the HVC
> instruction and setup the HYP mode HVBAR (while still secure).
> 
> The actual switch is done by dropping back from a HYP mode handler
> without actually leaving HYP mode, so we introduce a new handler
> routine in our new secure exception vector table.
> 
> In the assembly switching routine we save and restore the banked LR
> and SP registers around the hypercall to do the actual HYP mode
> switch.
> 
> The C routine first checks whether we are in HYP mode already and
> also whether the virtualization extensions are available. It also
> checks whether the HYP mode switch was finally successful.
> The bootm command part only adds and adjusts some error reporting.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>  arch/arm/cpu/armv7/Makefile      |  2 +-
>  arch/arm/cpu/armv7/nonsec_virt.S | 43 +++++++++++++++++++++++++++++++++++-----
>  arch/arm/cpu/armv7/virt-v7.c     | 37 ++++++++++++++++++++++++++++++++++
>  arch/arm/include/asm/armv7.h     |  6 ++++--
>  arch/arm/lib/bootm.c             |  7 ++++++-
>  5 files changed, 86 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/arm/cpu/armv7/Makefile b/arch/arm/cpu/armv7/Makefile
> index 5813e87..c20df3d 100644
> --- a/arch/arm/cpu/armv7/Makefile
> +++ b/arch/arm/cpu/armv7/Makefile
> @@ -36,7 +36,7 @@ ifneq ($(CONFIG_AM33XX)$(CONFIG_OMAP44XX)$(CONFIG_OMAP54XX)$(CONFIG_TEGRA)$(CONF
>  SOBJS	+= lowlevel_init.o
>  endif
>  
> -ifneq ($(CONFIG_ARMV7_NONSEC),)
> +ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
>  SOBJS	+= nonsec_virt.o
>  COBJS	+= virt-v7.o
>  endif
> diff --git a/arch/arm/cpu/armv7/nonsec_virt.S b/arch/arm/cpu/armv7/nonsec_virt.S
> index a88fa6b..fb1651d 100644
> --- a/arch/arm/cpu/armv7/nonsec_virt.S
> +++ b/arch/arm/cpu/armv7/nonsec_virt.S
> @@ -1,5 +1,5 @@
>  /*
> - * code for switching cores into non-secure state
> + * code for switching cores into non-secure state and into HYP mode
>   *
>   * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
>   *
> @@ -28,15 +28,16 @@
>  #include <asm/armv7.h>
>  
>  .arch_extension sec
> +.arch_extension virt
>  
> -/* the vector table for secure state */
> +/* the vector table for secure state and HYP mode */
>  _monitor_vectors:
>  	.word 0	/* reset */
>  	.word 0 /* undef */
>  	adr pc, _secure_monitor
>  	.word 0
>  	.word 0
> -	.word 0
> +	adr pc, _hyp_trap
>  	.word 0
>  	.word 0
>  	.word 0	/* pad */
> @@ -54,10 +55,27 @@ _secure_monitor:
>  	bic	r1, r1, #0x4e			@ clear IRQ, FIQ, EA, nET bits
>  	orr	r1, r1, #0x31			@ enable NS, AW, FW bits
>  
> +#ifdef CONFIG_ARMV7_VIRT
> +	mrc	p15, 0, r0, c0, c1, 1		@ read ID_PFR1
> +	and	r0, r0, #CPUID_ARM_VIRT_MASK	@ mask virtualization bits
> +	cmp	r0, #(1 << CPUID_ARM_VIRT_SHIFT)
> +	orreq	r1, r1, #0x100			@ allow HVC instruction
> +#endif
> +
>  	mcr	p15, 0, r1, c1, c1, 0		@ write SCR (with NS bit set)
>  
> +#ifdef CONFIG_ARMV7_VIRT
> +	mrceq	p15, 0, r0, c12, c0, 1		@ get MVBAR value
> +	mcreq	p15, 4, r0, c12, c0, 0		@ write HVBAR
> +#endif
> +
>  	movs	pc, lr				@ return to non-secure SVC
>  
> +_hyp_trap:
> +	mrs	lr, elr_hyp	@ for older asm: .byte 0x00, 0xe3, 0x0e, 0xe1

I see you kep this as is, oh well.

> +	mov pc, lr				@ do no switch modes, but
> +						@ return to caller
> +
>  /*
>   * Secondary CPUs start here and call the code for the core specific parts
>   * of the non-secure and HYP mode transition. The GIC distributor specific
> @@ -72,9 +90,13 @@ ENTRY(_smp_pen)
>  	mcr	p15, 0, r1, c12, c0, 0		@ set VBAR
>  
>  	bl	_nonsec_init
> +	mov	r12, r0				@ save GICC address
> +#ifdef CONFIG_ARMV7_VIRT
> +	bl	_switch_to_hyp
> +#endif
>  
> -	ldr	r1, [r0, #GICC_IAR]		@ acknowledge IPI
> -	str	r1, [r0, #GICC_EOIR]		@ signal end of interrupt
> +	ldr	r1, [r12, #GICC_IAR]		@ acknowledge IPI
> +	str	r1, [r12, #GICC_EOIR]		@ signal end of interrupt
>  
>  	adr	r0, _smp_pen			@ do not use this address again
>  	b	smp_waitloop			@ wait for IPIs, board specific
> @@ -161,3 +183,14 @@ ENTRY(_nonsec_init)
>  
>  	bx	lr
>  ENDPROC(_nonsec_init)
> +
> +ENTRY(_switch_to_hyp)
> +	mov	r0, lr
> +	mov	r1, sp				@ save SVC copy of LR and SP
> +	isb
> +	hvc #0			 @ for older asm: .byte 0x70, 0x00, 0x40, 0xe1
> +	mov	sp, r1
> +	mov	lr, r0				@ restore SVC copy of LR and SP
> +
> +	bx	lr
> +ENDPROC(_switch_to_hyp)
> diff --git a/arch/arm/cpu/armv7/virt-v7.c b/arch/arm/cpu/armv7/virt-v7.c
> index 50f0a3a..4dc8c45 100644
> --- a/arch/arm/cpu/armv7/virt-v7.c
> +++ b/arch/arm/cpu/armv7/virt-v7.c
> @@ -3,6 +3,7 @@
>   * Andre Przywara, Linaro
>   *
>   * Routines to transition ARMv7 processors from secure into non-secure state
> + * and from non-secure SVC into HYP mode
>   * needed to enable ARMv7 virtualization for current hypervisors
>   *
>   * See file CREDITS for list of people who contributed to this
> @@ -31,6 +32,14 @@
>  
>  unsigned long gic_dist_addr;
>  
> +static unsigned int read_cpsr(void)
> +{
> +	unsigned int reg;
> +
> +	asm volatile ("mrs %0, cpsr\n" : "=r" (reg));
> +	return reg;
> +}
> +
>  static unsigned int read_id_pfr1(void)
>  {
>  	unsigned int reg;
> @@ -90,6 +99,34 @@ void __weak smp_kick_all_cpus(void)
>  	kick_secondary_cpus_gic(gic_dist_addr);
>  }
>  
> +int armv7_switch_hyp(void)
> +{
> +	unsigned int reg;
> +
> +	/* check whether we are in HYP mode already */
> +	if ((read_cpsr() & 0x1f) == 0x1a) {
> +		debug("CPU already in HYP mode\n");
> +		return 0;
> +	}
> +
> +	/* check whether the CPU supports the virtualization extensions */
> +	reg = read_id_pfr1();
> +	if ((reg & CPUID_ARM_VIRT_MASK) != 1 << CPUID_ARM_VIRT_SHIFT) {
> +		printf("HYP mode: Virtualization extensions not implemented.\n");
> +		return -1;
> +	}
> +
> +	/* call the HYP switching code on this CPU also */
> +	_switch_to_hyp();
> +
> +	if ((read_cpsr() & 0x1F) != 0x1a) {
> +		printf("HYP mode: switch not successful.\n");
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
>  int armv7_switch_nonsec(void)
>  {
>  	unsigned int reg;
> diff --git a/arch/arm/include/asm/armv7.h b/arch/arm/include/asm/armv7.h
> index 06657fe..1a7b150 100644
> --- a/arch/arm/include/asm/armv7.h
> +++ b/arch/arm/include/asm/armv7.h
> @@ -92,14 +92,16 @@ void v7_outer_cache_inval_all(void);
>  void v7_outer_cache_flush_range(u32 start, u32 end);
>  void v7_outer_cache_inval_range(u32 start, u32 end);
>  
> -#ifdef CONFIG_ARMV7_NONSEC
> +#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
>  
>  int armv7_switch_nonsec(void);
> +int armv7_switch_hyp(void);
>  
>  /* defined in assembly file */
>  unsigned int _nonsec_init(void);
>  void _smp_pen(void);
> -#endif /* CONFIG_ARMV7_NONSEC */
> +void _switch_to_hyp(void);
> +#endif /* CONFIG_ARMV7_NONSEC || CONFIG_ARMV7_VIRT */
>  
>  #endif /* ! __ASSEMBLY__ */
>  
> diff --git a/arch/arm/lib/bootm.c b/arch/arm/lib/bootm.c
> index 6470eac..ff36319 100644
> --- a/arch/arm/lib/bootm.c
> +++ b/arch/arm/lib/bootm.c
> @@ -34,7 +34,7 @@
>  #include <asm/bootm.h>
>  #include <linux/compiler.h>
>  
> -#ifdef CONFIG_ARMV7_NONSEC
> +#if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
>  #include <asm/armv7.h>
>  #endif
>  
> @@ -201,8 +201,13 @@ static void do_nonsec_virt_switch(void)
>  {
>  #if defined(CONFIG_ARMV7_NONSEC) || defined(CONFIG_ARMV7_VIRT)
>  	if (armv7_switch_nonsec() == 0)
> +#ifdef CONFIG_ARMV7_VIRT
> +		if (armv7_switch_hyp() == 0)
> +			debug("entered HYP mode\n");
> +#else
>  		debug("entered non-secure state\n");
>  #endif
> +#endif
>  }
>  
>  /* Subcommand: PREP */
> -- 
> 1.7.12.1
> 

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Christoffer Dall]

On Fri, Aug 09, 2013 at 05:03:04PM +0200, Andre Przywara wrote:
> (for GIT URL and Changelog see below)
> 
> ARM CPUs with the virtualization extension have a new mode called
> HYP mode, which allows hypervisors to safely control and monitor
> guests. The current hypervisor implementations (KVM and Xen)
> require the kernel to be entered in that HYP mode.
> 
> This patch series introduces a configuration variable
> CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP
> mode. This is done automatically during execution of the bootm
> command.
> 
> The process of switching into HYP mode requires the CPU to be in
> secure state initially when entering u-boot, it will then setup some
> register and switch to non-secure state. This requires the GIC to be
> programmed properly first. Explanations about the details are in the
> commit messages of the respective patches.
> 
> The patches are structured like this:
> 1/8: prepare header file
> 2/8: add monitor handler (assembly)
> 3/8: add per CPU non-secure switch routine (assembly)
> 4/8: add system wide non-secure setup (C)
> 5/8: trigger non-secure switch during bootm command
> 6/8: add generic SMP functionality
> 7/8: add HYP mode switching
> 8/8: board specific code for ARM Versatile Express TC2
> 
> Since up to patch 6/8 this code works on non-virtualization capable
> CPUs also and there has been a request, there is now a second
> configuration variable CONFIG_ARMV7_NONSEC, which omits the final
> HYP mode switch and just goes into non-secure SVC state.
> You can specify either (or none) of them, the code cares about
> the dependency.
> 
> The code aims to be as generic as possible, though currently it has
> only been tested on the Versatile Express TC-2 board. The last patch
> thus enables the feature for that board and should serve as an
> example for supporting other boards.
> 
> For convenience there is a GIT tree which you can pull these patches
> from ("hypmode_v4" branch):
> git://git.linaro.org/people/aprzywara/u-boot.git
> 
> Changes RFC..v1
> * not a dedicated command anymore, code run by bootm & friends
> * protecting code with #ifdefs to avoid unnecessary inclusion and
>   accidental crashing (when accessing restricted registers)
> * moving prototypes to header file to meet checkpatch recommendation
> * adding comment as proposed by Christoffer
> 
> Changes v1..v2
> mostly style and code layout changes 
> * restructure assembly code to live in a new file and not start.S
> * split smp, nonsec_init and hyp_init to be separate functions
> * used named constants from common header files
> * split C function to be more readable
> * extend comments to be more precise and elaborate
> * add provision to override GIC base address (needed for Arndale?)
> * add configuration variable to enable VExpress specific SMP code
> * use writel/readl for MMIO GIC accesses
> * remove superfluous isb instructions
> * more minor fixes
> 
> Changes v2..v3
> * fix clobbering of GICC address actually spoiling the stack
> * do CNTFRQ setup in assembly per core (and not only once per SoC)
> * moving the new code files into arch/arm/cpu/armv7
> * add config variable for doing non-secure switch only
> * use actual HYP and secure instructions mnemonics instead of
>   the encoded byte sequence. This requires more recent compilers.
> * make the identification of the CPU core more robust and saner
> * use enum for error codes and rename them
> * lots of smaller layout and style fixes
> 
> Changes v3..v4
> * mask reserved bits in CBAR register
> * move the VExpress board specific SMP code into the board directory
> * embed error reporting in the respective functions and getting
>   rid of the error code enum at all (by popular demand ;-)
> * minor style fixes
> 
> Please review and comment!
> 
Only had those very few nits sent separately, which can be fixed later
if we want.  Otherwise looks good.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>

Thanks,
-Christoffer

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Andre Przywara]

On 08/09/2013 06:55 PM, Christoffer Dall wrote:
> On Fri, Aug 09, 2013 at 05:03:04PM +0200, Andre Przywara wrote:
>> (for GIT URL and Changelog see below)
>>
>> ARM CPUs with the virtualization extension have a new mode called
>> HYP mode, which allows hypervisors to safely control and monitor
>> guests. The current hypervisor implementations (KVM and Xen)
>> require the kernel to be entered in that HYP mode.
>>
>> This patch series introduces a configuration variable
>> CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP
>> mode. This is done automatically during execution of the bootm
>> command.
>>
>> The process of switching into HYP mode requires the CPU to be in
>> secure state initially when entering u-boot, it will then setup some
>> register and switch to non-secure state. This requires the GIC to be
>> programmed properly first. Explanations about the details are in the
>> commit messages of the respective patches.
>>
>> The patches are structured like this:
>> 1/8: prepare header file
>> 2/8: add monitor handler (assembly)
>> 3/8: add per CPU non-secure switch routine (assembly)
>> 4/8: add system wide non-secure setup (C)
>> 5/8: trigger non-secure switch during bootm command
>> 6/8: add generic SMP functionality
>> 7/8: add HYP mode switching
>> 8/8: board specific code for ARM Versatile Express TC2
>>
>> Since up to patch 6/8 this code works on non-virtualization capable
>> CPUs also and there has been a request, there is now a second
>> configuration variable CONFIG_ARMV7_NONSEC, which omits the final
>> HYP mode switch and just goes into non-secure SVC state.
>> You can specify either (or none) of them, the code cares about
>> the dependency.
>>
>> The code aims to be as generic as possible, though currently it has
>> only been tested on the Versatile Express TC-2 board. The last patch
>> thus enables the feature for that board and should serve as an
>> example for supporting other boards.
>>
>> For convenience there is a GIT tree which you can pull these patches
>> from ("hypmode_v4" branch):
>> git://git.linaro.org/people/aprzywara/u-boot.git
>>
>> Changes RFC..v1
>> * not a dedicated command anymore, code run by bootm & friends
>> * protecting code with #ifdefs to avoid unnecessary inclusion and
>>    accidental crashing (when accessing restricted registers)
>> * moving prototypes to header file to meet checkpatch recommendation
>> * adding comment as proposed by Christoffer
>>
>> Changes v1..v2
>> mostly style and code layout changes
>> * restructure assembly code to live in a new file and not start.S
>> * split smp, nonsec_init and hyp_init to be separate functions
>> * used named constants from common header files
>> * split C function to be more readable
>> * extend comments to be more precise and elaborate
>> * add provision to override GIC base address (needed for Arndale?)
>> * add configuration variable to enable VExpress specific SMP code
>> * use writel/readl for MMIO GIC accesses
>> * remove superfluous isb instructions
>> * more minor fixes
>>
>> Changes v2..v3
>> * fix clobbering of GICC address actually spoiling the stack
>> * do CNTFRQ setup in assembly per core (and not only once per SoC)
>> * moving the new code files into arch/arm/cpu/armv7
>> * add config variable for doing non-secure switch only
>> * use actual HYP and secure instructions mnemonics instead of
>>    the encoded byte sequence. This requires more recent compilers.
>> * make the identification of the CPU core more robust and saner
>> * use enum for error codes and rename them
>> * lots of smaller layout and style fixes
>>
>> Changes v3..v4
>> * mask reserved bits in CBAR register
>> * move the VExpress board specific SMP code into the board directory
>> * embed error reporting in the respective functions and getting
>>    rid of the error code enum at all (by popular demand ;-)
>> * minor style fixes
>>
>> Please review and comment!
>>
> Only had those very few nits sent separately, which can be fixed later
> if we want.  Otherwise looks good.
>
> Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>

Thanks, Christoffer.

Are there more concerns or comments about this?
Is there anything left I can do get these patches merged?

Albert, Tom,
do you need more ACKs or Reviewed-bys?

Kim, do you have time and expertise to look at these?

Thanks,
Andre.

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Nikolay Nikolaev]

Hello,


On Fri, Aug 9, 2013 at 6:03 PM, Andre Przywara <andre.przywara@linaro.org>wrote:

> (for GIT URL and Changelog see below)
>
> ARM CPUs with the virtualization extension have a new mode called
> HYP mode, which allows hypervisors to safely control and monitor
> guests. The current hypervisor implementations (KVM and Xen)
> require the kernel to be entered in that HYP mode.
>
> This patch series introduces a configuration variable
> CONFIG_ARMV7_VIRT which enables code to switch all cores into HYP
> mode. This is done automatically during execution of the bootm
> command.
>
> The process of switching into HYP mode requires the CPU to be in
> secure state initially when entering u-boot, it will then setup some
> register and switch to non-secure state. This requires the GIC to be
> programmed properly first. Explanations about the details are in the
> commit messages of the respective patches.
>
> The patches are structured like this:
> 1/8: prepare header file
> 2/8: add monitor handler (assembly)
> 3/8: add per CPU non-secure switch routine (assembly)
> 4/8: add system wide non-secure setup (C)
> 5/8: trigger non-secure switch during bootm command
> 6/8: add generic SMP functionality
> 7/8: add HYP mode switching
> 8/8: board specific code for ARM Versatile Express TC2
>
> Since up to patch 6/8 this code works on non-virtualization capable
> CPUs also and there has been a request, there is now a second
> configuration variable CONFIG_ARMV7_NONSEC, which omits the final
> HYP mode switch and just goes into non-secure SVC state.
> You can specify either (or none) of them, the code cares about
> the dependency.
>
> The code aims to be as generic as possible, though currently it has
> only been tested on the Versatile Express TC-2 board. The last patch
>

We have "backported" [1] these patches to U-Boot for the ARM Chromebook
(Samsung Exynos5250 SoC). They work without functional changes, including
this last v4.
We can consider the code tested on Exynos5 also.



> thus enables the feature for that board and should serve as an
> example for supporting other boards.
>
> For convenience there is a GIT tree which you can pull these patches
> from ("hypmode_v4" branch):
> git://git.linaro.org/people/aprzywara/u-boot.git
>
> Changes RFC..v1
> * not a dedicated command anymore, code run by bootm & friends
> * protecting code with #ifdefs to avoid unnecessary inclusion and
>   accidental crashing (when accessing restricted registers)
> * moving prototypes to header file to meet checkpatch recommendation
> * adding comment as proposed by Christoffer
>
> Changes v1..v2
> mostly style and code layout changes
> * restructure assembly code to live in a new file and not start.S
> * split smp, nonsec_init and hyp_init to be separate functions
> * used named constants from common header files
> * split C function to be more readable
> * extend comments to be more precise and elaborate
> * add provision to override GIC base address (needed for Arndale?)
> * add configuration variable to enable VExpress specific SMP code
> * use writel/readl for MMIO GIC accesses
> * remove superfluous isb instructions
> * more minor fixes
>
> Changes v2..v3
> * fix clobbering of GICC address actually spoiling the stack
> * do CNTFRQ setup in assembly per core (and not only once per SoC)
> * moving the new code files into arch/arm/cpu/armv7
> * add config variable for doing non-secure switch only
> * use actual HYP and secure instructions mnemonics instead of
>   the encoded byte sequence. This requires more recent compilers.
> * make the identification of the CPU core more robust and saner
> * use enum for error codes and rename them
> * lots of smaller layout and style fixes
>
> Changes v3..v4
> * mask reserved bits in CBAR register
> * move the VExpress board specific SMP code into the board directory
> * embed error reporting in the respective functions and getting
>   rid of the error code enum at all (by popular demand ;-)
> * minor style fixes
>
> Please review and comment!
>
> Contributions and comments to support other boards are welcome.
>
> Andre Przywara (8):
>   ARM: prepare armv7.h to be included from assembly source
>   ARM: add secure monitor handler to switch to non-secure state
>   ARM: add assembly routine to switch to non-secure state
>   ARM: add C function to switch to non-secure state
>   ARM: trigger non-secure state switch during bootm execution
>   ARM: add SMP support for non-secure switch
>   ARM: extend non-secure switch to also go into HYP mode
>   ARM: VExpress: enable ARMv7 virt support for VExpress A15
>
>  arch/arm/cpu/armv7/Makefile             |   5 +
>  arch/arm/cpu/armv7/nonsec_virt.S        | 196
> ++++++++++++++++++++++++++++++++
>  arch/arm/cpu/armv7/virt-v7.c            | 172 ++++++++++++++++++++++++++++
>  arch/arm/include/asm/armv7.h            |  33 +++++-
>  arch/arm/include/asm/gic.h              |  19 ++++
>  arch/arm/lib/bootm.c                    |  18 +++
>  board/armltd/vexpress/Makefile          |   7 +-
>  board/armltd/vexpress/vexpress_common.c |  13 +++
>  board/armltd/vexpress/vexpress_smp.S    |  36 ++++++
>  include/common.h                        |   2 +
>  include/configs/vexpress_ca15_tc2.h     |   4 +-
>  11 files changed, 501 insertions(+), 4 deletions(-)
>  create mode 100644 arch/arm/cpu/armv7/nonsec_virt.S
>  create mode 100644 arch/arm/cpu/armv7/virt-v7.c
>  create mode 100644 arch/arm/include/asm/gic.h
>  create mode 100644 board/armltd/vexpress/vexpress_smp.S
>
> --
> 1.7.12.1
>
> _______________________________________________
> kvmarm mailing list
> kvmarm at lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/kvmarm
>

regards,
Nikolay Nikolaev

[1] https://lists.cs.columbia.edu/pipermail/kvmarm/2013-July/006481.html

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Christoffer Dall]

On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara wrote:

[...]

> 
> Albert, Tom,
> do you need more ACKs or Reviewed-bys?
> 

Albert, Tom,

Can you let us know if you will accept a pull request for these patches?
They look to be in pretty good shape?

Thanks!
-Christoffer

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Tom Rini]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/26/2013 04:51 PM, Christoffer Dall wrote:
> On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara wrote:
> 
> [...]
> 
>> 
>> Albert, Tom, do you need more ACKs or Reviewed-bys?
>> 
> 
> Albert, Tom,
> 
> Can you let us know if you will accept a pull request for these
> patches? They look to be in pretty good shape?

It looks good to me, I just need Albert to pick them up.  Thanks!

- -- 
Tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSG8jmAAoJENk4IS6UOR1WAKoP/3bBH9a8SUy2ELpj2d1O0IJt
022KZVAiCNA31j26rxyqX8L78/0KujOnkaGaT8ge9W8U0Z/pxniiuPaKyLkcfdUi
ak5l3WtLbQRJZk7zOskaIZEwozHfnlzRv7ciXXhOkUbmfwwtog04tGLMKEpTBEpH
vDu65XyLlXwQ5poGasvGuExqrjJ+V7SE0KjYx+CnUEnJSX2tGfSkNXbaGE6s3Bur
0zYwucHQIqu43OuvxIFkHuGPN7oGqwNqj6ejfkcJzw215Zl93Yri0EireSizbfn2
LwXAUb67K6GdYjn2ZjwfXDeVPSjzzUzyYAc2KU1YBecBe6qhSenP8cKObOroeKd2
AWgFEDHi1thpi+ucPiFZDVHr2s7ZPZpJ49jEbFvqDxav98d2xOkcv2l1g2jMXkGt
nf7BFu2h8UlyW1Jn9dt3d8XH1OH0svEzqg/Tu/jpN2CUSahxCuMYOKp182R2O6LC
7GKmctOrHqnZNkgkemVegTRcUyrHS+52O1cHJTmWDVMwt96TsFK/ZpB23lL+04XE
FbhiaU19/vr5rCtG8kDR64Zw4Q2SjzzfpWB0KmS1Pv++ePy161csW7AYzXBgbHy2
ZJRDpOULyPJHtoHkBA1Ro79yNthySQhsz4iIy9Td+NyseOXkH21cWZd2FxM0Cs6m
y0+pEToN9pQVD5WqyPpv
=vFfX
-----END PGP SIGNATURE-----

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Christoffer Dall]

On Mon, Aug 26, 2013 at 05:30:14PM -0400, Tom Rini wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 08/26/2013 04:51 PM, Christoffer Dall wrote:
> > On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara wrote:
> > 
> > [...]
> > 
> >> 
> >> Albert, Tom, do you need more ACKs or Reviewed-bys?
> >> 
> > 
> > Albert, Tom,
> > 
> > Can you let us know if you will accept a pull request for these
> > patches? They look to be in pretty good shape?
> 
> It looks good to me, I just need Albert to pick them up.  Thanks!
> 
Great!  Do you need a pull request or do you just apply the patches from
the mailing list?

Thanks,
-Christoffer

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Tom Rini]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/26/2013 05:46 PM, Christoffer Dall wrote:
> On Mon, Aug 26, 2013 at 05:30:14PM -0400, Tom Rini wrote:
>> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
>> 
>> On 08/26/2013 04:51 PM, Christoffer Dall wrote:
>>> On Fri, Aug 16, 2013 at 03:53:01PM +0200, Andre Przywara
>>> wrote:
>>> 
>>> [...]
>>> 
>>>> 
>>>> Albert, Tom, do you need more ACKs or Reviewed-bys?
>>>> 
>>> 
>>> Albert, Tom,
>>> 
>>> Can you let us know if you will accept a pull request for
>>> these patches? They look to be in pretty good shape?
>> 
>> It looks good to me, I just need Albert to pick them up.
>> Thanks!
>> 
> Great!  Do you need a pull request or do you just apply the patches
> from the mailing list?

Albert will pick them up from patchwork or the mailing list, yes.

- -- 
Tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSG89IAAoJENk4IS6UOR1WJzQQAJE/YUevQ+2RHQt9sg2uNr1G
zludqaM76GG53r6LKVotMZ218ps0lYkRrMQZlIAjHvne1VvD+uXSuPEs6jnSpO0J
YaTdQaauLXiMh486EXVQmlu977AuB7yR31j5jUtB42K55SIZfqx+teMGXAau8OmZ
vbJamILfVLqTv70uk6zkmT63QTOWsQ01pyr4NLMzyZt9p7OqGFUYbbXpomd6byYU
1N/ZjUled2TKIccwp5Gy7+wUQRoHRnepYWxsbwpn2StB8LhXne7WHPiwQK00Dh4l
6NJwD0xK/KWk9TxJiqZZn92OF6nx6rLLsXdrE5Sx0FZ7Te6tHgJg9g3KDL8PTZny
zLlDdDbdi56Jz9yiK5oeBhgSMvQ1nH84BeqINVWrMvb4xiJP/FLrS4cBkCEqpgCl
w/oJ8klkbcFuEr2HiKxOcNDfMUqB/nb3sRnIIgSyuFwShhHm+DzrrNcQNJl0IpBJ
vM6aF3fEZaAthbubL0EH7Nn2WQY0/N34UXD6ZkDL9LVwy4fxVem+6bDOScCjI3de
7d/19o0Bga3siA3a5paXThnLwNKCNmeI81Ra2hBAJrPoDM1t4bGLe8UO3sSMy92Z
C4LTR2ymi0mRXzNHt7rjOr8PsOOz9qbiqtdGXqPyJ1hdlU+Brh0cgRLL94KSs5Xi
U58fWp39ZA3AfvNrJaoV
=CRbO
-----END PGP SIGNATURE-----

[U-Boot] [PATCH v4 2/8] ARM: add secure monitor handler to switch to non-secure state

[Masahiro Yamada]

Hello Andre,

> +/* the vector table for secure state */
> +_monitor_vectors:
> +	.word 0	/* reset */
> +	.word 0 /* undef */
> +	adr pc, _secure_monitor
> +	.word 0
> +	.word 0
> +	.word 0
> +	.word 0
> +	.word 0
> +	.word 0	/* pad */

Could you explain why the last line is needed?


Best Regards
Masahiro Yamada

[U-Boot] [PATCH v4 2/8] ARM: add secure monitor handler to switch to non-secure state

[Andre Przywara]

On 08/27/2013 02:23 AM, Masahiro Yamada wrote:
> Hello Andre,
>
>> +/* the vector table for secure state */
>> +_monitor_vectors:
>> +	.word 0	/* reset */
>> +	.word 0 /* undef */
>> +	adr pc, _secure_monitor
>> +	.word 0
>> +	.word 0
>> +	.word 0
>> +	.word 0
>> +	.word 0
>> +	.word 0	/* pad */
>
> Could you explain why the last line is needed?

I guess I cannot explain because it's probably not needed ;-)
I copied this from somewhere else and accidentally this padding somehow 
sneaked in (from the actual addresses array, where the start vector was 
missing and thus the array ended at 7 * 4 bytes)

Do we need another version or is a follow-up patch sufficient?
I'd like to avoid another review round for the sake of the reviewers.

Thanks for spotting this,
Andre.

[U-Boot] [PATCH v4 2/8] ARM: add secure monitor handler to switch to non-secure state

[Tom Rini]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 08/27/2013 05:51 AM, Andre Przywara wrote:
> On 08/27/2013 02:23 AM, Masahiro Yamada wrote:
>> Hello Andre,
>> 
>>> +/* the vector table for secure state */ +_monitor_vectors: +
>>> .word 0    /* reset */ +    .word 0 /* undef */ +    adr pc,
>>> _secure_monitor +    .word 0 +    .word 0 +    .word 0 +
>>> .word 0 +    .word 0 +    .word 0    /* pad */
>> 
>> Could you explain why the last line is needed?
> 
> I guess I cannot explain because it's probably not needed ;-) I
> copied this from somewhere else and accidentally this padding
> somehow sneaked in (from the actual addresses array, where the
> start vector was missing and thus the array ended at 7 * 4 bytes)
> 
> Do we need another version or is a follow-up patch sufficient? I'd
> like to avoid another review round for the sake of the reviewers.

Lets do a follow-up and Albert can squash it if desired.

- -- 
Tom
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJSHJdtAAoJENk4IS6UOR1WDgcP/1Rz+1U/La9brudy9OyjeVqT
FYdWMtFr/TGELkoaUX/saY/MZoIPBpCO4iW8y5UvQJ3cQAm5WqgPK4/ku3wcbYC6
yaVC91fQ9UQEqzmUBCRGZAGwJuZ+JiElP/9wmqEi7RvnQTYimRYkEpm+XgxRkuX0
l3HH5QLyTO8CbQ4FKm8XeE0JeMfpNqPflMo5oDNMVlWC7a4VXMP49uIGoYt6emPp
d7US4neOZRgL0ANiC/WoV807otl+llFRgZ0zqM7A60th+jJ2YqJIyZlp6QnL2r+Z
YDLlKSvwBliYlmdFWFxxfUoe+4qK27HrOA39PP/Cj5ocQCoEzIRZFTXhlV3AeB9L
CEhXgImSs0BS4AxN7CPdaJLdDGalphthkHOuGP66m+Xb9k7v6gmdNhphusX01FhC
UUgRRKVUT02dS+51tMRxk1wdy805eI6zo34hmvS2SmWWuxAM1kCQhphUl+Ib1ew6
H7V+XoDdOp+iQzABB86TL8J8KtoUgiVA81NqQsvbiW4pR5rUoxBhTTXEYOnZyMYA
qypDCyYepuQCLtz5kxxe0b1Y4tmZsNKtnVuMA55LtPVLHqX1uJ/vXXHBTsO3rGG3
Rz3BTq0o9GVYBg6YLJh496qD8kzqOfnhpu3U+5Fc10H0ldlDcs8YhLU+hzVD+nLD
dDuxa9OzgPsEK/Wc1mWg
=5MkM
-----END PGP SIGNATURE-----

[U-Boot] [PATCH v4 0/8] ARMv7: Add HYP mode switching support

[Albert ARIBAUD]

Hi Andre,

On Fri,  9 Aug 2013 17:03:04 +0200, Andre Przywara
<andre.przywara@linaro.org> wrote:

> (for GIT URL and Changelog see below)

Re: the changelog, I really prefer it when changelogs are per-patch
rather than per-series. Right now, if I want to know the history for,
say, patches 4/8 and 7/8, there is no way I can find it except by
painfully doing git diffs. I would thus suggest that in the future you
use patman, at least for patch series; based on the history you keep in
each individual commit on your branch, it generates both per-patch and
per-series history.

That said, I'll pick the patch series, but I would appreciate if you
could submit a standalone patch to address Christoffer's comment re
4/8 and 7/8.

Amicalement,
-- 
Albert.

[U-Boot] [PATCH v4 8/8] ARM: VExpress: enable ARMv7 virt support for VExpress A15

[Albert ARIBAUD]

Hi Andre,

On Fri,  9 Aug 2013 17:03:12 +0200, Andre Przywara
<andre.przywara@linaro.org> wrote:

> To enable hypervisors utilizing the ARMv7 virtualization extension
> on the Versatile Express board with the A15 core tile, we add the
> required configuration variable.
> Also we define the board specific functions to do the SMP bringup:
> smp_set_cpu_boot_addr() to set the start address for secondary cores
> and smp_waitloop() to wait for IPIs and jump to the start address.
> 
> This also serves as an example for what to do when adding support for
> new boards.
> 
> Signed-off-by: Andre Przywara <andre.przywara@linaro.org>
> ---
>  board/armltd/vexpress/Makefile          |  7 +++++--
>  board/armltd/vexpress/vexpress_common.c | 13 ++++++++++++
>  board/armltd/vexpress/vexpress_smp.S    | 36 +++++++++++++++++++++++++++++++++
>  include/configs/vexpress_ca15_tc2.h     |  4 ++++
>  4 files changed, 58 insertions(+), 2 deletions(-)
>  create mode 100644 board/armltd/vexpress/vexpress_smp.S
> 
> diff --git a/board/armltd/vexpress/Makefile b/board/armltd/vexpress/Makefile
> index 6719f3d..282ef6d 100644
> --- a/board/armltd/vexpress/Makefile
> +++ b/board/armltd/vexpress/Makefile
> @@ -26,9 +26,12 @@ include $(TOPDIR)/config.mk
>  LIB	= $(obj)lib$(BOARD).o
>  
>  COBJS	:= vexpress_common.o
> +ifneq ($(CONFIG_ARMV7_NONSEC)$(CONFIG_ARMV7_VIRT),)
> +SOBJS	:= vexpress_smp.o
> +endif
>  
> -SRCS	:= $(COBJS:.o=.c)
> -OBJS	:= $(addprefix $(obj),$(COBJS))
> +SRCS	:= $(COBJS:.o=.c) $(SOBJS:.o=.S)
> +OBJS	:= $(addprefix $(obj),$(COBJS) $(SOBJS))
>  
>  $(LIB):	$(obj).depend $(OBJS)
>  	$(call cmd_link_o_target, $(OBJS))
> diff --git a/board/armltd/vexpress/vexpress_common.c b/board/armltd/vexpress/vexpress_common.c
> index 2c54869..66b810d 100644
> --- a/board/armltd/vexpress/vexpress_common.c
> +++ b/board/armltd/vexpress/vexpress_common.c
> @@ -272,3 +272,16 @@ ulong get_tbclk(void)
>  {
>  	return (ulong)CONFIG_SYS_HZ;
>  }
> +
> +/* Setting the address at which secondary cores start from.
> + * Versatile Express uses one address for all cores, so ignore corenr
> + */
> +void smp_set_core_boot_addr(unsigned long addr, int corenr)
> +{
> +	/* The SYSFLAGS register on VExpress needs to be cleared first
> +	 * by writing to the next address, since any writes to the address
> +	 * at offset 0 will only be ORed in
> +	 */
> +	writel(~0, CONFIG_SYSFLAGS_ADDR + 4);
> +	writel(addr, CONFIG_SYSFLAGS_ADDR);
> +}
> diff --git a/board/armltd/vexpress/vexpress_smp.S b/board/armltd/vexpress/vexpress_smp.S
> new file mode 100644
> index 0000000..41be2e7
> --- /dev/null
> +++ b/board/armltd/vexpress/vexpress_smp.S
> @@ -0,0 +1,36 @@
> +/*
> + * code for redirecting secondary cores to their start address
> + *
> + * Copyright (c) 2013	Andre Przywara <andre.przywara@linaro.org>
> + *
> + * See file CREDITS for list of people who contributed to this
> + * project.
> + *
> + * This program is free software; you can redistribute it and/or
> + * modify it under the terms of the GNU General Public License as
> + * published by the Free Software Foundation; either version 2 of
> + * the License, or (at your option) any later version.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the
> + * GNU General Public License for more details.
> + *
> + * You should have received a copy of the GNU General Public License
> + * along with this program; if not, write to the Free Software
> + * Foundation, Inc., 59 Temple Place, Suite 330, Boston,
> + * MA 02111-1307 USA
> + */
> +
> +#include <config.h>
> +#include <linux/linkage.h>
> +
> +/* void _smp_waitloop(unsigned previous_address); */
> +ENTRY(smp_waitloop)
> +	wfi
> +	ldr	r1, =CONFIG_SYSFLAGS_ADDR	@ load start address
> +	ldr	r1, [r1]
> +	cmp	r0, r1			@ make sure we dont execute this code
> +	beq	smp_waitloop		@ again (due to a spurious wakeup)
> +	mov	pc, r1
> +ENDPROC(smp_waitloop)
> diff --git a/include/configs/vexpress_ca15_tc2.h b/include/configs/vexpress_ca15_tc2.h
> index 4f425ac..14aa78e 100644
> --- a/include/configs/vexpress_ca15_tc2.h
> +++ b/include/configs/vexpress_ca15_tc2.h
> @@ -31,4 +31,8 @@
>  #include "vexpress_common.h"
>  #define CONFIG_BOOTP_VCI_STRING     "U-boot.armv7.vexpress_ca15x2_tc2"
>  
> +#define CONFIG_SYSFLAGS_ADDR 0x1c010030
> +
> +#define CONFIG_ARMV7_VIRT
> +
>  #endif

This patch breaks vexpress_ca9x4 and vexpress_ca5x2:

vexpress_common.c: In function 'smp_set_core_boot_addr':
vexpress_common.c:269:2: error: 'CONFIG_SYSFLAGS_ADDR' undeclared
(first use in this function) vexpress_common.c:269:2: note: each
undeclared identifier is reported only once for each function it
appears in.

Please provide V5 series addressing this as well as Christoffer's and
Nikolay's comments.

Amicalement,
-- 
Albert.