[PATCH 00/10] CPS SMP fixes & debug improvements

[PATCH 00/10] CPS SMP fixes & debug improvements

[Paul Burton]

This series implements some fixes for the CPS SMP implementation on some
systems, and improves debug by allowing exceptions to be handled &
dumped to an ns16550-compatible UART.

Paul Burton (10):
  MIPS: CPS: set Status.BEV bit during early boot
  MIPS: CPS: set Status.KX on entry for MIPS64 kernels
  MIPS: CPS: early debug using an ns16550-compatible UART
  MIPS: CPS: read CM GCR base from cop0
  MIPS: CPS: skip Config1 presence check
  MIPS: CPS: warn if a core doesn't start
  MIPS: CM: fix GCR_Cx_CONFIG PVPE mask
  MIPS: CM: introduce core-other locking functions
  MIPS: CM: make use of mips_cm_{lock,unlock}_other
  MIPS: CM,CPC: ensure core-other GCRs reflect the correct core

 arch/mips/Kconfig.debug            |  26 +++++
 arch/mips/include/asm/mips-cm.h    |  34 ++++++-
 arch/mips/include/asm/mips-cpc.h   |   3 +-
 arch/mips/include/asm/mipsregs.h   |   3 +
 arch/mips/kernel/Makefile          |   1 +
 arch/mips/kernel/cps-vec-ns16550.S | 202 +++++++++++++++++++++++++++++++++++++
 arch/mips/kernel/cps-vec.S         |  44 ++++++--
 arch/mips/kernel/mips-cm.c         |  45 +++++++++
 arch/mips/kernel/mips-cpc.c        |   6 ++
 arch/mips/kernel/smp-cps.c         |  35 ++++++-
 arch/mips/kernel/smp-gic.c         |   2 +
 11 files changed, 386 insertions(+), 15 deletions(-)
 create mode 100644 arch/mips/kernel/cps-vec-ns16550.S

-- 
2.5.3

[PATCH 00/10] CPS SMP fixes & debug improvements

[Paul Burton]

This series implements some fixes for the CPS SMP implementation on some
systems, and improves debug by allowing exceptions to be handled &
dumped to an ns16550-compatible UART.

Paul Burton (10):
  MIPS: CPS: set Status.BEV bit during early boot
  MIPS: CPS: set Status.KX on entry for MIPS64 kernels
  MIPS: CPS: early debug using an ns16550-compatible UART
  MIPS: CPS: read CM GCR base from cop0
  MIPS: CPS: skip Config1 presence check
  MIPS: CPS: warn if a core doesn't start
  MIPS: CM: fix GCR_Cx_CONFIG PVPE mask
  MIPS: CM: introduce core-other locking functions
  MIPS: CM: make use of mips_cm_{lock,unlock}_other
  MIPS: CM,CPC: ensure core-other GCRs reflect the correct core

 arch/mips/Kconfig.debug            |  26 +++++
 arch/mips/include/asm/mips-cm.h    |  34 ++++++-
 arch/mips/include/asm/mips-cpc.h   |   3 +-
 arch/mips/include/asm/mipsregs.h   |   3 +
 arch/mips/kernel/Makefile          |   1 +
 arch/mips/kernel/cps-vec-ns16550.S | 202 +++++++++++++++++++++++++++++++++++++
 arch/mips/kernel/cps-vec.S         |  44 ++++++--
 arch/mips/kernel/mips-cm.c         |  45 +++++++++
 arch/mips/kernel/mips-cpc.c        |   6 ++
 arch/mips/kernel/smp-cps.c         |  35 ++++++-
 arch/mips/kernel/smp-gic.c         |   2 +
 11 files changed, 386 insertions(+), 15 deletions(-)
 create mode 100644 arch/mips/kernel/cps-vec-ns16550.S

-- 
2.5.3

[PATCH 01/10] MIPS: CPS: set Status.BEV bit during early boot

[Paul Burton]

Set the Status.BEV bit throughout the early startup of a secondary core
such that if an exception occurs the core branches to one of the
exception vector entries from cps-vec.S, rather than branching to
whatever is set in EBase.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 9f71c06..9710db5 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -70,7 +70,7 @@ not_nmi:
 	mtc0	t0, CP0_CAUSE
 
 	/* Setup Status */
-	li	t0, ST0_CU1 | ST0_CU0
+	li	t0, ST0_CU1 | ST0_CU0 | ST0_BEV
 	mtc0	t0, CP0_STATUS
 
 	/*
-- 
2.5.3

[PATCH 01/10] MIPS: CPS: set Status.BEV bit during early boot

[Paul Burton]

Set the Status.BEV bit throughout the early startup of a secondary core
such that if an exception occurs the core branches to one of the
exception vector entries from cps-vec.S, rather than branching to
whatever is set in EBase.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 9f71c06..9710db5 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -70,7 +70,7 @@ not_nmi:
 	mtc0	t0, CP0_CAUSE
 
 	/* Setup Status */
-	li	t0, ST0_CU1 | ST0_CU0
+	li	t0, ST0_CU1 | ST0_CU0 | ST0_BEV
 	mtc0	t0, CP0_STATUS
 
 	/*
-- 
2.5.3

[PATCH 02/10] MIPS: CPS: set Status.KX on entry for MIPS64 kernels

[Paul Burton]

If the kernel may make use of 64 bit addresses outside of the
compatibility address space then we need to set KX such that those
accesses can succeed. Do so for MIPS64 kernels.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 9710db5..99bf6aa 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -25,6 +25,12 @@
 
 .set noreorder
 
+#ifdef CONFIG_64BIT
+# define STATUS_BITDEPS		ST0_KX
+#else
+# define STATUS_BITDEPS		0
+#endif
+
 	/*
 	 * Set dest to non-zero if the core supports the MT ASE, else zero. If
 	 * MT is not supported then branch to nomt.
@@ -70,7 +76,7 @@ not_nmi:
 	mtc0	t0, CP0_CAUSE
 
 	/* Setup Status */
-	li	t0, ST0_CU1 | ST0_CU0 | ST0_BEV
+	li	t0, ST0_CU1 | ST0_CU0 | ST0_BEV | STATUS_BITDEPS
 	mtc0	t0, CP0_STATUS
 
 	/*
-- 
2.5.3

[PATCH 02/10] MIPS: CPS: set Status.KX on entry for MIPS64 kernels

[Paul Burton]

If the kernel may make use of 64 bit addresses outside of the
compatibility address space then we need to set KX such that those
accesses can succeed. Do so for MIPS64 kernels.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 9710db5..99bf6aa 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -25,6 +25,12 @@
 
 .set noreorder
 
+#ifdef CONFIG_64BIT
+# define STATUS_BITDEPS		ST0_KX
+#else
+# define STATUS_BITDEPS		0
+#endif
+
 	/*
 	 * Set dest to non-zero if the core supports the MT ASE, else zero. If
 	 * MT is not supported then branch to nomt.
@@ -70,7 +76,7 @@ not_nmi:
 	mtc0	t0, CP0_CAUSE
 
 	/* Setup Status */
-	li	t0, ST0_CU1 | ST0_CU0 | ST0_BEV
+	li	t0, ST0_CU1 | ST0_CU0 | ST0_BEV | STATUS_BITDEPS
 	mtc0	t0, CP0_STATUS
 
 	/*
-- 
2.5.3

[PATCH 03/10] MIPS: CPS: early debug using an ns16550-compatible UART

[Paul Burton]

Provide support for outputting early debug information, in the form of
various register values should an exception occur, during the early
bringup of secondary cores. This code requires an ns16550-compatible
UART accessible from the secondary core, and is written in assembly due
to the environment in which such early exceptions occur where way may
not have a stack, be coherent or even have initialised caches.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/Kconfig.debug            |  26 +++++
 arch/mips/include/asm/mipsregs.h   |   3 +
 arch/mips/kernel/Makefile          |   1 +
 arch/mips/kernel/cps-vec-ns16550.S | 202 +++++++++++++++++++++++++++++++++++++
 arch/mips/kernel/cps-vec.S         |  20 ++++
 5 files changed, 252 insertions(+)
 create mode 100644 arch/mips/kernel/cps-vec-ns16550.S

diff --git a/arch/mips/Kconfig.debug b/arch/mips/Kconfig.debug
index e250524..7b109cd 100644
--- a/arch/mips/Kconfig.debug
+++ b/arch/mips/Kconfig.debug
@@ -113,4 +113,30 @@ config SPINLOCK_TEST
 	help
 	  Add several files to the debugfs to test spinlock speed.
 
+menuconfig MIPS_CPS_NS16550
+	bool "CPS SMP NS16550 UART output"
+	depends on MIPS_CPS
+	help
+	  Output debug information via an ns16550 compatible UART if exceptions
+	  occur early in the boot process of a secondary core.
+
+if MIPS_CPS_NS16550
+
+config MIPS_CPS_NS16550_BASE
+	hex "UART Base Address"
+	default 0x1b0003f8 if MIPS_MALTA
+	help
+	  The base address of the ns16550 compatible UART on which to output
+	  debug information from the early stages of core startup.
+
+config MIPS_CPS_NS16550_SHIFT
+	int "UART Register Shift"
+	default 0 if MIPS_MALTA
+	help
+	  The number of bits to shift ns16550 register indices by in order to
+	  form their addresses. That is, log base 2 of the span between
+	  adjacent ns16550 registers in the system.
+
+endif # MIPS_CPS_NS16550
+
 endmenu
diff --git a/arch/mips/include/asm/mipsregs.h b/arch/mips/include/asm/mipsregs.h
index d3cd8ea..d626d71 100644
--- a/arch/mips/include/asm/mipsregs.h
+++ b/arch/mips/include/asm/mipsregs.h
@@ -51,6 +51,7 @@
 #define CP0_WIRED $6
 #define CP0_INFO $7
 #define CP0_BADVADDR $8
+#define CP0_BADINSTR $8, 1
 #define CP0_COUNT $9
 #define CP0_ENTRYHI $10
 #define CP0_COMPARE $11
@@ -58,6 +59,8 @@
 #define CP0_CAUSE $13
 #define CP0_EPC $14
 #define CP0_PRID $15
+#define CP0_EBASE $15, 1
+#define CP0_CMGCRBASE $15, 3
 #define CP0_CONFIG $16
 #define CP0_LLADDR $17
 #define CP0_WATCHLO $18
diff --git a/arch/mips/kernel/Makefile b/arch/mips/kernel/Makefile
index d982be1..68e2b7d 100644
--- a/arch/mips/kernel/Makefile
+++ b/arch/mips/kernel/Makefile
@@ -51,6 +51,7 @@ obj-$(CONFIG_MIPS_MT_FPAFF)	+= mips-mt-fpaff.o
 obj-$(CONFIG_MIPS_MT_SMP)	+= smp-mt.o
 obj-$(CONFIG_MIPS_CMP)		+= smp-cmp.o
 obj-$(CONFIG_MIPS_CPS)		+= smp-cps.o cps-vec.o
+obj-$(CONFIG_MIPS_CPS_NS16550)	+= cps-vec-ns16550.o
 obj-$(CONFIG_MIPS_GIC_IPI)	+= smp-gic.o
 obj-$(CONFIG_MIPS_SPRAM)	+= spram.o
 
diff --git a/arch/mips/kernel/cps-vec-ns16550.S b/arch/mips/kernel/cps-vec-ns16550.S
new file mode 100644
index 0000000..6d246ad
--- /dev/null
+++ b/arch/mips/kernel/cps-vec-ns16550.S
@@ -0,0 +1,202 @@
+/*
+ * Copyright (C) 2015 Imagination Technologies
+ * Author: Paul Burton <paul.burton@imgtec.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation;  either version 2 of the  License, or (at your
+ * option) any later version.
+ */
+
+#include <asm/addrspace.h>
+#include <asm/asm.h>
+#include <asm/asm-offsets.h>
+#include <asm/mipsregs.h>
+#include <asm/regdef.h>
+#include <linux/serial_reg.h>
+
+#define UART_TX_OFS	(UART_TX << CONFIG_MIPS_CPS_NS16550_SHIFT)
+#define UART_LSR_OFS	(UART_LSR << CONFIG_MIPS_CPS_NS16550_SHIFT)
+
+/**
+ * _mips_cps_putc() - write a character to the UART
+ * @a0: ASCII character to write
+ * @t9: UART base address
+ */
+LEAF(_mips_cps_putc)
+1:	lw		t0, UART_LSR_OFS(t9)
+	andi		t0, t0, UART_LSR_TEMT
+	beqz		t0, 1b
+	sb		a0, UART_TX_OFS(t9)
+	jr		ra
+	END(_mips_cps_putc)
+
+/**
+ * _mips_cps_puts() - write a string to the UART
+ * @a0: pointer to NULL-terminated ASCII string
+ * @t9: UART base address
+ *
+ * Write a null-terminated ASCII string to the UART.
+ */
+NESTED(_mips_cps_puts, 0, ra)
+	move		s7, ra
+	move		s6, a0
+
+1:	lb		a0, 0(s6)
+	beqz		a0, 2f
+	jal		_mips_cps_putc
+	PTR_ADDIU	s6, s6, 1
+	b		1b
+
+2:	jr		s7
+	END(_mips_cps_puts)
+
+/**
+ * _mips_cps_putx4 - write a 4b hex value to the UART
+ * @a0: the 4b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a single hexadecimal character to the UART.
+ */
+NESTED(_mips_cps_putx4, 0, ra)
+	andi		a0, a0, 0xf
+	li		t0, '0'
+	blt		a0, 10, 1f
+	li		t0, 'a'
+	addiu		a0, a0, -10
+1:	addu		a0, a0, t0
+	b		_mips_cps_putc
+	END(_mips_cps_putx4)
+
+/**
+ * _mips_cps_putx8 - write an 8b hex value to the UART
+ * @a0: the 8b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write an 8 bit value (ie. 2 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx8, 0, ra)
+	move		s3, ra
+	move		s2, a0
+	srl		a0, a0, 4
+	jal		_mips_cps_putx4
+	move		a0, s2
+	move		ra, s3
+	b		_mips_cps_putx4
+	END(_mips_cps_putx8)
+
+/**
+ * _mips_cps_putx16 - write a 16b hex value to the UART
+ * @a0: the 16b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a 16 bit value (ie. 4 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx16, 0, ra)
+	move		s5, ra
+	move		s4, a0
+	srl		a0, a0, 8
+	jal		_mips_cps_putx8
+	move		a0, s4
+	move		ra, s5
+	b		_mips_cps_putx8
+	END(_mips_cps_putx16)
+
+/**
+ * _mips_cps_putx32 - write a 32b hex value to the UART
+ * @a0: the 32b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a 32 bit value (ie. 8 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx32, 0, ra)
+	move		s7, ra
+	move		s6, a0
+	srl		a0, a0, 16
+	jal		_mips_cps_putx16
+	move		a0, s6
+	move		ra, s7
+	b		_mips_cps_putx16
+	END(_mips_cps_putx32)
+
+#ifdef CONFIG_64BIT
+
+/**
+ * _mips_cps_putx64 - write a 64b hex value to the UART
+ * @a0: the 64b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a 64 bit value (ie. 16 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx64, 0, ra)
+	move		sp, ra
+	move		s8, a0
+	dsrl32		a0, a0, 0
+	jal		_mips_cps_putx32
+	move		a0, s8
+	move		ra, sp
+	b		_mips_cps_putx32
+	END(_mips_cps_putx64)
+
+#define _mips_cps_putxlong _mips_cps_putx64
+
+#else /* !CONFIG_64BIT */
+
+#define _mips_cps_putxlong _mips_cps_putx32
+
+#endif /* !CONFIG_64BIT */
+
+/**
+ * mips_cps_bev_dump() - dump relevant exception state to UART
+ * @a0: pointer to NULL-terminated ASCII string naming the exception
+ *
+ * Write information that may be useful in debugging an exception to the
+ * UART configured by CONFIG_MIPS_CPS_NS16550_*. As this BEV exception
+ * will only be run if something goes horribly wrong very early during
+ * the bringup of a core and it is very likely to be unsafe to perform
+ * memory accesses at that point (cache state indeterminate, EVA may not
+ * be configured, coherence may be disabled) let alone have a stack,
+ * this is all written in assembly using only registers & unmapped
+ * uncached access to the UART registers.
+ */
+LEAF(mips_cps_bev_dump)
+	move		s0, ra
+	move		s1, a0
+
+	li		t9, CKSEG1ADDR(CONFIG_MIPS_CPS_NS16550_BASE)
+
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+	PTR_LA		a0, str_bev
+	jal		_mips_cps_puts
+	move		a0, s1
+	jal		_mips_cps_puts
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+
+#define DUMP_COP0_REG(reg, name, sz, _mfc0)		\
+	PTR_LA		a0, 8f;				\
+	jal		_mips_cps_puts;			\
+	_mfc0		a0, reg;			\
+	jal		_mips_cps_putx##sz;		\
+	PTR_LA		a0, str_newline;		\
+	jal		_mips_cps_puts;			\
+	TEXT(name)
+
+	DUMP_COP0_REG(CP0_CAUSE,    "Cause:    0x", 32, mfc0)
+	DUMP_COP0_REG(CP0_STATUS,   "Status:   0x", 32, mfc0)
+	DUMP_COP0_REG(CP0_EBASE,    "EBase:    0x", long, MFC0)
+	DUMP_COP0_REG(CP0_BADVADDR, "BadVAddr: 0x", long, MFC0)
+	DUMP_COP0_REG(CP0_BADINSTR, "BadInstr: 0x", 32, mfc0)
+
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+	jr		s0
+	END(mips_cps_bev_dump)
+
+.pushsection	.data
+str_bev: .asciiz "BEV Exception: "
+str_newline: .asciiz "\r\n"
+.popsection
diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 99bf6aa..a9cc16a 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -31,6 +31,20 @@
 # define STATUS_BITDEPS		0
 #endif
 
+#ifdef CONFIG_MIPS_CPS_NS16550
+
+#define DUMP_EXCEP(name)		\
+	PTR_LA	a0, 8f;			\
+	jal	mips_cps_bev_dump;	\
+	 nop;				\
+	TEXT(name)
+
+#else /* !CONFIG_MIPS_CPS_NS16550 */
+
+#define DUMP_EXCEP(name)
+
+#endif /* !CONFIG_MIPS_CPS_NS16550 */
+
 	/*
 	 * Set dest to non-zero if the core supports the MT ASE, else zero. If
 	 * MT is not supported then branch to nomt.
@@ -193,36 +207,42 @@ dcache_done:
 
 .org 0x200
 LEAF(excep_tlbfill)
+	DUMP_EXCEP("TLB Fill")
 	b	.
 	 nop
 	END(excep_tlbfill)
 
 .org 0x280
 LEAF(excep_xtlbfill)
+	DUMP_EXCEP("XTLB Fill")
 	b	.
 	 nop
 	END(excep_xtlbfill)
 
 .org 0x300
 LEAF(excep_cache)
+	DUMP_EXCEP("Cache")
 	b	.
 	 nop
 	END(excep_cache)
 
 .org 0x380
 LEAF(excep_genex)
+	DUMP_EXCEP("General")
 	b	.
 	 nop
 	END(excep_genex)
 
 .org 0x400
 LEAF(excep_intex)
+	DUMP_EXCEP("Interrupt")
 	b	.
 	 nop
 	END(excep_intex)
 
 .org 0x480
 LEAF(excep_ejtag)
+	DUMP_EXCEP("EJTAG")
 	PTR_LA	k0, ejtag_debug_handler
 	jr	k0
 	 nop
-- 
2.5.3

[PATCH 03/10] MIPS: CPS: early debug using an ns16550-compatible UART

[Paul Burton]

Provide support for outputting early debug information, in the form of
various register values should an exception occur, during the early
bringup of secondary cores. This code requires an ns16550-compatible
UART accessible from the secondary core, and is written in assembly due
to the environment in which such early exceptions occur where way may
not have a stack, be coherent or even have initialised caches.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/Kconfig.debug            |  26 +++++
 arch/mips/include/asm/mipsregs.h   |   3 +
 arch/mips/kernel/Makefile          |   1 +
 arch/mips/kernel/cps-vec-ns16550.S | 202 +++++++++++++++++++++++++++++++++++++
 arch/mips/kernel/cps-vec.S         |  20 ++++
 5 files changed, 252 insertions(+)
 create mode 100644 arch/mips/kernel/cps-vec-ns16550.S

diff --git a/arch/mips/Kconfig.debug b/arch/mips/Kconfig.debug
index e250524..7b109cd 100644
--- a/arch/mips/Kconfig.debug
+++ b/arch/mips/Kconfig.debug
@@ -113,4 +113,30 @@ config SPINLOCK_TEST
 	help
 	  Add several files to the debugfs to test spinlock speed.
 
+menuconfig MIPS_CPS_NS16550
+	bool "CPS SMP NS16550 UART output"
+	depends on MIPS_CPS
+	help
+	  Output debug information via an ns16550 compatible UART if exceptions
+	  occur early in the boot process of a secondary core.
+
+if MIPS_CPS_NS16550
+
+config MIPS_CPS_NS16550_BASE
+	hex "UART Base Address"
+	default 0x1b0003f8 if MIPS_MALTA
+	help
+	  The base address of the ns16550 compatible UART on which to output
+	  debug information from the early stages of core startup.
+
+config MIPS_CPS_NS16550_SHIFT
+	int "UART Register Shift"
+	default 0 if MIPS_MALTA
+	help
+	  The number of bits to shift ns16550 register indices by in order to
+	  form their addresses. That is, log base 2 of the span between
+	  adjacent ns16550 registers in the system.
+
+endif # MIPS_CPS_NS16550
+
 endmenu
diff --git a/arch/mips/include/asm/mipsregs.h b/arch/mips/include/asm/mipsregs.h
index d3cd8ea..d626d71 100644
--- a/arch/mips/include/asm/mipsregs.h
+++ b/arch/mips/include/asm/mipsregs.h
@@ -51,6 +51,7 @@
 #define CP0_WIRED $6
 #define CP0_INFO $7
 #define CP0_BADVADDR $8
+#define CP0_BADINSTR $8, 1
 #define CP0_COUNT $9
 #define CP0_ENTRYHI $10
 #define CP0_COMPARE $11
@@ -58,6 +59,8 @@
 #define CP0_CAUSE $13
 #define CP0_EPC $14
 #define CP0_PRID $15
+#define CP0_EBASE $15, 1
+#define CP0_CMGCRBASE $15, 3
 #define CP0_CONFIG $16
 #define CP0_LLADDR $17
 #define CP0_WATCHLO $18
diff --git a/arch/mips/kernel/Makefile b/arch/mips/kernel/Makefile
index d982be1..68e2b7d 100644
--- a/arch/mips/kernel/Makefile
+++ b/arch/mips/kernel/Makefile
@@ -51,6 +51,7 @@ obj-$(CONFIG_MIPS_MT_FPAFF)	+= mips-mt-fpaff.o
 obj-$(CONFIG_MIPS_MT_SMP)	+= smp-mt.o
 obj-$(CONFIG_MIPS_CMP)		+= smp-cmp.o
 obj-$(CONFIG_MIPS_CPS)		+= smp-cps.o cps-vec.o
+obj-$(CONFIG_MIPS_CPS_NS16550)	+= cps-vec-ns16550.o
 obj-$(CONFIG_MIPS_GIC_IPI)	+= smp-gic.o
 obj-$(CONFIG_MIPS_SPRAM)	+= spram.o
 
diff --git a/arch/mips/kernel/cps-vec-ns16550.S b/arch/mips/kernel/cps-vec-ns16550.S
new file mode 100644
index 0000000..6d246ad
--- /dev/null
+++ b/arch/mips/kernel/cps-vec-ns16550.S
@@ -0,0 +1,202 @@
+/*
+ * Copyright (C) 2015 Imagination Technologies
+ * Author: Paul Burton <paul.burton@imgtec.com>
+ *
+ * This program is free software; you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License as published by the
+ * Free Software Foundation;  either version 2 of the  License, or (at your
+ * option) any later version.
+ */
+
+#include <asm/addrspace.h>
+#include <asm/asm.h>
+#include <asm/asm-offsets.h>
+#include <asm/mipsregs.h>
+#include <asm/regdef.h>
+#include <linux/serial_reg.h>
+
+#define UART_TX_OFS	(UART_TX << CONFIG_MIPS_CPS_NS16550_SHIFT)
+#define UART_LSR_OFS	(UART_LSR << CONFIG_MIPS_CPS_NS16550_SHIFT)
+
+/**
+ * _mips_cps_putc() - write a character to the UART
+ * @a0: ASCII character to write
+ * @t9: UART base address
+ */
+LEAF(_mips_cps_putc)
+1:	lw		t0, UART_LSR_OFS(t9)
+	andi		t0, t0, UART_LSR_TEMT
+	beqz		t0, 1b
+	sb		a0, UART_TX_OFS(t9)
+	jr		ra
+	END(_mips_cps_putc)
+
+/**
+ * _mips_cps_puts() - write a string to the UART
+ * @a0: pointer to NULL-terminated ASCII string
+ * @t9: UART base address
+ *
+ * Write a null-terminated ASCII string to the UART.
+ */
+NESTED(_mips_cps_puts, 0, ra)
+	move		s7, ra
+	move		s6, a0
+
+1:	lb		a0, 0(s6)
+	beqz		a0, 2f
+	jal		_mips_cps_putc
+	PTR_ADDIU	s6, s6, 1
+	b		1b
+
+2:	jr		s7
+	END(_mips_cps_puts)
+
+/**
+ * _mips_cps_putx4 - write a 4b hex value to the UART
+ * @a0: the 4b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a single hexadecimal character to the UART.
+ */
+NESTED(_mips_cps_putx4, 0, ra)
+	andi		a0, a0, 0xf
+	li		t0, '0'
+	blt		a0, 10, 1f
+	li		t0, 'a'
+	addiu		a0, a0, -10
+1:	addu		a0, a0, t0
+	b		_mips_cps_putc
+	END(_mips_cps_putx4)
+
+/**
+ * _mips_cps_putx8 - write an 8b hex value to the UART
+ * @a0: the 8b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write an 8 bit value (ie. 2 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx8, 0, ra)
+	move		s3, ra
+	move		s2, a0
+	srl		a0, a0, 4
+	jal		_mips_cps_putx4
+	move		a0, s2
+	move		ra, s3
+	b		_mips_cps_putx4
+	END(_mips_cps_putx8)
+
+/**
+ * _mips_cps_putx16 - write a 16b hex value to the UART
+ * @a0: the 16b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a 16 bit value (ie. 4 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx16, 0, ra)
+	move		s5, ra
+	move		s4, a0
+	srl		a0, a0, 8
+	jal		_mips_cps_putx8
+	move		a0, s4
+	move		ra, s5
+	b		_mips_cps_putx8
+	END(_mips_cps_putx16)
+
+/**
+ * _mips_cps_putx32 - write a 32b hex value to the UART
+ * @a0: the 32b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a 32 bit value (ie. 8 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx32, 0, ra)
+	move		s7, ra
+	move		s6, a0
+	srl		a0, a0, 16
+	jal		_mips_cps_putx16
+	move		a0, s6
+	move		ra, s7
+	b		_mips_cps_putx16
+	END(_mips_cps_putx32)
+
+#ifdef CONFIG_64BIT
+
+/**
+ * _mips_cps_putx64 - write a 64b hex value to the UART
+ * @a0: the 64b value to write to the UART
+ * @t9: UART base address
+ *
+ * Write a 64 bit value (ie. 16 hexadecimal characters) to the UART.
+ */
+NESTED(_mips_cps_putx64, 0, ra)
+	move		sp, ra
+	move		s8, a0
+	dsrl32		a0, a0, 0
+	jal		_mips_cps_putx32
+	move		a0, s8
+	move		ra, sp
+	b		_mips_cps_putx32
+	END(_mips_cps_putx64)
+
+#define _mips_cps_putxlong _mips_cps_putx64
+
+#else /* !CONFIG_64BIT */
+
+#define _mips_cps_putxlong _mips_cps_putx32
+
+#endif /* !CONFIG_64BIT */
+
+/**
+ * mips_cps_bev_dump() - dump relevant exception state to UART
+ * @a0: pointer to NULL-terminated ASCII string naming the exception
+ *
+ * Write information that may be useful in debugging an exception to the
+ * UART configured by CONFIG_MIPS_CPS_NS16550_*. As this BEV exception
+ * will only be run if something goes horribly wrong very early during
+ * the bringup of a core and it is very likely to be unsafe to perform
+ * memory accesses at that point (cache state indeterminate, EVA may not
+ * be configured, coherence may be disabled) let alone have a stack,
+ * this is all written in assembly using only registers & unmapped
+ * uncached access to the UART registers.
+ */
+LEAF(mips_cps_bev_dump)
+	move		s0, ra
+	move		s1, a0
+
+	li		t9, CKSEG1ADDR(CONFIG_MIPS_CPS_NS16550_BASE)
+
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+	PTR_LA		a0, str_bev
+	jal		_mips_cps_puts
+	move		a0, s1
+	jal		_mips_cps_puts
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+
+#define DUMP_COP0_REG(reg, name, sz, _mfc0)		\
+	PTR_LA		a0, 8f;				\
+	jal		_mips_cps_puts;			\
+	_mfc0		a0, reg;			\
+	jal		_mips_cps_putx##sz;		\
+	PTR_LA		a0, str_newline;		\
+	jal		_mips_cps_puts;			\
+	TEXT(name)
+
+	DUMP_COP0_REG(CP0_CAUSE,    "Cause:    0x", 32, mfc0)
+	DUMP_COP0_REG(CP0_STATUS,   "Status:   0x", 32, mfc0)
+	DUMP_COP0_REG(CP0_EBASE,    "EBase:    0x", long, MFC0)
+	DUMP_COP0_REG(CP0_BADVADDR, "BadVAddr: 0x", long, MFC0)
+	DUMP_COP0_REG(CP0_BADINSTR, "BadInstr: 0x", 32, mfc0)
+
+	PTR_LA		a0, str_newline
+	jal		_mips_cps_puts
+	jr		s0
+	END(mips_cps_bev_dump)
+
+.pushsection	.data
+str_bev: .asciiz "BEV Exception: "
+str_newline: .asciiz "\r\n"
+.popsection
diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 99bf6aa..a9cc16a 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -31,6 +31,20 @@
 # define STATUS_BITDEPS		0
 #endif
 
+#ifdef CONFIG_MIPS_CPS_NS16550
+
+#define DUMP_EXCEP(name)		\
+	PTR_LA	a0, 8f;			\
+	jal	mips_cps_bev_dump;	\
+	 nop;				\
+	TEXT(name)
+
+#else /* !CONFIG_MIPS_CPS_NS16550 */
+
+#define DUMP_EXCEP(name)
+
+#endif /* !CONFIG_MIPS_CPS_NS16550 */
+
 	/*
 	 * Set dest to non-zero if the core supports the MT ASE, else zero. If
 	 * MT is not supported then branch to nomt.
@@ -193,36 +207,42 @@ dcache_done:
 
 .org 0x200
 LEAF(excep_tlbfill)
+	DUMP_EXCEP("TLB Fill")
 	b	.
 	 nop
 	END(excep_tlbfill)
 
 .org 0x280
 LEAF(excep_xtlbfill)
+	DUMP_EXCEP("XTLB Fill")
 	b	.
 	 nop
 	END(excep_xtlbfill)
 
 .org 0x300
 LEAF(excep_cache)
+	DUMP_EXCEP("Cache")
 	b	.
 	 nop
 	END(excep_cache)
 
 .org 0x380
 LEAF(excep_genex)
+	DUMP_EXCEP("General")
 	b	.
 	 nop
 	END(excep_genex)
 
 .org 0x400
 LEAF(excep_intex)
+	DUMP_EXCEP("Interrupt")
 	b	.
 	 nop
 	END(excep_intex)
 
 .org 0x480
 LEAF(excep_ejtag)
+	DUMP_EXCEP("EJTAG")
 	PTR_LA	k0, ejtag_debug_handler
 	jr	k0
 	 nop
-- 
2.5.3

[PATCH 04/10] MIPS: CPS: read CM GCR base from cop0

[Paul Burton]

Rather than patching the start of mips_cps_core_entry to provide the
base address of the CM GCRs, simply read that base address from the cop0
CMGCRBase register, converting from the physical address to an uncached
virtual address.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 12 ++++++++----
 arch/mips/kernel/smp-cps.c |  2 --
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index a9cc16a..5407ce4 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -66,11 +66,9 @@
 
 LEAF(mips_cps_core_entry)
 	/*
-	 * These first 12 bytes will be patched by cps_smp_setup to load the
-	 * base address of the CM GCRs into register v1 and the CCA to use into
-	 * register s0.
+	 * These first 4 bytes will be patched by cps_smp_setup to load the
+	 * CCA to use into register s0.
 	 */
-	.quad	0
 	.word	0
 
 	/* Check whether we're here due to an NMI */
@@ -170,6 +168,12 @@ dcache_done:
 	mtc0	t0, CP0_CONFIG
 	ehb
 
+	/* Calculate an uncached address for the CM GCRs */
+	MFC0	v1, CP0_CMGCRBASE
+	PTR_SLL	v1, v1, 4
+	PTR_LI	t0, UNCAC_BASE
+	PTR_ADDU v1, v1, t0
+
 	/* Enter the coherent domain */
 	li	t0, 0xff
 	sw	t0, GCR_CL_COHERENCE_OFS(v1)
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index c889377..8b96750 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -133,11 +133,9 @@ static void __init cps_prepare_cpus(unsigned int max_cpus)
 	/*
 	 * Patch the start of mips_cps_core_entry to provide:
 	 *
-	 * v1 = CM base address
 	 * s0 = kseg0 CCA
 	 */
 	entry_code = (u32 *)&mips_cps_core_entry;
-	UASM_i_LA(&entry_code, 3, (long)mips_cm_base);
 	uasm_i_addiu(&entry_code, 16, 0, cca);
 	blast_dcache_range((unsigned long)&mips_cps_core_entry,
 			   (unsigned long)entry_code);
-- 
2.5.3

[PATCH 04/10] MIPS: CPS: read CM GCR base from cop0

[Paul Burton]

Rather than patching the start of mips_cps_core_entry to provide the
base address of the CM GCRs, simply read that base address from the cop0
CMGCRBase register, converting from the physical address to an uncached
virtual address.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 12 ++++++++----
 arch/mips/kernel/smp-cps.c |  2 --
 2 files changed, 8 insertions(+), 6 deletions(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index a9cc16a..5407ce4 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -66,11 +66,9 @@
 
 LEAF(mips_cps_core_entry)
 	/*
-	 * These first 12 bytes will be patched by cps_smp_setup to load the
-	 * base address of the CM GCRs into register v1 and the CCA to use into
-	 * register s0.
+	 * These first 4 bytes will be patched by cps_smp_setup to load the
+	 * CCA to use into register s0.
 	 */
-	.quad	0
 	.word	0
 
 	/* Check whether we're here due to an NMI */
@@ -170,6 +168,12 @@ dcache_done:
 	mtc0	t0, CP0_CONFIG
 	ehb
 
+	/* Calculate an uncached address for the CM GCRs */
+	MFC0	v1, CP0_CMGCRBASE
+	PTR_SLL	v1, v1, 4
+	PTR_LI	t0, UNCAC_BASE
+	PTR_ADDU v1, v1, t0
+
 	/* Enter the coherent domain */
 	li	t0, 0xff
 	sw	t0, GCR_CL_COHERENCE_OFS(v1)
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index c889377..8b96750 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -133,11 +133,9 @@ static void __init cps_prepare_cpus(unsigned int max_cpus)
 	/*
 	 * Patch the start of mips_cps_core_entry to provide:
 	 *
-	 * v1 = CM base address
 	 * s0 = kseg0 CCA
 	 */
 	entry_code = (u32 *)&mips_cps_core_entry;
-	UASM_i_LA(&entry_code, 3, (long)mips_cm_base);
 	uasm_i_addiu(&entry_code, 16, 0, cca);
 	blast_dcache_range((unsigned long)&mips_cps_core_entry,
 			   (unsigned long)entry_code);
-- 
2.5.3

[PATCH 05/10] MIPS: CPS: skip Config1 presence check

[Paul Burton]

The Config1 register is architecturally defined as required, and is thus
present in all systems which may make use of cps-vec.S. Skip the check
for its presence via the Config.M bit.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 5407ce4..84af91a 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -50,9 +50,7 @@
 	 * MT is not supported then branch to nomt.
 	 */
 	.macro	has_mt	dest, nomt
-	mfc0	\dest, CP0_CONFIG
-	bgez	\dest, \nomt
-	 mfc0	\dest, CP0_CONFIG, 1
+	mfc0	\dest, CP0_CONFIG, 1
 	bgez	\dest, \nomt
 	 mfc0	\dest, CP0_CONFIG, 2
 	bgez	\dest, \nomt
-- 
2.5.3

[PATCH 05/10] MIPS: CPS: skip Config1 presence check

[Paul Burton]

The Config1 register is architecturally defined as required, and is thus
present in all systems which may make use of cps-vec.S. Skip the check
for its presence via the Config.M bit.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/cps-vec.S | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/arch/mips/kernel/cps-vec.S b/arch/mips/kernel/cps-vec.S
index 5407ce4..84af91a 100644
--- a/arch/mips/kernel/cps-vec.S
+++ b/arch/mips/kernel/cps-vec.S
@@ -50,9 +50,7 @@
 	 * MT is not supported then branch to nomt.
 	 */
 	.macro	has_mt	dest, nomt
-	mfc0	\dest, CP0_CONFIG
-	bgez	\dest, \nomt
-	 mfc0	\dest, CP0_CONFIG, 1
+	mfc0	\dest, CP0_CONFIG, 1
 	bgez	\dest, \nomt
 	 mfc0	\dest, CP0_CONFIG, 2
 	bgez	\dest, \nomt
-- 
2.5.3

[PATCH 06/10] MIPS: CPS: warn if a core doesn't start

[Paul Burton]

When debugging core bringup it is useful to see the state of the CPC
sequencer, so output that value if the core hasn't started within a
reasonable amount of time (1 second). This avoids simply appearing to
the user to hang if a secondary core fails to start.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/smp-cps.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 8b96750..48b1b75 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -8,6 +8,7 @@
  * option) any later version.
  */
 
+#include <linux/delay.h>
 #include <linux/io.h>
 #include <linux/irqchip/mips-gic.h>
 #include <linux/sched.h>
@@ -188,7 +189,8 @@ err_out:
 
 static void boot_core(unsigned core)
 {
-	u32 access;
+	u32 access, stat, seq_state;
+	unsigned timeout;
 
 	/* Select the appropriate core */
 	write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF);
@@ -208,6 +210,28 @@ static void boot_core(unsigned core)
 		/* Reset the core */
 		mips_cpc_lock_other(core);
 		write_cpc_co_cmd(CPC_Cx_CMD_RESET);
+
+		timeout = 100;
+		while (true) {
+			stat = read_cpc_co_stat_conf();
+			seq_state = stat & CPC_Cx_STAT_CONF_SEQSTATE_MSK;
+
+			/* U6 == coherent execution, ie. the core is up */
+			if (seq_state == CPC_Cx_STAT_CONF_SEQSTATE_U6)
+				break;
+
+			/* Delay a little while before we start warning */
+			if (timeout) {
+				timeout--;
+				mdelay(10);
+				continue;
+			}
+
+			pr_warn("Waiting for core %u to start... STAT_CONF=0x%x\n",
+				core, stat);
+			mdelay(1000);
+		}
+
 		mips_cpc_unlock_other();
 	} else {
 		/* Take the core out of reset */
-- 
2.5.3

[PATCH 06/10] MIPS: CPS: warn if a core doesn't start

[Paul Burton]

When debugging core bringup it is useful to see the state of the CPC
sequencer, so output that value if the core hasn't started within a
reasonable amount of time (1 second). This avoids simply appearing to
the user to hang if a secondary core fails to start.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/smp-cps.c | 26 +++++++++++++++++++++++++-
 1 file changed, 25 insertions(+), 1 deletion(-)

diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 8b96750..48b1b75 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -8,6 +8,7 @@
  * option) any later version.
  */
 
+#include <linux/delay.h>
 #include <linux/io.h>
 #include <linux/irqchip/mips-gic.h>
 #include <linux/sched.h>
@@ -188,7 +189,8 @@ err_out:
 
 static void boot_core(unsigned core)
 {
-	u32 access;
+	u32 access, stat, seq_state;
+	unsigned timeout;
 
 	/* Select the appropriate core */
 	write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF);
@@ -208,6 +210,28 @@ static void boot_core(unsigned core)
 		/* Reset the core */
 		mips_cpc_lock_other(core);
 		write_cpc_co_cmd(CPC_Cx_CMD_RESET);
+
+		timeout = 100;
+		while (true) {
+			stat = read_cpc_co_stat_conf();
+			seq_state = stat & CPC_Cx_STAT_CONF_SEQSTATE_MSK;
+
+			/* U6 == coherent execution, ie. the core is up */
+			if (seq_state == CPC_Cx_STAT_CONF_SEQSTATE_U6)
+				break;
+
+			/* Delay a little while before we start warning */
+			if (timeout) {
+				timeout--;
+				mdelay(10);
+				continue;
+			}
+
+			pr_warn("Waiting for core %u to start... STAT_CONF=0x%x\n",
+				core, stat);
+			mdelay(1000);
+		}
+
 		mips_cpc_unlock_other();
 	} else {
 		/* Take the core out of reset */
-- 
2.5.3

[PATCH 07/10] MIPS: CM: fix GCR_Cx_CONFIG PVPE mask

[Paul Burton]

The PVPE (or PVP in >= CM3) field is 10 bits wide, but the mask
previously only covered the bottom 9 bits. Extend the mask to cover all
10 bits of the field.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/include/asm/mips-cm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/mips-cm.h b/arch/mips/include/asm/mips-cm.h
index d75b75e..2796424 100644
--- a/arch/mips/include/asm/mips-cm.h
+++ b/arch/mips/include/asm/mips-cm.h
@@ -324,7 +324,7 @@ BUILD_CM_Cx_R_(tcid_8_priority,	0x80)
 #define CM_GCR_Cx_CONFIG_IOCUTYPE_SHF		10
 #define CM_GCR_Cx_CONFIG_IOCUTYPE_MSK		(_ULCAST_(0x3) << 10)
 #define CM_GCR_Cx_CONFIG_PVPE_SHF		0
-#define CM_GCR_Cx_CONFIG_PVPE_MSK		(_ULCAST_(0x1ff) << 0)
+#define CM_GCR_Cx_CONFIG_PVPE_MSK		(_ULCAST_(0x3ff) << 0)
 
 /* GCR_Cx_OTHER register fields */
 #define CM_GCR_Cx_OTHER_CORENUM_SHF		16
-- 
2.5.3

[PATCH 07/10] MIPS: CM: fix GCR_Cx_CONFIG PVPE mask

[Paul Burton]

The PVPE (or PVP in >= CM3) field is 10 bits wide, but the mask
previously only covered the bottom 9 bits. Extend the mask to cover all
10 bits of the field.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/include/asm/mips-cm.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/mips/include/asm/mips-cm.h b/arch/mips/include/asm/mips-cm.h
index d75b75e..2796424 100644
--- a/arch/mips/include/asm/mips-cm.h
+++ b/arch/mips/include/asm/mips-cm.h
@@ -324,7 +324,7 @@ BUILD_CM_Cx_R_(tcid_8_priority,	0x80)
 #define CM_GCR_Cx_CONFIG_IOCUTYPE_SHF		10
 #define CM_GCR_Cx_CONFIG_IOCUTYPE_MSK		(_ULCAST_(0x3) << 10)
 #define CM_GCR_Cx_CONFIG_PVPE_SHF		0
-#define CM_GCR_Cx_CONFIG_PVPE_MSK		(_ULCAST_(0x1ff) << 0)
+#define CM_GCR_Cx_CONFIG_PVPE_MSK		(_ULCAST_(0x3ff) << 0)
 
 /* GCR_Cx_OTHER register fields */
 #define CM_GCR_Cx_OTHER_CORENUM_SHF		16
-- 
2.5.3

[PATCH 08/10] MIPS: CM: introduce core-other locking functions

[Paul Burton]

Introduce mips_cm_lock_other & mips_cm_unlock_other, mirroring the
existing CPC equivalents, in order to lock access from the current core
to another via the core-other GCR region. This hasn't been required in
the past but with CM3 the CPC starts using GCR_CL_OTHER rather than
CPC_CL_OTHER and this will be required for safety.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/include/asm/mips-cm.h | 32 ++++++++++++++++++++++++++++++++
 arch/mips/kernel/mips-cm.c      | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)

diff --git a/arch/mips/include/asm/mips-cm.h b/arch/mips/include/asm/mips-cm.h
index 2796424..29e6b9d 100644
--- a/arch/mips/include/asm/mips-cm.h
+++ b/arch/mips/include/asm/mips-cm.h
@@ -329,6 +329,10 @@ BUILD_CM_Cx_R_(tcid_8_priority,	0x80)
 /* GCR_Cx_OTHER register fields */
 #define CM_GCR_Cx_OTHER_CORENUM_SHF		16
 #define CM_GCR_Cx_OTHER_CORENUM_MSK		(_ULCAST_(0xffff) << 16)
+#define CM3_GCR_Cx_OTHER_CORE_SHF		8
+#define CM3_GCR_Cx_OTHER_CORE_MSK		(_ULCAST_(0x3f) << 8)
+#define CM3_GCR_Cx_OTHER_VP_SHF			0
+#define CM3_GCR_Cx_OTHER_VP_MSK			(_ULCAST_(0x7) << 0)
 
 /* GCR_Cx_RESET_BASE register fields */
 #define CM_GCR_Cx_RESET_BASE_BEVEXCBASE_SHF	12
@@ -405,4 +409,32 @@ static inline int mips_cm_revision(void)
 	return read_gcr_rev();
 }
 
+#ifdef CONFIG_MIPS_CM
+
+/**
+ * mips_cm_lock_other - lock access to another core
+ * @core: the other core to be accessed
+ * @vp: the VP within the other core to be accessed
+ *
+ * Call before operating upon a core via the 'other' register region in
+ * order to prevent the region being moved during access. Must be followed
+ * by a call to mips_cm_unlock_other.
+ */
+extern void mips_cm_lock_other(unsigned int core, unsigned int vp);
+
+/**
+ * mips_cm_unlock_other - unlock access to another core
+ *
+ * Call after operating upon another core via the 'other' register region.
+ * Must be called after mips_cm_lock_other.
+ */
+extern void mips_cm_unlock_other(void);
+
+#else /* !CONFIG_MIPS_CM */
+
+static inline void mips_cm_lock_other(unsigned int core) { }
+static inline void mips_cm_unlock_other(void) { }
+
+#endif /* !CONFIG_MIPS_CM */
+
 #endif /* __MIPS_ASM_MIPS_CM_H__ */
diff --git a/arch/mips/kernel/mips-cm.c b/arch/mips/kernel/mips-cm.c
index b8ceee5..fef2647 100644
--- a/arch/mips/kernel/mips-cm.c
+++ b/arch/mips/kernel/mips-cm.c
@@ -9,6 +9,8 @@
  */
 
 #include <linux/errno.h>
+#include <linux/percpu.h>
+#include <linux/spinlock.h>
 
 #include <asm/mips-cm.h>
 #include <asm/mipsregs.h>
@@ -136,6 +138,9 @@ static char *cm3_causes[32] = {
 	"0x19", "0x1a", "0x1b", "0x1c", "0x1d", "0x1e", "0x1f"
 };
 
+static DEFINE_PER_CPU_ALIGNED(spinlock_t, cm_core_lock);
+static DEFINE_PER_CPU_ALIGNED(unsigned long, cm_core_lock_flags);
+
 phys_addr_t __mips_cm_phys_base(void)
 {
 	u32 config3 = read_c0_config3();
@@ -200,6 +205,7 @@ int mips_cm_probe(void)
 {
 	phys_addr_t addr;
 	u32 base_reg;
+	unsigned cpu;
 
 	/*
 	 * No need to probe again if we have already been
@@ -247,9 +253,42 @@ int mips_cm_probe(void)
 	/* determine register width for this CM */
 	mips_cm_is64 = config_enabled(CONFIG_64BIT) && (mips_cm_revision() >= CM_REV_CM3);
 
+	for_each_possible_cpu(cpu)
+		spin_lock_init(&per_cpu(cm_core_lock, cpu));
+
 	return 0;
 }
 
+void mips_cm_lock_other(unsigned int core, unsigned int vp)
+{
+	unsigned curr_core;
+	u32 val;
+
+	preempt_disable();
+	curr_core = current_cpu_data.core;
+	spin_lock_irqsave(&per_cpu(cm_core_lock, curr_core),
+			  per_cpu(cm_core_lock_flags, curr_core));
+
+	if (mips_cm_revision() >= CM_REV_CM3) {
+		val = core << CM3_GCR_Cx_OTHER_CORE_SHF;
+		val |= vp << CM3_GCR_Cx_OTHER_VP_SHF;
+	} else {
+		BUG_ON(vp != 0);
+		val = core << CM_GCR_Cx_OTHER_CORENUM_SHF;
+	}
+
+	write_gcr_cl_other(val);
+}
+
+void mips_cm_unlock_other(void)
+{
+	unsigned curr_core = current_cpu_data.core;
+
+	spin_unlock_irqrestore(&per_cpu(cm_core_lock, curr_core),
+			       per_cpu(cm_core_lock_flags, curr_core));
+	preempt_enable();
+}
+
 void mips_cm_error_report(void)
 {
 	unsigned long revision = mips_cm_revision();
-- 
2.5.3

[PATCH 08/10] MIPS: CM: introduce core-other locking functions

[Paul Burton]

Introduce mips_cm_lock_other & mips_cm_unlock_other, mirroring the
existing CPC equivalents, in order to lock access from the current core
to another via the core-other GCR region. This hasn't been required in
the past but with CM3 the CPC starts using GCR_CL_OTHER rather than
CPC_CL_OTHER and this will be required for safety.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/include/asm/mips-cm.h | 32 ++++++++++++++++++++++++++++++++
 arch/mips/kernel/mips-cm.c      | 39 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 71 insertions(+)

diff --git a/arch/mips/include/asm/mips-cm.h b/arch/mips/include/asm/mips-cm.h
index 2796424..29e6b9d 100644
--- a/arch/mips/include/asm/mips-cm.h
+++ b/arch/mips/include/asm/mips-cm.h
@@ -329,6 +329,10 @@ BUILD_CM_Cx_R_(tcid_8_priority,	0x80)
 /* GCR_Cx_OTHER register fields */
 #define CM_GCR_Cx_OTHER_CORENUM_SHF		16
 #define CM_GCR_Cx_OTHER_CORENUM_MSK		(_ULCAST_(0xffff) << 16)
+#define CM3_GCR_Cx_OTHER_CORE_SHF		8
+#define CM3_GCR_Cx_OTHER_CORE_MSK		(_ULCAST_(0x3f) << 8)
+#define CM3_GCR_Cx_OTHER_VP_SHF			0
+#define CM3_GCR_Cx_OTHER_VP_MSK			(_ULCAST_(0x7) << 0)
 
 /* GCR_Cx_RESET_BASE register fields */
 #define CM_GCR_Cx_RESET_BASE_BEVEXCBASE_SHF	12
@@ -405,4 +409,32 @@ static inline int mips_cm_revision(void)
 	return read_gcr_rev();
 }
 
+#ifdef CONFIG_MIPS_CM
+
+/**
+ * mips_cm_lock_other - lock access to another core
+ * @core: the other core to be accessed
+ * @vp: the VP within the other core to be accessed
+ *
+ * Call before operating upon a core via the 'other' register region in
+ * order to prevent the region being moved during access. Must be followed
+ * by a call to mips_cm_unlock_other.
+ */
+extern void mips_cm_lock_other(unsigned int core, unsigned int vp);
+
+/**
+ * mips_cm_unlock_other - unlock access to another core
+ *
+ * Call after operating upon another core via the 'other' register region.
+ * Must be called after mips_cm_lock_other.
+ */
+extern void mips_cm_unlock_other(void);
+
+#else /* !CONFIG_MIPS_CM */
+
+static inline void mips_cm_lock_other(unsigned int core) { }
+static inline void mips_cm_unlock_other(void) { }
+
+#endif /* !CONFIG_MIPS_CM */
+
 #endif /* __MIPS_ASM_MIPS_CM_H__ */
diff --git a/arch/mips/kernel/mips-cm.c b/arch/mips/kernel/mips-cm.c
index b8ceee5..fef2647 100644
--- a/arch/mips/kernel/mips-cm.c
+++ b/arch/mips/kernel/mips-cm.c
@@ -9,6 +9,8 @@
  */
 
 #include <linux/errno.h>
+#include <linux/percpu.h>
+#include <linux/spinlock.h>
 
 #include <asm/mips-cm.h>
 #include <asm/mipsregs.h>
@@ -136,6 +138,9 @@ static char *cm3_causes[32] = {
 	"0x19", "0x1a", "0x1b", "0x1c", "0x1d", "0x1e", "0x1f"
 };
 
+static DEFINE_PER_CPU_ALIGNED(spinlock_t, cm_core_lock);
+static DEFINE_PER_CPU_ALIGNED(unsigned long, cm_core_lock_flags);
+
 phys_addr_t __mips_cm_phys_base(void)
 {
 	u32 config3 = read_c0_config3();
@@ -200,6 +205,7 @@ int mips_cm_probe(void)
 {
 	phys_addr_t addr;
 	u32 base_reg;
+	unsigned cpu;
 
 	/*
 	 * No need to probe again if we have already been
@@ -247,9 +253,42 @@ int mips_cm_probe(void)
 	/* determine register width for this CM */
 	mips_cm_is64 = config_enabled(CONFIG_64BIT) && (mips_cm_revision() >= CM_REV_CM3);
 
+	for_each_possible_cpu(cpu)
+		spin_lock_init(&per_cpu(cm_core_lock, cpu));
+
 	return 0;
 }
 
+void mips_cm_lock_other(unsigned int core, unsigned int vp)
+{
+	unsigned curr_core;
+	u32 val;
+
+	preempt_disable();
+	curr_core = current_cpu_data.core;
+	spin_lock_irqsave(&per_cpu(cm_core_lock, curr_core),
+			  per_cpu(cm_core_lock_flags, curr_core));
+
+	if (mips_cm_revision() >= CM_REV_CM3) {
+		val = core << CM3_GCR_Cx_OTHER_CORE_SHF;
+		val |= vp << CM3_GCR_Cx_OTHER_VP_SHF;
+	} else {
+		BUG_ON(vp != 0);
+		val = core << CM_GCR_Cx_OTHER_CORENUM_SHF;
+	}
+
+	write_gcr_cl_other(val);
+}
+
+void mips_cm_unlock_other(void)
+{
+	unsigned curr_core = current_cpu_data.core;
+
+	spin_unlock_irqrestore(&per_cpu(cm_core_lock, curr_core),
+			       per_cpu(cm_core_lock_flags, curr_core));
+	preempt_enable();
+}
+
 void mips_cm_error_report(void)
 {
 	unsigned long revision = mips_cm_revision();
-- 
2.5.3

[PATCH 09/10] MIPS: CM: make use of mips_cm_{lock,unlock}_other

[Paul Burton]

Document that CPC core-other accesses must take place within the bounds
of the CM lock, and begin using the CM lock functions where we access
the GCRs of other cores. This is required because with CM3 the CPC began
using GCR_CL_OTHER instead of CPC_CL_OTHER.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/include/asm/mips-cpc.h | 3 ++-
 arch/mips/kernel/smp-cps.c       | 7 +++++--
 arch/mips/kernel/smp-gic.c       | 2 ++
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/mips/include/asm/mips-cpc.h b/arch/mips/include/asm/mips-cpc.h
index f386f32..e090352 100644
--- a/arch/mips/include/asm/mips-cpc.h
+++ b/arch/mips/include/asm/mips-cpc.h
@@ -149,7 +149,8 @@ BUILD_CPC_Cx_RW(other,		0x10)
  * core: the other core to be accessed
  *
  * Call before operating upon a core via the 'other' register region in
- * order to prevent the region being moved during access. Must be followed
+ * order to prevent the region being moved during access. Must be called
+ * within the bounds of a mips_cm_{lock,unlock}_other pair, and followed
  * by a call to mips_cpc_unlock_other.
  */
 extern void mips_cpc_lock_other(unsigned int core);
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 48b1b75..e04c805 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -38,8 +38,9 @@ static unsigned core_vpe_count(unsigned core)
 	if (!config_enabled(CONFIG_MIPS_MT_SMP) || !cpu_has_mipsmt)
 		return 1;
 
-	write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF);
+	mips_cm_lock_other(core, 0);
 	cfg = read_gcr_co_config() & CM_GCR_Cx_CONFIG_PVPE_MSK;
+	mips_cm_unlock_other();
 	return (cfg >> CM_GCR_Cx_CONFIG_PVPE_SHF) + 1;
 }
 
@@ -193,7 +194,7 @@ static void boot_core(unsigned core)
 	unsigned timeout;
 
 	/* Select the appropriate core */
-	write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF);
+	mips_cm_lock_other(core, 0);
 
 	/* Set its reset vector */
 	write_gcr_co_reset_base(CKSEG1ADDR((unsigned long)mips_cps_core_entry));
@@ -238,6 +239,8 @@ static void boot_core(unsigned core)
 		write_gcr_co_reset_release(0);
 	}
 
+	mips_cm_unlock_other();
+
 	/* The core is now powered up */
 	bitmap_set(core_power, core, 1);
 }
diff --git a/arch/mips/kernel/smp-gic.c b/arch/mips/kernel/smp-gic.c
index 5f0ab5b..9b63829 100644
--- a/arch/mips/kernel/smp-gic.c
+++ b/arch/mips/kernel/smp-gic.c
@@ -46,9 +46,11 @@ void gic_send_ipi_single(int cpu, unsigned int action)
 
 	if (mips_cpc_present() && (core != current_cpu_data.core)) {
 		while (!cpumask_test_cpu(cpu, &cpu_coherent_mask)) {
+			mips_cm_lock_other(core, 0);
 			mips_cpc_lock_other(core);
 			write_cpc_co_cmd(CPC_Cx_CMD_PWRUP);
 			mips_cpc_unlock_other();
+			mips_cm_unlock_other();
 		}
 	}
 
-- 
2.5.3

[PATCH 09/10] MIPS: CM: make use of mips_cm_{lock,unlock}_other

[Paul Burton]

Document that CPC core-other accesses must take place within the bounds
of the CM lock, and begin using the CM lock functions where we access
the GCRs of other cores. This is required because with CM3 the CPC began
using GCR_CL_OTHER instead of CPC_CL_OTHER.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/include/asm/mips-cpc.h | 3 ++-
 arch/mips/kernel/smp-cps.c       | 7 +++++--
 arch/mips/kernel/smp-gic.c       | 2 ++
 3 files changed, 9 insertions(+), 3 deletions(-)

diff --git a/arch/mips/include/asm/mips-cpc.h b/arch/mips/include/asm/mips-cpc.h
index f386f32..e090352 100644
--- a/arch/mips/include/asm/mips-cpc.h
+++ b/arch/mips/include/asm/mips-cpc.h
@@ -149,7 +149,8 @@ BUILD_CPC_Cx_RW(other,		0x10)
  * core: the other core to be accessed
  *
  * Call before operating upon a core via the 'other' register region in
- * order to prevent the region being moved during access. Must be followed
+ * order to prevent the region being moved during access. Must be called
+ * within the bounds of a mips_cm_{lock,unlock}_other pair, and followed
  * by a call to mips_cpc_unlock_other.
  */
 extern void mips_cpc_lock_other(unsigned int core);
diff --git a/arch/mips/kernel/smp-cps.c b/arch/mips/kernel/smp-cps.c
index 48b1b75..e04c805 100644
--- a/arch/mips/kernel/smp-cps.c
+++ b/arch/mips/kernel/smp-cps.c
@@ -38,8 +38,9 @@ static unsigned core_vpe_count(unsigned core)
 	if (!config_enabled(CONFIG_MIPS_MT_SMP) || !cpu_has_mipsmt)
 		return 1;
 
-	write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF);
+	mips_cm_lock_other(core, 0);
 	cfg = read_gcr_co_config() & CM_GCR_Cx_CONFIG_PVPE_MSK;
+	mips_cm_unlock_other();
 	return (cfg >> CM_GCR_Cx_CONFIG_PVPE_SHF) + 1;
 }
 
@@ -193,7 +194,7 @@ static void boot_core(unsigned core)
 	unsigned timeout;
 
 	/* Select the appropriate core */
-	write_gcr_cl_other(core << CM_GCR_Cx_OTHER_CORENUM_SHF);
+	mips_cm_lock_other(core, 0);
 
 	/* Set its reset vector */
 	write_gcr_co_reset_base(CKSEG1ADDR((unsigned long)mips_cps_core_entry));
@@ -238,6 +239,8 @@ static void boot_core(unsigned core)
 		write_gcr_co_reset_release(0);
 	}
 
+	mips_cm_unlock_other();
+
 	/* The core is now powered up */
 	bitmap_set(core_power, core, 1);
 }
diff --git a/arch/mips/kernel/smp-gic.c b/arch/mips/kernel/smp-gic.c
index 5f0ab5b..9b63829 100644
--- a/arch/mips/kernel/smp-gic.c
+++ b/arch/mips/kernel/smp-gic.c
@@ -46,9 +46,11 @@ void gic_send_ipi_single(int cpu, unsigned int action)
 
 	if (mips_cpc_present() && (core != current_cpu_data.core)) {
 		while (!cpumask_test_cpu(cpu, &cpu_coherent_mask)) {
+			mips_cm_lock_other(core, 0);
 			mips_cpc_lock_other(core);
 			write_cpc_co_cmd(CPC_Cx_CMD_PWRUP);
 			mips_cpc_unlock_other();
+			mips_cm_unlock_other();
 		}
 	}
 
-- 
2.5.3

[PATCH 10/10] MIPS: CM,CPC: ensure core-other GCRs reflect the correct core

[Paul Burton]

Ensure the update to which core the core-other GCR regions reflect has
taken place before any core-other GCRs are accessed by placing a memory
barrier (sync instruction) between the write to the core-other registers
and any such GCR accesses.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/mips-cm.c  | 6 ++++++
 arch/mips/kernel/mips-cpc.c | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/arch/mips/kernel/mips-cm.c b/arch/mips/kernel/mips-cm.c
index fef2647..485e441 100644
--- a/arch/mips/kernel/mips-cm.c
+++ b/arch/mips/kernel/mips-cm.c
@@ -278,6 +278,12 @@ void mips_cm_lock_other(unsigned int core, unsigned int vp)
 	}
 
 	write_gcr_cl_other(val);
+
+	/*
+	 * Ensure the core-other region reflects the appropriate core &
+	 * VP before any accesses to it occur.
+	 */
+	mb();
 }
 
 void mips_cm_unlock_other(void)
diff --git a/arch/mips/kernel/mips-cpc.c b/arch/mips/kernel/mips-cpc.c
index 8af4d62..566b8d2 100644
--- a/arch/mips/kernel/mips-cpc.c
+++ b/arch/mips/kernel/mips-cpc.c
@@ -76,6 +76,12 @@ void mips_cpc_lock_other(unsigned int core)
 	spin_lock_irqsave(&per_cpu(cpc_core_lock, curr_core),
 			  per_cpu(cpc_core_lock_flags, curr_core));
 	write_cpc_cl_other(core << CPC_Cx_OTHER_CORENUM_SHF);
+
+	/*
+	 * Ensure the core-other region reflects the appropriate core &
+	 * VP before any accesses to it occur.
+	 */
+	mb();
 }
 
 void mips_cpc_unlock_other(void)
-- 
2.5.3

[PATCH 10/10] MIPS: CM,CPC: ensure core-other GCRs reflect the correct core

[Paul Burton]

Ensure the update to which core the core-other GCR regions reflect has
taken place before any core-other GCRs are accessed by placing a memory
barrier (sync instruction) between the write to the core-other registers
and any such GCR accesses.

Signed-off-by: Paul Burton <paul.burton@imgtec.com>
---

 arch/mips/kernel/mips-cm.c  | 6 ++++++
 arch/mips/kernel/mips-cpc.c | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/arch/mips/kernel/mips-cm.c b/arch/mips/kernel/mips-cm.c
index fef2647..485e441 100644
--- a/arch/mips/kernel/mips-cm.c
+++ b/arch/mips/kernel/mips-cm.c
@@ -278,6 +278,12 @@ void mips_cm_lock_other(unsigned int core, unsigned int vp)
 	}
 
 	write_gcr_cl_other(val);
+
+	/*
+	 * Ensure the core-other region reflects the appropriate core &
+	 * VP before any accesses to it occur.
+	 */
+	mb();
 }
 
 void mips_cm_unlock_other(void)
diff --git a/arch/mips/kernel/mips-cpc.c b/arch/mips/kernel/mips-cpc.c
index 8af4d62..566b8d2 100644
--- a/arch/mips/kernel/mips-cpc.c
+++ b/arch/mips/kernel/mips-cpc.c
@@ -76,6 +76,12 @@ void mips_cpc_lock_other(unsigned int core)
 	spin_lock_irqsave(&per_cpu(cpc_core_lock, curr_core),
 			  per_cpu(cpc_core_lock_flags, curr_core));
 	write_cpc_cl_other(core << CPC_Cx_OTHER_CORENUM_SHF);
+
+	/*
+	 * Ensure the core-other region reflects the appropriate core &
+	 * VP before any accesses to it occur.
+	 */
+	mb();
 }
 
 void mips_cpc_unlock_other(void)
-- 
2.5.3