[PATCH/RFC 0/6] ARM: r8a73a4: Add SMP support

[PATCH/RFC 0/6] ARM: r8a73a4: Add SMP support

[Geert Uytterhoeven]

	Hi all,

This RFC patch series leverages the existing APMU support for R-Car Gen2
and RZ/G1 SoCs to enable SMP support on R-Mobile APE6.  This allows me
to enjoy the 4 Cortex-A15 CPU cores on the APE6-EVM development board.
The core APE6-specific code was lifted from an old series by Magnus[1].

Known issues:
  - While initial secondary CPU bring-up during boot works, CPU cores
    fail to come online after offlining/onlining them using the sysfs
    interface ("echo [01] > /sys/*/*/cpu/cpu[0-9]*{off,on}line").

  - This breaks detection of the CFI FLASH.  I traced this back to
    setting the SBAR_BAREN bit in the SYSC.CA15BAR register.
    Presumably this affects the LBSC in some way?

To do:
  - Call into rmobile-sysc instead of hardcoding the SYSC mapping,
  - Describe CCI-400 in DT and call cci_enable_port_by_index()?
  - Let pm-rcar-gen2.c call into rcar-rst instead of hardcoding the RST
    mapping,
  - Abstract SYSC vs. RST access and merge with pm-rcar-gen2.c?

Questions:
  - Do we want to bring up the Cortex-A7 cores, too? We don't on R-Car
    H2 neither.

Thanks for your comments!

[1] [PATCH 00/02] ARM: shmobile: Initial r8a73a4 APMU SMP code
    https://lore.kernel.org/linux-arm-kernel/20130807225531.9856.18974.sendpatchset@w520/

Geert Uytterhoeven (6):
  soc: renesas: Introduce RENESAS_APMU Kconfig option
  dt-bindings: power: renesas,apmu: Document R-Mobile APE6 support
  ARM: dts: r8a73a4: Add secondary CPU nodes
  ARM: dts: r8a73a4: Add Media RAM for SMP jump stub
  ARM: dts: r8a73a4: Add APMU nodes
  [WIP] ARM: shmobile: r8a73a4: Add SMP support

 .../bindings/power/renesas,apmu.yaml          |   6 +-
 arch/arm/boot/dts/r8a73a4.dtsi                |  96 +++++++++++
 arch/arm/mach-shmobile/Makefile               |   6 +-
 arch/arm/mach-shmobile/platsmp-apmu.c         |  15 +-
 arch/arm/mach-shmobile/pm-r8a73a4.c           | 152 ++++++++++++++++++
 arch/arm/mach-shmobile/r8a73a4.h              |   7 +
 drivers/soc/renesas/Kconfig                   |   5 +
 7 files changed, 281 insertions(+), 6 deletions(-)
 create mode 100644 arch/arm/mach-shmobile/pm-r8a73a4.c
 create mode 100644 arch/arm/mach-shmobile/r8a73a4.h

-- 
2.25.1

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

[PATCH/RFC 1/6] soc: renesas: Introduce RENESAS_APMU Kconfig option

[Geert Uytterhoeven]

The Renesas Advanced Power Management Unit for AP-System Core (APMU) is
not limited to R-Car Gen2 and RZ/G1 SoCs, but also present on R-Mobile
APE6.

Hence introduce a separate Kconfig symbol to control the build of
APMU-related files, currently selected by ARCH_RCAR_GEN2.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm/mach-shmobile/Makefile | 5 +++--
 drivers/soc/renesas/Kconfig     | 4 ++++
 2 files changed, 7 insertions(+), 2 deletions(-)

diff --git a/arch/arm/mach-shmobile/Makefile b/arch/arm/mach-shmobile/Makefile
index f7bf17b7abaef7a9..bce16aa04c5c1bb2 100644
--- a/arch/arm/mach-shmobile/Makefile
+++ b/arch/arm/mach-shmobile/Makefile
@@ -20,7 +20,8 @@ obj-$(CONFIG_ARCH_R7S9210)	+= setup-r7s9210.o
 cpu-y				:= platsmp.o headsmp.o
 
 # Shared SoC family objects
-obj-$(CONFIG_ARCH_RCAR_GEN2)	+= setup-rcar-gen2.o platsmp-apmu.o $(cpu-y)
+obj-$(CONFIG_ARCH_RCAR_GEN2)	+= setup-rcar-gen2.o
+obj-$(CONFIG_RENESAS_APMU)	+= platsmp-apmu.o $(cpu-y)
 CFLAGS_setup-rcar-gen2.o	+= -march=armv7-a
 obj-$(CONFIG_ARCH_R8A7790)	+= regulator-quirk-rcar-gen2.o
 obj-$(CONFIG_ARCH_R8A7791)	+= regulator-quirk-rcar-gen2.o
@@ -28,7 +29,7 @@ obj-$(CONFIG_ARCH_R8A7793)	+= regulator-quirk-rcar-gen2.o
 
 # SMP objects
 smp-y				:= $(cpu-y)
-smp-$(CONFIG_ARCH_RCAR_GEN2)	+= headsmp-apmu.o
+smp-$(CONFIG_RENESAS_APMU)	+= headsmp-apmu.o
 smp-$(CONFIG_ARCH_SH73A0)	+= smp-sh73a0.o headsmp-scu.o platsmp-scu.o
 smp-$(CONFIG_ARCH_R8A7779)	+= smp-r8a7779.o headsmp-scu.o platsmp-scu.o
 smp-$(CONFIG_ARCH_EMEV2)	+= smp-emev2.o headsmp-scu.o platsmp-scu.o
diff --git a/drivers/soc/renesas/Kconfig b/drivers/soc/renesas/Kconfig
index b70bbc38efc6771d..f92b6cdff8d6ca25 100644
--- a/drivers/soc/renesas/Kconfig
+++ b/drivers/soc/renesas/Kconfig
@@ -6,6 +6,9 @@ menuconfig SOC_RENESAS
 
 if SOC_RENESAS
 
+config RENESAS_APMU
+	bool
+
 config ARCH_RCAR_GEN1
 	bool
 	select PM
@@ -19,6 +22,7 @@ config ARCH_RCAR_GEN2
 	select HAVE_ARM_ARCH_TIMER
 	select PM
 	select PM_GENERIC_DOMAINS
+	select RENESAS_APMU
 	select RENESAS_IRQC
 	select RST_RCAR
 	select SYS_SUPPORTS_SH_CMT
-- 
2.25.1

[PATCH/RFC 2/6] dt-bindings: power: renesas,apmu: Document R-Mobile APE6 support

[Geert Uytterhoeven]

R-Mobile APE6 has an Advanced Power Management Unit (APMU) similar to
R-Car Gen2 SoCs.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 Documentation/devicetree/bindings/power/renesas,apmu.yaml | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/Documentation/devicetree/bindings/power/renesas,apmu.yaml b/Documentation/devicetree/bindings/power/renesas,apmu.yaml
index 391897d897f241c9..a4717b6561782443 100644
--- a/Documentation/devicetree/bindings/power/renesas,apmu.yaml
+++ b/Documentation/devicetree/bindings/power/renesas,apmu.yaml
@@ -11,13 +11,15 @@ maintainers:
   - Magnus Damm <magnus.damm@gmail.com>
 
 description:
-  Renesas R-Car Gen2 and RZ/G1 SoCs utilize one or more APMU hardware units for
-  CPU core power domain control including SMP boot and CPU Hotplug.
+  Renesas R-Mobile APE6, R-Car Gen2, and RZ/G1 SoCs utilize one or more APMU
+  hardware units for CPU core power domain control including SMP boot and CPU
+  Hotplug.
 
 properties:
   compatible:
     items:
       - enum:
+          - renesas,r8a73a4-apmu  # R-Mobile APE6
           - renesas,r8a7742-apmu  # RZ/G1H
           - renesas,r8a7743-apmu  # RZ/G1M
           - renesas,r8a7744-apmu  # RZ/G1N
-- 
2.25.1

[PATCH/RFC 3/6] ARM: dts: r8a73a4: Add secondary CPU nodes

[Geert Uytterhoeven]

Add nodes for:
  - CPU1-3, which are Cortex A15 at 1.5 GHz, living in PM domain A2SL,
  - CPU4-7, which are Cortex A7 at 1.0 GHz, living in PM domain A2KL.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm/boot/dts/r8a73a4.dtsi | 70 ++++++++++++++++++++++++++++++++++
 1 file changed, 70 insertions(+)

diff --git a/arch/arm/boot/dts/r8a73a4.dtsi b/arch/arm/boot/dts/r8a73a4.dtsi
index e5fb1ce261f72f2d..d498f848d34ab6ed 100644
--- a/arch/arm/boot/dts/r8a73a4.dtsi
+++ b/arch/arm/boot/dts/r8a73a4.dtsi
@@ -30,6 +30,76 @@ cpu0: cpu@0 {
 			next-level-cache = <&L2_CA15>;
 		};
 
+		cpu1: cpu@1 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a15";
+			reg = <1>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z>;
+			clock-frequency = <1500000000>;
+			power-domains = <&pd_a2sl>;
+			next-level-cache = <&L2_CA15>;
+		};
+
+		cpu2: cpu@2 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a15";
+			reg = <2>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z>;
+			clock-frequency = <1500000000>;
+			power-domains = <&pd_a2sl>;
+			next-level-cache = <&L2_CA15>;
+		};
+
+		cpu3: cpu@3 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a15";
+			reg = <3>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z>;
+			clock-frequency = <1500000000>;
+			power-domains = <&pd_a2sl>;
+			next-level-cache = <&L2_CA15>;
+		};
+
+		cpu4: cpu@100 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a7";
+			reg = <0x100>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z2>;
+			clock-frequency = <1000000000>;
+			power-domains = <&pd_a2kl>;
+			next-level-cache = <&L2_CA7>;
+		};
+
+		cpu5: cpu@101 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a7";
+			reg = <0x101>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z2>;
+			clock-frequency = <1000000000>;
+			power-domains = <&pd_a2kl>;
+			next-level-cache = <&L2_CA7>;
+		};
+
+		cpu6: cpu@102 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a7";
+			reg = <0x102>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z2>;
+			clock-frequency = <1000000000>;
+			power-domains = <&pd_a2kl>;
+			next-level-cache = <&L2_CA7>;
+		};
+
+		cpu7: cpu@103 {
+			device_type = "cpu";
+			compatible = "arm,cortex-a7";
+			reg = <0x103>;
+			clocks = <&cpg_clocks R8A73A4_CLK_Z2>;
+			clock-frequency = <1000000000>;
+			power-domains = <&pd_a2kl>;
+			next-level-cache = <&L2_CA7>;
+		};
+
 		L2_CA15: cache-controller-0 {
 			compatible = "cache";
 			clocks = <&cpg_clocks R8A73A4_CLK_Z>;
-- 
2.25.1

[PATCH/RFC 4/6] ARM: dts: r8a73a4: Add Media RAM for SMP jump stub

[Geert Uytterhoeven]

R-Mobile APE6 has 2 MiB of Media RAM.
Reserve 256 bytes for the jump stub for CPU core bringup.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm/boot/dts/r8a73a4.dtsi | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm/boot/dts/r8a73a4.dtsi b/arch/arm/boot/dts/r8a73a4.dtsi
index d498f848d34ab6ed..9abf5e9bb240f0a2 100644
--- a/arch/arm/boot/dts/r8a73a4.dtsi
+++ b/arch/arm/boot/dts/r8a73a4.dtsi
@@ -479,6 +479,19 @@ scifb3: serial@e6cf0000 {
 		status = "disabled";
 	};
 
+	meram: sram@e8080000 {
+		compatible = "mmio-sram";
+		reg = <0 0xe8080000 0 0x200000>;
+		#address-cells = <1>;
+		#size-cells = <1>;
+		ranges = <0 0 0xe8080000 0x200000>;
+
+		smp-sram@0 {
+			compatible = "renesas,smp-sram";
+			reg = <0 0x100>;
+		};
+	};
+
 	sdhi0: mmc@ee100000 {
 		compatible = "renesas,sdhi-r8a73a4";
 		reg = <0 0xee100000 0 0x100>;
-- 
2.25.1

[PATCH/RFC 5/6] ARM: dts: r8a73a4: Add APMU nodes

[Geert Uytterhoeven]

Add DT nodes for the Advanced Power Management Units (APMU), and use the
enable-method to point out that the APMU should be used for SMP support.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm/boot/dts/r8a73a4.dtsi | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/arch/arm/boot/dts/r8a73a4.dtsi b/arch/arm/boot/dts/r8a73a4.dtsi
index 9abf5e9bb240f0a2..f9c08c1bdcd687ba 100644
--- a/arch/arm/boot/dts/r8a73a4.dtsi
+++ b/arch/arm/boot/dts/r8a73a4.dtsi
@@ -19,6 +19,7 @@ / {
 	cpus {
 		#address-cells = <1>;
 		#size-cells = <0>;
+		enable-method = "renesas,apmu";
 
 		cpu0: cpu@0 {
 			device_type = "cpu";
@@ -130,6 +131,18 @@ timer {
 			     <GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(8) | IRQ_TYPE_LEVEL_LOW)>;
 	};
 
+	apmu@e6151000 {
+		compatible = "renesas,r8a73a4-apmu", "renesas,apmu";
+		reg = <0 0xe6151000 0 0x188>;
+		cpus = <&cpu4>, <&cpu5>, <&cpu6>, <&cpu7>;
+	};
+
+	apmu@e6152000 {
+		compatible = "renesas,r8a73a4-apmu", "renesas,apmu";
+		reg = <0 0xe6152000 0 0x188>;
+		cpus = <&cpu0>, <&cpu1>, <&cpu2>, <&cpu3>;
+	};
+
 	dbsc1: memory-controller@e6790000 {
 		compatible = "renesas,dbsc-r8a73a4";
 		reg = <0 0xe6790000 0 0x10000>;
-- 
2.25.1

[PATCH/RFC 6/6] [WIP] ARM: shmobile: r8a73a4: Add SMP support

[Geert Uytterhoeven]

Hoop up APMU-based secondary CPU support for R-Mobile APE6.
This brings up the 3 other Cortex-A15 CPU cores.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
 arch/arm/mach-shmobile/Makefile       |   1 +
 arch/arm/mach-shmobile/platsmp-apmu.c |  15 ++-
 arch/arm/mach-shmobile/pm-r8a73a4.c   | 152 ++++++++++++++++++++++++++
 arch/arm/mach-shmobile/r8a73a4.h      |   7 ++
 drivers/soc/renesas/Kconfig           |   1 +
 5 files changed, 174 insertions(+), 2 deletions(-)
 create mode 100644 arch/arm/mach-shmobile/pm-r8a73a4.c
 create mode 100644 arch/arm/mach-shmobile/r8a73a4.h

diff --git a/arch/arm/mach-shmobile/Makefile b/arch/arm/mach-shmobile/Makefile
index bce16aa04c5c1bb2..78e9fd2bab847cb6 100644
--- a/arch/arm/mach-shmobile/Makefile
+++ b/arch/arm/mach-shmobile/Makefile
@@ -37,6 +37,7 @@ smp-$(CONFIG_ARCH_EMEV2)	+= smp-emev2.o headsmp-scu.o platsmp-scu.o
 # PM objects
 obj-$(CONFIG_SUSPEND)		+= suspend.o
 obj-$(CONFIG_ARCH_RCAR_GEN2)	+= pm-rcar-gen2.o
+obj-$(CONFIG_ARCH_R8A73A4)	+= pm-r8a73a4.o
 
 # Framework support
 obj-$(CONFIG_SMP)		+= $(smp-y)
diff --git a/arch/arm/mach-shmobile/platsmp-apmu.c b/arch/arm/mach-shmobile/platsmp-apmu.c
index e771ce70e132ff53..f2af3edc16a02808 100644
--- a/arch/arm/mach-shmobile/platsmp-apmu.c
+++ b/arch/arm/mach-shmobile/platsmp-apmu.c
@@ -21,6 +21,7 @@
 #include <asm/suspend.h>
 #include "common.h"
 #include "rcar-gen2.h"
+#include "r8a73a4.h"
 
 static struct {
 	void __iomem *iomem;
@@ -194,7 +195,7 @@ static void apmu_init_cpu(struct resource *res, int cpu, int bit)
 
 	pr_debug("apmu ioremap %d %d %pr\n", cpu, bit, res);
 
-	/* Setup for debug mode */
+	/* Setup for debug mode */	// FIXME Not documented on APE6?
 	x = readl(apmu_cpus[cpu].iomem + DBGRCR_OFFS);
 	x |= DBGCPUREN | DBGCPUNREN(bit) | DBGCPUPREN;
 	writel(x, apmu_cpus[cpu].iomem + DBGRCR_OFFS);
@@ -253,7 +254,9 @@ static void __init shmobile_smp_apmu_setup_boot(void)
 {
 	/* install boot code shared by all CPUs */
 	shmobile_boot_fn = __pa_symbol(shmobile_smp_boot);
+#ifdef CONFIG_ARCH_RCAR_GEN2	// FIXME
 	shmobile_boot_fn_gen2 = shmobile_boot_fn;
+#endif
 }
 
 static int shmobile_smp_apmu_boot_secondary(unsigned int cpu,
@@ -269,7 +272,15 @@ static void __init shmobile_smp_apmu_prepare_cpus_dt(unsigned int max_cpus)
 {
 	shmobile_smp_apmu_setup_boot();
 	apmu_parse_dt(apmu_init_cpu);
-	rcar_gen2_pm_init();
+	if (of_machine_is_compatible("renesas,r8a73a4")) {	// FIXME
+#ifdef CONFIG_ARCH_R8A73A4
+		r8a73a4_pm_init();
+#endif
+	} else {
+#ifdef CONFIG_ARCH_RCAR_GEN2
+		rcar_gen2_pm_init();
+#endif
+	}
 }
 
 static struct smp_operations apmu_smp_ops __initdata = {
diff --git a/arch/arm/mach-shmobile/pm-r8a73a4.c b/arch/arm/mach-shmobile/pm-r8a73a4.c
new file mode 100644
index 0000000000000000..3ee270063ecf9f9b
--- /dev/null
+++ b/arch/arm/mach-shmobile/pm-r8a73a4.c
@@ -0,0 +1,152 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * R-Mobile APE6 Power management support
+ * Based on R-Car Generation 2 Power management support
+ *
+ * Copyright (C) 2013 - 2015  Renesas Electronics Corporation
+ * Copyright (C) 2011 - 2013  Renesas Solutions Corp.
+ * Copyright (C) 2012 Takashi Yoshii <takashi.yoshii.ze@renesas.com>
+ * Copyright (C) 2011  Magnus Damm
+ */
+
+#include <linux/kernel.h>
+#include <linux/ioport.h>
+#include <linux/of.h>
+#include <linux/of_address.h>
+#include <linux/smp.h>
+#include <asm/io.h>
+#include <asm/cputype.h>
+#include "common.h"
+#include "r8a73a4.h"
+
+/* System Controller */
+#define SYSC		0xe6180000	// FIXME
+
+/* Boot Address Registers */
+#define CA7BAR		0x4020		/* CA7 Boot Address Register */
+#define CA15BAR		0x6020		/* CA15 Boot Address Register */
+#define SBAR_BAREN	BIT(4)		/* SBAR is valid */
+
+/* Reset Control Register */
+#define RESCNT		0x801c		/* Reset Control Register */
+#define RESCNT_MSTPCA15	BIT(10)		/* AP-System Core(CA15) Module stop */
+#define RESCNT_MSTPCA7	BIT(9)		/* AP-System Core(CA9) Module stop */
+
+/* CCI-400 */
+#define CCI_BASE		0xf0190000	// FIXME
+
+#define CCI_SLAVE3		0x4000		/* S3: CA15 */
+#define CCI_SLAVE4		0x5000		/* S4: CA7 */
+
+#define CCI_PORT_CTRL		0x0000
+
+#define CCI_ENABLE_SNOOP_REQ	BIT(0)
+#define CCI_ENABLE_DVM_REQ	BIT(1)
+#define CCI_ENABLE_REQ		(CCI_ENABLE_SNOOP_REQ | CCI_ENABLE_DVM_REQ)
+
+#define CCI_CTRL_STATUS		0x000c
+
+static inline u32 phys_to_sbar(phys_addr_t addr)
+{
+	return (addr >> 8) & 0xfffffc00;
+}
+
+void __init r8a73a4_pm_init(void)
+{
+	void __iomem *p;
+	u32 bar;
+	static int once;
+	struct device_node *np;
+	bool has_a7 = false;
+	bool has_a15 = false;
+	struct resource res;
+	int error;
+
+	if (once++)
+		return;
+
+	for_each_of_cpu_node(np) {
+		if (of_device_is_compatible(np, "arm,cortex-a15"))
+			has_a15 = true;
+		else if (of_device_is_compatible(np, "arm,cortex-a7"))
+			has_a7 = true;
+	}
+
+	np = of_find_compatible_node(NULL, NULL, "renesas,smp-sram");
+	if (!np)
+		return;
+
+	error = of_address_to_resource(np, 0, &res);
+	of_node_put(np);
+	if (error) {
+		pr_err("Failed to get smp-sram address: %d\n", error);
+		return;
+	}
+
+	/* RAM for jump stub, because BAR requires 256KB aligned address */
+	if (res.start & (256 * 1024 - 1) ||
+	    resource_size(&res) < shmobile_boot_size) {
+		pr_err("Invalid smp-sram region\n");
+		return;
+	}
+
+	/* install reset vector */
+	p = ioremap(res.start, resource_size(&res));
+	if (!p)
+		return;
+
+	memcpy_toio(p, shmobile_boot_vector, shmobile_boot_size);
+	iounmap(p);
+
+	/* setup reset vector and disable reset */
+	p = ioremap(SYSC, 0x9000);
+	bar = phys_to_sbar(res.start);
+	if (has_a15) {
+		writel_relaxed(bar, p + CA15BAR);
+		// FIXME The register write below breaks the CFI FLASH:
+		// -0.flash: Found 1 x16 devices at 0x0 in 16-bit bank. Manufacturer ID 0x000089 Chip ID 0x0088b1
+		// -Intel/Sharp Extended Query Table at 0x010A
+		// -Intel/Sharp Extended Query Table at 0x010A
+		// -Intel/Sharp Extended Query Table at 0x010A
+		// -Intel/Sharp Extended Query Table at 0x010A
+		// -Intel/Sharp Extended Query Table at 0x010A
+		// -Using buffer write method
+		// -Using auto-unlock on power-up/resume
+		// -cfi_cmdset_0001: Erase suspend on write enabled
+		// -erase region 0: offset=0x0,size=0x40000,blocks=512
+		// -0.flash: program region size/ctrl_valid/ctrl_inval = 1024/16/16
+		// -0.flash: 1 set(s) of 1 interleaved chips --> 8 partitions of 16384 KiB
+		// -3 fixed-partitions partitions found on MTD device 0.flash
+		// -Creating 3 MTD partitions on "0.flash":
+		// -0x000000000000-0x000000040000 : "uboot"
+		// -0x000000040000-0x000000080000 : "uboot-env"
+		// -0x000000080000-0x000008000000 : "flash"
+		// +physmap-flash 0.flash: map_probe failed
+		writel_relaxed(bar | SBAR_BAREN, p + CA15BAR);
+
+		/* de-assert reset for CA15 CPUs */
+		writel_relaxed(readl_relaxed(p + RESCNT) & ~RESCNT_MSTPCA15,
+			       p + RESCNT);
+	}
+	if (has_a7) {
+		writel_relaxed(bar, p + CA7BAR);
+		writel_relaxed(bar | SBAR_BAREN, p + CA7BAR);
+
+		/* de-assert reset for CA7 CPUs */
+		writel_relaxed(readl_relaxed(p + RESCNT) & ~RESCNT_MSTPCA7,
+			       p + RESCNT);
+	}
+	iounmap(p);
+
+	/* enable snoop and DVM */
+	p = ioremap(CCI_BASE, 0x8000);
+	if (has_a15)
+		writel_relaxed(CCI_ENABLE_REQ, p + CCI_SLAVE3 + CCI_PORT_CTRL);
+	if (has_a7)
+		writel_relaxed(CCI_ENABLE_REQ, p + CCI_SLAVE4 + CCI_PORT_CTRL);
+	while (readl_relaxed(p + CCI_CTRL_STATUS))
+		/* wait for pending bit low */;
+	iounmap(p);
+
+	shmobile_smp_apmu_suspend_init();
+}
diff --git a/arch/arm/mach-shmobile/r8a73a4.h b/arch/arm/mach-shmobile/r8a73a4.h
new file mode 100644
index 0000000000000000..d2ca57225ac5e5a3
--- /dev/null
+++ b/arch/arm/mach-shmobile/r8a73a4.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __ASM_R8A73A4_H__
+#define __ASM_R8A73A4_H__
+
+void r8a73a4_pm_init(void);
+
+#endif /* __ASM_R8A73A4_H__ */
diff --git a/drivers/soc/renesas/Kconfig b/drivers/soc/renesas/Kconfig
index f92b6cdff8d6ca25..4fe0247189a615b0 100644
--- a/drivers/soc/renesas/Kconfig
+++ b/drivers/soc/renesas/Kconfig
@@ -117,6 +117,7 @@ config ARCH_R8A73A4
 	select ARM_ERRATA_798181 if SMP
 	select ARM_ERRATA_814220
 	select HAVE_ARM_ARCH_TIMER
+	select RENESAS_APMU
 	select RENESAS_IRQC
 
 config ARCH_R7S72100
-- 
2.25.1

Re: [PATCH/RFC 0/6] ARM: r8a73a4: Add SMP support

[Niklas Söderlund]

Hi Geert,

Thanks for your work.

On 2021-02-04 14:54:03 +0100, Geert Uytterhoeven wrote:
> 	Hi all,
> 
> This RFC patch series leverages the existing APMU support for R-Car Gen2
> and RZ/G1 SoCs to enable SMP support on R-Mobile APE6.  This allows me
> to enjoy the 4 Cortex-A15 CPU cores on the APE6-EVM development board.
> The core APE6-specific code was lifted from an old series by Magnus[1].

I'm also able to enjoy the 4 Cortex-A15 CPU cores, nice :-)

Tested-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

> 
> Known issues:
>   - While initial secondary CPU bring-up during boot works, CPU cores
>     fail to come online after offlining/onlining them using the sysfs
>     interface ("echo [01] > /sys/*/*/cpu/cpu[0-9]*{off,on}line").
> 
>   - This breaks detection of the CFI FLASH.  I traced this back to
>     setting the SBAR_BAREN bit in the SYSC.CA15BAR register.
>     Presumably this affects the LBSC in some way?
> 
> To do:
>   - Call into rmobile-sysc instead of hardcoding the SYSC mapping,
>   - Describe CCI-400 in DT and call cci_enable_port_by_index()?
>   - Let pm-rcar-gen2.c call into rcar-rst instead of hardcoding the RST
>     mapping,
>   - Abstract SYSC vs. RST access and merge with pm-rcar-gen2.c?

Some of these to do's sounds rather fun and educational. If you have 
something you want done that is not time critical and does not require a 
100h time investment I would be happy to help out.

> 
> Questions:
>   - Do we want to bring up the Cortex-A7 cores, too? We don't on R-Car
>     H2 neither.
> 
> Thanks for your comments!
> 
> [1] [PATCH 00/02] ARM: shmobile: Initial r8a73a4 APMU SMP code
>     https://lore.kernel.org/linux-arm-kernel/20130807225531.9856.18974.sendpatchset@w520/
> 
> Geert Uytterhoeven (6):
>   soc: renesas: Introduce RENESAS_APMU Kconfig option
>   dt-bindings: power: renesas,apmu: Document R-Mobile APE6 support
>   ARM: dts: r8a73a4: Add secondary CPU nodes
>   ARM: dts: r8a73a4: Add Media RAM for SMP jump stub
>   ARM: dts: r8a73a4: Add APMU nodes
>   [WIP] ARM: shmobile: r8a73a4: Add SMP support
> 
>  .../bindings/power/renesas,apmu.yaml          |   6 +-
>  arch/arm/boot/dts/r8a73a4.dtsi                |  96 +++++++++++
>  arch/arm/mach-shmobile/Makefile               |   6 +-
>  arch/arm/mach-shmobile/platsmp-apmu.c         |  15 +-
>  arch/arm/mach-shmobile/pm-r8a73a4.c           | 152 ++++++++++++++++++
>  arch/arm/mach-shmobile/r8a73a4.h              |   7 +
>  drivers/soc/renesas/Kconfig                   |   5 +
>  7 files changed, 281 insertions(+), 6 deletions(-)
>  create mode 100644 arch/arm/mach-shmobile/pm-r8a73a4.c
>  create mode 100644 arch/arm/mach-shmobile/r8a73a4.h
> 
> -- 
> 2.25.1
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

-- 
Regards,
Niklas Söderlund

Re: [PATCH/RFC 0/6] ARM: r8a73a4: Add SMP support

[Ulrich Hecht]

> On 02/04/2021 2:54 PM Geert Uytterhoeven <geert+renesas@glider.be> wrote:
> Questions:
>   - Do we want to bring up the Cortex-A7 cores, too? We don't on R-Car
>     H2 neither.

What still needs to be done for that?

[I recall having issues bringing up the A7 cores on my old APE6 board (that has since failed and been returned to Renesas). The kernel failed to bring up the A7 cores if enabled at boot, but I was able to enable up to three (any three) of them one-by-one via sysfs. I'm not sure if that was specific to that board, though.]

CU
Uli

Re: [PATCH/RFC 0/6] ARM: r8a73a4: Add SMP support

[Geert Uytterhoeven]

Hi Uli,

On Wed, Feb 10, 2021 at 10:17 AM Ulrich Hecht <uli@fpond.eu> wrote:
> > On 02/04/2021 2:54 PM Geert Uytterhoeven <geert+renesas@glider.be> wrote:
> > Questions:
> >   - Do we want to bring up the Cortex-A7 cores, too? We don't on R-Car
> >     H2 neither.
>
> What still needs to be done for that?

Revert ee490bcc4f2d456c ("ARM: shmobile: Extend APMU code to allow
single cluster only")?  Looks like Magnus only wanted to enable the boot
cluster, due to lack of big.LITTLE scheduling support?

> [I recall having issues bringing up the A7 cores on my old APE6 board (that has since failed and been returned to Renesas). The kernel failed to bring up the A7 cores if enabled at boot, but I was able to enable up to three (any three) of them one-by-one via sysfs. I'm not sure if that was specific to that board, though.]

Disabling the check makes the A7 cores start on my APE6-EVM.
Userland (Debian nfsroot) boot fails in the middle though, when systemd
fails to send messages to one of its brethren.

With a ramdisk, everything seems fine.  I can even offline/online CPU
cores from sysfs at will, which failed before!
Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds