[RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[Chen-Yu Tsai]

Hi everyone,

This is my attempt to support SMP and CPU hot plugging on the Allwinner
A80 SoC. The A80 is a big.Little processor with 2 clusters of 4x Cortex-A7
and 4x Cortex-A15 cores.

Much of the sunxi-specific MCPM code is derived from Allwinner code and
documentation, with some references to the other MCPM implementations,
as well as the Cortex's Technical Reference Manuals for the power
sequencing info.

One major difference compared to other platforms is we currently do not
have a standalone PMU or other embedded firmware to do the actually power
sequencing. All power/reset control is done by the kernel. As such, I
couldn't figure out where to put the code to power off the outbound
processor. I'm putting it in the .wait_for_powerdown() callback for now.
This does not get called by the big.Little switcher. But since we lack
cpufreq support at the moment, big.Little switcher is probably not going
to work anyway.

The code has been tested on my A80 Optimus, and reliably brings up all
cores. CPU hotplugging works as well. One issue I have is the processors
in cluster 0 do not stay in WFI after they are signaled to go offline.
I haven't tested the CCI-400 PMU bits yet.

I've done the best I could to fit the code into the new MCPM callbacks,
unlike the Allwinner code which uses the old .power_up()/.power_down()
ones. However my knowledge of ARM internals is limited, so it is quite
possible I got something wrong. Reviews are highly appreciated.

The actual work is split into 3 phases:

Patch 1 adds basic SMP bringup code using the common MCPM code.
No hotplugging is supported.

Patch 2 ~ 4 add the required DT device nodes.

Patch 5 adds support for hotplugging processor cores 1~7.

Patch 6 adds support for cpu0 hotplugging. The BROM checks a region
of secure SRAM for special flags. If they are set, execution is
diverted to the configured secondary startup address, just like it
would be for all the other processor cores.

Patch 7 adds the DT nodes for the secure SRAM.


Regards
ChenYu


Chen-Yu Tsai (7):
  ARM: sun9i: Support SMP on A80 with Multi-Cluster Power Management
    (MCPM)
  ARM: dts: sun9i: Add CCI-400 device nodes for A80
  ARM: dts: sun9i: Add CPUCFG device node for A80 dtsi
  ARM: dts: sun9i: Add PRCM device node for the A80 dtsi
  ARM: sunxi: mcpm: Support CPU/cluster power down and hotplugging for
    cpu1~7
  ARM: sunxi: mcpm: Support cpu0 hotplug
  ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

 arch/arm/boot/dts/sun9i-a80.dtsi |  76 ++++++
 arch/arm/mach-sunxi/Kconfig      |   8 +
 arch/arm/mach-sunxi/Makefile     |   1 +
 arch/arm/mach-sunxi/mcpm.c       | 497 +++++++++++++++++++++++++++++++++++++++
 4 files changed, 582 insertions(+)
 create mode 100644 arch/arm/mach-sunxi/mcpm.c

-- 
2.1.4

[RFC 1/7] ARM: sun9i: Support SMP on A80 with Multi-Cluster Power Management (MCPM)

[Chen-Yu Tsai]

The A80 is a big.LITTLE SoC with 1 cluster of 4 Cortex-A7s and
1 cluster of 4 Cortex-A15s.

This patch adds support to bring up the second cluster and thus all
cores using the common MCPM code. Core/cluster power down has not
been implemented, thus CPU hotplugging and big.LITTLE switcher is
not supported.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/mach-sunxi/Kconfig  |   8 +
 arch/arm/mach-sunxi/Makefile |   1 +
 arch/arm/mach-sunxi/mcpm.c   | 370 +++++++++++++++++++++++++++++++++++++++++++
 3 files changed, 379 insertions(+)
 create mode 100644 arch/arm/mach-sunxi/mcpm.c

diff --git a/arch/arm/mach-sunxi/Kconfig b/arch/arm/mach-sunxi/Kconfig
index 81502b90dd91..8c596cfb4039 100644
--- a/arch/arm/mach-sunxi/Kconfig
+++ b/arch/arm/mach-sunxi/Kconfig
@@ -45,4 +45,12 @@ config MACH_SUN9I
 	default ARCH_SUNXI
 	select ARM_GIC
 
+config SUN9I_A80_MCPM
+	bool "Allwinner A80 Multi-Cluster PM support"
+	depends on MCPM && MACH_SUN9I
+	select ARM_CCI400_PORT_CTRL
+	help
+	  This is needed to provide CPU and cluster power management
+	  on Allwinner A80 implementing big.LITTLE.
+
 endif
diff --git a/arch/arm/mach-sunxi/Makefile b/arch/arm/mach-sunxi/Makefile
index 27b168f121a1..e8558912c714 100644
--- a/arch/arm/mach-sunxi/Makefile
+++ b/arch/arm/mach-sunxi/Makefile
@@ -1,2 +1,3 @@
 obj-$(CONFIG_ARCH_SUNXI) += sunxi.o
 obj-$(CONFIG_SMP) += platsmp.o
+obj-$(CONFIG_SUN9I_A80_MCPM) += mcpm.o
diff --git a/arch/arm/mach-sunxi/mcpm.c b/arch/arm/mach-sunxi/mcpm.c
new file mode 100644
index 000000000000..cf9cbf268d29
--- /dev/null
+++ b/arch/arm/mach-sunxi/mcpm.c
@@ -0,0 +1,370 @@
+/*
+ * Copyright (c) 2015 Chen-Yu Tsai
+ *
+ * Chen-Yu Tsai <wens@csie.org>
+ *
+ * arch/arm/mach-sunxi/mcpm.c
+ *
+ * Based on arch/arm/mach-exynos/mcpm-exynos.c and Allwinner code
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#include <linux/arm-cci.h>
+#include <linux/delay.h>
+#include <linux/io.h>
+#include <linux/of_address.h>
+
+#include <asm/cputype.h>
+#include <asm/cp15.h>
+#include <asm/mcpm.h>
+
+#define SUNXI_CPUS_PER_CLUSTER		4
+#define SUNXI_NR_CLUSTERS		2
+
+#define SUN9I_A80_A15_CLUSTER		1
+
+#define CPUCFG_CX_CTRL_REG0(c)		(0x10 * (c))
+#define CPUCFG_CX_CTRL_REG0_L1_RST_DISABLE(n)	BIT(n)
+#define CPUCFG_CX_CTRL_REG0_L1_RST_DISABLE_ALL	0xf
+#define CPUCFG_CX_CTRL_REG0_L2_RST_DISABLE_A7	BIT(4)
+#define CPUCFG_CX_CTRL_REG0_L2_RST_DISABLE_A15	BIT(0)
+#define CPUCFG_CX_CTRL_REG1(c)		(0x10 * (c) + 0x4)
+#define CPUCFG_CX_CTRL_REG1_ACINACTM	BIT(0)
+#define CPUCFG_CX_RST_CTRL(c)		(0x80 + 0x4 * (c))
+#define CPUCFG_CX_RST_CTRL_DBG_SOC_RST	BIT(24)
+#define CPUCFG_CX_RST_CTRL_ETM_RST(n)	BIT(20 + (n))
+#define CPUCFG_CX_RST_CTRL_ETM_RST_ALL	(0xf << 20)
+#define CPUCFG_CX_RST_CTRL_DBG_RST(n)	BIT(16 + (n))
+#define CPUCFG_CX_RST_CTRL_DBG_RST_ALL	(0xf << 16)
+#define CPUCFG_CX_RST_CTRL_H_RST	BIT(12)
+#define CPUCFG_CX_RST_CTRL_L2_RST	BIT(8)
+#define CPUCFG_CX_RST_CTRL_CX_RST(n)	BIT(4 + (n))
+#define CPUCFG_CX_RST_CTRL_CORE_RST(n)	BIT(n)
+
+#define PRCM_CPU_PO_RST_CTRL(c)		(0x4 + 0x4 * (c))
+#define PRCM_CPU_PO_RST_CTRL_CORE(n)	BIT(n)
+#define PRCM_CPU_PO_RST_CTRL_CORE_ALL	0xf
+#define PRCM_PWROFF_GATING_REG(c)	(0x100 + 0x4 * (c))
+#define PRCM_PWROFF_GATING_REG_CLUSTER	BIT(4)
+#define PRCM_PWROFF_GATING_REG_CORE(n)	BIT(n)
+#define PRCM_PWR_SWITCH_REG(c, cpu)	(0x140 + 0x10 * (c) + 0x4 * (cpu))
+#define PRCM_CPU_SOFT_ENTRY_REG		0x164
+
+static void __iomem *cpucfg_base;
+static void __iomem *prcm_base;
+
+static int sunxi_cpu_power_switch_set(unsigned int cpu, unsigned int cluster,
+				      bool enable)
+{
+	u32 reg;
+
+	/* control sequence from Allwinner A80 user manual v1.2 PRCM section */
+	reg = readl(prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+	if (enable) {
+		if (reg == 0x00) {
+			pr_debug("power clamp for cluster %u cpu %u already open",
+				 cluster, cpu);
+			return 0;
+		}
+
+		writel(0xff, prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+		udelay(10);
+		writel(0xfe, prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+		udelay(10);
+		writel(0xf8, prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+		udelay(10);
+		writel(0xf0, prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+		udelay(10);
+		writel(0x00, prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+		udelay(10);
+	} else {
+		writel(0xff, prcm_base + PRCM_PWR_SWITCH_REG(cluster, cpu));
+		udelay(10);
+	}
+
+	return 0;
+}
+
+static int sunxi_cpu_powerup(unsigned int cpu, unsigned int cluster)
+{
+	u32 reg;
+
+	pr_debug("%s: cpu %u cluster %u\n", __func__, cpu, cluster);
+	if (cpu >= SUNXI_CPUS_PER_CLUSTER || cluster >= SUNXI_NR_CLUSTERS)
+		return -EINVAL;
+
+	/* assert processor power-on reset */
+	reg = readl(prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
+	reg &= ~PRCM_CPU_PO_RST_CTRL_CORE(cpu);
+	writel(reg, prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
+
+	/* Cortex-A7: hold L1 reset disable signal low */
+	if (!(of_machine_is_compatible("allwinner,sun9i-a80") &&
+			cluster == SUN9I_A80_A15_CLUSTER)) {
+		reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG0(cluster));
+		reg &= ~CPUCFG_CX_CTRL_REG0_L1_RST_DISABLE(cpu);
+		writel(reg, cpucfg_base + CPUCFG_CX_CTRL_REG0(cluster));
+	}
+
+	/* assert processor related resets */
+	reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+	reg &= ~CPUCFG_CX_RST_CTRL_DBG_RST(cpu);
+
+	/*
+	 * Allwinner code also asserts resets for NEON on A15. According
+	 * to ARM manuals, asserting power-on reset is sufficient.
+	 */
+	if (!(of_machine_is_compatible("allwinner,sun9i-a80") &&
+			cluster == SUN9I_A80_A15_CLUSTER)) {
+		reg &= ~CPUCFG_CX_RST_CTRL_ETM_RST(cpu);
+	}
+	writel(reg, cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+
+	/* open power switch */
+	sunxi_cpu_power_switch_set(cpu, cluster, true);
+
+	/* clear processor power gate */
+	reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	reg &= ~PRCM_PWROFF_GATING_REG_CORE(cpu);
+	writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	udelay(20);
+
+	/* de-assert processor power-on reset */
+	reg = readl(prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
+	reg |= PRCM_CPU_PO_RST_CTRL_CORE(cpu);
+	writel(reg, prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
+
+	/* de-assert all processor resets */
+	reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+	reg |= CPUCFG_CX_RST_CTRL_DBG_RST(cpu);
+	reg |= CPUCFG_CX_RST_CTRL_CORE_RST(cpu);
+	if (!(of_machine_is_compatible("allwinner,sun9i-a80") &&
+			cluster == SUN9I_A80_A15_CLUSTER)) {
+		reg |= CPUCFG_CX_RST_CTRL_ETM_RST(cpu);
+	} else {
+		reg |= CPUCFG_CX_RST_CTRL_CX_RST(cpu); /* NEON */
+	}
+	writel(reg, cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+
+	return 0;
+}
+
+static int sunxi_cluster_powerup(unsigned int cluster)
+{
+	u32 reg;
+
+	pr_debug("%s: cluster %u\n", __func__, cluster);
+	if (cluster >= SUNXI_NR_CLUSTERS)
+		return -EINVAL;
+
+	/* assert ACINACTM */
+	reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster));
+	reg |= CPUCFG_CX_CTRL_REG1_ACINACTM;
+	writel(reg, cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster));
+
+	/* assert cluster processor power-on resets */
+	reg = readl(prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
+	reg &= ~PRCM_CPU_PO_RST_CTRL_CORE_ALL;
+	writel(reg, prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
+
+	/* assert cluster resets */
+	reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+	reg &= ~CPUCFG_CX_RST_CTRL_DBG_SOC_RST;
+	reg &= ~CPUCFG_CX_RST_CTRL_DBG_RST_ALL;
+	reg &= ~CPUCFG_CX_RST_CTRL_H_RST;
+	reg &= ~CPUCFG_CX_RST_CTRL_L2_RST;
+
+	/*
+	 * Allwinner code also asserts resets for NEON on A15. According
+	 * to ARM manuals, asserting power-on reset is sufficient.
+	 */
+	if (!(of_machine_is_compatible("allwinner,sun9i-a80") &&
+			cluster == SUN9I_A80_A15_CLUSTER)) {
+		reg &= ~CPUCFG_CX_RST_CTRL_ETM_RST_ALL;
+	}
+	writel(reg, cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+
+	/* hold L1/L2 reset disable signals low */
+	reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG0(cluster));
+	if (of_machine_is_compatible("allwinner,sun9i-a80") &&
+			cluster == SUN9I_A80_A15_CLUSTER) {
+		/* Cortex-A15: hold L2RSTDISABLE low */
+		reg &= ~CPUCFG_CX_CTRL_REG0_L2_RST_DISABLE_A15;
+	} else {
+		/* Cortex-A7: hold L1RSTDISABLE and L2RSTDISABLE low */
+		reg &= ~CPUCFG_CX_CTRL_REG0_L1_RST_DISABLE_ALL;
+		reg &= ~CPUCFG_CX_CTRL_REG0_L2_RST_DISABLE_A7;
+	}
+	writel(reg, cpucfg_base + CPUCFG_CX_CTRL_REG0(cluster));
+
+	/* clear cluster power gate */
+	reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	reg &= ~PRCM_PWROFF_GATING_REG_CLUSTER;
+	writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	udelay(20);
+
+	/* de-assert cluster resets */
+	reg = readl(cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+	reg |= CPUCFG_CX_RST_CTRL_DBG_SOC_RST;
+	reg |= CPUCFG_CX_RST_CTRL_H_RST;
+	reg |= CPUCFG_CX_RST_CTRL_L2_RST;
+	writel(reg, cpucfg_base + CPUCFG_CX_RST_CTRL(cluster));
+
+	/* de-assert ACINACTM */
+	reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster));
+	reg &= ~CPUCFG_CX_CTRL_REG1_ACINACTM;
+	writel(reg, cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster));
+
+	return 0;
+}
+
+static void sunxi_cpu_cache_disable(void)
+{
+	/* Disable and flush the local CPU cache. */
+	v7_exit_coherency_flush(louis);
+}
+
+static void sunxi_cluster_cache_disable(void)
+{
+	if (read_cpuid_part() == ARM_CPU_PART_CORTEX_A15) {
+		/*
+		 * On the Cortex-A15 we need to disable
+		 * L2 prefetching before flushing the cache.
+		 */
+		asm volatile(
+		"mcr	p15, 1, %0, c15, c0, 3\n"
+		"isb\n"
+		"dsb"
+		: : "r" (0x400));
+	}
+
+	/* Flush all cache levels for this cluster. */
+	v7_exit_coherency_flush(all);
+
+	/* Are we supposed to wait for STANDBYWFIL2? */
+
+	/*
+	 * Disable cluster-level coherency by masking
+	 * incoming snoops and DVM messages:
+	 */
+	cci_disable_port_by_cpu(read_cpuid_mpidr());
+}
+
+static const struct mcpm_platform_ops sunxi_power_ops = {
+	.cpu_powerup		= sunxi_cpu_powerup,
+	.cluster_powerup	= sunxi_cluster_powerup,
+	.cpu_cache_disable	= sunxi_cpu_cache_disable,
+	.cluster_cache_disable	= sunxi_cluster_cache_disable,
+};
+
+/*
+ * Enable cluster-level coherency, in preparation for turning on the MMU.
+ *
+ * Also enable regional clock gating and L2 data latency settings for
+ * Cortex-A15.
+ */
+static void __naked sunxi_power_up_setup(unsigned int affinity_level)
+{
+	asm volatile (
+		"mrc	p15, 0, r1, c0, c0, 0\n"
+		"movw	r2, #" __stringify(ARM_CPU_PART_MASK & 0xffff) "\n"
+		"movt	r2, #" __stringify(ARM_CPU_PART_MASK >> 16) "\n"
+		"and	r1, r1, r2\n"
+		"movw	r2, #" __stringify(ARM_CPU_PART_CORTEX_A15 & 0xffff) "\n"
+		"movt	r2, #" __stringify(ARM_CPU_PART_CORTEX_A15 >> 16) "\n"
+		"cmp	r1, r2\n"
+		"bne	not_a15\n"
+
+		/* L2CTRL: Enable CPU regional clock gates */
+		"mrc p15, 1, r1, c15, c0, 4\n"
+		"orr r1, r1, #(0x1<<31)\n"
+		"mcr p15, 1, r1, c15, c0, 4\n"
+
+		/* L2ACTLR */
+		"mrc p15, 1, r1, c15, c0, 0\n"
+		/* Enable L2, GIC, and Timer regional clock gates */
+		"orr r1, r1, #(0x1<<26)\n"
+		/* Disable clean/evict from being pushed to external */
+		"orr r1, r1, #(0x1<<3)\n"
+		"mcr p15, 1, r1, c15, c0, 0\n"
+
+		/* L2 data RAM latency */
+		"mrc p15, 1, r1, c9, c0, 2\n"
+		"bic r1, r1, #(0x7<<0)\n"
+		"orr r1, r1, #(0x3<<0)\n"
+		"mcr p15, 1, r1, c9, c0, 2\n"
+		"not_a15:\n"
+
+		"cmp	r0, #1\n"
+		"bxne	lr\n"
+		"b	cci_enable_port_for_self"
+	);
+}
+
+static void sunxi_mcpm_setup_entry_point(void)
+{
+	__raw_writel(virt_to_phys(mcpm_entry_point),
+		     prcm_base + PRCM_CPU_SOFT_ENTRY_REG);
+}
+
+static int __init sunxi_mcpm_init(void)
+{
+	struct device_node *node;
+	int ret;
+
+	if (!of_machine_is_compatible("allwinner,sun9i-a80"))
+		return -ENODEV;
+
+	if (!cci_probed())
+		return -ENODEV;
+
+	node = of_find_compatible_node(NULL, NULL,
+			"allwinner,sun9i-a80-cpucfg");
+	if (!node)
+		return -ENODEV;
+
+	cpucfg_base = of_iomap(node, 0);
+	of_node_put(node);
+	if (!cpucfg_base) {
+		pr_err("%s: failed to map CPUCFG registers\n", __func__);
+		return -ENOMEM;
+	}
+
+	node = of_find_compatible_node(NULL, NULL,
+			"allwinner,sun9i-a80-prcm");
+	if (!node)
+		return -ENODEV;
+
+	prcm_base = of_iomap(node, 0);
+	of_node_put(node);
+	if (!prcm_base) {
+		pr_err("%s: failed to map PRCM registers\n", __func__);
+		iounmap(prcm_base);
+		return -ENOMEM;
+	}
+
+	ret = mcpm_platform_register(&sunxi_power_ops);
+	if (!ret)
+		ret = mcpm_sync_init(sunxi_power_up_setup);
+	/* turn on the CCI */
+	if (!ret)
+		ret = mcpm_loopback(sunxi_cluster_cache_disable);
+	if (ret) {
+		iounmap(cpucfg_base);
+		iounmap(prcm_base);
+		return ret;
+	}
+
+	mcpm_smp_set_ops();
+
+	pr_info("sunxi MCPM support installed\n");
+
+	sunxi_mcpm_setup_entry_point();
+
+	return ret;
+}
+
+early_initcall(sunxi_mcpm_init);
-- 
2.1.4

[RFC 2/7] ARM: dts: sun9i: Add CCI-400 device nodes for A80

[Chen-Yu Tsai]

The A80 includes an ARM CCI-400 interconnect to support multi-cluster
CPU caches.

Also add the default clock frequency for the CPUs.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/boot/dts/sun9i-a80.dtsi | 46 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
index ca272e92b85d..200e712fbf0e 100644
--- a/arch/arm/boot/dts/sun9i-a80.dtsi
+++ b/arch/arm/boot/dts/sun9i-a80.dtsi
@@ -58,48 +58,64 @@
 		cpu0: cpu@0 {
 			compatible = "arm,cortex-a7";
 			device_type = "cpu";
+			cci-control-port = <&cci_control0>;
+			clock-frequency = <12000000>;
 			reg = <0x0>;
 		};
 
 		cpu1: cpu@1 {
 			compatible = "arm,cortex-a7";
 			device_type = "cpu";
+			cci-control-port = <&cci_control0>;
+			clock-frequency = <12000000>;
 			reg = <0x1>;
 		};
 
 		cpu2: cpu@2 {
 			compatible = "arm,cortex-a7";
 			device_type = "cpu";
+			cci-control-port = <&cci_control0>;
+			clock-frequency = <12000000>;
 			reg = <0x2>;
 		};
 
 		cpu3: cpu@3 {
 			compatible = "arm,cortex-a7";
 			device_type = "cpu";
+			cci-control-port = <&cci_control0>;
+			clock-frequency = <12000000>;
 			reg = <0x3>;
 		};
 
 		cpu4: cpu@100 {
 			compatible = "arm,cortex-a15";
 			device_type = "cpu";
+			cci-control-port = <&cci_control1>;
+			clock-frequency = <9000000>;
 			reg = <0x100>;
 		};
 
 		cpu5: cpu@101 {
 			compatible = "arm,cortex-a15";
 			device_type = "cpu";
+			cci-control-port = <&cci_control1>;
+			clock-frequency = <9000000>;
 			reg = <0x101>;
 		};
 
 		cpu6: cpu@102 {
 			compatible = "arm,cortex-a15";
 			device_type = "cpu";
+			cci-control-port = <&cci_control1>;
+			clock-frequency = <9000000>;
 			reg = <0x102>;
 		};
 
 		cpu7: cpu@103 {
 			compatible = "arm,cortex-a15";
 			device_type = "cpu";
+			cci-control-port = <&cci_control1>;
+			clock-frequency = <9000000>;
 			reg = <0x103>;
 		};
 	};
@@ -522,6 +538,36 @@
 			interrupts = <GIC_PPI 9 (GIC_CPU_MASK_SIMPLE(4) | IRQ_TYPE_LEVEL_HIGH)>;
 		};
 
+		cci: cci@01c90000 {
+			compatible = "arm,cci-400";
+			#address-cells = <1>;
+			#size-cells = <1>;
+			reg = <0x01c90000 0x1000>;
+			ranges = <0x0 0x01c90000 0x10000>;
+
+			cci_control0: slave-if@4000 {
+				compatible = "arm,cci-400-ctrl-if";
+				interface-type = "ace";
+				reg = <0x4000 0x1000>;
+			};
+
+			cci_control1: slave-if@5000 {
+				compatible = "arm,cci-400-ctrl-if";
+				interface-type = "ace";
+				reg = <0x5000 0x1000>;
+			};
+
+			pmu@9000 {
+				 compatible = "arm,cci-400-pmu,r1";
+				 reg = <0x9000 0x5000>;
+				 interrupts = <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+					      <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+					      <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+					      <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>,
+					      <GIC_SPI 134 IRQ_TYPE_LEVEL_HIGH>;
+			};
+		};
+
 		ahb0_resets: reset@060005a0 {
 			#reset-cells = <1>;
 			compatible = "allwinner,sun6i-a31-clock-reset";
-- 
2.1.4

[RFC 3/7] ARM: dts: sun9i: Add CPUCFG device node for A80 dtsi

[Chen-Yu Tsai]

CPUCFG is a collection of registers that are mapped to the SoC's signals
from each individual processor core and associated peripherals, such as
resets for processors, L1/L2 cache and other things.

These registers are used for SMP bringup and CPU hotplugging.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/boot/dts/sun9i-a80.dtsi | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
index 200e712fbf0e..c45735398f6a 100644
--- a/arch/arm/boot/dts/sun9i-a80.dtsi
+++ b/arch/arm/boot/dts/sun9i-a80.dtsi
@@ -458,6 +458,11 @@
 			#phy-cells = <0>;
 		};
 
+		cpucfg@01700000 {
+			compatible = "allwinner,sun9i-a80-cpucfg";
+			reg = <0x01700000 0x100>;
+		};
+
 		mmc0: mmc@01c0f000 {
 			compatible = "allwinner,sun5i-a13-mmc";
 			reg = <0x01c0f000 0x1000>;
-- 
2.1.4

[RFC 4/7] ARM: dts: sun9i: Add PRCM device node for the A80 dtsi

[Chen-Yu Tsai]

The PRCM is a collection of clock controls, reset controls, and various
power switches/gates. Some of these can be independently listed and
supported, while a number of CPU related ones are used in tandem with
CPUCFG for SMP bringup and CPU hotplugging.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/boot/dts/sun9i-a80.dtsi | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
index c45735398f6a..1507bd2a88f0 100644
--- a/arch/arm/boot/dts/sun9i-a80.dtsi
+++ b/arch/arm/boot/dts/sun9i-a80.dtsi
@@ -797,6 +797,11 @@
 			interrupts = <GIC_SPI 36 IRQ_TYPE_LEVEL_HIGH>;
 		};
 
+		prcm@01700000 {
+			compatible = "allwinner,sun9i-a80-prcm";
+			reg = <0x08001400 0x200>;
+		};
+
 		r_uart: serial@08002800 {
 			compatible = "snps,dw-apb-uart";
 			reg = <0x08002800 0x400>;
-- 
2.1.4

[RFC 5/7] ARM: sunxi: mcpm: Support CPU/cluster power down and hotplugging for cpu1~7

[Chen-Yu Tsai]

The primary core (cpu0) requires setting flags to have the BROM bounce
execution to the SMP software entry code.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/mach-sunxi/mcpm.c | 103 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 99 insertions(+), 4 deletions(-)

diff --git a/arch/arm/mach-sunxi/mcpm.c b/arch/arm/mach-sunxi/mcpm.c
index cf9cbf268d29..5ea4b488890c 100644
--- a/arch/arm/mach-sunxi/mcpm.c
+++ b/arch/arm/mach-sunxi/mcpm.c
@@ -15,7 +15,9 @@
 #include <linux/arm-cci.h>
 #include <linux/delay.h>
 #include <linux/io.h>
+#include <linux/iopoll.h>
 #include <linux/of_address.h>
+#include <linux/irqchip/arm-gic.h>
 
 #include <asm/cputype.h>
 #include <asm/cp15.h>
@@ -33,6 +35,9 @@
 #define CPUCFG_CX_CTRL_REG0_L2_RST_DISABLE_A15	BIT(0)
 #define CPUCFG_CX_CTRL_REG1(c)		(0x10 * (c) + 0x4)
 #define CPUCFG_CX_CTRL_REG1_ACINACTM	BIT(0)
+#define CPUCFG_CX_STATUS(c)		(0x30 * (c) + 0x4)
+#define CPUCFG_CX_STATUS_STANDBYWFI(n)	BIT(16 + (n))
+#define CPUCFG_CX_STATUS_STANDBYWFIL2	BIT(0)
 #define CPUCFG_CX_RST_CTRL(c)		(0x80 + 0x4 * (c))
 #define CPUCFG_CX_RST_CTRL_DBG_SOC_RST	BIT(24)
 #define CPUCFG_CX_RST_CTRL_ETM_RST(n)	BIT(20 + (n))
@@ -221,6 +226,15 @@ static int sunxi_cluster_powerup(unsigned int cluster)
 	return 0;
 }
 
+static void sunxi_cpu_powerdown_prepare(unsigned int cpu, unsigned int cluster)
+{
+	gic_cpu_if_down();
+}
+
+static void sunxi_cluster_powerdown_prepare(unsigned int cluster)
+{
+}
+
 static void sunxi_cpu_cache_disable(void)
 {
 	/* Disable and flush the local CPU cache. */
@@ -253,11 +267,92 @@ static void sunxi_cluster_cache_disable(void)
 	cci_disable_port_by_cpu(read_cpuid_mpidr());
 }
 
+static int sunxi_cpu_powerdown(unsigned int cpu, unsigned int cluster)
+{
+	u32 reg;
+
+	pr_debug("%s: cpu %u cluster %u\n", __func__, cpu, cluster);
+	if (cpu >= SUNXI_CPUS_PER_CLUSTER || cluster >= SUNXI_NR_CLUSTERS)
+		return -EINVAL;
+
+	/* gate processor power */
+	reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	reg |= PRCM_PWROFF_GATING_REG_CORE(cpu);
+	writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	udelay(20);
+
+	/* close power switch */
+	sunxi_cpu_power_switch_set(cpu, cluster, false);
+
+	return 0;
+}
+
+static int sunxi_cluster_powerdown(unsigned int cluster)
+{
+	u32 reg;
+
+	pr_debug("%s: cluster %u\n", __func__, cluster);
+	if (cluster >= SUNXI_NR_CLUSTERS)
+		return -EINVAL;
+
+	/* clear cluster power gate */
+	reg = readl(prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	reg &= ~PRCM_PWROFF_GATING_REG_CLUSTER;
+	writel(reg, prcm_base + PRCM_PWROFF_GATING_REG(cluster));
+	udelay(20);
+
+	return 0;
+}
+
+static int sunxi_wait_for_powerdown(unsigned int cpu, unsigned int cluster)
+{
+	int ret;
+	u32 reg;
+
+	pr_debug("%s: cpu %u cluster %u\n", __func__, cpu, cluster);
+
+	/* wait for CPU core to enter WFI */
+	ret = readl_poll_timeout(cpucfg_base + CPUCFG_CX_STATUS(cluster), reg,
+				 reg & CPUCFG_CX_STATUS_STANDBYWFI(cpu),
+				 1000, 100000);
+
+	if (ret)
+		return ret;
+
+	/* power down CPU core */
+	sunxi_cpu_powerdown(cpu, cluster);
+
+	if (__mcpm_cluster_state(cluster) != CLUSTER_DOWN)
+		return 0;
+
+	/* last man standing, assert ACINACTM */
+	reg = readl(cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster));
+	reg |= CPUCFG_CX_CTRL_REG1_ACINACTM;
+	writel(reg, cpucfg_base + CPUCFG_CX_CTRL_REG1(cluster));
+
+	/* wait for cluster L2 WFI */
+	ret = readl_poll_timeout(cpucfg_base + CPUCFG_CX_STATUS(cluster), reg,
+				 reg & CPUCFG_CX_STATUS_STANDBYWFIL2,
+				 1000, 100000);
+	if (ret) {
+		pr_warn("%s: cluster %u time out waiting for STANDBYWFIL2\n",
+				__func__, cluster);
+		return ret;
+	}
+
+	sunxi_cluster_powerdown(cluster);
+
+	return 0;
+}
+
 static const struct mcpm_platform_ops sunxi_power_ops = {
-	.cpu_powerup		= sunxi_cpu_powerup,
-	.cluster_powerup	= sunxi_cluster_powerup,
-	.cpu_cache_disable	= sunxi_cpu_cache_disable,
-	.cluster_cache_disable	= sunxi_cluster_cache_disable,
+	.cpu_powerup		   = sunxi_cpu_powerup,
+	.cpu_powerdown_prepare	   = sunxi_cpu_powerdown_prepare,
+	.cluster_powerup	   = sunxi_cluster_powerup,
+	.cluster_powerdown_prepare = sunxi_cluster_powerdown_prepare,
+	.cpu_cache_disable	   = sunxi_cpu_cache_disable,
+	.cluster_cache_disable	   = sunxi_cluster_cache_disable,
+	.wait_for_powerdown	   = sunxi_wait_for_powerdown,
 };
 
 /*
-- 
2.1.4

[RFC 6/7] ARM: sunxi: mcpm: Support cpu0 hotplug

[Chen-Yu Tsai]

This sets the flags that makes BROM jump execution on the
primary core (cpu0)to the SMP software entry code.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/mach-sunxi/mcpm.c | 32 ++++++++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)

diff --git a/arch/arm/mach-sunxi/mcpm.c b/arch/arm/mach-sunxi/mcpm.c
index 5ea4b488890c..e74e8aab2560 100644
--- a/arch/arm/mach-sunxi/mcpm.c
+++ b/arch/arm/mach-sunxi/mcpm.c
@@ -58,8 +58,12 @@
 #define PRCM_PWR_SWITCH_REG(c, cpu)	(0x140 + 0x10 * (c) + 0x4 * (cpu))
 #define PRCM_CPU_SOFT_ENTRY_REG		0x164
 
+#define CPU0_SUPPORT_HOTPLUG_MAGIC0	0xFA50392F
+#define CPU0_SUPPORT_HOTPLUG_MAGIC1	0x790DCA3A
+
 static void __iomem *cpucfg_base;
 static void __iomem *prcm_base;
+static void __iomem *sram_b_smp_base;
 
 static int sunxi_cpu_power_switch_set(unsigned int cpu, unsigned int cluster,
 				      bool enable)
@@ -93,6 +97,17 @@ static int sunxi_cpu_power_switch_set(unsigned int cpu, unsigned int cluster,
 	return 0;
 }
 
+static void sunxi_cpu0_hotplug_support_set(bool enable)
+{
+	if (enable) {
+		writel(CPU0_SUPPORT_HOTPLUG_MAGIC0, sram_b_smp_base);
+		writel(CPU0_SUPPORT_HOTPLUG_MAGIC1, sram_b_smp_base + 0x4);
+	} else {
+		writel(0x0, sram_b_smp_base);
+		writel(0x0, sram_b_smp_base + 0x4);
+	}
+}
+
 static int sunxi_cpu_powerup(unsigned int cpu, unsigned int cluster)
 {
 	u32 reg;
@@ -101,6 +116,10 @@ static int sunxi_cpu_powerup(unsigned int cpu, unsigned int cluster)
 	if (cpu >= SUNXI_CPUS_PER_CLUSTER || cluster >= SUNXI_NR_CLUSTERS)
 		return -EINVAL;
 
+	/* Set hotplug support magic flags for cpu0 */
+	if (cluster == 0 && cpu == 0)
+		sunxi_cpu0_hotplug_support_set(true);
+
 	/* assert processor power-on reset */
 	reg = readl(prcm_base + PRCM_CPU_PO_RST_CTRL(cluster));
 	reg &= ~PRCM_CPU_PO_RST_CTRL_CORE(cpu);
@@ -441,6 +460,19 @@ static int __init sunxi_mcpm_init(void)
 		return -ENOMEM;
 	}
 
+	node = of_find_compatible_node(NULL, NULL,
+			"allwinner,sun9i-smp-sram");
+	if (!node)
+		return -ENODEV;
+
+	sram_b_smp_base = of_iomap(node, 0);
+	of_node_put(node);
+	if (!sram_b_smp_base) {
+		pr_err("%s: failed to map secure SRAM\n", __func__);
+		iounmap(sram_b_smp_base);
+		return -ENOMEM;
+	}
+
 	ret = mcpm_platform_register(&sunxi_power_ops);
 	if (!ret)
 		ret = mcpm_sync_init(sunxi_power_up_setup);
-- 
2.1.4

[RFC 7/7] ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

[Chen-Yu Tsai]

The A80 stores some magic flags in a portion of the secure SRAM. The
BROM jumps directly to the software entry point set by the SMP code
if the flags are set. This is required for CPU0 hotplugging.

Signed-off-by: Chen-Yu Tsai <wens@csie.org>
---
 arch/arm/boot/dts/sun9i-a80.dtsi | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
index 1507bd2a88f0..0695215634d4 100644
--- a/arch/arm/boot/dts/sun9i-a80.dtsi
+++ b/arch/arm/boot/dts/sun9i-a80.dtsi
@@ -366,6 +366,26 @@
 		 */
 		ranges = <0 0 0 0x20000000>;
 
+		sram_b: sram@00020000 {
+			/* 256 KiB secure SRAM at 0x20000 */
+			compatible = "mmio-sram";
+			reg = <0x00020000 0x40000>;
+
+			#address-cells = <1>;
+			#size-cells = <1>;
+			ranges = <0 0x00020000 0x40000>;
+
+			smp-sram@1000 {
+				/*
+				 * This is checked by BROM to determine if
+				 * cpu0 should jump to SMP entry vector
+				 */
+				compatible = "allwinner,sun9i-smp-sram";
+				reg = <0x1000 0x8>;
+			};
+		};
+
+
 		ehci0: usb@00a00000 {
 			compatible = "allwinner,sun9i-a80-ehci", "generic-ehci";
 			reg = <0x00a00000 0x100>;
-- 
2.1.4

Re: [linux-sunxi] [RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[Ian Campbell]

On Thu, 2015-05-14 at 14:10 +0800, Chen-Yu Tsai wrote:
> This is my attempt to support SMP and CPU hot plugging on the Allwinner
> A80 SoC. The A80 is a big.Little processor with 2 clusters of 4x Cortex-A7
> and 4x Cortex-A15 cores.

I thought there was a preference these days to support this sort of
thing via support PSCI in the firmware, which allows for other things
such as non-secure-world etc.

Ian.

Re: [linux-sunxi] [RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

Hi Ian,

On Sat, May 16, 2015 at 11:08:46AM +0100, Ian Campbell wrote:
> On Thu, 2015-05-14 at 14:10 +0800, Chen-Yu Tsai wrote:
> > This is my attempt to support SMP and CPU hot plugging on the Allwinner
> > A80 SoC. The A80 is a big.Little processor with 2 clusters of 4x Cortex-A7
> > and 4x Cortex-A15 cores.
> 
> I thought there was a preference these days to support this sort of
> thing via support PSCI in the firmware, which allows for other things
> such as non-secure-world etc.

Yes, it is the preferred way. Meaning that if someone wants to do that
work, he's very much welcome and encouraged to do so. But if no one's
doing it, then we still have to have a way to bringup the secondary
CPUs.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC 2/7] ARM: dts: sun9i: Add CCI-400 device nodes for A80

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1890 bytes --]

On Thu, May 14, 2015 at 02:10:06PM +0800, Chen-Yu Tsai wrote:
> The A80 includes an ARM CCI-400 interconnect to support multi-cluster
> CPU caches.
> 
> Also add the default clock frequency for the CPUs.
> 
> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
> ---
>  arch/arm/boot/dts/sun9i-a80.dtsi | 46 ++++++++++++++++++++++++++++++++++++++++
>  1 file changed, 46 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
> index ca272e92b85d..200e712fbf0e 100644
> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
> @@ -58,48 +58,64 @@
>  		cpu0: cpu@0 {
>  			compatible = "arm,cortex-a7";
>  			device_type = "cpu";
> +			cci-control-port = <&cci_control0>;
> +			clock-frequency = <12000000>;
>  			reg = <0x0>;
>  		};
>  
>  		cpu1: cpu@1 {
>  			compatible = "arm,cortex-a7";
>  			device_type = "cpu";
> +			cci-control-port = <&cci_control0>;
> +			clock-frequency = <12000000>;
>  			reg = <0x1>;
>  		};
>  
>  		cpu2: cpu@2 {
>  			compatible = "arm,cortex-a7";
>  			device_type = "cpu";
> +			cci-control-port = <&cci_control0>;
> +			clock-frequency = <12000000>;
>  			reg = <0x2>;
>  		};
>  
>  		cpu3: cpu@3 {
>  			compatible = "arm,cortex-a7";
>  			device_type = "cpu";
> +			cci-control-port = <&cci_control0>;
> +			clock-frequency = <12000000>;
>  			reg = <0x3>;
>  		};
>  
>  		cpu4: cpu@100 {
>  			compatible = "arm,cortex-a15";
>  			device_type = "cpu";
> +			cci-control-port = <&cci_control1>;
> +			clock-frequency = <9000000>;

Isn't the clock frequency property is supposed to be the maximum
frequency of that CPU in Linux?

It looks odd that the A15 are clocked at a lower frequency than the
A7...

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [RFC 4/7] ARM: dts: sun9i: Add PRCM device node for the A80 dtsi

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]

On Thu, May 14, 2015 at 02:10:08PM +0800, Chen-Yu Tsai wrote:
> The PRCM is a collection of clock controls, reset controls, and various
> power switches/gates. Some of these can be independently listed and
> supported, while a number of CPU related ones are used in tandem with
> CPUCFG for SMP bringup and CPU hotplugging.
> 
> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
> ---
>  arch/arm/boot/dts/sun9i-a80.dtsi | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
> index c45735398f6a..1507bd2a88f0 100644
> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
> @@ -797,6 +797,11 @@
>  			interrupts = <GIC_SPI 36 IRQ_TYPE_LEVEL_HIGH>;
>  		};
>  
> +		prcm@01700000 {
> +			compatible = "allwinner,sun9i-a80-prcm";
> +			reg = <0x08001400 0x200>;
> +		};
> +

The unit address doesn't match the content of the reg property.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [linux-sunxi] [RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[Nicolas Pitre]

On Sun, 17 May 2015, Maxime Ripard wrote:

> Hi Ian,
> 
> On Sat, May 16, 2015 at 11:08:46AM +0100, Ian Campbell wrote:
> > On Thu, 2015-05-14 at 14:10 +0800, Chen-Yu Tsai wrote:
> > > This is my attempt to support SMP and CPU hot plugging on the Allwinner
> > > A80 SoC. The A80 is a big.Little processor with 2 clusters of 4x Cortex-A7
> > > and 4x Cortex-A15 cores.
> > 
> > I thought there was a preference these days to support this sort of
> > thing via support PSCI in the firmware, which allows for other things
> > such as non-secure-world etc.
> 
> Yes, it is the preferred way. Meaning that if someone wants to do that
> work, he's very much welcome and encouraged to do so. But if no one's
> doing it, then we still have to have a way to bringup the secondary
> CPUs.

And doing so in the kernel (at least initially) is simpler, and so much 
easier to fix when it is broken.  We've seen a few systems already where 
power management is crippled because no one is able/allowed/willing to 
fix the broken firmware.


Nicolas

Re: [linux-sunxi] [RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1551 bytes --]

Hi Nicolas,

On Mon, May 18, 2015 at 01:19:18AM -0400, Nicolas Pitre wrote:
> On Sun, 17 May 2015, Maxime Ripard wrote:
> 
> > Hi Ian,
> > 
> > On Sat, May 16, 2015 at 11:08:46AM +0100, Ian Campbell wrote:
> > > On Thu, 2015-05-14 at 14:10 +0800, Chen-Yu Tsai wrote:
> > > > This is my attempt to support SMP and CPU hot plugging on the Allwinner
> > > > A80 SoC. The A80 is a big.Little processor with 2 clusters of 4x Cortex-A7
> > > > and 4x Cortex-A15 cores.
> > > 
> > > I thought there was a preference these days to support this sort of
> > > thing via support PSCI in the firmware, which allows for other things
> > > such as non-secure-world etc.
> > 
> > Yes, it is the preferred way. Meaning that if someone wants to do that
> > work, he's very much welcome and encouraged to do so. But if no one's
> > doing it, then we still have to have a way to bringup the secondary
> > CPUs.
> 
> And doing so in the kernel (at least initially) is simpler, and so much 
> easier to fix when it is broken.  We've seen a few systems already where 
> power management is crippled because no one is able/allowed/willing to 
> fix the broken firmware.

To be fair, our platform support is done by hobbyist, and our only
implementation of PSCI is done through mainline u-boot, which means we
have an easy way to fix any issue we might have, and that it's
possible, otherwise we wouldn't have it at all.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [linux-sunxi] [RFC 0/7] ARM: sun9i: SMP support with Multi-Cluster Power Management

[Chen-Yu Tsai]

On Sat, May 16, 2015 at 6:08 PM, Ian Campbell <ijc@hellion.org.uk> wrote:
> On Thu, 2015-05-14 at 14:10 +0800, Chen-Yu Tsai wrote:
>> This is my attempt to support SMP and CPU hot plugging on the Allwinner
>> A80 SoC. The A80 is a big.Little processor with 2 clusters of 4x Cortex-A7
>> and 4x Cortex-A15 cores.
>
> I thought there was a preference these days to support this sort of
> thing via support PSCI in the firmware, which allows for other things
> such as non-secure-world etc.

Ideally, yes. However, U-boot only has minimal A80 support for now.
Also, multi-cluster support is not as simple as what we had for the
A20. The kernel already has common multi-cluster code. With it we
can do full SMP/hotplug support. Kernel code is also easier to write
and debug than PSCI, which ATM is all per-platform assembly.

Also, my A31 and A23 PSCI implementations still need some work.

ChenYu

Re: [RFC 4/7] ARM: dts: sun9i: Add PRCM device node for the A80 dtsi

[Chen-Yu Tsai]

On Sun, May 17, 2015 at 10:54 PM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> On Thu, May 14, 2015 at 02:10:08PM +0800, Chen-Yu Tsai wrote:
>> The PRCM is a collection of clock controls, reset controls, and various
>> power switches/gates. Some of these can be independently listed and
>> supported, while a number of CPU related ones are used in tandem with
>> CPUCFG for SMP bringup and CPU hotplugging.
>>
>> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
>> ---
>>  arch/arm/boot/dts/sun9i-a80.dtsi | 5 +++++
>>  1 file changed, 5 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
>> index c45735398f6a..1507bd2a88f0 100644
>> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
>> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
>> @@ -797,6 +797,11 @@
>>                       interrupts = <GIC_SPI 36 IRQ_TYPE_LEVEL_HIGH>;
>>               };
>>
>> +             prcm@01700000 {
>> +                     compatible = "allwinner,sun9i-a80-prcm";
>> +                     reg = <0x08001400 0x200>;
>> +             };
>> +
>
> The unit address doesn't match the content of the reg property.

Obvious copy-paste error (facepalm)

ChenYu

Re: [RFC 2/7] ARM: dts: sun9i: Add CCI-400 device nodes for A80

[Chen-Yu Tsai]

On Sun, May 17, 2015 at 10:51 PM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> On Thu, May 14, 2015 at 02:10:06PM +0800, Chen-Yu Tsai wrote:
>> The A80 includes an ARM CCI-400 interconnect to support multi-cluster
>> CPU caches.
>>
>> Also add the default clock frequency for the CPUs.
>>
>> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
>> ---
>>  arch/arm/boot/dts/sun9i-a80.dtsi | 46 ++++++++++++++++++++++++++++++++++++++++
>>  1 file changed, 46 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
>> index ca272e92b85d..200e712fbf0e 100644
>> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
>> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
>> @@ -58,48 +58,64 @@
>>               cpu0: cpu@0 {
>>                       compatible = "arm,cortex-a7";
>>                       device_type = "cpu";
>> +                     cci-control-port = <&cci_control0>;
>> +                     clock-frequency = <12000000>;
>>                       reg = <0x0>;
>>               };
>>
>>               cpu1: cpu@1 {
>>                       compatible = "arm,cortex-a7";
>>                       device_type = "cpu";
>> +                     cci-control-port = <&cci_control0>;
>> +                     clock-frequency = <12000000>;
>>                       reg = <0x1>;
>>               };
>>
>>               cpu2: cpu@2 {
>>                       compatible = "arm,cortex-a7";
>>                       device_type = "cpu";
>> +                     cci-control-port = <&cci_control0>;
>> +                     clock-frequency = <12000000>;
>>                       reg = <0x2>;
>>               };
>>
>>               cpu3: cpu@3 {
>>                       compatible = "arm,cortex-a7";
>>                       device_type = "cpu";
>> +                     cci-control-port = <&cci_control0>;
>> +                     clock-frequency = <12000000>;
>>                       reg = <0x3>;
>>               };
>>
>>               cpu4: cpu@100 {
>>                       compatible = "arm,cortex-a15";
>>                       device_type = "cpu";
>> +                     cci-control-port = <&cci_control1>;
>> +                     clock-frequency = <9000000>;
>
> Isn't the clock frequency property is supposed to be the maximum
> frequency of that CPU in Linux?
>
> It looks odd that the A15 are clocked at a lower frequency than the
> A7...

You're right. Looking at the FEX file, the A15s can go up to 1.8 GHz.
I'll update the numbers.

ChenYu

Re: [RFC 7/7] ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 1149 bytes --]

On Thu, May 14, 2015 at 02:10:11PM +0800, Chen-Yu Tsai wrote:
> The A80 stores some magic flags in a portion of the secure SRAM. The
> BROM jumps directly to the software entry point set by the SMP code
> if the flags are set. This is required for CPU0 hotplugging.
> 
> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
> ---
>  arch/arm/boot/dts/sun9i-a80.dtsi | 20 ++++++++++++++++++++
>  1 file changed, 20 insertions(+)
> 
> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
> index 1507bd2a88f0..0695215634d4 100644
> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
> @@ -366,6 +366,26 @@
>  		 */
>  		ranges = <0 0 0 0x20000000>;
>  
> +		sram_b: sram@00020000 {
> +			/* 256 KiB secure SRAM at 0x20000 */
> +			compatible = "mmio-sram";
> +			reg = <0x00020000 0x40000>;
> +

We should probably add a property to that SRAM to tell the driver not
to access it if we're not booted in secure mode.

Otherwise, bad things might happen.

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [linux-sunxi] Re: [RFC 7/7] ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

[Chen-Yu Tsai]

On Wed, May 20, 2015 at 6:08 PM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> On Thu, May 14, 2015 at 02:10:11PM +0800, Chen-Yu Tsai wrote:
>> The A80 stores some magic flags in a portion of the secure SRAM. The
>> BROM jumps directly to the software entry point set by the SMP code
>> if the flags are set. This is required for CPU0 hotplugging.
>>
>> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
>> ---
>>  arch/arm/boot/dts/sun9i-a80.dtsi | 20 ++++++++++++++++++++
>>  1 file changed, 20 insertions(+)
>>
>> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
>> index 1507bd2a88f0..0695215634d4 100644
>> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
>> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
>> @@ -366,6 +366,26 @@
>>                */
>>               ranges = <0 0 0 0x20000000>;
>>
>> +             sram_b: sram@00020000 {
>> +                     /* 256 KiB secure SRAM at 0x20000 */
>> +                     compatible = "mmio-sram";
>> +                     reg = <0x00020000 0x40000>;
>> +
>
> We should probably add a property to that SRAM to tell the driver not
> to access it if we're not booted in secure mode.

(CC-ing Heiko...)

That kind of puts architecture (ARM) dependent code into a platform
driver. Furthermore, AFAIK there isn't a safe way to check if we're
in secure mode or not. Checking SCR raises an undefined instruction
exception in non-secure mode. Can the kernel handle that? I really
don't understand this bit well.

> Otherwise, bad things might happen.

The kernel (or rather the bootloader) boots in secure mode by default,
and we don't have any bootloader support to boot into non-secure mode
ATM. Couldn't we have the bootloader mark the SRAM as disabled when
booting into non-secure when we add that support?


ChenYu

Re: [linux-sunxi] Re: [RFC 7/7] ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 3167 bytes --]

On Sun, May 24, 2015 at 11:55:22PM +0800, Chen-Yu Tsai wrote:
> On Wed, May 20, 2015 at 6:08 PM, Maxime Ripard
> <maxime.ripard@free-electrons.com> wrote:
> > On Thu, May 14, 2015 at 02:10:11PM +0800, Chen-Yu Tsai wrote:
> >> The A80 stores some magic flags in a portion of the secure SRAM. The
> >> BROM jumps directly to the software entry point set by the SMP code
> >> if the flags are set. This is required for CPU0 hotplugging.
> >>
> >> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
> >> ---
> >>  arch/arm/boot/dts/sun9i-a80.dtsi | 20 ++++++++++++++++++++
> >>  1 file changed, 20 insertions(+)
> >>
> >> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
> >> index 1507bd2a88f0..0695215634d4 100644
> >> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
> >> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
> >> @@ -366,6 +366,26 @@
> >>                */
> >>               ranges = <0 0 0 0x20000000>;
> >>
> >> +             sram_b: sram@00020000 {
> >> +                     /* 256 KiB secure SRAM at 0x20000 */
> >> +                     compatible = "mmio-sram";
> >> +                     reg = <0x00020000 0x40000>;
> >> +
> >
> > We should probably add a property to that SRAM to tell the driver not
> > to access it if we're not booted in secure mode.
> 
> (CC-ing Heiko...)
> 
> That kind of puts architecture (ARM) dependent code into a platform
> driver. Furthermore, AFAIK there isn't a safe way to check if we're
> in secure mode or not. Checking SCR raises an undefined instruction
> exception in non-secure mode. Can the kernel handle that? I really
> don't understand this bit well.

That's a very good question. I'm pretty sure the kernel can know that,
since it actually prints the execution mode, and must be able to know
whether it can use the virtualization extensions or not I assume.

> > Otherwise, bad things might happen.
> 
> The kernel (or rather the bootloader) boots in secure mode by default,
> and we don't have any bootloader support to boot into non-secure mode
> ATM.

That's not really true. We do have some U-Boot patches, and U-Boot
might very well be setup to boot into HYP, even though it doesn't do
anything else.

And even so, the fact that we have no implementation yet doesn't mean
that we won't have one in a few month. So just sweeping it under the
carpet doesn't seem to be a very good solution.

> Couldn't we have the bootloader mark the SRAM as disabled when
> booting into non-secure when we add that support?

It actually changes the kernel requirements to be able to
boot. Changing that is not an option, especially since it's something
that: 1) isn't dynamic in any way, 2) will possibly break the kernel
if not done, 3) require synchronisation between the bootloader and the
kernel when a new secure SRAM is added to the DT, and require an
upgrade of the bootloader, 4) we might not be able to replace the
bootloader in the first place.

All of these issues make it look like a rather bad solution :/

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [linux-sunxi] Re: [RFC 7/7] ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

[Chen-Yu Tsai]

On Tue, May 26, 2015 at 5:24 AM, Maxime Ripard
<maxime.ripard@free-electrons.com> wrote:
> On Sun, May 24, 2015 at 11:55:22PM +0800, Chen-Yu Tsai wrote:
>> On Wed, May 20, 2015 at 6:08 PM, Maxime Ripard
>> <maxime.ripard@free-electrons.com> wrote:
>> > On Thu, May 14, 2015 at 02:10:11PM +0800, Chen-Yu Tsai wrote:
>> >> The A80 stores some magic flags in a portion of the secure SRAM. The
>> >> BROM jumps directly to the software entry point set by the SMP code
>> >> if the flags are set. This is required for CPU0 hotplugging.
>> >>
>> >> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
>> >> ---
>> >>  arch/arm/boot/dts/sun9i-a80.dtsi | 20 ++++++++++++++++++++
>> >>  1 file changed, 20 insertions(+)
>> >>
>> >> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
>> >> index 1507bd2a88f0..0695215634d4 100644
>> >> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
>> >> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
>> >> @@ -366,6 +366,26 @@
>> >>                */
>> >>               ranges = <0 0 0 0x20000000>;
>> >>
>> >> +             sram_b: sram@00020000 {
>> >> +                     /* 256 KiB secure SRAM at 0x20000 */
>> >> +                     compatible = "mmio-sram";
>> >> +                     reg = <0x00020000 0x40000>;
>> >> +
>> >
>> > We should probably add a property to that SRAM to tell the driver not
>> > to access it if we're not booted in secure mode.
>>
>> (CC-ing Heiko...)
>>
>> That kind of puts architecture (ARM) dependent code into a platform
>> driver. Furthermore, AFAIK there isn't a safe way to check if we're
>> in secure mode or not. Checking SCR raises an undefined instruction
>> exception in non-secure mode. Can the kernel handle that? I really
>> don't understand this bit well.
>
> That's a very good question. I'm pretty sure the kernel can know that,
> since it actually prints the execution mode, and must be able to know
> whether it can use the virtualization extensions or not I assume.

What you are referring to is HYP vs SVC mode. IIUC, this is not the
same as secure vs non-secure. The kernel can be booted into non-secure
SVC mode. Just disable CONFIG_CPU_V7_HAS_VIRT on mainline u-boot for
sun7i, and you get a system booted in non-secure mode. The architected
timer's non-secure interrupt firing is proof.

Which brings us back to my original question. Is there a way the kernel
can reliably detect whether it is in secure mode or not?

>> > Otherwise, bad things might happen.
>>
>> The kernel (or rather the bootloader) boots in secure mode by default,
>> and we don't have any bootloader support to boot into non-secure mode
>> ATM.
>
> That's not really true. We do have some U-Boot patches, and U-Boot
> might very well be setup to boot into HYP, even though it doesn't do
> anything else.

Same thing can be done for sun6i/sun8i. I should probably test this
and see if it has any impact on the current SMP bring-up.

> And even so, the fact that we have no implementation yet doesn't mean
> that we won't have one in a few month. So just sweeping it under the
> carpet doesn't seem to be a very good solution.
>
>> Couldn't we have the bootloader mark the SRAM as disabled when
>> booting into non-secure when we add that support?
>
> It actually changes the kernel requirements to be able to
> boot. Changing that is not an option, especially since it's something
> that: 1) isn't dynamic in any way, 2) will possibly break the kernel
> if not done, 3) require synchronisation between the bootloader and the
> kernel when a new secure SRAM is added to the DT, and require an
> upgrade of the bootloader, 4) we might not be able to replace the
> bootloader in the first place.
>
> All of these issues make it look like a rather bad solution :/

I think the discussions here depends on the question I asked earlier.
If the bootloader left the kernel in non-secure without any way to
signal this fact, then SMP probably won't work either.

Looking at the diagram on page 291 of the user manual, many critical
hardware blocks are marked as trusted (probably secure?), I don't
think we can boot Linux non-secure with the same DT...


Regards
ChenYu

Re: [linux-sunxi] Re: [RFC 7/7] ARM: dts: sun9i: Add secure SRAM node used for MCPM SMP hotplug

[Maxime Ripard]

[-- Attachment #1: Type: text/plain, Size: 5233 bytes --]

On Wed, May 27, 2015 at 12:47:49AM +0800, Chen-Yu Tsai wrote:
> On Tue, May 26, 2015 at 5:24 AM, Maxime Ripard
> <maxime.ripard@free-electrons.com> wrote:
> > On Sun, May 24, 2015 at 11:55:22PM +0800, Chen-Yu Tsai wrote:
> >> On Wed, May 20, 2015 at 6:08 PM, Maxime Ripard
> >> <maxime.ripard@free-electrons.com> wrote:
> >> > On Thu, May 14, 2015 at 02:10:11PM +0800, Chen-Yu Tsai wrote:
> >> >> The A80 stores some magic flags in a portion of the secure SRAM. The
> >> >> BROM jumps directly to the software entry point set by the SMP code
> >> >> if the flags are set. This is required for CPU0 hotplugging.
> >> >>
> >> >> Signed-off-by: Chen-Yu Tsai <wens@csie.org>
> >> >> ---
> >> >>  arch/arm/boot/dts/sun9i-a80.dtsi | 20 ++++++++++++++++++++
> >> >>  1 file changed, 20 insertions(+)
> >> >>
> >> >> diff --git a/arch/arm/boot/dts/sun9i-a80.dtsi b/arch/arm/boot/dts/sun9i-a80.dtsi
> >> >> index 1507bd2a88f0..0695215634d4 100644
> >> >> --- a/arch/arm/boot/dts/sun9i-a80.dtsi
> >> >> +++ b/arch/arm/boot/dts/sun9i-a80.dtsi
> >> >> @@ -366,6 +366,26 @@
> >> >>                */
> >> >>               ranges = <0 0 0 0x20000000>;
> >> >>
> >> >> +             sram_b: sram@00020000 {
> >> >> +                     /* 256 KiB secure SRAM at 0x20000 */
> >> >> +                     compatible = "mmio-sram";
> >> >> +                     reg = <0x00020000 0x40000>;
> >> >> +
> >> >
> >> > We should probably add a property to that SRAM to tell the driver not
> >> > to access it if we're not booted in secure mode.
> >>
> >> (CC-ing Heiko...)
> >>
> >> That kind of puts architecture (ARM) dependent code into a platform
> >> driver. Furthermore, AFAIK there isn't a safe way to check if we're
> >> in secure mode or not. Checking SCR raises an undefined instruction
> >> exception in non-secure mode. Can the kernel handle that? I really
> >> don't understand this bit well.
> >
> > That's a very good question. I'm pretty sure the kernel can know that,
> > since it actually prints the execution mode, and must be able to know
> > whether it can use the virtualization extensions or not I assume.
> 
> What you are referring to is HYP vs SVC mode. IIUC, this is not the
> same as secure vs non-secure. The kernel can be booted into non-secure
> SVC mode. Just disable CONFIG_CPU_V7_HAS_VIRT on mainline u-boot for
> sun7i, and you get a system booted in non-secure mode. The architected
> timer's non-secure interrupt firing is proof.
> 
> Which brings us back to my original question. Is there a way the kernel
> can reliably detect whether it is in secure mode or not?

Apparently, none at the architecture level.

You might have some hints at the SoC level (like we do by writing to
the secure RAM and read it back), but other SoCs might behave
differently (like an abort, if we follow the same example).

So one way might be to simply add a machine hook to determine whether
we are in secure or not.


> >> > Otherwise, bad things might happen.
> >>
> >> The kernel (or rather the bootloader) boots in secure mode by default,
> >> and we don't have any bootloader support to boot into non-secure mode
> >> ATM.
> >
> > That's not really true. We do have some U-Boot patches, and U-Boot
> > might very well be setup to boot into HYP, even though it doesn't do
> > anything else.
> 
> Same thing can be done for sun6i/sun8i. I should probably test this
> and see if it has any impact on the current SMP bring-up.

That's true, even though now we do have PSCI ;)

> > And even so, the fact that we have no implementation yet doesn't mean
> > that we won't have one in a few month. So just sweeping it under the
> > carpet doesn't seem to be a very good solution.
> >
> >> Couldn't we have the bootloader mark the SRAM as disabled when
> >> booting into non-secure when we add that support?
> >
> > It actually changes the kernel requirements to be able to
> > boot. Changing that is not an option, especially since it's something
> > that: 1) isn't dynamic in any way, 2) will possibly break the kernel
> > if not done, 3) require synchronisation between the bootloader and the
> > kernel when a new secure SRAM is added to the DT, and require an
> > upgrade of the bootloader, 4) we might not be able to replace the
> > bootloader in the first place.
> >
> > All of these issues make it look like a rather bad solution :/
> 
> I think the discussions here depends on the question I asked earlier.
> If the bootloader left the kernel in non-secure without any way to
> signal this fact, then SMP probably won't work either.
> 
> Looking at the diagram on page 291 of the user manual, many critical
> hardware blocks are marked as trusted (probably secure?), I don't
> think we can boot Linux non-secure with the same DT...

Maybe not, but our current way of working doesn't really care about
that. I haven't really seen other platforms taking that into account
either...

Maybe adding a generic property to signal that a particular device is
accessible in secure mode only would fix that?

Maxime

-- 
Maxime Ripard, Free Electrons
Embedded Linux, Kernel and Android engineering
http://free-electrons.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]