[PATCH v3 0/6] coresight: etm4x: save/restore ETMv4 context across CPU low power states

[PATCH v3 0/6] coresight: etm4x: save/restore ETMv4 context across CPU low power states

[Andrew Murray]

Some hardware will ignore bit TRCPDCR.PU which is used to signal
to hardware that power should not be removed from the trace unit.
Let's mitigate against this by conditionally saving and restoring
the trace unit state when the CPU enters low power states.

This patchset introduces a firmware property named
'arm,coresight-needs-save-restore' - when this is present the
hardware state will be conditionally saved and restored.

A module parameter 'pm_save_enable' is also introduced which can
be configured to override the firmware property.

The hardware state is only ever saved and restored when the claim
tags indicate that coresight is in use.

Changes since v2:

 - Move the PM notifier block from drvdata to file static

 - Add section names to document references

 - Add additional information to commit messages

 - Remove trcdvcvr and trcdvcmr from saved state and add a comment to
   describe why

 - Ensure TRCPDCR_PU is set after restore and add a comment to explain
   why we bother toggling TRCPDCR_PU on save/restore

 - Reword the pm_save_enable options and add comments

 - Miscellaneous style changes

 - Move device tree binding documentation to its own patch

Changes since v1:

 - Rebased onto coresight/next

 - Correcly pass bit number rather than BIT macro to coresight_timeout

 - Abort saving state if a timeout occurs

 - Fix completely broken pm_notify handling and unregister handler on error

 - Use state_needs_restore to ensure state is restored only once

 - Add module parameter description to existing boot_enable parameter
   and use module_param instead of module_param_named

 - Add firmware bindings for coresight-needs-save-restore

 - Rename 'disable_pm_save' to 'pm_save_enable' which allows for
   disabled, enabled or firmware

 - Update comment on etm4_os_lock, it incorrectly indicated that
   the code unlocks the trace registers

 - Add comments to explain use of OS lock during save/restore

 - Fix incorrect error description whilst waiting for PM stable

 - Add WARN_ON_ONCE when cpu isn't as expected during save/restore

 - Various updates to commit messages


Andrew Murray (6):
  coresight: etm4x: remove superfluous setting of os_unlock
  coresight: etm4x: use explicit barriers on enable/disable
  coresight: etm4x: use module_param instead of module_param_named
  coresight: etm4x: improve clarity of etm4_os_unlock comment
  coresight: etm4x: save/restore state across CPU low power states
  dt-bindings: arm: coresight: Add support for
    coresight-needs-save-restore

 .../devicetree/bindings/arm/coresight.txt     |   3 +
 drivers/hwtracing/coresight/coresight-etm4x.c | 348 +++++++++++++++++-
 drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
 drivers/hwtracing/coresight/coresight.c       |   2 +-
 include/linux/coresight.h                     |   8 +
 5 files changed, 415 insertions(+), 8 deletions(-)

-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v3 1/6] coresight: etm4x: remove superfluous setting of os_unlock

[Andrew Murray]

In addition to unlocking the OS lock, etm4_os_unlock will also
set the os_unlock flag. Therefore let's avoid unnecessarily
setting os_unlock flag outside of this function.

Signed-off-by: Andrew Murray <andrew.murray@arm.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 drivers/hwtracing/coresight/coresight-etm4x.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
index 7fe266194ab5..c89190d464ab 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x.c
@@ -1047,10 +1047,8 @@ static int etm4_starting_cpu(unsigned int cpu)
 		return 0;
 
 	spin_lock(&etmdrvdata[cpu]->spinlock);
-	if (!etmdrvdata[cpu]->os_unlock) {
+	if (!etmdrvdata[cpu]->os_unlock)
 		etm4_os_unlock(etmdrvdata[cpu]);
-		etmdrvdata[cpu]->os_unlock = true;
-	}
 
 	if (local_read(&etmdrvdata[cpu]->mode))
 		etm4_enable_hw(etmdrvdata[cpu]);
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v3 2/6] coresight: etm4x: use explicit barriers on enable/disable

[Andrew Murray]

Synchronization is recommended before disabling the trace registers
to prevent any start or stop points being speculative at the point
of disabling the unit (section 7.3.77 of ARM IHI 0064D).

Synchronization is also recommended after programming the trace
registers to ensure all updates are committed prior to normal code
resuming (section 4.3.7 of ARM IHI 0064D).

Let's ensure these syncronization points are present in the code
and clearly commented.

Note that we could rely on the barriers in CS_LOCK and
coresight_disclaim_device_unlocked or the context switch to user
space - however coresight may be of use in the kernel.

On armv8 the mb macro is defined as dsb(sy) - Given that the etm4x is
only used on armv8 let's directly use dsb(sy) instead of mb(). This
removes some ambiguity and makes it easier to correlate the code with
the TRM.

Signed-off-by: Andrew Murray <andrew.murray@arm.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
CC: stable@vger.kernel.org
---
 drivers/hwtracing/coresight/coresight-etm4x.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
index c89190d464ab..3825a39e9a49 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x.c
@@ -188,6 +188,13 @@ static int etm4_enable_hw(struct etmv4_drvdata *drvdata)
 		dev_err(etm_dev,
 			"timeout while waiting for Idle Trace Status\n");
 
+	/*
+	 * As recommended by section 4.3.7 ("Synchronization when using the
+	 * memory-mapped interface") of ARM IHI 0064D
+	 */
+	dsb(sy);
+	isb();
+
 done:
 	CS_LOCK(drvdata->base);
 
@@ -453,8 +460,12 @@ static void etm4_disable_hw(void *info)
 	/* EN, bit[0] Trace unit enable bit */
 	control &= ~0x1;
 
-	/* make sure everything completes before disabling */
-	mb();
+	/*
+	 * Make sure everything completes before disabling, as recommended
+	 * by section 7.3.77 ("TRCVICTLR, ViewInst Main Control Register,
+	 * SSTATUS") of ARM IHI 0064D
+	 */
+	dsb(sy);
 	isb();
 	writel_relaxed(control, drvdata->base + TRCPRGCTLR);
 
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v3 3/6] coresight: etm4x: use module_param instead of module_param_named

[Andrew Murray]

Given that the user-exposed module parameter for 'boot_enable' matches
the variable that it sets, let's use module_param instead of
module_param_named.

Let's also use octal permissions (checkpatch recommends this) and
provide a module parameter description.

Signed-off-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/hwtracing/coresight/coresight-etm4x.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
index 3825a39e9a49..451f2b24cc03 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x.c
@@ -34,7 +34,8 @@
 #include "coresight-etm-perf.h"
 
 static int boot_enable;
-module_param_named(boot_enable, boot_enable, int, S_IRUGO);
+module_param(boot_enable, int, 0444);
+MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
 
 /* The number of ETMv4 currently registered */
 static int etm4_count;
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v3 4/6] coresight: etm4x: improve clarity of etm4_os_unlock comment

[Andrew Murray]

To improve clarity, let's update the comment for etm4_os_unlock
to use the name of the register as per the ETM architecture
specification.

The existing comment is also misleading as it suggests any value
written to TRCOSLAR unlocks the trace registers, however it must
be '0' - let's also correct this.

Signed-off-by: Andrew Murray <andrew.murray@arm.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
---
 drivers/hwtracing/coresight/coresight-etm4x.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
index 451f2b24cc03..d993359e2117 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x.c
@@ -48,7 +48,7 @@ static enum cpuhp_state hp_online;
 
 static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
 {
-	/* Writing any value to ETMOSLAR unlocks the trace registers */
+	/* Writing 0 to TRCOSLAR unlocks the trace registers */
 	writel_relaxed(0x0, drvdata->base + TRCOSLAR);
 	drvdata->os_unlock = true;
 	isb();
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v3 5/6] coresight: etm4x: save/restore state across CPU low power states

[Andrew Murray]

Some hardware will ignore bit TRCPDCR.PU which is used to signal
to hardware that power should not be removed from the trace unit.
Let's mitigate against this by conditionally saving and restoring
the trace unit state when the CPU enters low power states.

This patchset introduces a firmware property named
'arm,coresight-needs-save-restore' - when this is present the
hardware state will be conditionally saved and restored.

A module parameter 'pm_save_enable' is also introduced which can
be configured to override the firmware property. This can be set
to never allow save/restore, to conditionally allow it, or to
do as the firmware indicates (default).

The hardware state is only ever saved and restored when the claim
tags indicate that coresight is in use.

Signed-off-by: Andrew Murray <andrew.murray@arm.com>
---
 drivers/hwtracing/coresight/coresight-etm4x.c | 324 ++++++++++++++++++
 drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
 drivers/hwtracing/coresight/coresight.c       |   2 +-
 include/linux/coresight.h                     |   8 +
 4 files changed, 395 insertions(+), 1 deletion(-)

diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
index d993359e2117..b0bd8158bf13 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x.c
@@ -18,6 +18,7 @@
 #include <linux/stat.h>
 #include <linux/clk.h>
 #include <linux/cpu.h>
+#include <linux/cpu_pm.h>
 #include <linux/coresight.h>
 #include <linux/coresight-pmu.h>
 #include <linux/pm_wakeup.h>
@@ -26,6 +27,7 @@
 #include <linux/uaccess.h>
 #include <linux/perf_event.h>
 #include <linux/pm_runtime.h>
+#include <linux/property.h>
 #include <asm/sections.h>
 #include <asm/local.h>
 #include <asm/virt.h>
@@ -37,6 +39,15 @@ static int boot_enable;
 module_param(boot_enable, int, 0444);
 MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
 
+#define PARAM_PM_SAVE_NEVER	0 /* never save/restore state */
+#define PARAM_PM_SAVE_ALWAYS	1 /* always save/restore state */
+#define PARAM_PM_SAVE_FIRMWARE	2 /* only save/restore if firmware flag set */
+
+static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
+module_param(pm_save_enable, int, 0644);
+MODULE_PARM_DESC(pm_save_enable,
+	"Save/restore state on power down: 0 = never, 1 = always, 2 = firmware");
+
 /* The number of ETMv4 currently registered */
 static int etm4_count;
 static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
@@ -54,6 +65,14 @@ static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
 	isb();
 }
 
+static void etm4_os_lock(struct etmv4_drvdata *drvdata)
+{
+	/* Writing 0x1 to TRCOSLAR locks the trace registers */
+	writel_relaxed(0x1, drvdata->base + TRCOSLAR);
+	drvdata->os_unlock = false;
+	isb();
+}
+
 static bool etm4_arch_supported(u8 arch)
 {
 	/* Mask out the minor version number */
@@ -1085,6 +1104,303 @@ static void etm4_init_trace_id(struct etmv4_drvdata *drvdata)
 	drvdata->trcid = coresight_get_trace_id(drvdata->cpu);
 }
 
+#ifdef CONFIG_CPU_PM
+static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
+{
+	int i, ret = 0;
+	u32 control;
+	struct etmv4_save_state *state;
+	struct device *etm_dev = &drvdata->csdev->dev;
+
+	/*
+	 * As recommended by 3.4.1 ("The procedure when powering down the PE")
+	 * of ARM IHI 0064D
+	 */
+	dsb(sy);
+	isb();
+
+	CS_UNLOCK(drvdata->base);
+
+	/* Lock the OS lock to disable trace and external debugger access */
+	etm4_os_lock(drvdata);
+
+	/* wait for TRCSTATR.PMSTABLE to go up */
+	if (coresight_timeout(drvdata->base, TRCSTATR,
+					TRCSTATR_PMSTABLE_BIT, 1)) {
+		dev_err(etm_dev,
+			"timeout while waiting for PM Stable Status\n");
+		etm4_os_unlock(drvdata);
+		ret = -EBUSY;
+		goto out;
+	}
+
+	state = &drvdata->save_state;
+
+	state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
+	state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
+	state->trcconfigr = readl(drvdata->base + TRCCONFIGR);
+	state->trcauxctlr = readl(drvdata->base + TRCAUXCTLR);
+	state->trceventctl0r = readl(drvdata->base + TRCEVENTCTL0R);
+	state->trceventctl1r = readl(drvdata->base + TRCEVENTCTL1R);
+	state->trcstallctlr = readl(drvdata->base + TRCSTALLCTLR);
+	state->trctsctlr = readl(drvdata->base + TRCTSCTLR);
+	state->trcsyncpr = readl(drvdata->base + TRCSYNCPR);
+	state->trcccctlr = readl(drvdata->base + TRCCCCTLR);
+	state->trcbbctlr = readl(drvdata->base + TRCBBCTLR);
+	state->trctraceidr = readl(drvdata->base + TRCTRACEIDR);
+	state->trcqctlr = readl(drvdata->base + TRCQCTLR);
+
+	state->trcvictlr = readl(drvdata->base + TRCVICTLR);
+	state->trcviiectlr = readl(drvdata->base + TRCVIIECTLR);
+	state->trcvissctlr = readl(drvdata->base + TRCVISSCTLR);
+	state->trcvipcssctlr = readl(drvdata->base + TRCVIPCSSCTLR);
+	state->trcvdctlr = readl(drvdata->base + TRCVDCTLR);
+	state->trcvdsacctlr = readl(drvdata->base + TRCVDSACCTLR);
+	state->trcvdarcctlr = readl(drvdata->base + TRCVDARCCTLR);
+
+	for (i = 0; i < drvdata->nrseqstate; i++)
+		state->trcseqevr[i] = readl(drvdata->base + TRCSEQEVRn(i));
+
+	state->trcseqrstevr = readl(drvdata->base + TRCSEQRSTEVR);
+	state->trcseqstr = readl(drvdata->base + TRCSEQSTR);
+	state->trcextinselr = readl(drvdata->base + TRCEXTINSELR);
+
+	for (i = 0; i < drvdata->nr_cntr; i++) {
+		state->trccntrldvr[i] = readl(drvdata->base + TRCCNTRLDVRn(i));
+		state->trccntctlr[i] = readl(drvdata->base + TRCCNTCTLRn(i));
+		state->trccntvr[i] = readl(drvdata->base + TRCCNTVRn(i));
+	}
+
+	for (i = 0; i < drvdata->nr_resource * 2; i++)
+		state->trcrsctlr[i] = readl(drvdata->base + TRCRSCTLRn(i));
+
+	for (i = 0; i < drvdata->nr_ss_cmp; i++) {
+		state->trcssccr[i] = readl(drvdata->base + TRCSSCCRn(i));
+		state->trcsscsr[i] = readl(drvdata->base + TRCSSCSRn(i));
+		state->trcsspcicr[i] = readl(drvdata->base + TRCSSPCICRn(i));
+	}
+
+	for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
+		state->trcacvr[i] = readl(drvdata->base + TRCACVRn(i));
+		state->trcacatr[i] = readl(drvdata->base + TRCACATRn(i));
+	}
+
+	/*
+	 * Data trace stream is architecturally prohibited for A profile cores
+	 * so we don't save (or later restore) trcdvcvr and trcdvcmr - As per
+	 * section 1.3.4 ("Possible functional configurations of an ETMv4 trace
+	 * unit") of ARM IHI 0064D.
+	 */
+
+	for (i = 0; i < drvdata->numcidc; i++)
+		state->trccidcvr[i] = readl(drvdata->base + TRCCIDCVRn(i));
+
+	for (i = 0; i < drvdata->numvmidc; i++)
+		state->trcvmidcvr[i] = readl(drvdata->base + TRCVMIDCVRn(i));
+
+	state->trccidcctlr0 = readl(drvdata->base + TRCCIDCCTLR0);
+	state->trccidcctlr1 = readl(drvdata->base + TRCCIDCCTLR1);
+
+	state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR0);
+	state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR1);
+
+	state->trcclaimset = readl(drvdata->base + TRCCLAIMCLR);
+
+	/* wait for TRCSTATR.IDLE to go up */
+	if (coresight_timeout(drvdata->base, TRCSTATR, TRCSTATR_IDLE_BIT, 1)) {
+		dev_err(etm_dev,
+			"timeout while waiting for Idle Trace Status\n");
+		etm4_os_unlock(drvdata);
+		ret = -EBUSY;
+		goto out;
+	}
+
+	drvdata->state_needs_restore = true;
+
+	/*
+	 * Power can be removed from the trace unit now. We do this to
+	 * potentially save power on systems that respect the TRCPDCR_PU
+	 * despite requesting software to save/restore state.
+	 */
+	control = readl_relaxed(drvdata->base + TRCPDCR);
+	control &= ~TRCPDCR_PU;
+	writel_relaxed(control, drvdata->base + TRCPDCR);
+
+out:
+	CS_LOCK(drvdata->base);
+	return ret;
+}
+
+static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
+{
+	int i;
+	struct etmv4_save_state *state;
+
+	state = &drvdata->save_state;
+
+	CS_UNLOCK(drvdata->base);
+
+	writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
+
+	writel_relaxed(state->trcprgctlr, drvdata->base + TRCPRGCTLR);
+	writel_relaxed(state->trcprocselr, drvdata->base + TRCPROCSELR);
+	writel_relaxed(state->trcconfigr, drvdata->base + TRCCONFIGR);
+	writel_relaxed(state->trcauxctlr, drvdata->base + TRCAUXCTLR);
+	writel_relaxed(state->trceventctl0r, drvdata->base + TRCEVENTCTL0R);
+	writel_relaxed(state->trceventctl1r, drvdata->base + TRCEVENTCTL1R);
+	writel_relaxed(state->trcstallctlr, drvdata->base + TRCSTALLCTLR);
+	writel_relaxed(state->trctsctlr, drvdata->base + TRCTSCTLR);
+	writel_relaxed(state->trcsyncpr, drvdata->base + TRCSYNCPR);
+	writel_relaxed(state->trcccctlr, drvdata->base + TRCCCCTLR);
+	writel_relaxed(state->trcbbctlr, drvdata->base + TRCBBCTLR);
+	writel_relaxed(state->trctraceidr, drvdata->base + TRCTRACEIDR);
+	writel_relaxed(state->trcqctlr, drvdata->base + TRCQCTLR);
+
+	writel_relaxed(state->trcvictlr, drvdata->base + TRCVICTLR);
+	writel_relaxed(state->trcviiectlr, drvdata->base + TRCVIIECTLR);
+	writel_relaxed(state->trcvissctlr, drvdata->base + TRCVISSCTLR);
+	writel_relaxed(state->trcvipcssctlr, drvdata->base + TRCVIPCSSCTLR);
+	writel_relaxed(state->trcvdctlr, drvdata->base + TRCVDCTLR);
+	writel_relaxed(state->trcvdsacctlr, drvdata->base + TRCVDSACCTLR);
+	writel_relaxed(state->trcvdarcctlr, drvdata->base + TRCVDARCCTLR);
+
+	for (i = 0; i < drvdata->nrseqstate; i++)
+		writel_relaxed(state->trcseqevr[i],
+					drvdata->base + TRCSEQEVRn(i));
+
+	writel_relaxed(state->trcseqrstevr, drvdata->base + TRCSEQRSTEVR);
+	writel_relaxed(state->trcseqstr, drvdata->base + TRCSEQSTR);
+	writel_relaxed(state->trcextinselr, drvdata->base + TRCEXTINSELR);
+
+	for (i = 0; i < drvdata->nr_cntr; i++) {
+		writel_relaxed(state->trccntrldvr[i],
+					drvdata->base + TRCCNTRLDVRn(i));
+		writel_relaxed(state->trccntctlr[i],
+					drvdata->base + TRCCNTCTLRn(i));
+		writel_relaxed(state->trccntvr[i],
+					drvdata->base + TRCCNTVRn(i));
+	}
+
+	for (i = 0; i < drvdata->nr_resource * 2; i++)
+		writel_relaxed(state->trcrsctlr[i],
+					drvdata->base + TRCRSCTLRn(i));
+
+	for (i = 0; i < drvdata->nr_ss_cmp; i++) {
+		writel_relaxed(state->trcssccr[i],
+					drvdata->base + TRCSSCCRn(i));
+		writel_relaxed(state->trcsscsr[i],
+					drvdata->base + TRCSSCSRn(i));
+		writel_relaxed(state->trcsspcicr[i],
+					drvdata->base + TRCSSPCICRn(i));
+	}
+
+	for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
+		writel_relaxed(state->trcacvr[i],
+					drvdata->base + TRCACVRn(i));
+		writel_relaxed(state->trcacatr[i],
+					drvdata->base + TRCACATRn(i));
+	}
+
+	for (i = 0; i < drvdata->numcidc; i++)
+		writel_relaxed(state->trccidcvr[i],
+					drvdata->base + TRCCIDCVRn(i));
+
+	for (i = 0; i < drvdata->numvmidc; i++)
+		writel_relaxed(state->trcvmidcvr[i],
+					drvdata->base + TRCVMIDCVRn(i));
+
+	writel_relaxed(state->trccidcctlr0, drvdata->base + TRCCIDCCTLR0);
+	writel_relaxed(state->trccidcctlr1, drvdata->base + TRCCIDCCTLR1);
+
+	writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR0);
+	writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR1);
+
+	writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
+
+	drvdata->state_needs_restore = false;
+
+	/*
+	 * Request to keep the trace unit powered and also
+	 * emulation of powerdown
+	 */
+	writel_relaxed(readl_relaxed(drvdata->base + TRCPDCR) | TRCPDCR_PU,
+		       drvdata->base + TRCPDCR);
+
+	/*
+	 * As recommended by section 4.3.7 ("Synchronization when using the
+	 * memory-mapped interface") of ARM IHI 0064D
+	 */
+	dsb(sy);
+	isb();
+
+	/* Unlock the OS lock to re-enable trace and external debug access */
+	etm4_os_unlock(drvdata);
+	CS_LOCK(drvdata->base);
+}
+
+static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd,
+			      void *v)
+{
+	struct etmv4_drvdata *drvdata;
+	unsigned int cpu = smp_processor_id();
+
+	if (!etmdrvdata[cpu])
+		return 0;
+
+	drvdata = etmdrvdata[cpu];
+
+	if (pm_save_enable == PARAM_PM_SAVE_NEVER ||
+	    (pm_save_enable == PARAM_PM_SAVE_FIRMWARE &&
+	     !drvdata->pm_save_enable))
+		return NOTIFY_OK;
+
+	if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
+		return NOTIFY_BAD;
+
+	switch (cmd) {
+	case CPU_PM_ENTER:
+		/* save the state if coresight is in use */
+		if (coresight_is_claimed_any(drvdata->base))
+			if (etm4_cpu_save(drvdata))
+				return NOTIFY_BAD;
+		break;
+	case CPU_PM_EXIT:
+	case CPU_PM_ENTER_FAILED:
+		/* trcclaimset is set when there is state to restore */
+		if (drvdata->state_needs_restore)
+			etm4_cpu_restore(drvdata);
+		break;
+	default:
+		return NOTIFY_DONE;
+	}
+
+	return NOTIFY_OK;
+}
+
+static struct notifier_block etm4_cpu_pm_nb = {
+	.notifier_call = etm4_cpu_pm_notify,
+};
+
+static int etm4_cpu_pm_register(void)
+{
+	return cpu_pm_register_notifier(&etm4_cpu_pm_nb);
+}
+
+static void etm4_cpu_pm_unregister(void)
+{
+	cpu_pm_unregister_notifier(&etm4_cpu_pm_nb);
+}
+#else
+static int etm4_cpu_pm_register(void) { return 0; }
+static void etm4_cpu_pm_unregister(void) { }
+#endif
+
+static inline bool etm4_needs_save_restore(struct device *dev)
+{
+	return fwnode_property_present(dev->fwnode,
+				       "arm,coresight-needs-save-restore");
+}
+
 static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
 {
 	int ret;
@@ -1101,6 +1417,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
 
 	dev_set_drvdata(dev, drvdata);
 
+	drvdata->pm_save_enable = etm4_needs_save_restore(dev);
+
 	/* Validity for the resource is already checked by the AMBA core */
 	base = devm_ioremap_resource(dev, res);
 	if (IS_ERR(base))
@@ -1132,6 +1450,10 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
 		if (ret < 0)
 			goto err_arch_supported;
 		hp_online = ret;
+
+		ret = etm4_cpu_pm_register();
+		if (ret)
+			goto err_arch_supported;
 	}
 
 	cpus_read_unlock();
@@ -1182,6 +1504,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
 
 err_arch_supported:
 	if (--etm4_count == 0) {
+		etm4_cpu_pm_unregister();
+
 		cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING);
 		if (hp_online)
 			cpuhp_remove_state_nocalls(hp_online);
diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
index 4523f10ddd0f..c31634c64f87 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.h
+++ b/drivers/hwtracing/coresight/coresight-etm4x.h
@@ -175,6 +175,7 @@
 					 ETM_MODE_EXCL_USER)
 
 #define TRCSTATR_IDLE_BIT		0
+#define TRCSTATR_PMSTABLE_BIT		1
 #define ETM_DEFAULT_ADDR_COMP		0
 
 /* PowerDown Control Register bits */
@@ -281,6 +282,63 @@ struct etmv4_config {
 	u32				ext_inp;
 };
 
+/**
+ * struct etm4_save_state - state to be preserved when ETM is without power
+ */
+struct etmv4_save_state {
+	u32	trcprgctlr;
+	u32	trcprocselr;
+	u32	trcconfigr;
+	u32	trcauxctlr;
+	u32	trceventctl0r;
+	u32	trceventctl1r;
+	u32	trcstallctlr;
+	u32	trctsctlr;
+	u32	trcsyncpr;
+	u32	trcccctlr;
+	u32	trcbbctlr;
+	u32	trctraceidr;
+	u32	trcqctlr;
+
+	u32	trcvictlr;
+	u32	trcviiectlr;
+	u32	trcvissctlr;
+	u32	trcvipcssctlr;
+	u32	trcvdctlr;
+	u32	trcvdsacctlr;
+	u32	trcvdarcctlr;
+
+	u32	trcseqevr[ETM_MAX_SEQ_STATES];
+	u32	trcseqrstevr;
+	u32	trcseqstr;
+	u32	trcextinselr;
+	u32	trccntrldvr[ETMv4_MAX_CNTR];
+	u32	trccntctlr[ETMv4_MAX_CNTR];
+	u32	trccntvr[ETMv4_MAX_CNTR];
+
+	u32	trcrsctlr[ETM_MAX_RES_SEL * 2];
+
+	u32	trcssccr[ETM_MAX_SS_CMP];
+	u32	trcsscsr[ETM_MAX_SS_CMP];
+	u32	trcsspcicr[ETM_MAX_SS_CMP];
+
+	u64	trcacvr[ETM_MAX_SINGLE_ADDR_CMP];
+	u64	trcacatr[ETM_MAX_SINGLE_ADDR_CMP];
+	u64	trccidcvr[ETMv4_MAX_CTXID_CMP];
+	u32	trcvmidcvr[ETM_MAX_VMID_CMP];
+	u32	trccidcctlr0;
+	u32	trccidcctlr1;
+	u32	trcvmidcctlr0;
+	u32	trcvmidcctlr1;
+
+	u32	trcclaimset;
+
+	u32	cntr_val[ETMv4_MAX_CNTR];
+	u32	seq_state;
+	u32	vinst_ctrl;
+	u32	ss_status[ETM_MAX_SS_CMP];
+};
+
 /**
  * struct etm4_drvdata - specifics associated to an ETM component
  * @base:       Memory mapped base address for this component.
@@ -336,6 +394,7 @@ struct etmv4_config {
  * @atbtrig:	If the implementation can support ATB triggers
  * @lpoverride:	If the implementation can support low-power state over.
  * @config:	structure holding configuration parameters.
+ * @save_state:	State to be preserved across power loss
  */
 struct etmv4_drvdata {
 	void __iomem			*base;
@@ -367,6 +426,7 @@ struct etmv4_drvdata {
 	u8				q_support;
 	bool				sticky_enable;
 	bool				boot_enable;
+	bool				pm_save_enable;
 	bool				os_unlock;
 	bool				instrp0;
 	bool				trcbb;
@@ -381,6 +441,8 @@ struct etmv4_drvdata {
 	bool				atbtrig;
 	bool				lpoverride;
 	struct etmv4_config		config;
+	struct etmv4_save_state		save_state;
+	bool				state_needs_restore;
 };
 
 /* Address comparator access types */
diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
index 86d1fc2c1bd4..aba71a5a025f 100644
--- a/drivers/hwtracing/coresight/coresight.c
+++ b/drivers/hwtracing/coresight/coresight.c
@@ -140,7 +140,7 @@ static inline bool coresight_is_claimed_self_hosted(void __iomem *base)
 	return coresight_read_claim_tags(base) == CORESIGHT_CLAIM_SELF_HOSTED;
 }
 
-static inline bool coresight_is_claimed_any(void __iomem *base)
+bool coresight_is_claimed_any(void __iomem *base)
 {
 	return coresight_read_claim_tags(base) != 0;
 }
diff --git a/include/linux/coresight.h b/include/linux/coresight.h
index a2b68823717b..c3a875dffe65 100644
--- a/include/linux/coresight.h
+++ b/include/linux/coresight.h
@@ -285,6 +285,9 @@ extern void coresight_disclaim_device(void __iomem *base);
 extern void coresight_disclaim_device_unlocked(void __iomem *base);
 extern char *coresight_alloc_device_name(struct coresight_dev_list *devs,
 					 struct device *dev);
+
+extern bool coresight_is_claimed_any(void __iomem *base);
+
 #else
 static inline struct coresight_device *
 coresight_register(struct coresight_desc *desc) { return NULL; }
@@ -307,6 +310,11 @@ static inline int coresight_claim_device(void __iomem *base)
 static inline void coresight_disclaim_device(void __iomem *base) {}
 static inline void coresight_disclaim_device_unlocked(void __iomem *base) {}
 
+static inline bool coresight_is_claimed_any(void __iomem *base)
+{
+	return false;
+}
+
 #endif
 
 extern int coresight_get_cpu(struct device *dev);
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH v3 6/6] dt-bindings: arm: coresight: Add support for coresight-needs-save-restore

[Andrew Murray]

Some coresight components, because of choices made during hardware
integration, require their state to be saved and restored across CPU low
power states.

The software has no reliable method of detecting when save/restore is
required thus let's add a binding to inform the kernel.

Signed-off-by: Andrew Murray <andrew.murray@arm.com>
---
 Documentation/devicetree/bindings/arm/coresight.txt | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/Documentation/devicetree/bindings/arm/coresight.txt b/Documentation/devicetree/bindings/arm/coresight.txt
index 8a88ddebc1a2..e0dc0a93312f 100644
--- a/Documentation/devicetree/bindings/arm/coresight.txt
+++ b/Documentation/devicetree/bindings/arm/coresight.txt
@@ -90,6 +90,9 @@ its hardware characteristcs.
 	* cpu: the cpu phandle this ETM/PTM is affined to. When omitted the
 	  source is considered to belong to CPU0.
 
+	* arm,coresight-needs-save-restore: boolean. Indicates that software
+	  should save/restore state across power down.
+
 * Optional property for TMC:
 
 	* arm,buffer-size: size of contiguous buffer space for TMC ETR
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

[PATCH] coresight: etm4x: lazily allocate memory for save_state

[Andrew Murray]

I had intended to lazily allocate memory for the save_state structure when
it is first used. Following is a patch that I will squash into "[PATCH v3 5/6]
coresight: etm4x: save/restore state across CPU low power states" on my
next respin. I thought I'd share it here to get some feedback along with
the rest of v3.

Thanks,

Andrew Murray

---
 drivers/hwtracing/coresight/coresight-etm4x.c | 14 +++++++++++---
 drivers/hwtracing/coresight/coresight-etm4x.h |  2 +-
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
index b0bd8158bf13..cd02372194bc 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x.c
@@ -1112,6 +1112,13 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
 	struct etmv4_save_state *state;
 	struct device *etm_dev = &drvdata->csdev->dev;
 
+	if (!drvdata->save_state) {
+		drvdata->save_state = devm_kmalloc(etm_dev,
+				sizeof(struct etmv4_save_state), GFP_KERNEL);
+		if (!drvdata->save_state)
+			return -ENOMEM;
+	}
+
 	/*
 	 * As recommended by 3.4.1 ("The procedure when powering down the PE")
 	 * of ARM IHI 0064D
@@ -1134,7 +1141,7 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
 		goto out;
 	}
 
-	state = &drvdata->save_state;
+	state = drvdata->save_state;
 
 	state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
 	state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
@@ -1234,9 +1241,10 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
 static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
 {
 	int i;
-	struct etmv4_save_state *state;
+	struct etmv4_save_state *state = drvdata->save_state;
 
-	state = &drvdata->save_state;
+	if (WARN_ON_ONCE(!state))
+		return;
 
 	CS_UNLOCK(drvdata->base);
 
diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
index c31634c64f87..a70cafbbb8cf 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x.h
+++ b/drivers/hwtracing/coresight/coresight-etm4x.h
@@ -441,7 +441,7 @@ struct etmv4_drvdata {
 	bool				atbtrig;
 	bool				lpoverride;
 	struct etmv4_config		config;
-	struct etmv4_save_state		save_state;
+	struct etmv4_save_state		*save_state;
 	bool				state_needs_restore;
 };
 
-- 
2.21.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v3 5/6] coresight: etm4x: save/restore state across CPU low power states

[Mike Leach]

HI Andrew,

On Thu, 11 Jul 2019 at 17:01, Andrew Murray <andrew.murray@arm.com> wrote:
>
> Some hardware will ignore bit TRCPDCR.PU which is used to signal
> to hardware that power should not be removed from the trace unit.
> Let's mitigate against this by conditionally saving and restoring
> the trace unit state when the CPU enters low power states.
>
> This patchset introduces a firmware property named
> 'arm,coresight-needs-save-restore' - when this is present the
> hardware state will be conditionally saved and restored.
>
> A module parameter 'pm_save_enable' is also introduced which can
> be configured to override the firmware property. This can be set
> to never allow save/restore, to conditionally allow it, or to
> do as the firmware indicates (default).
>
> The hardware state is only ever saved and restored when the claim
> tags indicate that coresight is in use.
>
> Signed-off-by: Andrew Murray <andrew.murray@arm.com>
> ---
>  drivers/hwtracing/coresight/coresight-etm4x.c | 324 ++++++++++++++++++
>  drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
>  drivers/hwtracing/coresight/coresight.c       |   2 +-
>  include/linux/coresight.h                     |   8 +
>  4 files changed, 395 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> index d993359e2117..b0bd8158bf13 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> @@ -18,6 +18,7 @@
>  #include <linux/stat.h>
>  #include <linux/clk.h>
>  #include <linux/cpu.h>
> +#include <linux/cpu_pm.h>
>  #include <linux/coresight.h>
>  #include <linux/coresight-pmu.h>
>  #include <linux/pm_wakeup.h>
> @@ -26,6 +27,7 @@
>  #include <linux/uaccess.h>
>  #include <linux/perf_event.h>
>  #include <linux/pm_runtime.h>
> +#include <linux/property.h>
>  #include <asm/sections.h>
>  #include <asm/local.h>
>  #include <asm/virt.h>
> @@ -37,6 +39,15 @@ static int boot_enable;
>  module_param(boot_enable, int, 0444);
>  MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
>
> +#define PARAM_PM_SAVE_NEVER    0 /* never save/restore state */
> +#define PARAM_PM_SAVE_ALWAYS   1 /* always save/restore state */
> +#define PARAM_PM_SAVE_FIRMWARE 2 /* only save/restore if firmware flag set */
> +
> +static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
> +module_param(pm_save_enable, int, 0644);
> +MODULE_PARM_DESC(pm_save_enable,
> +       "Save/restore state on power down: 0 = never, 1 = always, 2 = firmware");
> +
>  /* The number of ETMv4 currently registered */
>  static int etm4_count;
>  static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
> @@ -54,6 +65,14 @@ static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
>         isb();
>  }
>
> +static void etm4_os_lock(struct etmv4_drvdata *drvdata)
> +{
> +       /* Writing 0x1 to TRCOSLAR locks the trace registers */
> +       writel_relaxed(0x1, drvdata->base + TRCOSLAR);
> +       drvdata->os_unlock = false;
> +       isb();
> +}
> +
>  static bool etm4_arch_supported(u8 arch)
>  {
>         /* Mask out the minor version number */
> @@ -1085,6 +1104,303 @@ static void etm4_init_trace_id(struct etmv4_drvdata *drvdata)
>         drvdata->trcid = coresight_get_trace_id(drvdata->cpu);
>  }
>
> +#ifdef CONFIG_CPU_PM
> +static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> +{
> +       int i, ret = 0;
> +       u32 control;
> +       struct etmv4_save_state *state;
> +       struct device *etm_dev = &drvdata->csdev->dev;
> +
> +       /*
> +        * As recommended by 3.4.1 ("The procedure when powering down the PE")
> +        * of ARM IHI 0064D
> +        */
> +       dsb(sy);
> +       isb();
> +
> +       CS_UNLOCK(drvdata->base);
> +
> +       /* Lock the OS lock to disable trace and external debugger access */
> +       etm4_os_lock(drvdata);
> +
> +       /* wait for TRCSTATR.PMSTABLE to go up */
> +       if (coresight_timeout(drvdata->base, TRCSTATR,
> +                                       TRCSTATR_PMSTABLE_BIT, 1)) {
> +               dev_err(etm_dev,
> +                       "timeout while waiting for PM Stable Status\n");
> +               etm4_os_unlock(drvdata);
> +               ret = -EBUSY;
> +               goto out;
> +       }
> +
> +       state = &drvdata->save_state;
> +
> +       state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
> +       state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> +       state->trcconfigr = readl(drvdata->base + TRCCONFIGR);
> +       state->trcauxctlr = readl(drvdata->base + TRCAUXCTLR);
> +       state->trceventctl0r = readl(drvdata->base + TRCEVENTCTL0R);
> +       state->trceventctl1r = readl(drvdata->base + TRCEVENTCTL1R);
> +       state->trcstallctlr = readl(drvdata->base + TRCSTALLCTLR);
> +       state->trctsctlr = readl(drvdata->base + TRCTSCTLR);
> +       state->trcsyncpr = readl(drvdata->base + TRCSYNCPR);
> +       state->trcccctlr = readl(drvdata->base + TRCCCCTLR);
> +       state->trcbbctlr = readl(drvdata->base + TRCBBCTLR);
> +       state->trctraceidr = readl(drvdata->base + TRCTRACEIDR);
> +       state->trcqctlr = readl(drvdata->base + TRCQCTLR);
> +
> +       state->trcvictlr = readl(drvdata->base + TRCVICTLR);
> +       state->trcviiectlr = readl(drvdata->base + TRCVIIECTLR);
> +       state->trcvissctlr = readl(drvdata->base + TRCVISSCTLR);
> +       state->trcvipcssctlr = readl(drvdata->base + TRCVIPCSSCTLR);
> +       state->trcvdctlr = readl(drvdata->base + TRCVDCTLR);
> +       state->trcvdsacctlr = readl(drvdata->base + TRCVDSACCTLR);
> +       state->trcvdarcctlr = readl(drvdata->base + TRCVDARCCTLR);
> +
> +       for (i = 0; i < drvdata->nrseqstate; i++)
> +               state->trcseqevr[i] = readl(drvdata->base + TRCSEQEVRn(i));
> +
> +       state->trcseqrstevr = readl(drvdata->base + TRCSEQRSTEVR);
> +       state->trcseqstr = readl(drvdata->base + TRCSEQSTR);
> +       state->trcextinselr = readl(drvdata->base + TRCEXTINSELR);
> +
> +       for (i = 0; i < drvdata->nr_cntr; i++) {
> +               state->trccntrldvr[i] = readl(drvdata->base + TRCCNTRLDVRn(i));
> +               state->trccntctlr[i] = readl(drvdata->base + TRCCNTCTLRn(i));
> +               state->trccntvr[i] = readl(drvdata->base + TRCCNTVRn(i));
> +       }
> +
> +       for (i = 0; i < drvdata->nr_resource * 2; i++)
> +               state->trcrsctlr[i] = readl(drvdata->base + TRCRSCTLRn(i));
> +
> +       for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> +               state->trcssccr[i] = readl(drvdata->base + TRCSSCCRn(i));
> +               state->trcsscsr[i] = readl(drvdata->base + TRCSSCSRn(i));
> +               state->trcsspcicr[i] = readl(drvdata->base + TRCSSPCICRn(i));
> +       }
> +
> +       for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> +               state->trcacvr[i] = readl(drvdata->base + TRCACVRn(i));
> +               state->trcacatr[i] = readl(drvdata->base + TRCACATRn(i));
> +       }
> +
> +       /*
> +        * Data trace stream is architecturally prohibited for A profile cores
> +        * so we don't save (or later restore) trcdvcvr and trcdvcmr - As per
> +        * section 1.3.4 ("Possible functional configurations of an ETMv4 trace
> +        * unit") of ARM IHI 0064D.
> +        */
> +
> +       for (i = 0; i < drvdata->numcidc; i++)
> +               state->trccidcvr[i] = readl(drvdata->base + TRCCIDCVRn(i));
> +
> +       for (i = 0; i < drvdata->numvmidc; i++)
> +               state->trcvmidcvr[i] = readl(drvdata->base + TRCVMIDCVRn(i));
> +
> +       state->trccidcctlr0 = readl(drvdata->base + TRCCIDCCTLR0);
> +       state->trccidcctlr1 = readl(drvdata->base + TRCCIDCCTLR1);
> +
> +       state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR0);
> +       state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR1);
> +
> +       state->trcclaimset = readl(drvdata->base + TRCCLAIMCLR);
> +
> +       /* wait for TRCSTATR.IDLE to go up */
> +       if (coresight_timeout(drvdata->base, TRCSTATR, TRCSTATR_IDLE_BIT, 1)) {
> +               dev_err(etm_dev,
> +                       "timeout while waiting for Idle Trace Status\n");
> +               etm4_os_unlock(drvdata);
> +               ret = -EBUSY;
> +               goto out;
> +       }
> +
> +       drvdata->state_needs_restore = true;
> +
> +       /*
> +        * Power can be removed from the trace unit now. We do this to
> +        * potentially save power on systems that respect the TRCPDCR_PU
> +        * despite requesting software to save/restore state.
> +        */
> +       control = readl_relaxed(drvdata->base + TRCPDCR);
> +       control &= ~TRCPDCR_PU;
> +       writel_relaxed(control, drvdata->base + TRCPDCR);
> +

Removing the power is fine, except you are not remembering the state
for restore later.

> +out:
> +       CS_LOCK(drvdata->base);
> +       return ret;
> +}
> +
> +static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
> +{
> +       int i;
> +       struct etmv4_save_state *state;
> +
> +       state = &drvdata->save_state;
> +
> +       CS_UNLOCK(drvdata->base);
> +
> +       writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> +
> +       writel_relaxed(state->trcprgctlr, drvdata->base + TRCPRGCTLR);
> +       writel_relaxed(state->trcprocselr, drvdata->base + TRCPROCSELR);
> +       writel_relaxed(state->trcconfigr, drvdata->base + TRCCONFIGR);
> +       writel_relaxed(state->trcauxctlr, drvdata->base + TRCAUXCTLR);
> +       writel_relaxed(state->trceventctl0r, drvdata->base + TRCEVENTCTL0R);
> +       writel_relaxed(state->trceventctl1r, drvdata->base + TRCEVENTCTL1R);
> +       writel_relaxed(state->trcstallctlr, drvdata->base + TRCSTALLCTLR);
> +       writel_relaxed(state->trctsctlr, drvdata->base + TRCTSCTLR);
> +       writel_relaxed(state->trcsyncpr, drvdata->base + TRCSYNCPR);
> +       writel_relaxed(state->trcccctlr, drvdata->base + TRCCCCTLR);
> +       writel_relaxed(state->trcbbctlr, drvdata->base + TRCBBCTLR);
> +       writel_relaxed(state->trctraceidr, drvdata->base + TRCTRACEIDR);
> +       writel_relaxed(state->trcqctlr, drvdata->base + TRCQCTLR);
> +
> +       writel_relaxed(state->trcvictlr, drvdata->base + TRCVICTLR);
> +       writel_relaxed(state->trcviiectlr, drvdata->base + TRCVIIECTLR);
> +       writel_relaxed(state->trcvissctlr, drvdata->base + TRCVISSCTLR);
> +       writel_relaxed(state->trcvipcssctlr, drvdata->base + TRCVIPCSSCTLR);
> +       writel_relaxed(state->trcvdctlr, drvdata->base + TRCVDCTLR);
> +       writel_relaxed(state->trcvdsacctlr, drvdata->base + TRCVDSACCTLR);
> +       writel_relaxed(state->trcvdarcctlr, drvdata->base + TRCVDARCCTLR);
> +
> +       for (i = 0; i < drvdata->nrseqstate; i++)
> +               writel_relaxed(state->trcseqevr[i],
> +                                       drvdata->base + TRCSEQEVRn(i));
> +
> +       writel_relaxed(state->trcseqrstevr, drvdata->base + TRCSEQRSTEVR);
> +       writel_relaxed(state->trcseqstr, drvdata->base + TRCSEQSTR);
> +       writel_relaxed(state->trcextinselr, drvdata->base + TRCEXTINSELR);
> +
> +       for (i = 0; i < drvdata->nr_cntr; i++) {
> +               writel_relaxed(state->trccntrldvr[i],
> +                                       drvdata->base + TRCCNTRLDVRn(i));
> +               writel_relaxed(state->trccntctlr[i],
> +                                       drvdata->base + TRCCNTCTLRn(i));
> +               writel_relaxed(state->trccntvr[i],
> +                                       drvdata->base + TRCCNTVRn(i));
> +       }
> +
> +       for (i = 0; i < drvdata->nr_resource * 2; i++)
> +               writel_relaxed(state->trcrsctlr[i],
> +                                       drvdata->base + TRCRSCTLRn(i));
> +
> +       for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> +               writel_relaxed(state->trcssccr[i],
> +                                       drvdata->base + TRCSSCCRn(i));
> +               writel_relaxed(state->trcsscsr[i],
> +                                       drvdata->base + TRCSSCSRn(i));
> +               writel_relaxed(state->trcsspcicr[i],
> +                                       drvdata->base + TRCSSPCICRn(i));
> +       }
> +
> +       for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> +               writel_relaxed(state->trcacvr[i],
> +                                       drvdata->base + TRCACVRn(i));
> +               writel_relaxed(state->trcacatr[i],
> +                                       drvdata->base + TRCACATRn(i));
> +       }
> +
> +       for (i = 0; i < drvdata->numcidc; i++)
> +               writel_relaxed(state->trccidcvr[i],
> +                                       drvdata->base + TRCCIDCVRn(i));
> +
> +       for (i = 0; i < drvdata->numvmidc; i++)
> +               writel_relaxed(state->trcvmidcvr[i],
> +                                       drvdata->base + TRCVMIDCVRn(i));
> +
> +       writel_relaxed(state->trccidcctlr0, drvdata->base + TRCCIDCCTLR0);
> +       writel_relaxed(state->trccidcctlr1, drvdata->base + TRCCIDCCTLR1);
> +
> +       writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR0);
> +       writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR1);
> +
> +       writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> +
> +       drvdata->state_needs_restore = false;
> +
> +       /*
> +        * Request to keep the trace unit powered and also
> +        * emulation of powerdown
> +        */
> +       writel_relaxed(readl_relaxed(drvdata->base + TRCPDCR) | TRCPDCR_PU,
> +                      drvdata->base + TRCPDCR);
> +

This sets the PU bit irrespective of the value on power down. The bit
should be restored to the state on power down not unconditionally set.

Regards

Mike


> +       /*
> +        * As recommended by section 4.3.7 ("Synchronization when using the
> +        * memory-mapped interface") of ARM IHI 0064D
> +        */
> +       dsb(sy);
> +       isb();
> +
> +       /* Unlock the OS lock to re-enable trace and external debug access */
> +       etm4_os_unlock(drvdata);
> +       CS_LOCK(drvdata->base);
> +}
> +
> +static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd,
> +                             void *v)
> +{
> +       struct etmv4_drvdata *drvdata;
> +       unsigned int cpu = smp_processor_id();
> +
> +       if (!etmdrvdata[cpu])
> +               return 0;
> +
> +       drvdata = etmdrvdata[cpu];
> +
> +       if (pm_save_enable == PARAM_PM_SAVE_NEVER ||
> +           (pm_save_enable == PARAM_PM_SAVE_FIRMWARE &&
> +            !drvdata->pm_save_enable))
> +               return NOTIFY_OK;
> +
> +       if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
> +               return NOTIFY_BAD;
> +
> +       switch (cmd) {
> +       case CPU_PM_ENTER:
> +               /* save the state if coresight is in use */
> +               if (coresight_is_claimed_any(drvdata->base))
> +                       if (etm4_cpu_save(drvdata))
> +                               return NOTIFY_BAD;
> +               break;
> +       case CPU_PM_EXIT:
> +       case CPU_PM_ENTER_FAILED:
> +               /* trcclaimset is set when there is state to restore */
> +               if (drvdata->state_needs_restore)
> +                       etm4_cpu_restore(drvdata);
> +               break;
> +       default:
> +               return NOTIFY_DONE;
> +       }
> +
> +       return NOTIFY_OK;
> +}
> +
> +static struct notifier_block etm4_cpu_pm_nb = {
> +       .notifier_call = etm4_cpu_pm_notify,
> +};
> +
> +static int etm4_cpu_pm_register(void)
> +{
> +       return cpu_pm_register_notifier(&etm4_cpu_pm_nb);
> +}
> +
> +static void etm4_cpu_pm_unregister(void)
> +{
> +       cpu_pm_unregister_notifier(&etm4_cpu_pm_nb);
> +}
> +#else
> +static int etm4_cpu_pm_register(void) { return 0; }
> +static void etm4_cpu_pm_unregister(void) { }
> +#endif
> +
> +static inline bool etm4_needs_save_restore(struct device *dev)
> +{
> +       return fwnode_property_present(dev->fwnode,
> +                                      "arm,coresight-needs-save-restore");
> +}
> +
>  static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>  {
>         int ret;
> @@ -1101,6 +1417,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>
>         dev_set_drvdata(dev, drvdata);
>
> +       drvdata->pm_save_enable = etm4_needs_save_restore(dev);
> +
>         /* Validity for the resource is already checked by the AMBA core */
>         base = devm_ioremap_resource(dev, res);
>         if (IS_ERR(base))
> @@ -1132,6 +1450,10 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>                 if (ret < 0)
>                         goto err_arch_supported;
>                 hp_online = ret;
> +
> +               ret = etm4_cpu_pm_register();
> +               if (ret)
> +                       goto err_arch_supported;
>         }
>
>         cpus_read_unlock();
> @@ -1182,6 +1504,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>
>  err_arch_supported:
>         if (--etm4_count == 0) {
> +               etm4_cpu_pm_unregister();
> +
>                 cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING);
>                 if (hp_online)
>                         cpuhp_remove_state_nocalls(hp_online);
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> index 4523f10ddd0f..c31634c64f87 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> @@ -175,6 +175,7 @@
>                                          ETM_MODE_EXCL_USER)
>
>  #define TRCSTATR_IDLE_BIT              0
> +#define TRCSTATR_PMSTABLE_BIT          1
>  #define ETM_DEFAULT_ADDR_COMP          0
>
>  /* PowerDown Control Register bits */
> @@ -281,6 +282,63 @@ struct etmv4_config {
>         u32                             ext_inp;
>  };
>
> +/**
> + * struct etm4_save_state - state to be preserved when ETM is without power
> + */
> +struct etmv4_save_state {
> +       u32     trcprgctlr;
> +       u32     trcprocselr;
> +       u32     trcconfigr;
> +       u32     trcauxctlr;
> +       u32     trceventctl0r;
> +       u32     trceventctl1r;
> +       u32     trcstallctlr;
> +       u32     trctsctlr;
> +       u32     trcsyncpr;
> +       u32     trcccctlr;
> +       u32     trcbbctlr;
> +       u32     trctraceidr;
> +       u32     trcqctlr;
> +
> +       u32     trcvictlr;
> +       u32     trcviiectlr;
> +       u32     trcvissctlr;
> +       u32     trcvipcssctlr;
> +       u32     trcvdctlr;
> +       u32     trcvdsacctlr;
> +       u32     trcvdarcctlr;
> +
> +       u32     trcseqevr[ETM_MAX_SEQ_STATES];
> +       u32     trcseqrstevr;
> +       u32     trcseqstr;
> +       u32     trcextinselr;
> +       u32     trccntrldvr[ETMv4_MAX_CNTR];
> +       u32     trccntctlr[ETMv4_MAX_CNTR];
> +       u32     trccntvr[ETMv4_MAX_CNTR];
> +
> +       u32     trcrsctlr[ETM_MAX_RES_SEL * 2];
> +
> +       u32     trcssccr[ETM_MAX_SS_CMP];
> +       u32     trcsscsr[ETM_MAX_SS_CMP];
> +       u32     trcsspcicr[ETM_MAX_SS_CMP];
> +
> +       u64     trcacvr[ETM_MAX_SINGLE_ADDR_CMP];
> +       u64     trcacatr[ETM_MAX_SINGLE_ADDR_CMP];
> +       u64     trccidcvr[ETMv4_MAX_CTXID_CMP];
> +       u32     trcvmidcvr[ETM_MAX_VMID_CMP];
> +       u32     trccidcctlr0;
> +       u32     trccidcctlr1;
> +       u32     trcvmidcctlr0;
> +       u32     trcvmidcctlr1;
> +
> +       u32     trcclaimset;
> +
> +       u32     cntr_val[ETMv4_MAX_CNTR];
> +       u32     seq_state;
> +       u32     vinst_ctrl;
> +       u32     ss_status[ETM_MAX_SS_CMP];
> +};
> +
>  /**
>   * struct etm4_drvdata - specifics associated to an ETM component
>   * @base:       Memory mapped base address for this component.
> @@ -336,6 +394,7 @@ struct etmv4_config {
>   * @atbtrig:   If the implementation can support ATB triggers
>   * @lpoverride:        If the implementation can support low-power state over.
>   * @config:    structure holding configuration parameters.
> + * @save_state:        State to be preserved across power loss
>   */
>  struct etmv4_drvdata {
>         void __iomem                    *base;
> @@ -367,6 +426,7 @@ struct etmv4_drvdata {
>         u8                              q_support;
>         bool                            sticky_enable;
>         bool                            boot_enable;
> +       bool                            pm_save_enable;
>         bool                            os_unlock;
>         bool                            instrp0;
>         bool                            trcbb;
> @@ -381,6 +441,8 @@ struct etmv4_drvdata {
>         bool                            atbtrig;
>         bool                            lpoverride;
>         struct etmv4_config             config;
> +       struct etmv4_save_state         save_state;
> +       bool                            state_needs_restore;
>  };
>
>  /* Address comparator access types */
> diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> index 86d1fc2c1bd4..aba71a5a025f 100644
> --- a/drivers/hwtracing/coresight/coresight.c
> +++ b/drivers/hwtracing/coresight/coresight.c
> @@ -140,7 +140,7 @@ static inline bool coresight_is_claimed_self_hosted(void __iomem *base)
>         return coresight_read_claim_tags(base) == CORESIGHT_CLAIM_SELF_HOSTED;
>  }
>
> -static inline bool coresight_is_claimed_any(void __iomem *base)
> +bool coresight_is_claimed_any(void __iomem *base)
>  {
>         return coresight_read_claim_tags(base) != 0;
>  }
> diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> index a2b68823717b..c3a875dffe65 100644
> --- a/include/linux/coresight.h
> +++ b/include/linux/coresight.h
> @@ -285,6 +285,9 @@ extern void coresight_disclaim_device(void __iomem *base);
>  extern void coresight_disclaim_device_unlocked(void __iomem *base);
>  extern char *coresight_alloc_device_name(struct coresight_dev_list *devs,
>                                          struct device *dev);
> +
> +extern bool coresight_is_claimed_any(void __iomem *base);
> +
>  #else
>  static inline struct coresight_device *
>  coresight_register(struct coresight_desc *desc) { return NULL; }
> @@ -307,6 +310,11 @@ static inline int coresight_claim_device(void __iomem *base)
>  static inline void coresight_disclaim_device(void __iomem *base) {}
>  static inline void coresight_disclaim_device_unlocked(void __iomem *base) {}
>
> +static inline bool coresight_is_claimed_any(void __iomem *base)
> +{
> +       return false;
> +}
> +
>  #endif
>
>  extern int coresight_get_cpu(struct device *dev);
> --
> 2.21.0
>


-- 
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v3 5/6] coresight: etm4x: save/restore state across CPU low power states

[Mathieu Poirier]

Hi Andrew,

Please see comments below.

On Thu, Jul 11, 2019 at 05:01:13PM +0100, Andrew Murray wrote:
> Some hardware will ignore bit TRCPDCR.PU which is used to signal
> to hardware that power should not be removed from the trace unit.
> Let's mitigate against this by conditionally saving and restoring
> the trace unit state when the CPU enters low power states.
> 
> This patchset introduces a firmware property named
> 'arm,coresight-needs-save-restore' - when this is present the
> hardware state will be conditionally saved and restored.
> 
> A module parameter 'pm_save_enable' is also introduced which can
> be configured to override the firmware property. This can be set
> to never allow save/restore, to conditionally allow it, or to
> do as the firmware indicates (default).
> 
> The hardware state is only ever saved and restored when the claim
> tags indicate that coresight is in use.
> 
> Signed-off-by: Andrew Murray <andrew.murray@arm.com>
> ---
>  drivers/hwtracing/coresight/coresight-etm4x.c | 324 ++++++++++++++++++
>  drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
>  drivers/hwtracing/coresight/coresight.c       |   2 +-
>  include/linux/coresight.h                     |   8 +
>  4 files changed, 395 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> index d993359e2117..b0bd8158bf13 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> @@ -18,6 +18,7 @@
>  #include <linux/stat.h>
>  #include <linux/clk.h>
>  #include <linux/cpu.h>
> +#include <linux/cpu_pm.h>
>  #include <linux/coresight.h>
>  #include <linux/coresight-pmu.h>
>  #include <linux/pm_wakeup.h>
> @@ -26,6 +27,7 @@
>  #include <linux/uaccess.h>
>  #include <linux/perf_event.h>
>  #include <linux/pm_runtime.h>
> +#include <linux/property.h>
>  #include <asm/sections.h>
>  #include <asm/local.h>
>  #include <asm/virt.h>
> @@ -37,6 +39,15 @@ static int boot_enable;
>  module_param(boot_enable, int, 0444);
>  MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
>  
> +#define PARAM_PM_SAVE_NEVER	0 /* never save/restore state */
> +#define PARAM_PM_SAVE_ALWAYS	1 /* always save/restore state */
> +#define PARAM_PM_SAVE_FIRMWARE	2 /* only save/restore if firmware flag set */
> +
> +static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
> +module_param(pm_save_enable, int, 0644);
> +MODULE_PARM_DESC(pm_save_enable,
> +	"Save/restore state on power down: 0 = never, 1 = always, 2 = firmware");
> +
>  /* The number of ETMv4 currently registered */
>  static int etm4_count;
>  static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
> @@ -54,6 +65,14 @@ static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
>  	isb();
>  }
>  
> +static void etm4_os_lock(struct etmv4_drvdata *drvdata)
> +{
> +	/* Writing 0x1 to TRCOSLAR locks the trace registers */
> +	writel_relaxed(0x1, drvdata->base + TRCOSLAR);
> +	drvdata->os_unlock = false;
> +	isb();
> +}
> +
>  static bool etm4_arch_supported(u8 arch)
>  {
>  	/* Mask out the minor version number */
> @@ -1085,6 +1104,303 @@ static void etm4_init_trace_id(struct etmv4_drvdata *drvdata)
>  	drvdata->trcid = coresight_get_trace_id(drvdata->cpu);
>  }
>  
> +#ifdef CONFIG_CPU_PM
> +static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> +{
> +	int i, ret = 0;
> +	u32 control;
> +	struct etmv4_save_state *state;
> +	struct device *etm_dev = &drvdata->csdev->dev;
> +
> +	/*
> +	 * As recommended by 3.4.1 ("The procedure when powering down the PE")
> +	 * of ARM IHI 0064D
> +	 */
> +	dsb(sy);
> +	isb();
> +
> +	CS_UNLOCK(drvdata->base);
> +
> +	/* Lock the OS lock to disable trace and external debugger access */
> +	etm4_os_lock(drvdata);
> +
> +	/* wait for TRCSTATR.PMSTABLE to go up */
> +	if (coresight_timeout(drvdata->base, TRCSTATR,
> +					TRCSTATR_PMSTABLE_BIT, 1)) {
> +		dev_err(etm_dev,
> +			"timeout while waiting for PM Stable Status\n");
> +		etm4_os_unlock(drvdata);
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	state = &drvdata->save_state;
> +
> +	state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
> +	state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> +	state->trcconfigr = readl(drvdata->base + TRCCONFIGR);
> +	state->trcauxctlr = readl(drvdata->base + TRCAUXCTLR);
> +	state->trceventctl0r = readl(drvdata->base + TRCEVENTCTL0R);
> +	state->trceventctl1r = readl(drvdata->base + TRCEVENTCTL1R);
> +	state->trcstallctlr = readl(drvdata->base + TRCSTALLCTLR);
> +	state->trctsctlr = readl(drvdata->base + TRCTSCTLR);
> +	state->trcsyncpr = readl(drvdata->base + TRCSYNCPR);
> +	state->trcccctlr = readl(drvdata->base + TRCCCCTLR);
> +	state->trcbbctlr = readl(drvdata->base + TRCBBCTLR);
> +	state->trctraceidr = readl(drvdata->base + TRCTRACEIDR);
> +	state->trcqctlr = readl(drvdata->base + TRCQCTLR);
> +
> +	state->trcvictlr = readl(drvdata->base + TRCVICTLR);
> +	state->trcviiectlr = readl(drvdata->base + TRCVIIECTLR);
> +	state->trcvissctlr = readl(drvdata->base + TRCVISSCTLR);
> +	state->trcvipcssctlr = readl(drvdata->base + TRCVIPCSSCTLR);
> +	state->trcvdctlr = readl(drvdata->base + TRCVDCTLR);
> +	state->trcvdsacctlr = readl(drvdata->base + TRCVDSACCTLR);
> +	state->trcvdarcctlr = readl(drvdata->base + TRCVDARCCTLR);
> +
> +	for (i = 0; i < drvdata->nrseqstate; i++)
> +		state->trcseqevr[i] = readl(drvdata->base + TRCSEQEVRn(i));
> +
> +	state->trcseqrstevr = readl(drvdata->base + TRCSEQRSTEVR);
> +	state->trcseqstr = readl(drvdata->base + TRCSEQSTR);
> +	state->trcextinselr = readl(drvdata->base + TRCEXTINSELR);
> +
> +	for (i = 0; i < drvdata->nr_cntr; i++) {
> +		state->trccntrldvr[i] = readl(drvdata->base + TRCCNTRLDVRn(i));
> +		state->trccntctlr[i] = readl(drvdata->base + TRCCNTCTLRn(i));
> +		state->trccntvr[i] = readl(drvdata->base + TRCCNTVRn(i));
> +	}
> +
> +	for (i = 0; i < drvdata->nr_resource * 2; i++)
> +		state->trcrsctlr[i] = readl(drvdata->base + TRCRSCTLRn(i));
> +
> +	for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> +		state->trcssccr[i] = readl(drvdata->base + TRCSSCCRn(i));
> +		state->trcsscsr[i] = readl(drvdata->base + TRCSSCSRn(i));
> +		state->trcsspcicr[i] = readl(drvdata->base + TRCSSPCICRn(i));
> +	}
> +
> +	for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> +		state->trcacvr[i] = readl(drvdata->base + TRCACVRn(i));
> +		state->trcacatr[i] = readl(drvdata->base + TRCACATRn(i));
> +	}
> +
> +	/*
> +	 * Data trace stream is architecturally prohibited for A profile cores
> +	 * so we don't save (or later restore) trcdvcvr and trcdvcmr - As per
> +	 * section 1.3.4 ("Possible functional configurations of an ETMv4 trace
> +	 * unit") of ARM IHI 0064D.
> +	 */
> +
> +	for (i = 0; i < drvdata->numcidc; i++)
> +		state->trccidcvr[i] = readl(drvdata->base + TRCCIDCVRn(i));
> +
> +	for (i = 0; i < drvdata->numvmidc; i++)
> +		state->trcvmidcvr[i] = readl(drvdata->base + TRCVMIDCVRn(i));
> +
> +	state->trccidcctlr0 = readl(drvdata->base + TRCCIDCCTLR0);
> +	state->trccidcctlr1 = readl(drvdata->base + TRCCIDCCTLR1);
> +
> +	state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR0);
> +	state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR1);
> +
> +	state->trcclaimset = readl(drvdata->base + TRCCLAIMCLR);
> +
> +	/* wait for TRCSTATR.IDLE to go up */
> +	if (coresight_timeout(drvdata->base, TRCSTATR, TRCSTATR_IDLE_BIT, 1)) {
> +		dev_err(etm_dev,
> +			"timeout while waiting for Idle Trace Status\n");
> +		etm4_os_unlock(drvdata);
> +		ret = -EBUSY;
> +		goto out;
> +	}
> +
> +	drvdata->state_needs_restore = true;
> +
> +	/*
> +	 * Power can be removed from the trace unit now. We do this to
> +	 * potentially save power on systems that respect the TRCPDCR_PU
> +	 * despite requesting software to save/restore state.
> +	 */
> +	control = readl_relaxed(drvdata->base + TRCPDCR);
> +	control &= ~TRCPDCR_PU;
> +	writel_relaxed(control, drvdata->base + TRCPDCR);
> +
> +out:
> +	CS_LOCK(drvdata->base);
> +	return ret;
> +}
> +
> +static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
> +{
> +	int i;
> +	struct etmv4_save_state *state;
> +
> +	state = &drvdata->save_state;
> +
> +	CS_UNLOCK(drvdata->base);
> +
> +	writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> +
> +	writel_relaxed(state->trcprgctlr, drvdata->base + TRCPRGCTLR);
> +	writel_relaxed(state->trcprocselr, drvdata->base + TRCPROCSELR);
> +	writel_relaxed(state->trcconfigr, drvdata->base + TRCCONFIGR);
> +	writel_relaxed(state->trcauxctlr, drvdata->base + TRCAUXCTLR);
> +	writel_relaxed(state->trceventctl0r, drvdata->base + TRCEVENTCTL0R);
> +	writel_relaxed(state->trceventctl1r, drvdata->base + TRCEVENTCTL1R);
> +	writel_relaxed(state->trcstallctlr, drvdata->base + TRCSTALLCTLR);
> +	writel_relaxed(state->trctsctlr, drvdata->base + TRCTSCTLR);
> +	writel_relaxed(state->trcsyncpr, drvdata->base + TRCSYNCPR);
> +	writel_relaxed(state->trcccctlr, drvdata->base + TRCCCCTLR);
> +	writel_relaxed(state->trcbbctlr, drvdata->base + TRCBBCTLR);
> +	writel_relaxed(state->trctraceidr, drvdata->base + TRCTRACEIDR);
> +	writel_relaxed(state->trcqctlr, drvdata->base + TRCQCTLR);
> +
> +	writel_relaxed(state->trcvictlr, drvdata->base + TRCVICTLR);
> +	writel_relaxed(state->trcviiectlr, drvdata->base + TRCVIIECTLR);
> +	writel_relaxed(state->trcvissctlr, drvdata->base + TRCVISSCTLR);
> +	writel_relaxed(state->trcvipcssctlr, drvdata->base + TRCVIPCSSCTLR);
> +	writel_relaxed(state->trcvdctlr, drvdata->base + TRCVDCTLR);
> +	writel_relaxed(state->trcvdsacctlr, drvdata->base + TRCVDSACCTLR);
> +	writel_relaxed(state->trcvdarcctlr, drvdata->base + TRCVDARCCTLR);
> +
> +	for (i = 0; i < drvdata->nrseqstate; i++)
> +		writel_relaxed(state->trcseqevr[i],
> +					drvdata->base + TRCSEQEVRn(i));
> +
> +	writel_relaxed(state->trcseqrstevr, drvdata->base + TRCSEQRSTEVR);
> +	writel_relaxed(state->trcseqstr, drvdata->base + TRCSEQSTR);
> +	writel_relaxed(state->trcextinselr, drvdata->base + TRCEXTINSELR);
> +
> +	for (i = 0; i < drvdata->nr_cntr; i++) {
> +		writel_relaxed(state->trccntrldvr[i],
> +					drvdata->base + TRCCNTRLDVRn(i));
> +		writel_relaxed(state->trccntctlr[i],
> +					drvdata->base + TRCCNTCTLRn(i));
> +		writel_relaxed(state->trccntvr[i],
> +					drvdata->base + TRCCNTVRn(i));
> +	}
> +
> +	for (i = 0; i < drvdata->nr_resource * 2; i++)
> +		writel_relaxed(state->trcrsctlr[i],
> +					drvdata->base + TRCRSCTLRn(i));
> +
> +	for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> +		writel_relaxed(state->trcssccr[i],
> +					drvdata->base + TRCSSCCRn(i));
> +		writel_relaxed(state->trcsscsr[i],
> +					drvdata->base + TRCSSCSRn(i));
> +		writel_relaxed(state->trcsspcicr[i],
> +					drvdata->base + TRCSSPCICRn(i));
> +	}
> +
> +	for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> +		writel_relaxed(state->trcacvr[i],
> +					drvdata->base + TRCACVRn(i));
> +		writel_relaxed(state->trcacatr[i],
> +					drvdata->base + TRCACATRn(i));
> +	}
> +
> +	for (i = 0; i < drvdata->numcidc; i++)
> +		writel_relaxed(state->trccidcvr[i],
> +					drvdata->base + TRCCIDCVRn(i));
> +
> +	for (i = 0; i < drvdata->numvmidc; i++)
> +		writel_relaxed(state->trcvmidcvr[i],
> +					drvdata->base + TRCVMIDCVRn(i));
> +
> +	writel_relaxed(state->trccidcctlr0, drvdata->base + TRCCIDCCTLR0);
> +	writel_relaxed(state->trccidcctlr1, drvdata->base + TRCCIDCCTLR1);
> +
> +	writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR0);
> +	writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR1);
> +
> +	writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> +
> +	drvdata->state_needs_restore = false;
> +
> +	/*
> +	 * Request to keep the trace unit powered and also
> +	 * emulation of powerdown
> +	 */
> +	writel_relaxed(readl_relaxed(drvdata->base + TRCPDCR) | TRCPDCR_PU,
> +		       drvdata->base + TRCPDCR);
> +
> +	/*
> +	 * As recommended by section 4.3.7 ("Synchronization when using the
> +	 * memory-mapped interface") of ARM IHI 0064D
> +	 */
> +	dsb(sy);
> +	isb();
> +
> +	/* Unlock the OS lock to re-enable trace and external debug access */
> +	etm4_os_unlock(drvdata);
> +	CS_LOCK(drvdata->base);
> +}
> +
> +static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd,
> +			      void *v)
> +{
> +	struct etmv4_drvdata *drvdata;
> +	unsigned int cpu = smp_processor_id();
> +
> +	if (!etmdrvdata[cpu])
> +		return 0;
> +
> +	drvdata = etmdrvdata[cpu];
> +
> +	if (pm_save_enable == PARAM_PM_SAVE_NEVER ||
> +	    (pm_save_enable == PARAM_PM_SAVE_FIRMWARE &&
> +	     !drvdata->pm_save_enable))
> +		return NOTIFY_OK;
> +
> +	if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
> +		return NOTIFY_BAD;
> +
> +	switch (cmd) {
> +	case CPU_PM_ENTER:
> +		/* save the state if coresight is in use */
> +		if (coresight_is_claimed_any(drvdata->base))
> +			if (etm4_cpu_save(drvdata))
> +				return NOTIFY_BAD;
> +		break;
> +	case CPU_PM_EXIT:
> +	case CPU_PM_ENTER_FAILED:
> +		/* trcclaimset is set when there is state to restore */
> +		if (drvdata->state_needs_restore)
> +			etm4_cpu_restore(drvdata);
> +		break;
> +	default:
> +		return NOTIFY_DONE;
> +	}
> +
> +	return NOTIFY_OK;
> +}
> +
> +static struct notifier_block etm4_cpu_pm_nb = {
> +	.notifier_call = etm4_cpu_pm_notify,
> +};
> +
> +static int etm4_cpu_pm_register(void)
> +{
> +	return cpu_pm_register_notifier(&etm4_cpu_pm_nb);
> +}
> +
> +static void etm4_cpu_pm_unregister(void)
> +{
> +	cpu_pm_unregister_notifier(&etm4_cpu_pm_nb);
> +}
> +#else
> +static int etm4_cpu_pm_register(void) { return 0; }
> +static void etm4_cpu_pm_unregister(void) { }
> +#endif
> +
> +static inline bool etm4_needs_save_restore(struct device *dev)
> +{
> +	return fwnode_property_present(dev->fwnode,
> +				       "arm,coresight-needs-save-restore");
> +}
> +
>  static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>  {
>  	int ret;
> @@ -1101,6 +1417,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>  
>  	dev_set_drvdata(dev, drvdata);
>  
> +	drvdata->pm_save_enable = etm4_needs_save_restore(dev);
> +
>  	/* Validity for the resource is already checked by the AMBA core */
>  	base = devm_ioremap_resource(dev, res);
>  	if (IS_ERR(base))
> @@ -1132,6 +1450,10 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>  		if (ret < 0)
>  			goto err_arch_supported;
>  		hp_online = ret;
> +
> +		ret = etm4_cpu_pm_register();
> +		if (ret)
> +			goto err_arch_supported;
>  	}
>  
>  	cpus_read_unlock();
> @@ -1182,6 +1504,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
>  
>  err_arch_supported:
>  	if (--etm4_count == 0) {
> +		etm4_cpu_pm_unregister();
> +
>  		cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING);
>  		if (hp_online)
>  			cpuhp_remove_state_nocalls(hp_online);
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> index 4523f10ddd0f..c31634c64f87 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> @@ -175,6 +175,7 @@
>  					 ETM_MODE_EXCL_USER)
>  
>  #define TRCSTATR_IDLE_BIT		0
> +#define TRCSTATR_PMSTABLE_BIT		1
>  #define ETM_DEFAULT_ADDR_COMP		0
>  
>  /* PowerDown Control Register bits */
> @@ -281,6 +282,63 @@ struct etmv4_config {
>  	u32				ext_inp;
>  };
>  
> +/**
> + * struct etm4_save_state - state to be preserved when ETM is without power
> + */
> +struct etmv4_save_state {
> +	u32	trcprgctlr;
> +	u32	trcprocselr;
> +	u32	trcconfigr;
> +	u32	trcauxctlr;
> +	u32	trceventctl0r;
> +	u32	trceventctl1r;
> +	u32	trcstallctlr;
> +	u32	trctsctlr;
> +	u32	trcsyncpr;
> +	u32	trcccctlr;
> +	u32	trcbbctlr;
> +	u32	trctraceidr;
> +	u32	trcqctlr;
> +
> +	u32	trcvictlr;
> +	u32	trcviiectlr;
> +	u32	trcvissctlr;
> +	u32	trcvipcssctlr;
> +	u32	trcvdctlr;
> +	u32	trcvdsacctlr;
> +	u32	trcvdarcctlr;
> +
> +	u32	trcseqevr[ETM_MAX_SEQ_STATES];
> +	u32	trcseqrstevr;
> +	u32	trcseqstr;
> +	u32	trcextinselr;
> +	u32	trccntrldvr[ETMv4_MAX_CNTR];
> +	u32	trccntctlr[ETMv4_MAX_CNTR];
> +	u32	trccntvr[ETMv4_MAX_CNTR];
> +
> +	u32	trcrsctlr[ETM_MAX_RES_SEL * 2];
> +
> +	u32	trcssccr[ETM_MAX_SS_CMP];
> +	u32	trcsscsr[ETM_MAX_SS_CMP];
> +	u32	trcsspcicr[ETM_MAX_SS_CMP];
> +
> +	u64	trcacvr[ETM_MAX_SINGLE_ADDR_CMP];
> +	u64	trcacatr[ETM_MAX_SINGLE_ADDR_CMP];
> +	u64	trccidcvr[ETMv4_MAX_CTXID_CMP];
> +	u32	trcvmidcvr[ETM_MAX_VMID_CMP];
> +	u32	trccidcctlr0;
> +	u32	trccidcctlr1;
> +	u32	trcvmidcctlr0;
> +	u32	trcvmidcctlr1;
> +
> +	u32	trcclaimset;
> +
> +	u32	cntr_val[ETMv4_MAX_CNTR];
> +	u32	seq_state;
> +	u32	vinst_ctrl;
> +	u32	ss_status[ETM_MAX_SS_CMP];
> +};
> +
>  /**
>   * struct etm4_drvdata - specifics associated to an ETM component
>   * @base:       Memory mapped base address for this component.
> @@ -336,6 +394,7 @@ struct etmv4_config {
>   * @atbtrig:	If the implementation can support ATB triggers
>   * @lpoverride:	If the implementation can support low-power state over.
>   * @config:	structure holding configuration parameters.
> + * @save_state:	State to be preserved across power loss
>   */
>  struct etmv4_drvdata {
>  	void __iomem			*base;
> @@ -367,6 +426,7 @@ struct etmv4_drvdata {
>  	u8				q_support;
>  	bool				sticky_enable;
>  	bool				boot_enable;
> +	bool				pm_save_enable;

Please document - otherwise we have to guess what this is for.

>  	bool				os_unlock;
>  	bool				instrp0;
>  	bool				trcbb;
> @@ -381,6 +441,8 @@ struct etmv4_drvdata {
>  	bool				atbtrig;
>  	bool				lpoverride;
>  	struct etmv4_config		config;
> +	struct etmv4_save_state		save_state;
> +	bool				state_needs_restore;

Same as above.

Did you get a chance to test your code on something else than Juno?  I gave it a
spin on my dragonboard and I can't get the kernel to finish booting - my board
resets just after the ETMv4 have initialized.  I'll try to debug this futher tomorrow.

Thanks,
Mathieu

>  };
>  
>  /* Address comparator access types */
> diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> index 86d1fc2c1bd4..aba71a5a025f 100644
> --- a/drivers/hwtracing/coresight/coresight.c
> +++ b/drivers/hwtracing/coresight/coresight.c
> @@ -140,7 +140,7 @@ static inline bool coresight_is_claimed_self_hosted(void __iomem *base)
>  	return coresight_read_claim_tags(base) == CORESIGHT_CLAIM_SELF_HOSTED;
>  }
>  
> -static inline bool coresight_is_claimed_any(void __iomem *base)
> +bool coresight_is_claimed_any(void __iomem *base)
>  {
>  	return coresight_read_claim_tags(base) != 0;
>  }
> diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> index a2b68823717b..c3a875dffe65 100644
> --- a/include/linux/coresight.h
> +++ b/include/linux/coresight.h
> @@ -285,6 +285,9 @@ extern void coresight_disclaim_device(void __iomem *base);
>  extern void coresight_disclaim_device_unlocked(void __iomem *base);
>  extern char *coresight_alloc_device_name(struct coresight_dev_list *devs,
>  					 struct device *dev);
> +
> +extern bool coresight_is_claimed_any(void __iomem *base);
> +
>  #else
>  static inline struct coresight_device *
>  coresight_register(struct coresight_desc *desc) { return NULL; }
> @@ -307,6 +310,11 @@ static inline int coresight_claim_device(void __iomem *base)
>  static inline void coresight_disclaim_device(void __iomem *base) {}
>  static inline void coresight_disclaim_device_unlocked(void __iomem *base) {}
>  
> +static inline bool coresight_is_claimed_any(void __iomem *base)
> +{
> +	return false;
> +}
> +
>  #endif
>  
>  extern int coresight_get_cpu(struct device *dev);
> -- 
> 2.21.0
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v3 5/6] coresight: etm4x: save/restore state across CPU low power states

[Andrew Murray]

On Mon, Jul 15, 2019 at 03:22:47PM +0100, Mike Leach wrote:
> HI Andrew,
> 
> On Thu, 11 Jul 2019 at 17:01, Andrew Murray <andrew.murray@arm.com> wrote:
> >
> > Some hardware will ignore bit TRCPDCR.PU which is used to signal
> > to hardware that power should not be removed from the trace unit.
> > Let's mitigate against this by conditionally saving and restoring
> > the trace unit state when the CPU enters low power states.
> >
> > This patchset introduces a firmware property named
> > 'arm,coresight-needs-save-restore' - when this is present the
> > hardware state will be conditionally saved and restored.
> >
> > A module parameter 'pm_save_enable' is also introduced which can
> > be configured to override the firmware property. This can be set
> > to never allow save/restore, to conditionally allow it, or to
> > do as the firmware indicates (default).
> >
> > The hardware state is only ever saved and restored when the claim
> > tags indicate that coresight is in use.
> >
> > Signed-off-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/hwtracing/coresight/coresight-etm4x.c | 324 ++++++++++++++++++
> >  drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
> >  drivers/hwtracing/coresight/coresight.c       |   2 +-
> >  include/linux/coresight.h                     |   8 +
> >  4 files changed, 395 insertions(+), 1 deletion(-)
> >
> > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> > index d993359e2117..b0bd8158bf13 100644
> > --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> > +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> > @@ -18,6 +18,7 @@
> >  #include <linux/stat.h>
> >  #include <linux/clk.h>
> >  #include <linux/cpu.h>
> > +#include <linux/cpu_pm.h>
> >  #include <linux/coresight.h>
> >  #include <linux/coresight-pmu.h>
> >  #include <linux/pm_wakeup.h>
> > @@ -26,6 +27,7 @@
> >  #include <linux/uaccess.h>
> >  #include <linux/perf_event.h>
> >  #include <linux/pm_runtime.h>
> > +#include <linux/property.h>
> >  #include <asm/sections.h>
> >  #include <asm/local.h>
> >  #include <asm/virt.h>
> > @@ -37,6 +39,15 @@ static int boot_enable;
> >  module_param(boot_enable, int, 0444);
> >  MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
> >
> > +#define PARAM_PM_SAVE_NEVER    0 /* never save/restore state */
> > +#define PARAM_PM_SAVE_ALWAYS   1 /* always save/restore state */
> > +#define PARAM_PM_SAVE_FIRMWARE 2 /* only save/restore if firmware flag set */
> > +
> > +static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
> > +module_param(pm_save_enable, int, 0644);
> > +MODULE_PARM_DESC(pm_save_enable,
> > +       "Save/restore state on power down: 0 = never, 1 = always, 2 = firmware");
> > +
> >  /* The number of ETMv4 currently registered */
> >  static int etm4_count;
> >  static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
> > @@ -54,6 +65,14 @@ static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
> >         isb();
> >  }
> >
> > +static void etm4_os_lock(struct etmv4_drvdata *drvdata)
> > +{
> > +       /* Writing 0x1 to TRCOSLAR locks the trace registers */
> > +       writel_relaxed(0x1, drvdata->base + TRCOSLAR);
> > +       drvdata->os_unlock = false;
> > +       isb();
> > +}
> > +
> >  static bool etm4_arch_supported(u8 arch)
> >  {
> >         /* Mask out the minor version number */
> > @@ -1085,6 +1104,303 @@ static void etm4_init_trace_id(struct etmv4_drvdata *drvdata)
> >         drvdata->trcid = coresight_get_trace_id(drvdata->cpu);
> >  }
> >
> > +#ifdef CONFIG_CPU_PM
> > +static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> > +{
> > +       int i, ret = 0;
> > +       u32 control;
> > +       struct etmv4_save_state *state;
> > +       struct device *etm_dev = &drvdata->csdev->dev;
> > +
> > +       /*
> > +        * As recommended by 3.4.1 ("The procedure when powering down the PE")
> > +        * of ARM IHI 0064D
> > +        */
> > +       dsb(sy);
> > +       isb();
> > +
> > +       CS_UNLOCK(drvdata->base);
> > +
> > +       /* Lock the OS lock to disable trace and external debugger access */
> > +       etm4_os_lock(drvdata);
> > +
> > +       /* wait for TRCSTATR.PMSTABLE to go up */
> > +       if (coresight_timeout(drvdata->base, TRCSTATR,
> > +                                       TRCSTATR_PMSTABLE_BIT, 1)) {
> > +               dev_err(etm_dev,
> > +                       "timeout while waiting for PM Stable Status\n");
> > +               etm4_os_unlock(drvdata);
> > +               ret = -EBUSY;
> > +               goto out;
> > +       }
> > +
> > +       state = &drvdata->save_state;
> > +
> > +       state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
> > +       state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> > +       state->trcconfigr = readl(drvdata->base + TRCCONFIGR);
> > +       state->trcauxctlr = readl(drvdata->base + TRCAUXCTLR);
> > +       state->trceventctl0r = readl(drvdata->base + TRCEVENTCTL0R);
> > +       state->trceventctl1r = readl(drvdata->base + TRCEVENTCTL1R);
> > +       state->trcstallctlr = readl(drvdata->base + TRCSTALLCTLR);
> > +       state->trctsctlr = readl(drvdata->base + TRCTSCTLR);
> > +       state->trcsyncpr = readl(drvdata->base + TRCSYNCPR);
> > +       state->trcccctlr = readl(drvdata->base + TRCCCCTLR);
> > +       state->trcbbctlr = readl(drvdata->base + TRCBBCTLR);
> > +       state->trctraceidr = readl(drvdata->base + TRCTRACEIDR);
> > +       state->trcqctlr = readl(drvdata->base + TRCQCTLR);
> > +
> > +       state->trcvictlr = readl(drvdata->base + TRCVICTLR);
> > +       state->trcviiectlr = readl(drvdata->base + TRCVIIECTLR);
> > +       state->trcvissctlr = readl(drvdata->base + TRCVISSCTLR);
> > +       state->trcvipcssctlr = readl(drvdata->base + TRCVIPCSSCTLR);
> > +       state->trcvdctlr = readl(drvdata->base + TRCVDCTLR);
> > +       state->trcvdsacctlr = readl(drvdata->base + TRCVDSACCTLR);
> > +       state->trcvdarcctlr = readl(drvdata->base + TRCVDARCCTLR);
> > +
> > +       for (i = 0; i < drvdata->nrseqstate; i++)
> > +               state->trcseqevr[i] = readl(drvdata->base + TRCSEQEVRn(i));
> > +
> > +       state->trcseqrstevr = readl(drvdata->base + TRCSEQRSTEVR);
> > +       state->trcseqstr = readl(drvdata->base + TRCSEQSTR);
> > +       state->trcextinselr = readl(drvdata->base + TRCEXTINSELR);
> > +
> > +       for (i = 0; i < drvdata->nr_cntr; i++) {
> > +               state->trccntrldvr[i] = readl(drvdata->base + TRCCNTRLDVRn(i));
> > +               state->trccntctlr[i] = readl(drvdata->base + TRCCNTCTLRn(i));
> > +               state->trccntvr[i] = readl(drvdata->base + TRCCNTVRn(i));
> > +       }
> > +
> > +       for (i = 0; i < drvdata->nr_resource * 2; i++)
> > +               state->trcrsctlr[i] = readl(drvdata->base + TRCRSCTLRn(i));
> > +
> > +       for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> > +               state->trcssccr[i] = readl(drvdata->base + TRCSSCCRn(i));
> > +               state->trcsscsr[i] = readl(drvdata->base + TRCSSCSRn(i));
> > +               state->trcsspcicr[i] = readl(drvdata->base + TRCSSPCICRn(i));
> > +       }
> > +
> > +       for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> > +               state->trcacvr[i] = readl(drvdata->base + TRCACVRn(i));
> > +               state->trcacatr[i] = readl(drvdata->base + TRCACATRn(i));
> > +       }
> > +
> > +       /*
> > +        * Data trace stream is architecturally prohibited for A profile cores
> > +        * so we don't save (or later restore) trcdvcvr and trcdvcmr - As per
> > +        * section 1.3.4 ("Possible functional configurations of an ETMv4 trace
> > +        * unit") of ARM IHI 0064D.
> > +        */
> > +
> > +       for (i = 0; i < drvdata->numcidc; i++)
> > +               state->trccidcvr[i] = readl(drvdata->base + TRCCIDCVRn(i));
> > +
> > +       for (i = 0; i < drvdata->numvmidc; i++)
> > +               state->trcvmidcvr[i] = readl(drvdata->base + TRCVMIDCVRn(i));
> > +
> > +       state->trccidcctlr0 = readl(drvdata->base + TRCCIDCCTLR0);
> > +       state->trccidcctlr1 = readl(drvdata->base + TRCCIDCCTLR1);
> > +
> > +       state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR0);
> > +       state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR1);
> > +
> > +       state->trcclaimset = readl(drvdata->base + TRCCLAIMCLR);
> > +
> > +       /* wait for TRCSTATR.IDLE to go up */
> > +       if (coresight_timeout(drvdata->base, TRCSTATR, TRCSTATR_IDLE_BIT, 1)) {
> > +               dev_err(etm_dev,
> > +                       "timeout while waiting for Idle Trace Status\n");
> > +               etm4_os_unlock(drvdata);
> > +               ret = -EBUSY;
> > +               goto out;
> > +       }
> > +
> > +       drvdata->state_needs_restore = true;
> > +
> > +       /*
> > +        * Power can be removed from the trace unit now. We do this to
> > +        * potentially save power on systems that respect the TRCPDCR_PU
> > +        * despite requesting software to save/restore state.
> > +        */
> > +       control = readl_relaxed(drvdata->base + TRCPDCR);
> > +       control &= ~TRCPDCR_PU;
> > +       writel_relaxed(control, drvdata->base + TRCPDCR);
> > +
> 
> Removing the power is fine, except you are not remembering the state
> for restore later.
> 
> > +out:
> > +       CS_LOCK(drvdata->base);
> > +       return ret;
> > +}
> > +
> > +static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
> > +{
> > +       int i;
> > +       struct etmv4_save_state *state;
> > +
> > +       state = &drvdata->save_state;
> > +
> > +       CS_UNLOCK(drvdata->base);
> > +
> > +       writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> > +
> > +       writel_relaxed(state->trcprgctlr, drvdata->base + TRCPRGCTLR);
> > +       writel_relaxed(state->trcprocselr, drvdata->base + TRCPROCSELR);
> > +       writel_relaxed(state->trcconfigr, drvdata->base + TRCCONFIGR);
> > +       writel_relaxed(state->trcauxctlr, drvdata->base + TRCAUXCTLR);
> > +       writel_relaxed(state->trceventctl0r, drvdata->base + TRCEVENTCTL0R);
> > +       writel_relaxed(state->trceventctl1r, drvdata->base + TRCEVENTCTL1R);
> > +       writel_relaxed(state->trcstallctlr, drvdata->base + TRCSTALLCTLR);
> > +       writel_relaxed(state->trctsctlr, drvdata->base + TRCTSCTLR);
> > +       writel_relaxed(state->trcsyncpr, drvdata->base + TRCSYNCPR);
> > +       writel_relaxed(state->trcccctlr, drvdata->base + TRCCCCTLR);
> > +       writel_relaxed(state->trcbbctlr, drvdata->base + TRCBBCTLR);
> > +       writel_relaxed(state->trctraceidr, drvdata->base + TRCTRACEIDR);
> > +       writel_relaxed(state->trcqctlr, drvdata->base + TRCQCTLR);
> > +
> > +       writel_relaxed(state->trcvictlr, drvdata->base + TRCVICTLR);
> > +       writel_relaxed(state->trcviiectlr, drvdata->base + TRCVIIECTLR);
> > +       writel_relaxed(state->trcvissctlr, drvdata->base + TRCVISSCTLR);
> > +       writel_relaxed(state->trcvipcssctlr, drvdata->base + TRCVIPCSSCTLR);
> > +       writel_relaxed(state->trcvdctlr, drvdata->base + TRCVDCTLR);
> > +       writel_relaxed(state->trcvdsacctlr, drvdata->base + TRCVDSACCTLR);
> > +       writel_relaxed(state->trcvdarcctlr, drvdata->base + TRCVDARCCTLR);
> > +
> > +       for (i = 0; i < drvdata->nrseqstate; i++)
> > +               writel_relaxed(state->trcseqevr[i],
> > +                                       drvdata->base + TRCSEQEVRn(i));
> > +
> > +       writel_relaxed(state->trcseqrstevr, drvdata->base + TRCSEQRSTEVR);
> > +       writel_relaxed(state->trcseqstr, drvdata->base + TRCSEQSTR);
> > +       writel_relaxed(state->trcextinselr, drvdata->base + TRCEXTINSELR);
> > +
> > +       for (i = 0; i < drvdata->nr_cntr; i++) {
> > +               writel_relaxed(state->trccntrldvr[i],
> > +                                       drvdata->base + TRCCNTRLDVRn(i));
> > +               writel_relaxed(state->trccntctlr[i],
> > +                                       drvdata->base + TRCCNTCTLRn(i));
> > +               writel_relaxed(state->trccntvr[i],
> > +                                       drvdata->base + TRCCNTVRn(i));
> > +       }
> > +
> > +       for (i = 0; i < drvdata->nr_resource * 2; i++)
> > +               writel_relaxed(state->trcrsctlr[i],
> > +                                       drvdata->base + TRCRSCTLRn(i));
> > +
> > +       for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> > +               writel_relaxed(state->trcssccr[i],
> > +                                       drvdata->base + TRCSSCCRn(i));
> > +               writel_relaxed(state->trcsscsr[i],
> > +                                       drvdata->base + TRCSSCSRn(i));
> > +               writel_relaxed(state->trcsspcicr[i],
> > +                                       drvdata->base + TRCSSPCICRn(i));
> > +       }
> > +
> > +       for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> > +               writel_relaxed(state->trcacvr[i],
> > +                                       drvdata->base + TRCACVRn(i));
> > +               writel_relaxed(state->trcacatr[i],
> > +                                       drvdata->base + TRCACATRn(i));
> > +       }
> > +
> > +       for (i = 0; i < drvdata->numcidc; i++)
> > +               writel_relaxed(state->trccidcvr[i],
> > +                                       drvdata->base + TRCCIDCVRn(i));
> > +
> > +       for (i = 0; i < drvdata->numvmidc; i++)
> > +               writel_relaxed(state->trcvmidcvr[i],
> > +                                       drvdata->base + TRCVMIDCVRn(i));
> > +
> > +       writel_relaxed(state->trccidcctlr0, drvdata->base + TRCCIDCCTLR0);
> > +       writel_relaxed(state->trccidcctlr1, drvdata->base + TRCCIDCCTLR1);
> > +
> > +       writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR0);
> > +       writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR1);
> > +
> > +       writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> > +
> > +       drvdata->state_needs_restore = false;
> > +
> > +       /*
> > +        * Request to keep the trace unit powered and also
> > +        * emulation of powerdown
> > +        */
> > +       writel_relaxed(readl_relaxed(drvdata->base + TRCPDCR) | TRCPDCR_PU,
> > +                      drvdata->base + TRCPDCR);
> > +
> 
> This sets the PU bit irrespective of the value on power down. The bit
> should be restored to the state on power down not unconditionally set.

My rationale here was to save/restore only the registers listed in 3.4.1
of the spec (ARM IHI 0064D, "The procedure when powering down the PE") -
though the TRCPDCR isn't listed there.

I originally added the toggle of the PU bit to keep the state inline with
that set by etm4_[enable,disable]_hw. I.e. without save/restore, when
self-hosted these functions will ensure the PU bit is always set anyway.
Thus making the change you suggest has only an impact on externally-hosted
debug when the debugger doesn't set the PU bit.

I'm happy to make this change, however is there any reason why an external
debugger wouldn't set this bit? What harm does it do by always setting it
here?

Thanks,

Andrew Murray

> 
> Regards
> 
> Mike
> 
> 
> > +       /*
> > +        * As recommended by section 4.3.7 ("Synchronization when using the
> > +        * memory-mapped interface") of ARM IHI 0064D
> > +        */
> > +       dsb(sy);
> > +       isb();
> > +
> > +       /* Unlock the OS lock to re-enable trace and external debug access */
> > +       etm4_os_unlock(drvdata);
> > +       CS_LOCK(drvdata->base);
> > +}
> > +
> > +static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd,
> > +                             void *v)
> > +{
> > +       struct etmv4_drvdata *drvdata;
> > +       unsigned int cpu = smp_processor_id();
> > +
> > +       if (!etmdrvdata[cpu])
> > +               return 0;
> > +
> > +       drvdata = etmdrvdata[cpu];
> > +
> > +       if (pm_save_enable == PARAM_PM_SAVE_NEVER ||
> > +           (pm_save_enable == PARAM_PM_SAVE_FIRMWARE &&
> > +            !drvdata->pm_save_enable))
> > +               return NOTIFY_OK;
> > +
> > +       if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
> > +               return NOTIFY_BAD;
> > +
> > +       switch (cmd) {
> > +       case CPU_PM_ENTER:
> > +               /* save the state if coresight is in use */
> > +               if (coresight_is_claimed_any(drvdata->base))
> > +                       if (etm4_cpu_save(drvdata))
> > +                               return NOTIFY_BAD;
> > +               break;
> > +       case CPU_PM_EXIT:
> > +       case CPU_PM_ENTER_FAILED:
> > +               /* trcclaimset is set when there is state to restore */
> > +               if (drvdata->state_needs_restore)
> > +                       etm4_cpu_restore(drvdata);
> > +               break;
> > +       default:
> > +               return NOTIFY_DONE;
> > +       }
> > +
> > +       return NOTIFY_OK;
> > +}
> > +
> > +static struct notifier_block etm4_cpu_pm_nb = {
> > +       .notifier_call = etm4_cpu_pm_notify,
> > +};
> > +
> > +static int etm4_cpu_pm_register(void)
> > +{
> > +       return cpu_pm_register_notifier(&etm4_cpu_pm_nb);
> > +}
> > +
> > +static void etm4_cpu_pm_unregister(void)
> > +{
> > +       cpu_pm_unregister_notifier(&etm4_cpu_pm_nb);
> > +}
> > +#else
> > +static int etm4_cpu_pm_register(void) { return 0; }
> > +static void etm4_cpu_pm_unregister(void) { }
> > +#endif
> > +
> > +static inline bool etm4_needs_save_restore(struct device *dev)
> > +{
> > +       return fwnode_property_present(dev->fwnode,
> > +                                      "arm,coresight-needs-save-restore");
> > +}
> > +
> >  static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >  {
> >         int ret;
> > @@ -1101,6 +1417,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >
> >         dev_set_drvdata(dev, drvdata);
> >
> > +       drvdata->pm_save_enable = etm4_needs_save_restore(dev);
> > +
> >         /* Validity for the resource is already checked by the AMBA core */
> >         base = devm_ioremap_resource(dev, res);
> >         if (IS_ERR(base))
> > @@ -1132,6 +1450,10 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >                 if (ret < 0)
> >                         goto err_arch_supported;
> >                 hp_online = ret;
> > +
> > +               ret = etm4_cpu_pm_register();
> > +               if (ret)
> > +                       goto err_arch_supported;
> >         }
> >
> >         cpus_read_unlock();
> > @@ -1182,6 +1504,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >
> >  err_arch_supported:
> >         if (--etm4_count == 0) {
> > +               etm4_cpu_pm_unregister();
> > +
> >                 cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING);
> >                 if (hp_online)
> >                         cpuhp_remove_state_nocalls(hp_online);
> > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> > index 4523f10ddd0f..c31634c64f87 100644
> > --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> > +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> > @@ -175,6 +175,7 @@
> >                                          ETM_MODE_EXCL_USER)
> >
> >  #define TRCSTATR_IDLE_BIT              0
> > +#define TRCSTATR_PMSTABLE_BIT          1
> >  #define ETM_DEFAULT_ADDR_COMP          0
> >
> >  /* PowerDown Control Register bits */
> > @@ -281,6 +282,63 @@ struct etmv4_config {
> >         u32                             ext_inp;
> >  };
> >
> > +/**
> > + * struct etm4_save_state - state to be preserved when ETM is without power
> > + */
> > +struct etmv4_save_state {
> > +       u32     trcprgctlr;
> > +       u32     trcprocselr;
> > +       u32     trcconfigr;
> > +       u32     trcauxctlr;
> > +       u32     trceventctl0r;
> > +       u32     trceventctl1r;
> > +       u32     trcstallctlr;
> > +       u32     trctsctlr;
> > +       u32     trcsyncpr;
> > +       u32     trcccctlr;
> > +       u32     trcbbctlr;
> > +       u32     trctraceidr;
> > +       u32     trcqctlr;
> > +
> > +       u32     trcvictlr;
> > +       u32     trcviiectlr;
> > +       u32     trcvissctlr;
> > +       u32     trcvipcssctlr;
> > +       u32     trcvdctlr;
> > +       u32     trcvdsacctlr;
> > +       u32     trcvdarcctlr;
> > +
> > +       u32     trcseqevr[ETM_MAX_SEQ_STATES];
> > +       u32     trcseqrstevr;
> > +       u32     trcseqstr;
> > +       u32     trcextinselr;
> > +       u32     trccntrldvr[ETMv4_MAX_CNTR];
> > +       u32     trccntctlr[ETMv4_MAX_CNTR];
> > +       u32     trccntvr[ETMv4_MAX_CNTR];
> > +
> > +       u32     trcrsctlr[ETM_MAX_RES_SEL * 2];
> > +
> > +       u32     trcssccr[ETM_MAX_SS_CMP];
> > +       u32     trcsscsr[ETM_MAX_SS_CMP];
> > +       u32     trcsspcicr[ETM_MAX_SS_CMP];
> > +
> > +       u64     trcacvr[ETM_MAX_SINGLE_ADDR_CMP];
> > +       u64     trcacatr[ETM_MAX_SINGLE_ADDR_CMP];
> > +       u64     trccidcvr[ETMv4_MAX_CTXID_CMP];
> > +       u32     trcvmidcvr[ETM_MAX_VMID_CMP];
> > +       u32     trccidcctlr0;
> > +       u32     trccidcctlr1;
> > +       u32     trcvmidcctlr0;
> > +       u32     trcvmidcctlr1;
> > +
> > +       u32     trcclaimset;
> > +
> > +       u32     cntr_val[ETMv4_MAX_CNTR];
> > +       u32     seq_state;
> > +       u32     vinst_ctrl;
> > +       u32     ss_status[ETM_MAX_SS_CMP];
> > +};
> > +
> >  /**
> >   * struct etm4_drvdata - specifics associated to an ETM component
> >   * @base:       Memory mapped base address for this component.
> > @@ -336,6 +394,7 @@ struct etmv4_config {
> >   * @atbtrig:   If the implementation can support ATB triggers
> >   * @lpoverride:        If the implementation can support low-power state over.
> >   * @config:    structure holding configuration parameters.
> > + * @save_state:        State to be preserved across power loss
> >   */
> >  struct etmv4_drvdata {
> >         void __iomem                    *base;
> > @@ -367,6 +426,7 @@ struct etmv4_drvdata {
> >         u8                              q_support;
> >         bool                            sticky_enable;
> >         bool                            boot_enable;
> > +       bool                            pm_save_enable;
> >         bool                            os_unlock;
> >         bool                            instrp0;
> >         bool                            trcbb;
> > @@ -381,6 +441,8 @@ struct etmv4_drvdata {
> >         bool                            atbtrig;
> >         bool                            lpoverride;
> >         struct etmv4_config             config;
> > +       struct etmv4_save_state         save_state;
> > +       bool                            state_needs_restore;
> >  };
> >
> >  /* Address comparator access types */
> > diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> > index 86d1fc2c1bd4..aba71a5a025f 100644
> > --- a/drivers/hwtracing/coresight/coresight.c
> > +++ b/drivers/hwtracing/coresight/coresight.c
> > @@ -140,7 +140,7 @@ static inline bool coresight_is_claimed_self_hosted(void __iomem *base)
> >         return coresight_read_claim_tags(base) == CORESIGHT_CLAIM_SELF_HOSTED;
> >  }
> >
> > -static inline bool coresight_is_claimed_any(void __iomem *base)
> > +bool coresight_is_claimed_any(void __iomem *base)
> >  {
> >         return coresight_read_claim_tags(base) != 0;
> >  }
> > diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> > index a2b68823717b..c3a875dffe65 100644
> > --- a/include/linux/coresight.h
> > +++ b/include/linux/coresight.h
> > @@ -285,6 +285,9 @@ extern void coresight_disclaim_device(void __iomem *base);
> >  extern void coresight_disclaim_device_unlocked(void __iomem *base);
> >  extern char *coresight_alloc_device_name(struct coresight_dev_list *devs,
> >                                          struct device *dev);
> > +
> > +extern bool coresight_is_claimed_any(void __iomem *base);
> > +
> >  #else
> >  static inline struct coresight_device *
> >  coresight_register(struct coresight_desc *desc) { return NULL; }
> > @@ -307,6 +310,11 @@ static inline int coresight_claim_device(void __iomem *base)
> >  static inline void coresight_disclaim_device(void __iomem *base) {}
> >  static inline void coresight_disclaim_device_unlocked(void __iomem *base) {}
> >
> > +static inline bool coresight_is_claimed_any(void __iomem *base)
> > +{
> > +       return false;
> > +}
> > +
> >  #endif
> >
> >  extern int coresight_get_cpu(struct device *dev);
> > --
> > 2.21.0
> >
> 
> 
> -- 
> Mike Leach
> Principal Engineer, ARM Ltd.
> Manchester Design Centre. UK

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v3 5/6] coresight: etm4x: save/restore state across CPU low power states

[Andrew Murray]

On Mon, Jul 15, 2019 at 05:04:30PM -0600, Mathieu Poirier wrote:
> Hi Andrew,
> 
> Please see comments below.
> 
> On Thu, Jul 11, 2019 at 05:01:13PM +0100, Andrew Murray wrote:
> > Some hardware will ignore bit TRCPDCR.PU which is used to signal
> > to hardware that power should not be removed from the trace unit.
> > Let's mitigate against this by conditionally saving and restoring
> > the trace unit state when the CPU enters low power states.
> > 
> > This patchset introduces a firmware property named
> > 'arm,coresight-needs-save-restore' - when this is present the
> > hardware state will be conditionally saved and restored.
> > 
> > A module parameter 'pm_save_enable' is also introduced which can
> > be configured to override the firmware property. This can be set
> > to never allow save/restore, to conditionally allow it, or to
> > do as the firmware indicates (default).
> > 
> > The hardware state is only ever saved and restored when the claim
> > tags indicate that coresight is in use.
> > 
> > Signed-off-by: Andrew Murray <andrew.murray@arm.com>
> > ---
> >  drivers/hwtracing/coresight/coresight-etm4x.c | 324 ++++++++++++++++++
> >  drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
> >  drivers/hwtracing/coresight/coresight.c       |   2 +-
> >  include/linux/coresight.h                     |   8 +
> >  4 files changed, 395 insertions(+), 1 deletion(-)
> > 
> > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> > index d993359e2117..b0bd8158bf13 100644
> > --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> > +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> > @@ -18,6 +18,7 @@
> >  #include <linux/stat.h>
> >  #include <linux/clk.h>
> >  #include <linux/cpu.h>
> > +#include <linux/cpu_pm.h>
> >  #include <linux/coresight.h>
> >  #include <linux/coresight-pmu.h>
> >  #include <linux/pm_wakeup.h>
> > @@ -26,6 +27,7 @@
> >  #include <linux/uaccess.h>
> >  #include <linux/perf_event.h>
> >  #include <linux/pm_runtime.h>
> > +#include <linux/property.h>
> >  #include <asm/sections.h>
> >  #include <asm/local.h>
> >  #include <asm/virt.h>
> > @@ -37,6 +39,15 @@ static int boot_enable;
> >  module_param(boot_enable, int, 0444);
> >  MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
> >  
> > +#define PARAM_PM_SAVE_NEVER	0 /* never save/restore state */
> > +#define PARAM_PM_SAVE_ALWAYS	1 /* always save/restore state */
> > +#define PARAM_PM_SAVE_FIRMWARE	2 /* only save/restore if firmware flag set */
> > +
> > +static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
> > +module_param(pm_save_enable, int, 0644);
> > +MODULE_PARM_DESC(pm_save_enable,
> > +	"Save/restore state on power down: 0 = never, 1 = always, 2 = firmware");
> > +
> >  /* The number of ETMv4 currently registered */
> >  static int etm4_count;
> >  static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
> > @@ -54,6 +65,14 @@ static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
> >  	isb();
> >  }
> >  
> > +static void etm4_os_lock(struct etmv4_drvdata *drvdata)
> > +{
> > +	/* Writing 0x1 to TRCOSLAR locks the trace registers */
> > +	writel_relaxed(0x1, drvdata->base + TRCOSLAR);
> > +	drvdata->os_unlock = false;
> > +	isb();
> > +}
> > +
> >  static bool etm4_arch_supported(u8 arch)
> >  {
> >  	/* Mask out the minor version number */
> > @@ -1085,6 +1104,303 @@ static void etm4_init_trace_id(struct etmv4_drvdata *drvdata)
> >  	drvdata->trcid = coresight_get_trace_id(drvdata->cpu);
> >  }
> >  
> > +#ifdef CONFIG_CPU_PM
> > +static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> > +{
> > +	int i, ret = 0;
> > +	u32 control;
> > +	struct etmv4_save_state *state;
> > +	struct device *etm_dev = &drvdata->csdev->dev;
> > +
> > +	/*
> > +	 * As recommended by 3.4.1 ("The procedure when powering down the PE")
> > +	 * of ARM IHI 0064D
> > +	 */
> > +	dsb(sy);
> > +	isb();
> > +
> > +	CS_UNLOCK(drvdata->base);
> > +
> > +	/* Lock the OS lock to disable trace and external debugger access */
> > +	etm4_os_lock(drvdata);
> > +
> > +	/* wait for TRCSTATR.PMSTABLE to go up */
> > +	if (coresight_timeout(drvdata->base, TRCSTATR,
> > +					TRCSTATR_PMSTABLE_BIT, 1)) {
> > +		dev_err(etm_dev,
> > +			"timeout while waiting for PM Stable Status\n");
> > +		etm4_os_unlock(drvdata);
> > +		ret = -EBUSY;
> > +		goto out;
> > +	}
> > +
> > +	state = &drvdata->save_state;
> > +
> > +	state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
> > +	state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> > +	state->trcconfigr = readl(drvdata->base + TRCCONFIGR);
> > +	state->trcauxctlr = readl(drvdata->base + TRCAUXCTLR);
> > +	state->trceventctl0r = readl(drvdata->base + TRCEVENTCTL0R);
> > +	state->trceventctl1r = readl(drvdata->base + TRCEVENTCTL1R);
> > +	state->trcstallctlr = readl(drvdata->base + TRCSTALLCTLR);
> > +	state->trctsctlr = readl(drvdata->base + TRCTSCTLR);
> > +	state->trcsyncpr = readl(drvdata->base + TRCSYNCPR);
> > +	state->trcccctlr = readl(drvdata->base + TRCCCCTLR);
> > +	state->trcbbctlr = readl(drvdata->base + TRCBBCTLR);
> > +	state->trctraceidr = readl(drvdata->base + TRCTRACEIDR);
> > +	state->trcqctlr = readl(drvdata->base + TRCQCTLR);
> > +
> > +	state->trcvictlr = readl(drvdata->base + TRCVICTLR);
> > +	state->trcviiectlr = readl(drvdata->base + TRCVIIECTLR);
> > +	state->trcvissctlr = readl(drvdata->base + TRCVISSCTLR);
> > +	state->trcvipcssctlr = readl(drvdata->base + TRCVIPCSSCTLR);
> > +	state->trcvdctlr = readl(drvdata->base + TRCVDCTLR);
> > +	state->trcvdsacctlr = readl(drvdata->base + TRCVDSACCTLR);
> > +	state->trcvdarcctlr = readl(drvdata->base + TRCVDARCCTLR);
> > +
> > +	for (i = 0; i < drvdata->nrseqstate; i++)
> > +		state->trcseqevr[i] = readl(drvdata->base + TRCSEQEVRn(i));
> > +
> > +	state->trcseqrstevr = readl(drvdata->base + TRCSEQRSTEVR);
> > +	state->trcseqstr = readl(drvdata->base + TRCSEQSTR);
> > +	state->trcextinselr = readl(drvdata->base + TRCEXTINSELR);
> > +
> > +	for (i = 0; i < drvdata->nr_cntr; i++) {
> > +		state->trccntrldvr[i] = readl(drvdata->base + TRCCNTRLDVRn(i));
> > +		state->trccntctlr[i] = readl(drvdata->base + TRCCNTCTLRn(i));
> > +		state->trccntvr[i] = readl(drvdata->base + TRCCNTVRn(i));
> > +	}
> > +
> > +	for (i = 0; i < drvdata->nr_resource * 2; i++)
> > +		state->trcrsctlr[i] = readl(drvdata->base + TRCRSCTLRn(i));
> > +
> > +	for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> > +		state->trcssccr[i] = readl(drvdata->base + TRCSSCCRn(i));
> > +		state->trcsscsr[i] = readl(drvdata->base + TRCSSCSRn(i));
> > +		state->trcsspcicr[i] = readl(drvdata->base + TRCSSPCICRn(i));
> > +	}
> > +
> > +	for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> > +		state->trcacvr[i] = readl(drvdata->base + TRCACVRn(i));
> > +		state->trcacatr[i] = readl(drvdata->base + TRCACATRn(i));
> > +	}
> > +
> > +	/*
> > +	 * Data trace stream is architecturally prohibited for A profile cores
> > +	 * so we don't save (or later restore) trcdvcvr and trcdvcmr - As per
> > +	 * section 1.3.4 ("Possible functional configurations of an ETMv4 trace
> > +	 * unit") of ARM IHI 0064D.
> > +	 */
> > +
> > +	for (i = 0; i < drvdata->numcidc; i++)
> > +		state->trccidcvr[i] = readl(drvdata->base + TRCCIDCVRn(i));
> > +
> > +	for (i = 0; i < drvdata->numvmidc; i++)
> > +		state->trcvmidcvr[i] = readl(drvdata->base + TRCVMIDCVRn(i));
> > +
> > +	state->trccidcctlr0 = readl(drvdata->base + TRCCIDCCTLR0);
> > +	state->trccidcctlr1 = readl(drvdata->base + TRCCIDCCTLR1);
> > +
> > +	state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR0);
> > +	state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR1);
> > +
> > +	state->trcclaimset = readl(drvdata->base + TRCCLAIMCLR);
> > +
> > +	/* wait for TRCSTATR.IDLE to go up */
> > +	if (coresight_timeout(drvdata->base, TRCSTATR, TRCSTATR_IDLE_BIT, 1)) {
> > +		dev_err(etm_dev,
> > +			"timeout while waiting for Idle Trace Status\n");
> > +		etm4_os_unlock(drvdata);
> > +		ret = -EBUSY;
> > +		goto out;
> > +	}
> > +
> > +	drvdata->state_needs_restore = true;
> > +
> > +	/*
> > +	 * Power can be removed from the trace unit now. We do this to
> > +	 * potentially save power on systems that respect the TRCPDCR_PU
> > +	 * despite requesting software to save/restore state.
> > +	 */
> > +	control = readl_relaxed(drvdata->base + TRCPDCR);
> > +	control &= ~TRCPDCR_PU;
> > +	writel_relaxed(control, drvdata->base + TRCPDCR);
> > +
> > +out:
> > +	CS_LOCK(drvdata->base);
> > +	return ret;
> > +}
> > +
> > +static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
> > +{
> > +	int i;
> > +	struct etmv4_save_state *state;
> > +
> > +	state = &drvdata->save_state;
> > +
> > +	CS_UNLOCK(drvdata->base);
> > +
> > +	writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> > +
> > +	writel_relaxed(state->trcprgctlr, drvdata->base + TRCPRGCTLR);
> > +	writel_relaxed(state->trcprocselr, drvdata->base + TRCPROCSELR);
> > +	writel_relaxed(state->trcconfigr, drvdata->base + TRCCONFIGR);
> > +	writel_relaxed(state->trcauxctlr, drvdata->base + TRCAUXCTLR);
> > +	writel_relaxed(state->trceventctl0r, drvdata->base + TRCEVENTCTL0R);
> > +	writel_relaxed(state->trceventctl1r, drvdata->base + TRCEVENTCTL1R);
> > +	writel_relaxed(state->trcstallctlr, drvdata->base + TRCSTALLCTLR);
> > +	writel_relaxed(state->trctsctlr, drvdata->base + TRCTSCTLR);
> > +	writel_relaxed(state->trcsyncpr, drvdata->base + TRCSYNCPR);
> > +	writel_relaxed(state->trcccctlr, drvdata->base + TRCCCCTLR);
> > +	writel_relaxed(state->trcbbctlr, drvdata->base + TRCBBCTLR);
> > +	writel_relaxed(state->trctraceidr, drvdata->base + TRCTRACEIDR);
> > +	writel_relaxed(state->trcqctlr, drvdata->base + TRCQCTLR);
> > +
> > +	writel_relaxed(state->trcvictlr, drvdata->base + TRCVICTLR);
> > +	writel_relaxed(state->trcviiectlr, drvdata->base + TRCVIIECTLR);
> > +	writel_relaxed(state->trcvissctlr, drvdata->base + TRCVISSCTLR);
> > +	writel_relaxed(state->trcvipcssctlr, drvdata->base + TRCVIPCSSCTLR);
> > +	writel_relaxed(state->trcvdctlr, drvdata->base + TRCVDCTLR);
> > +	writel_relaxed(state->trcvdsacctlr, drvdata->base + TRCVDSACCTLR);
> > +	writel_relaxed(state->trcvdarcctlr, drvdata->base + TRCVDARCCTLR);
> > +
> > +	for (i = 0; i < drvdata->nrseqstate; i++)
> > +		writel_relaxed(state->trcseqevr[i],
> > +					drvdata->base + TRCSEQEVRn(i));
> > +
> > +	writel_relaxed(state->trcseqrstevr, drvdata->base + TRCSEQRSTEVR);
> > +	writel_relaxed(state->trcseqstr, drvdata->base + TRCSEQSTR);
> > +	writel_relaxed(state->trcextinselr, drvdata->base + TRCEXTINSELR);
> > +
> > +	for (i = 0; i < drvdata->nr_cntr; i++) {
> > +		writel_relaxed(state->trccntrldvr[i],
> > +					drvdata->base + TRCCNTRLDVRn(i));
> > +		writel_relaxed(state->trccntctlr[i],
> > +					drvdata->base + TRCCNTCTLRn(i));
> > +		writel_relaxed(state->trccntvr[i],
> > +					drvdata->base + TRCCNTVRn(i));
> > +	}
> > +
> > +	for (i = 0; i < drvdata->nr_resource * 2; i++)
> > +		writel_relaxed(state->trcrsctlr[i],
> > +					drvdata->base + TRCRSCTLRn(i));
> > +
> > +	for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> > +		writel_relaxed(state->trcssccr[i],
> > +					drvdata->base + TRCSSCCRn(i));
> > +		writel_relaxed(state->trcsscsr[i],
> > +					drvdata->base + TRCSSCSRn(i));
> > +		writel_relaxed(state->trcsspcicr[i],
> > +					drvdata->base + TRCSSPCICRn(i));
> > +	}
> > +
> > +	for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> > +		writel_relaxed(state->trcacvr[i],
> > +					drvdata->base + TRCACVRn(i));
> > +		writel_relaxed(state->trcacatr[i],
> > +					drvdata->base + TRCACATRn(i));
> > +	}
> > +
> > +	for (i = 0; i < drvdata->numcidc; i++)
> > +		writel_relaxed(state->trccidcvr[i],
> > +					drvdata->base + TRCCIDCVRn(i));
> > +
> > +	for (i = 0; i < drvdata->numvmidc; i++)
> > +		writel_relaxed(state->trcvmidcvr[i],
> > +					drvdata->base + TRCVMIDCVRn(i));
> > +
> > +	writel_relaxed(state->trccidcctlr0, drvdata->base + TRCCIDCCTLR0);
> > +	writel_relaxed(state->trccidcctlr1, drvdata->base + TRCCIDCCTLR1);
> > +
> > +	writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR0);
> > +	writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR1);
> > +
> > +	writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> > +
> > +	drvdata->state_needs_restore = false;
> > +
> > +	/*
> > +	 * Request to keep the trace unit powered and also
> > +	 * emulation of powerdown
> > +	 */
> > +	writel_relaxed(readl_relaxed(drvdata->base + TRCPDCR) | TRCPDCR_PU,
> > +		       drvdata->base + TRCPDCR);
> > +
> > +	/*
> > +	 * As recommended by section 4.3.7 ("Synchronization when using the
> > +	 * memory-mapped interface") of ARM IHI 0064D
> > +	 */
> > +	dsb(sy);
> > +	isb();
> > +
> > +	/* Unlock the OS lock to re-enable trace and external debug access */
> > +	etm4_os_unlock(drvdata);
> > +	CS_LOCK(drvdata->base);
> > +}
> > +
> > +static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd,
> > +			      void *v)
> > +{
> > +	struct etmv4_drvdata *drvdata;
> > +	unsigned int cpu = smp_processor_id();
> > +
> > +	if (!etmdrvdata[cpu])
> > +		return 0;
> > +
> > +	drvdata = etmdrvdata[cpu];
> > +
> > +	if (pm_save_enable == PARAM_PM_SAVE_NEVER ||
> > +	    (pm_save_enable == PARAM_PM_SAVE_FIRMWARE &&
> > +	     !drvdata->pm_save_enable))
> > +		return NOTIFY_OK;
> > +
> > +	if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
> > +		return NOTIFY_BAD;
> > +
> > +	switch (cmd) {
> > +	case CPU_PM_ENTER:
> > +		/* save the state if coresight is in use */
> > +		if (coresight_is_claimed_any(drvdata->base))
> > +			if (etm4_cpu_save(drvdata))
> > +				return NOTIFY_BAD;
> > +		break;
> > +	case CPU_PM_EXIT:
> > +	case CPU_PM_ENTER_FAILED:
> > +		/* trcclaimset is set when there is state to restore */
> > +		if (drvdata->state_needs_restore)
> > +			etm4_cpu_restore(drvdata);
> > +		break;
> > +	default:
> > +		return NOTIFY_DONE;
> > +	}
> > +
> > +	return NOTIFY_OK;
> > +}
> > +
> > +static struct notifier_block etm4_cpu_pm_nb = {
> > +	.notifier_call = etm4_cpu_pm_notify,
> > +};
> > +
> > +static int etm4_cpu_pm_register(void)
> > +{
> > +	return cpu_pm_register_notifier(&etm4_cpu_pm_nb);
> > +}
> > +
> > +static void etm4_cpu_pm_unregister(void)
> > +{
> > +	cpu_pm_unregister_notifier(&etm4_cpu_pm_nb);
> > +}
> > +#else
> > +static int etm4_cpu_pm_register(void) { return 0; }
> > +static void etm4_cpu_pm_unregister(void) { }
> > +#endif
> > +
> > +static inline bool etm4_needs_save_restore(struct device *dev)
> > +{
> > +	return fwnode_property_present(dev->fwnode,
> > +				       "arm,coresight-needs-save-restore");
> > +}
> > +
> >  static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >  {
> >  	int ret;
> > @@ -1101,6 +1417,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >  
> >  	dev_set_drvdata(dev, drvdata);
> >  
> > +	drvdata->pm_save_enable = etm4_needs_save_restore(dev);
> > +
> >  	/* Validity for the resource is already checked by the AMBA core */
> >  	base = devm_ioremap_resource(dev, res);
> >  	if (IS_ERR(base))
> > @@ -1132,6 +1450,10 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >  		if (ret < 0)
> >  			goto err_arch_supported;
> >  		hp_online = ret;
> > +
> > +		ret = etm4_cpu_pm_register();
> > +		if (ret)
> > +			goto err_arch_supported;
> >  	}
> >  
> >  	cpus_read_unlock();
> > @@ -1182,6 +1504,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> >  
> >  err_arch_supported:
> >  	if (--etm4_count == 0) {
> > +		etm4_cpu_pm_unregister();
> > +
> >  		cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING);
> >  		if (hp_online)
> >  			cpuhp_remove_state_nocalls(hp_online);
> > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> > index 4523f10ddd0f..c31634c64f87 100644
> > --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> > +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> > @@ -175,6 +175,7 @@
> >  					 ETM_MODE_EXCL_USER)
> >  
> >  #define TRCSTATR_IDLE_BIT		0
> > +#define TRCSTATR_PMSTABLE_BIT		1
> >  #define ETM_DEFAULT_ADDR_COMP		0
> >  
> >  /* PowerDown Control Register bits */
> > @@ -281,6 +282,63 @@ struct etmv4_config {
> >  	u32				ext_inp;
> >  };
> >  
> > +/**
> > + * struct etm4_save_state - state to be preserved when ETM is without power
> > + */
> > +struct etmv4_save_state {
> > +	u32	trcprgctlr;
> > +	u32	trcprocselr;
> > +	u32	trcconfigr;
> > +	u32	trcauxctlr;
> > +	u32	trceventctl0r;
> > +	u32	trceventctl1r;
> > +	u32	trcstallctlr;
> > +	u32	trctsctlr;
> > +	u32	trcsyncpr;
> > +	u32	trcccctlr;
> > +	u32	trcbbctlr;
> > +	u32	trctraceidr;
> > +	u32	trcqctlr;
> > +
> > +	u32	trcvictlr;
> > +	u32	trcviiectlr;
> > +	u32	trcvissctlr;
> > +	u32	trcvipcssctlr;
> > +	u32	trcvdctlr;
> > +	u32	trcvdsacctlr;
> > +	u32	trcvdarcctlr;
> > +
> > +	u32	trcseqevr[ETM_MAX_SEQ_STATES];
> > +	u32	trcseqrstevr;
> > +	u32	trcseqstr;
> > +	u32	trcextinselr;
> > +	u32	trccntrldvr[ETMv4_MAX_CNTR];
> > +	u32	trccntctlr[ETMv4_MAX_CNTR];
> > +	u32	trccntvr[ETMv4_MAX_CNTR];
> > +
> > +	u32	trcrsctlr[ETM_MAX_RES_SEL * 2];
> > +
> > +	u32	trcssccr[ETM_MAX_SS_CMP];
> > +	u32	trcsscsr[ETM_MAX_SS_CMP];
> > +	u32	trcsspcicr[ETM_MAX_SS_CMP];
> > +
> > +	u64	trcacvr[ETM_MAX_SINGLE_ADDR_CMP];
> > +	u64	trcacatr[ETM_MAX_SINGLE_ADDR_CMP];
> > +	u64	trccidcvr[ETMv4_MAX_CTXID_CMP];
> > +	u32	trcvmidcvr[ETM_MAX_VMID_CMP];
> > +	u32	trccidcctlr0;
> > +	u32	trccidcctlr1;
> > +	u32	trcvmidcctlr0;
> > +	u32	trcvmidcctlr1;
> > +
> > +	u32	trcclaimset;
> > +
> > +	u32	cntr_val[ETMv4_MAX_CNTR];
> > +	u32	seq_state;
> > +	u32	vinst_ctrl;
> > +	u32	ss_status[ETM_MAX_SS_CMP];
> > +};
> > +
> >  /**
> >   * struct etm4_drvdata - specifics associated to an ETM component
> >   * @base:       Memory mapped base address for this component.
> > @@ -336,6 +394,7 @@ struct etmv4_config {
> >   * @atbtrig:	If the implementation can support ATB triggers
> >   * @lpoverride:	If the implementation can support low-power state over.
> >   * @config:	structure holding configuration parameters.
> > + * @save_state:	State to be preserved across power loss
> >   */
> >  struct etmv4_drvdata {
> >  	void __iomem			*base;
> > @@ -367,6 +426,7 @@ struct etmv4_drvdata {
> >  	u8				q_support;
> >  	bool				sticky_enable;
> >  	bool				boot_enable;
> > +	bool				pm_save_enable;
> 
> Please document - otherwise we have to guess what this is for.

Ah yes I've missed these. I'll add the missing doc in the documentation
block before the struct.

> 
> >  	bool				os_unlock;
> >  	bool				instrp0;
> >  	bool				trcbb;
> > @@ -381,6 +441,8 @@ struct etmv4_drvdata {
> >  	bool				atbtrig;
> >  	bool				lpoverride;
> >  	struct etmv4_config		config;
> > +	struct etmv4_save_state		save_state;
> > +	bool				state_needs_restore;
> 
> Same as above.
> 
> Did you get a chance to test your code on something else than Juno?  I gave it a
> spin on my dragonboard and I can't get the kernel to finish booting - my board
> resets just after the ETMv4 have initialized.  I'll try to debug this futher tomorrow.

No I've only tested this on Juno.

Thanks for testing this, feel free to contact me on IRC, or share any debug,
I'll be happy to assist.

Thanks,

Andrew Murray

> 
> Thanks,
> Mathieu
> 
> >  };
> >  
> >  /* Address comparator access types */
> > diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> > index 86d1fc2c1bd4..aba71a5a025f 100644
> > --- a/drivers/hwtracing/coresight/coresight.c
> > +++ b/drivers/hwtracing/coresight/coresight.c
> > @@ -140,7 +140,7 @@ static inline bool coresight_is_claimed_self_hosted(void __iomem *base)
> >  	return coresight_read_claim_tags(base) == CORESIGHT_CLAIM_SELF_HOSTED;
> >  }
> >  
> > -static inline bool coresight_is_claimed_any(void __iomem *base)
> > +bool coresight_is_claimed_any(void __iomem *base)
> >  {
> >  	return coresight_read_claim_tags(base) != 0;
> >  }
> > diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> > index a2b68823717b..c3a875dffe65 100644
> > --- a/include/linux/coresight.h
> > +++ b/include/linux/coresight.h
> > @@ -285,6 +285,9 @@ extern void coresight_disclaim_device(void __iomem *base);
> >  extern void coresight_disclaim_device_unlocked(void __iomem *base);
> >  extern char *coresight_alloc_device_name(struct coresight_dev_list *devs,
> >  					 struct device *dev);
> > +
> > +extern bool coresight_is_claimed_any(void __iomem *base);
> > +
> >  #else
> >  static inline struct coresight_device *
> >  coresight_register(struct coresight_desc *desc) { return NULL; }
> > @@ -307,6 +310,11 @@ static inline int coresight_claim_device(void __iomem *base)
> >  static inline void coresight_disclaim_device(void __iomem *base) {}
> >  static inline void coresight_disclaim_device_unlocked(void __iomem *base) {}
> >  
> > +static inline bool coresight_is_claimed_any(void __iomem *base)
> > +{
> > +	return false;
> > +}
> > +
> >  #endif
> >  
> >  extern int coresight_get_cpu(struct device *dev);
> > -- 
> > 2.21.0
> > 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH v3 5/6] coresight: etm4x: save/restore state across CPU low power states

[Mike Leach]

Hi Andrew,

On Tue, 16 Jul 2019 at 12:50, Andrew Murray <andrew.murray@arm.com> wrote:
>
> On Mon, Jul 15, 2019 at 03:22:47PM +0100, Mike Leach wrote:
> > HI Andrew,
> >
> > On Thu, 11 Jul 2019 at 17:01, Andrew Murray <andrew.murray@arm.com> wrote:
> > >
> > > Some hardware will ignore bit TRCPDCR.PU which is used to signal
> > > to hardware that power should not be removed from the trace unit.
> > > Let's mitigate against this by conditionally saving and restoring
> > > the trace unit state when the CPU enters low power states.
> > >
> > > This patchset introduces a firmware property named
> > > 'arm,coresight-needs-save-restore' - when this is present the
> > > hardware state will be conditionally saved and restored.
> > >
> > > A module parameter 'pm_save_enable' is also introduced which can
> > > be configured to override the firmware property. This can be set
> > > to never allow save/restore, to conditionally allow it, or to
> > > do as the firmware indicates (default).
> > >
> > > The hardware state is only ever saved and restored when the claim
> > > tags indicate that coresight is in use.
> > >
> > > Signed-off-by: Andrew Murray <andrew.murray@arm.com>
> > > ---
> > >  drivers/hwtracing/coresight/coresight-etm4x.c | 324 ++++++++++++++++++
> > >  drivers/hwtracing/coresight/coresight-etm4x.h |  62 ++++
> > >  drivers/hwtracing/coresight/coresight.c       |   2 +-
> > >  include/linux/coresight.h                     |   8 +
> > >  4 files changed, 395 insertions(+), 1 deletion(-)
> > >
> > > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> > > index d993359e2117..b0bd8158bf13 100644
> > > --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> > > +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> > > @@ -18,6 +18,7 @@
> > >  #include <linux/stat.h>
> > >  #include <linux/clk.h>
> > >  #include <linux/cpu.h>
> > > +#include <linux/cpu_pm.h>
> > >  #include <linux/coresight.h>
> > >  #include <linux/coresight-pmu.h>
> > >  #include <linux/pm_wakeup.h>
> > > @@ -26,6 +27,7 @@
> > >  #include <linux/uaccess.h>
> > >  #include <linux/perf_event.h>
> > >  #include <linux/pm_runtime.h>
> > > +#include <linux/property.h>
> > >  #include <asm/sections.h>
> > >  #include <asm/local.h>
> > >  #include <asm/virt.h>
> > > @@ -37,6 +39,15 @@ static int boot_enable;
> > >  module_param(boot_enable, int, 0444);
> > >  MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
> > >
> > > +#define PARAM_PM_SAVE_NEVER    0 /* never save/restore state */
> > > +#define PARAM_PM_SAVE_ALWAYS   1 /* always save/restore state */
> > > +#define PARAM_PM_SAVE_FIRMWARE 2 /* only save/restore if firmware flag set */
> > > +
> > > +static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
> > > +module_param(pm_save_enable, int, 0644);
> > > +MODULE_PARM_DESC(pm_save_enable,
> > > +       "Save/restore state on power down: 0 = never, 1 = always, 2 = firmware");
> > > +
> > >  /* The number of ETMv4 currently registered */
> > >  static int etm4_count;
> > >  static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
> > > @@ -54,6 +65,14 @@ static void etm4_os_unlock(struct etmv4_drvdata *drvdata)
> > >         isb();
> > >  }
> > >
> > > +static void etm4_os_lock(struct etmv4_drvdata *drvdata)
> > > +{
> > > +       /* Writing 0x1 to TRCOSLAR locks the trace registers */
> > > +       writel_relaxed(0x1, drvdata->base + TRCOSLAR);
> > > +       drvdata->os_unlock = false;
> > > +       isb();
> > > +}
> > > +
> > >  static bool etm4_arch_supported(u8 arch)
> > >  {
> > >         /* Mask out the minor version number */
> > > @@ -1085,6 +1104,303 @@ static void etm4_init_trace_id(struct etmv4_drvdata *drvdata)
> > >         drvdata->trcid = coresight_get_trace_id(drvdata->cpu);
> > >  }
> > >
> > > +#ifdef CONFIG_CPU_PM
> > > +static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> > > +{
> > > +       int i, ret = 0;
> > > +       u32 control;
> > > +       struct etmv4_save_state *state;
> > > +       struct device *etm_dev = &drvdata->csdev->dev;
> > > +
> > > +       /*
> > > +        * As recommended by 3.4.1 ("The procedure when powering down the PE")
> > > +        * of ARM IHI 0064D
> > > +        */
> > > +       dsb(sy);
> > > +       isb();
> > > +
> > > +       CS_UNLOCK(drvdata->base);
> > > +
> > > +       /* Lock the OS lock to disable trace and external debugger access */
> > > +       etm4_os_lock(drvdata);
> > > +
> > > +       /* wait for TRCSTATR.PMSTABLE to go up */
> > > +       if (coresight_timeout(drvdata->base, TRCSTATR,
> > > +                                       TRCSTATR_PMSTABLE_BIT, 1)) {
> > > +               dev_err(etm_dev,
> > > +                       "timeout while waiting for PM Stable Status\n");
> > > +               etm4_os_unlock(drvdata);
> > > +               ret = -EBUSY;
> > > +               goto out;
> > > +       }
> > > +
> > > +       state = &drvdata->save_state;
> > > +
> > > +       state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
> > > +       state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> > > +       state->trcconfigr = readl(drvdata->base + TRCCONFIGR);
> > > +       state->trcauxctlr = readl(drvdata->base + TRCAUXCTLR);
> > > +       state->trceventctl0r = readl(drvdata->base + TRCEVENTCTL0R);
> > > +       state->trceventctl1r = readl(drvdata->base + TRCEVENTCTL1R);
> > > +       state->trcstallctlr = readl(drvdata->base + TRCSTALLCTLR);
> > > +       state->trctsctlr = readl(drvdata->base + TRCTSCTLR);
> > > +       state->trcsyncpr = readl(drvdata->base + TRCSYNCPR);
> > > +       state->trcccctlr = readl(drvdata->base + TRCCCCTLR);
> > > +       state->trcbbctlr = readl(drvdata->base + TRCBBCTLR);
> > > +       state->trctraceidr = readl(drvdata->base + TRCTRACEIDR);
> > > +       state->trcqctlr = readl(drvdata->base + TRCQCTLR);
> > > +
> > > +       state->trcvictlr = readl(drvdata->base + TRCVICTLR);
> > > +       state->trcviiectlr = readl(drvdata->base + TRCVIIECTLR);
> > > +       state->trcvissctlr = readl(drvdata->base + TRCVISSCTLR);
> > > +       state->trcvipcssctlr = readl(drvdata->base + TRCVIPCSSCTLR);
> > > +       state->trcvdctlr = readl(drvdata->base + TRCVDCTLR);
> > > +       state->trcvdsacctlr = readl(drvdata->base + TRCVDSACCTLR);
> > > +       state->trcvdarcctlr = readl(drvdata->base + TRCVDARCCTLR);
> > > +
> > > +       for (i = 0; i < drvdata->nrseqstate; i++)
> > > +               state->trcseqevr[i] = readl(drvdata->base + TRCSEQEVRn(i));
> > > +
> > > +       state->trcseqrstevr = readl(drvdata->base + TRCSEQRSTEVR);
> > > +       state->trcseqstr = readl(drvdata->base + TRCSEQSTR);
> > > +       state->trcextinselr = readl(drvdata->base + TRCEXTINSELR);
> > > +
> > > +       for (i = 0; i < drvdata->nr_cntr; i++) {
> > > +               state->trccntrldvr[i] = readl(drvdata->base + TRCCNTRLDVRn(i));
> > > +               state->trccntctlr[i] = readl(drvdata->base + TRCCNTCTLRn(i));
> > > +               state->trccntvr[i] = readl(drvdata->base + TRCCNTVRn(i));
> > > +       }
> > > +
> > > +       for (i = 0; i < drvdata->nr_resource * 2; i++)
> > > +               state->trcrsctlr[i] = readl(drvdata->base + TRCRSCTLRn(i));
> > > +
> > > +       for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> > > +               state->trcssccr[i] = readl(drvdata->base + TRCSSCCRn(i));
> > > +               state->trcsscsr[i] = readl(drvdata->base + TRCSSCSRn(i));
> > > +               state->trcsspcicr[i] = readl(drvdata->base + TRCSSPCICRn(i));
> > > +       }
> > > +
> > > +       for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> > > +               state->trcacvr[i] = readl(drvdata->base + TRCACVRn(i));
> > > +               state->trcacatr[i] = readl(drvdata->base + TRCACATRn(i));
> > > +       }
> > > +
> > > +       /*
> > > +        * Data trace stream is architecturally prohibited for A profile cores
> > > +        * so we don't save (or later restore) trcdvcvr and trcdvcmr - As per
> > > +        * section 1.3.4 ("Possible functional configurations of an ETMv4 trace
> > > +        * unit") of ARM IHI 0064D.
> > > +        */
> > > +
> > > +       for (i = 0; i < drvdata->numcidc; i++)
> > > +               state->trccidcvr[i] = readl(drvdata->base + TRCCIDCVRn(i));
> > > +
> > > +       for (i = 0; i < drvdata->numvmidc; i++)
> > > +               state->trcvmidcvr[i] = readl(drvdata->base + TRCVMIDCVRn(i));
> > > +
> > > +       state->trccidcctlr0 = readl(drvdata->base + TRCCIDCCTLR0);
> > > +       state->trccidcctlr1 = readl(drvdata->base + TRCCIDCCTLR1);
> > > +
> > > +       state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR0);
> > > +       state->trcvmidcctlr0 = readl(drvdata->base + TRCVMIDCCTLR1);
> > > +
> > > +       state->trcclaimset = readl(drvdata->base + TRCCLAIMCLR);
> > > +
> > > +       /* wait for TRCSTATR.IDLE to go up */
> > > +       if (coresight_timeout(drvdata->base, TRCSTATR, TRCSTATR_IDLE_BIT, 1)) {
> > > +               dev_err(etm_dev,
> > > +                       "timeout while waiting for Idle Trace Status\n");
> > > +               etm4_os_unlock(drvdata);
> > > +               ret = -EBUSY;
> > > +               goto out;
> > > +       }
> > > +
> > > +       drvdata->state_needs_restore = true;
> > > +
> > > +       /*
> > > +        * Power can be removed from the trace unit now. We do this to
> > > +        * potentially save power on systems that respect the TRCPDCR_PU
> > > +        * despite requesting software to save/restore state.
> > > +        */
> > > +       control = readl_relaxed(drvdata->base + TRCPDCR);
> > > +       control &= ~TRCPDCR_PU;
> > > +       writel_relaxed(control, drvdata->base + TRCPDCR);
> > > +
> >
> > Removing the power is fine, except you are not remembering the state
> > for restore later.
> >
> > > +out:
> > > +       CS_LOCK(drvdata->base);
> > > +       return ret;
> > > +}
> > > +
> > > +static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
> > > +{
> > > +       int i;
> > > +       struct etmv4_save_state *state;
> > > +
> > > +       state = &drvdata->save_state;
> > > +
> > > +       CS_UNLOCK(drvdata->base);
> > > +
> > > +       writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> > > +
> > > +       writel_relaxed(state->trcprgctlr, drvdata->base + TRCPRGCTLR);
> > > +       writel_relaxed(state->trcprocselr, drvdata->base + TRCPROCSELR);
> > > +       writel_relaxed(state->trcconfigr, drvdata->base + TRCCONFIGR);
> > > +       writel_relaxed(state->trcauxctlr, drvdata->base + TRCAUXCTLR);
> > > +       writel_relaxed(state->trceventctl0r, drvdata->base + TRCEVENTCTL0R);
> > > +       writel_relaxed(state->trceventctl1r, drvdata->base + TRCEVENTCTL1R);
> > > +       writel_relaxed(state->trcstallctlr, drvdata->base + TRCSTALLCTLR);
> > > +       writel_relaxed(state->trctsctlr, drvdata->base + TRCTSCTLR);
> > > +       writel_relaxed(state->trcsyncpr, drvdata->base + TRCSYNCPR);
> > > +       writel_relaxed(state->trcccctlr, drvdata->base + TRCCCCTLR);
> > > +       writel_relaxed(state->trcbbctlr, drvdata->base + TRCBBCTLR);
> > > +       writel_relaxed(state->trctraceidr, drvdata->base + TRCTRACEIDR);
> > > +       writel_relaxed(state->trcqctlr, drvdata->base + TRCQCTLR);
> > > +
> > > +       writel_relaxed(state->trcvictlr, drvdata->base + TRCVICTLR);
> > > +       writel_relaxed(state->trcviiectlr, drvdata->base + TRCVIIECTLR);
> > > +       writel_relaxed(state->trcvissctlr, drvdata->base + TRCVISSCTLR);
> > > +       writel_relaxed(state->trcvipcssctlr, drvdata->base + TRCVIPCSSCTLR);
> > > +       writel_relaxed(state->trcvdctlr, drvdata->base + TRCVDCTLR);
> > > +       writel_relaxed(state->trcvdsacctlr, drvdata->base + TRCVDSACCTLR);
> > > +       writel_relaxed(state->trcvdarcctlr, drvdata->base + TRCVDARCCTLR);
> > > +
> > > +       for (i = 0; i < drvdata->nrseqstate; i++)
> > > +               writel_relaxed(state->trcseqevr[i],
> > > +                                       drvdata->base + TRCSEQEVRn(i));
> > > +
> > > +       writel_relaxed(state->trcseqrstevr, drvdata->base + TRCSEQRSTEVR);
> > > +       writel_relaxed(state->trcseqstr, drvdata->base + TRCSEQSTR);
> > > +       writel_relaxed(state->trcextinselr, drvdata->base + TRCEXTINSELR);
> > > +
> > > +       for (i = 0; i < drvdata->nr_cntr; i++) {
> > > +               writel_relaxed(state->trccntrldvr[i],
> > > +                                       drvdata->base + TRCCNTRLDVRn(i));
> > > +               writel_relaxed(state->trccntctlr[i],
> > > +                                       drvdata->base + TRCCNTCTLRn(i));
> > > +               writel_relaxed(state->trccntvr[i],
> > > +                                       drvdata->base + TRCCNTVRn(i));
> > > +       }
> > > +
> > > +       for (i = 0; i < drvdata->nr_resource * 2; i++)
> > > +               writel_relaxed(state->trcrsctlr[i],
> > > +                                       drvdata->base + TRCRSCTLRn(i));
> > > +
> > > +       for (i = 0; i < drvdata->nr_ss_cmp; i++) {
> > > +               writel_relaxed(state->trcssccr[i],
> > > +                                       drvdata->base + TRCSSCCRn(i));
> > > +               writel_relaxed(state->trcsscsr[i],
> > > +                                       drvdata->base + TRCSSCSRn(i));
> > > +               writel_relaxed(state->trcsspcicr[i],
> > > +                                       drvdata->base + TRCSSPCICRn(i));
> > > +       }
> > > +
> > > +       for (i = 0; i < drvdata->nr_addr_cmp * 2; i++) {
> > > +               writel_relaxed(state->trcacvr[i],
> > > +                                       drvdata->base + TRCACVRn(i));
> > > +               writel_relaxed(state->trcacatr[i],
> > > +                                       drvdata->base + TRCACATRn(i));
> > > +       }
> > > +
> > > +       for (i = 0; i < drvdata->numcidc; i++)
> > > +               writel_relaxed(state->trccidcvr[i],
> > > +                                       drvdata->base + TRCCIDCVRn(i));
> > > +
> > > +       for (i = 0; i < drvdata->numvmidc; i++)
> > > +               writel_relaxed(state->trcvmidcvr[i],
> > > +                                       drvdata->base + TRCVMIDCVRn(i));
> > > +
> > > +       writel_relaxed(state->trccidcctlr0, drvdata->base + TRCCIDCCTLR0);
> > > +       writel_relaxed(state->trccidcctlr1, drvdata->base + TRCCIDCCTLR1);
> > > +
> > > +       writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR0);
> > > +       writel_relaxed(state->trcvmidcctlr0, drvdata->base + TRCVMIDCCTLR1);
> > > +
> > > +       writel_relaxed(state->trcclaimset, drvdata->base + TRCCLAIMSET);
> > > +
> > > +       drvdata->state_needs_restore = false;
> > > +
> > > +       /*
> > > +        * Request to keep the trace unit powered and also
> > > +        * emulation of powerdown
> > > +        */
> > > +       writel_relaxed(readl_relaxed(drvdata->base + TRCPDCR) | TRCPDCR_PU,
> > > +                      drvdata->base + TRCPDCR);
> > > +
> >
> > This sets the PU bit irrespective of the value on power down. The bit
> > should be restored to the state on power down not unconditionally set.
>
> My rationale here was to save/restore only the registers listed in 3.4.1
> of the spec (ARM IHI 0064D, "The procedure when powering down the PE") -
> though the TRCPDCR isn't listed there.
>
> I originally added the toggle of the PU bit to keep the state inline with
> that set by etm4_[enable,disable]_hw. I.e. without save/restore, when
> self-hosted these functions will ensure the PU bit is always set anyway.
> Thus making the change you suggest has only an impact on externally-hosted
> debug when the debugger doesn't set the PU bit.
>
> I'm happy to make this change, however is there any reason why an external
> debugger wouldn't set this bit? What harm does it do by always setting it
> here?
>

No harm for external debuggers - assuming it is always working
correctly. It is not just for external debuggers though.
We may have a platform that correctly follows the PU bit, but actually
only want to power the ETM when in use so use the on-board power
management.

If you alter the state of things in one place - especially where you
are claiming to save and restore - then you are potentially hiding
bugs in code that code elsewhere may trip up on.
Yes in many cases there is no harm, but there is less possibility for
hidden traps further down the line if the bit is restored rather than
set.

Regards

Mike

> Thanks,
>
> Andrew Murray
>
> >
> > Regards
> >
> > Mike
> >
> >
> > > +       /*
> > > +        * As recommended by section 4.3.7 ("Synchronization when using the
> > > +        * memory-mapped interface") of ARM IHI 0064D
> > > +        */
> > > +       dsb(sy);
> > > +       isb();
> > > +
> > > +       /* Unlock the OS lock to re-enable trace and external debug access */
> > > +       etm4_os_unlock(drvdata);
> > > +       CS_LOCK(drvdata->base);
> > > +}
> > > +
> > > +static int etm4_cpu_pm_notify(struct notifier_block *nb, unsigned long cmd,
> > > +                             void *v)
> > > +{
> > > +       struct etmv4_drvdata *drvdata;
> > > +       unsigned int cpu = smp_processor_id();
> > > +
> > > +       if (!etmdrvdata[cpu])
> > > +               return 0;
> > > +
> > > +       drvdata = etmdrvdata[cpu];
> > > +
> > > +       if (pm_save_enable == PARAM_PM_SAVE_NEVER ||
> > > +           (pm_save_enable == PARAM_PM_SAVE_FIRMWARE &&
> > > +            !drvdata->pm_save_enable))
> > > +               return NOTIFY_OK;
> > > +
> > > +       if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id()))
> > > +               return NOTIFY_BAD;
> > > +
> > > +       switch (cmd) {
> > > +       case CPU_PM_ENTER:
> > > +               /* save the state if coresight is in use */
> > > +               if (coresight_is_claimed_any(drvdata->base))
> > > +                       if (etm4_cpu_save(drvdata))
> > > +                               return NOTIFY_BAD;
> > > +               break;
> > > +       case CPU_PM_EXIT:
> > > +       case CPU_PM_ENTER_FAILED:
> > > +               /* trcclaimset is set when there is state to restore */
> > > +               if (drvdata->state_needs_restore)
> > > +                       etm4_cpu_restore(drvdata);
> > > +               break;
> > > +       default:
> > > +               return NOTIFY_DONE;
> > > +       }
> > > +
> > > +       return NOTIFY_OK;
> > > +}
> > > +
> > > +static struct notifier_block etm4_cpu_pm_nb = {
> > > +       .notifier_call = etm4_cpu_pm_notify,
> > > +};
> > > +
> > > +static int etm4_cpu_pm_register(void)
> > > +{
> > > +       return cpu_pm_register_notifier(&etm4_cpu_pm_nb);
> > > +}
> > > +
> > > +static void etm4_cpu_pm_unregister(void)
> > > +{
> > > +       cpu_pm_unregister_notifier(&etm4_cpu_pm_nb);
> > > +}
> > > +#else
> > > +static int etm4_cpu_pm_register(void) { return 0; }
> > > +static void etm4_cpu_pm_unregister(void) { }
> > > +#endif
> > > +
> > > +static inline bool etm4_needs_save_restore(struct device *dev)
> > > +{
> > > +       return fwnode_property_present(dev->fwnode,
> > > +                                      "arm,coresight-needs-save-restore");
> > > +}
> > > +
> > >  static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> > >  {
> > >         int ret;
> > > @@ -1101,6 +1417,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> > >
> > >         dev_set_drvdata(dev, drvdata);
> > >
> > > +       drvdata->pm_save_enable = etm4_needs_save_restore(dev);
> > > +
> > >         /* Validity for the resource is already checked by the AMBA core */
> > >         base = devm_ioremap_resource(dev, res);
> > >         if (IS_ERR(base))
> > > @@ -1132,6 +1450,10 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> > >                 if (ret < 0)
> > >                         goto err_arch_supported;
> > >                 hp_online = ret;
> > > +
> > > +               ret = etm4_cpu_pm_register();
> > > +               if (ret)
> > > +                       goto err_arch_supported;
> > >         }
> > >
> > >         cpus_read_unlock();
> > > @@ -1182,6 +1504,8 @@ static int etm4_probe(struct amba_device *adev, const struct amba_id *id)
> > >
> > >  err_arch_supported:
> > >         if (--etm4_count == 0) {
> > > +               etm4_cpu_pm_unregister();
> > > +
> > >                 cpuhp_remove_state_nocalls(CPUHP_AP_ARM_CORESIGHT_STARTING);
> > >                 if (hp_online)
> > >                         cpuhp_remove_state_nocalls(hp_online);
> > > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> > > index 4523f10ddd0f..c31634c64f87 100644
> > > --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> > > +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> > > @@ -175,6 +175,7 @@
> > >                                          ETM_MODE_EXCL_USER)
> > >
> > >  #define TRCSTATR_IDLE_BIT              0
> > > +#define TRCSTATR_PMSTABLE_BIT          1
> > >  #define ETM_DEFAULT_ADDR_COMP          0
> > >
> > >  /* PowerDown Control Register bits */
> > > @@ -281,6 +282,63 @@ struct etmv4_config {
> > >         u32                             ext_inp;
> > >  };
> > >
> > > +/**
> > > + * struct etm4_save_state - state to be preserved when ETM is without power
> > > + */
> > > +struct etmv4_save_state {
> > > +       u32     trcprgctlr;
> > > +       u32     trcprocselr;
> > > +       u32     trcconfigr;
> > > +       u32     trcauxctlr;
> > > +       u32     trceventctl0r;
> > > +       u32     trceventctl1r;
> > > +       u32     trcstallctlr;
> > > +       u32     trctsctlr;
> > > +       u32     trcsyncpr;
> > > +       u32     trcccctlr;
> > > +       u32     trcbbctlr;
> > > +       u32     trctraceidr;
> > > +       u32     trcqctlr;
> > > +
> > > +       u32     trcvictlr;
> > > +       u32     trcviiectlr;
> > > +       u32     trcvissctlr;
> > > +       u32     trcvipcssctlr;
> > > +       u32     trcvdctlr;
> > > +       u32     trcvdsacctlr;
> > > +       u32     trcvdarcctlr;
> > > +
> > > +       u32     trcseqevr[ETM_MAX_SEQ_STATES];
> > > +       u32     trcseqrstevr;
> > > +       u32     trcseqstr;
> > > +       u32     trcextinselr;
> > > +       u32     trccntrldvr[ETMv4_MAX_CNTR];
> > > +       u32     trccntctlr[ETMv4_MAX_CNTR];
> > > +       u32     trccntvr[ETMv4_MAX_CNTR];
> > > +
> > > +       u32     trcrsctlr[ETM_MAX_RES_SEL * 2];
> > > +
> > > +       u32     trcssccr[ETM_MAX_SS_CMP];
> > > +       u32     trcsscsr[ETM_MAX_SS_CMP];
> > > +       u32     trcsspcicr[ETM_MAX_SS_CMP];
> > > +
> > > +       u64     trcacvr[ETM_MAX_SINGLE_ADDR_CMP];
> > > +       u64     trcacatr[ETM_MAX_SINGLE_ADDR_CMP];
> > > +       u64     trccidcvr[ETMv4_MAX_CTXID_CMP];
> > > +       u32     trcvmidcvr[ETM_MAX_VMID_CMP];
> > > +       u32     trccidcctlr0;
> > > +       u32     trccidcctlr1;
> > > +       u32     trcvmidcctlr0;
> > > +       u32     trcvmidcctlr1;
> > > +
> > > +       u32     trcclaimset;
> > > +
> > > +       u32     cntr_val[ETMv4_MAX_CNTR];
> > > +       u32     seq_state;
> > > +       u32     vinst_ctrl;
> > > +       u32     ss_status[ETM_MAX_SS_CMP];
> > > +};
> > > +
> > >  /**
> > >   * struct etm4_drvdata - specifics associated to an ETM component
> > >   * @base:       Memory mapped base address for this component.
> > > @@ -336,6 +394,7 @@ struct etmv4_config {
> > >   * @atbtrig:   If the implementation can support ATB triggers
> > >   * @lpoverride:        If the implementation can support low-power state over.
> > >   * @config:    structure holding configuration parameters.
> > > + * @save_state:        State to be preserved across power loss
> > >   */
> > >  struct etmv4_drvdata {
> > >         void __iomem                    *base;
> > > @@ -367,6 +426,7 @@ struct etmv4_drvdata {
> > >         u8                              q_support;
> > >         bool                            sticky_enable;
> > >         bool                            boot_enable;
> > > +       bool                            pm_save_enable;
> > >         bool                            os_unlock;
> > >         bool                            instrp0;
> > >         bool                            trcbb;
> > > @@ -381,6 +441,8 @@ struct etmv4_drvdata {
> > >         bool                            atbtrig;
> > >         bool                            lpoverride;
> > >         struct etmv4_config             config;
> > > +       struct etmv4_save_state         save_state;
> > > +       bool                            state_needs_restore;
> > >  };
> > >
> > >  /* Address comparator access types */
> > > diff --git a/drivers/hwtracing/coresight/coresight.c b/drivers/hwtracing/coresight/coresight.c
> > > index 86d1fc2c1bd4..aba71a5a025f 100644
> > > --- a/drivers/hwtracing/coresight/coresight.c
> > > +++ b/drivers/hwtracing/coresight/coresight.c
> > > @@ -140,7 +140,7 @@ static inline bool coresight_is_claimed_self_hosted(void __iomem *base)
> > >         return coresight_read_claim_tags(base) == CORESIGHT_CLAIM_SELF_HOSTED;
> > >  }
> > >
> > > -static inline bool coresight_is_claimed_any(void __iomem *base)
> > > +bool coresight_is_claimed_any(void __iomem *base)
> > >  {
> > >         return coresight_read_claim_tags(base) != 0;
> > >  }
> > > diff --git a/include/linux/coresight.h b/include/linux/coresight.h
> > > index a2b68823717b..c3a875dffe65 100644
> > > --- a/include/linux/coresight.h
> > > +++ b/include/linux/coresight.h
> > > @@ -285,6 +285,9 @@ extern void coresight_disclaim_device(void __iomem *base);
> > >  extern void coresight_disclaim_device_unlocked(void __iomem *base);
> > >  extern char *coresight_alloc_device_name(struct coresight_dev_list *devs,
> > >                                          struct device *dev);
> > > +
> > > +extern bool coresight_is_claimed_any(void __iomem *base);
> > > +
> > >  #else
> > >  static inline struct coresight_device *
> > >  coresight_register(struct coresight_desc *desc) { return NULL; }
> > > @@ -307,6 +310,11 @@ static inline int coresight_claim_device(void __iomem *base)
> > >  static inline void coresight_disclaim_device(void __iomem *base) {}
> > >  static inline void coresight_disclaim_device_unlocked(void __iomem *base) {}
> > >
> > > +static inline bool coresight_is_claimed_any(void __iomem *base)
> > > +{
> > > +       return false;
> > > +}
> > > +
> > >  #endif
> > >
> > >  extern int coresight_get_cpu(struct device *dev);
> > > --
> > > 2.21.0
> > >
> >
> >
> > --
> > Mike Leach
> > Principal Engineer, ARM Ltd.
> > Manchester Design Centre. UK



-- 
Mike Leach
Principal Engineer, ARM Ltd.
Manchester Design Centre. UK

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] coresight: etm4x: lazily allocate memory for save_state

[Mathieu Poirier]

Hi Andrew,

Sorry for the late reply - you patch got lost under the pile.

On Fri, 12 Jul 2019 at 09:01, Andrew Murray <andrew.murray@arm.com> wrote:
>
> I had intended to lazily allocate memory for the save_state structure when
> it is first used. Following is a patch that I will squash into "[PATCH v3 5/6]
> coresight: etm4x: save/restore state across CPU low power states" on my
> next respin. I thought I'd share it here to get some feedback along with
> the rest of v3.
>
> Thanks,
>
> Andrew Murray
>
> ---
>  drivers/hwtracing/coresight/coresight-etm4x.c | 14 +++++++++++---
>  drivers/hwtracing/coresight/coresight-etm4x.h |  2 +-
>  2 files changed, 12 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> index b0bd8158bf13..cd02372194bc 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> @@ -1112,6 +1112,13 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>         struct etmv4_save_state *state;
>         struct device *etm_dev = &drvdata->csdev->dev;
>
> +       if (!drvdata->save_state) {
> +               drvdata->save_state = devm_kmalloc(etm_dev,
> +                               sizeof(struct etmv4_save_state), GFP_KERNEL);

GFP_KERNEL may sleep and will not work in the context where
etm4_cpu_save() is called.

Thanks,
Mathieu

> +               if (!drvdata->save_state)
> +                       return -ENOMEM;
> +       }
> +
>         /*
>          * As recommended by 3.4.1 ("The procedure when powering down the PE")
>          * of ARM IHI 0064D
> @@ -1134,7 +1141,7 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>                 goto out;
>         }
>
> -       state = &drvdata->save_state;
> +       state = drvdata->save_state;
>
>         state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
>         state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> @@ -1234,9 +1241,10 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>  static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
>  {
>         int i;
> -       struct etmv4_save_state *state;
> +       struct etmv4_save_state *state = drvdata->save_state;
>
> -       state = &drvdata->save_state;
> +       if (WARN_ON_ONCE(!state))
> +               return;
>
>         CS_UNLOCK(drvdata->base);
>
> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> index c31634c64f87..a70cafbbb8cf 100644
> --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> @@ -441,7 +441,7 @@ struct etmv4_drvdata {
>         bool                            atbtrig;
>         bool                            lpoverride;
>         struct etmv4_config             config;
> -       struct etmv4_save_state         save_state;
> +       struct etmv4_save_state         *save_state;
>         bool                            state_needs_restore;
>  };
>
> --
> 2.21.0
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] coresight: etm4x: lazily allocate memory for save_state

[Suzuki K Poulose]

On 22/07/2019 21:32, Mathieu Poirier wrote:
> Hi Andrew,
> 
> Sorry for the late reply - you patch got lost under the pile.
> 
> On Fri, 12 Jul 2019 at 09:01, Andrew Murray <andrew.murray@arm.com> wrote:
>>
>> I had intended to lazily allocate memory for the save_state structure when
>> it is first used. Following is a patch that I will squash into "[PATCH v3 5/6]
>> coresight: etm4x: save/restore state across CPU low power states" on my
>> next respin. I thought I'd share it here to get some feedback along with
>> the rest of v3.
>>
>> Thanks,
>>
>> Andrew Murray
>>
>> ---
>>   drivers/hwtracing/coresight/coresight-etm4x.c | 14 +++++++++++---
>>   drivers/hwtracing/coresight/coresight-etm4x.h |  2 +-
>>   2 files changed, 12 insertions(+), 4 deletions(-)
>>
>> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
>> index b0bd8158bf13..cd02372194bc 100644
>> --- a/drivers/hwtracing/coresight/coresight-etm4x.c
>> +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
>> @@ -1112,6 +1112,13 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>>          struct etmv4_save_state *state;
>>          struct device *etm_dev = &drvdata->csdev->dev;
>>
>> +       if (!drvdata->save_state) {
>> +               drvdata->save_state = devm_kmalloc(etm_dev,
>> +                               sizeof(struct etmv4_save_state), GFP_KERNEL);
> 
> GFP_KERNEL may sleep and will not work in the context where
> etm4_cpu_save() is called.

Thats right and it is not worth making this GFP_ATOMIC either. We could simply
decide this at probe time or when the save_restore is modified dynamically via
callbacks.

Suzuki

> 
> Thanks,
> Mathieu
> 
>> +               if (!drvdata->save_state)
>> +                       return -ENOMEM;
>> +       }
>> +
>>          /*
>>           * As recommended by 3.4.1 ("The procedure when powering down the PE")
>>           * of ARM IHI 0064D
>> @@ -1134,7 +1141,7 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>>                  goto out;
>>          }
>>
>> -       state = &drvdata->save_state;
>> +       state = drvdata->save_state;
>>
>>          state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
>>          state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
>> @@ -1234,9 +1241,10 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>>   static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
>>   {
>>          int i;
>> -       struct etmv4_save_state *state;
>> +       struct etmv4_save_state *state = drvdata->save_state;
>>
>> -       state = &drvdata->save_state;
>> +       if (WARN_ON_ONCE(!state))
>> +               return;
>>
>>          CS_UNLOCK(drvdata->base);
>>
>> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
>> index c31634c64f87..a70cafbbb8cf 100644
>> --- a/drivers/hwtracing/coresight/coresight-etm4x.h
>> +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
>> @@ -441,7 +441,7 @@ struct etmv4_drvdata {
>>          bool                            atbtrig;
>>          bool                            lpoverride;
>>          struct etmv4_config             config;
>> -       struct etmv4_save_state         save_state;
>> +       struct etmv4_save_state         *save_state;
>>          bool                            state_needs_restore;
>>   };
>>
>> --
>> 2.21.0
>>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] coresight: etm4x: lazily allocate memory for save_state

[Andrew Murray]

On Tue, Jul 23, 2019 at 10:05:37AM +0100, Suzuki K Poulose wrote:
> 
> 
> On 22/07/2019 21:32, Mathieu Poirier wrote:
> > Hi Andrew,
> > 
> > Sorry for the late reply - you patch got lost under the pile.
> > 
> > On Fri, 12 Jul 2019 at 09:01, Andrew Murray <andrew.murray@arm.com> wrote:
> > > 
> > > I had intended to lazily allocate memory for the save_state structure when
> > > it is first used. Following is a patch that I will squash into "[PATCH v3 5/6]
> > > coresight: etm4x: save/restore state across CPU low power states" on my
> > > next respin. I thought I'd share it here to get some feedback along with
> > > the rest of v3.
> > > 
> > > Thanks,
> > > 
> > > Andrew Murray
> > > 
> > > ---
> > >   drivers/hwtracing/coresight/coresight-etm4x.c | 14 +++++++++++---
> > >   drivers/hwtracing/coresight/coresight-etm4x.h |  2 +-
> > >   2 files changed, 12 insertions(+), 4 deletions(-)
> > > 
> > > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
> > > index b0bd8158bf13..cd02372194bc 100644
> > > --- a/drivers/hwtracing/coresight/coresight-etm4x.c
> > > +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
> > > @@ -1112,6 +1112,13 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> > >          struct etmv4_save_state *state;
> > >          struct device *etm_dev = &drvdata->csdev->dev;
> > > 
> > > +       if (!drvdata->save_state) {
> > > +               drvdata->save_state = devm_kmalloc(etm_dev,
> > > +                               sizeof(struct etmv4_save_state), GFP_KERNEL);
> > 
> > GFP_KERNEL may sleep and will not work in the context where
> > etm4_cpu_save() is called.
> 
> Thats right and it is not worth making this GFP_ATOMIC either. We could simply
> decide this at probe time or when the save_restore is modified dynamically via
> callbacks.

I think it is simpler to change this to GFP_ATOMIC and leave it where it is.

As pm_save_enable can change at run time, we can't rely solely on allocating
memory for this at probe time. We'd have to allocate memory in two places 1)
a module_parm_cb callback for when the pm_save_enable parameter changes at
run-time and 2) at probe time to find out the initial value of the
pm_save_enable which may be set by kernel command line.

As the module_parm_cb callback is file-static we wouldn't know which drvdata
to allocate the memory for. We could allocate it for any etmdrvdata member
that isn't NULL - but this seems to add a lot of complexity - is this worth
it to avoid a GFP_ATOMIC allocation? (If we fail the allocation we can return
NOTIFY_BAD and stop the PM event.)

Thanks,

Andrew Murray

> 
> Suzuki
> 
> > 
> > Thanks,
> > Mathieu
> > 
> > > +               if (!drvdata->save_state)
> > > +                       return -ENOMEM;
> > > +       }
> > > +
> > >          /*
> > >           * As recommended by 3.4.1 ("The procedure when powering down the PE")
> > >           * of ARM IHI 0064D
> > > @@ -1134,7 +1141,7 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> > >                  goto out;
> > >          }
> > > 
> > > -       state = &drvdata->save_state;
> > > +       state = drvdata->save_state;
> > > 
> > >          state->trcprgctlr = readl(drvdata->base + TRCPRGCTLR);
> > >          state->trcprocselr = readl(drvdata->base + TRCPROCSELR);
> > > @@ -1234,9 +1241,10 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
> > >   static void etm4_cpu_restore(struct etmv4_drvdata *drvdata)
> > >   {
> > >          int i;
> > > -       struct etmv4_save_state *state;
> > > +       struct etmv4_save_state *state = drvdata->save_state;
> > > 
> > > -       state = &drvdata->save_state;
> > > +       if (WARN_ON_ONCE(!state))
> > > +               return;
> > > 
> > >          CS_UNLOCK(drvdata->base);
> > > 
> > > diff --git a/drivers/hwtracing/coresight/coresight-etm4x.h b/drivers/hwtracing/coresight/coresight-etm4x.h
> > > index c31634c64f87..a70cafbbb8cf 100644
> > > --- a/drivers/hwtracing/coresight/coresight-etm4x.h
> > > +++ b/drivers/hwtracing/coresight/coresight-etm4x.h
> > > @@ -441,7 +441,7 @@ struct etmv4_drvdata {
> > >          bool                            atbtrig;
> > >          bool                            lpoverride;
> > >          struct etmv4_config             config;
> > > -       struct etmv4_save_state         save_state;
> > > +       struct etmv4_save_state         *save_state;
> > >          bool                            state_needs_restore;
> > >   };
> > > 
> > > --
> > > 2.21.0
> > > 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] coresight: etm4x: lazily allocate memory for save_state

[Suzuki K Poulose]

On 26/07/2019 12:24, Andrew Murray wrote:
> On Tue, Jul 23, 2019 at 10:05:37AM +0100, Suzuki K Poulose wrote:
>>
>>
>> On 22/07/2019 21:32, Mathieu Poirier wrote:
>>> Hi Andrew,
>>>
>>> Sorry for the late reply - you patch got lost under the pile.
>>>
>>> On Fri, 12 Jul 2019 at 09:01, Andrew Murray <andrew.murray@arm.com> wrote:
>>>>
>>>> I had intended to lazily allocate memory for the save_state structure when
>>>> it is first used. Following is a patch that I will squash into "[PATCH v3 5/6]
>>>> coresight: etm4x: save/restore state across CPU low power states" on my
>>>> next respin. I thought I'd share it here to get some feedback along with
>>>> the rest of v3.
>>>>
>>>> Thanks,
>>>>
>>>> Andrew Murray
>>>>
>>>> ---
>>>>    drivers/hwtracing/coresight/coresight-etm4x.c | 14 +++++++++++---
>>>>    drivers/hwtracing/coresight/coresight-etm4x.h |  2 +-
>>>>    2 files changed, 12 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/drivers/hwtracing/coresight/coresight-etm4x.c b/drivers/hwtracing/coresight/coresight-etm4x.c
>>>> index b0bd8158bf13..cd02372194bc 100644
>>>> --- a/drivers/hwtracing/coresight/coresight-etm4x.c
>>>> +++ b/drivers/hwtracing/coresight/coresight-etm4x.c
>>>> @@ -1112,6 +1112,13 @@ static int etm4_cpu_save(struct etmv4_drvdata *drvdata)
>>>>           struct etmv4_save_state *state;
>>>>           struct device *etm_dev = &drvdata->csdev->dev;
>>>>
>>>> +       if (!drvdata->save_state) {
>>>> +               drvdata->save_state = devm_kmalloc(etm_dev,
>>>> +                               sizeof(struct etmv4_save_state), GFP_KERNEL);
>>>
>>> GFP_KERNEL may sleep and will not work in the context where
>>> etm4_cpu_save() is called.
>>
>> Thats right and it is not worth making this GFP_ATOMIC either. We could simply
>> decide this at probe time or when the save_restore is modified dynamically via
>> callbacks.
> 
> I think it is simpler to change this to GFP_ATOMIC and leave it where it is.
> 
> As pm_save_enable can change at run time, we can't rely solely on allocating

Or we could make it static. You either need it on the system, irrespective of
what else happens or you don't need it at all. I agree that it helps with
testing.

> memory for this at probe time. We'd have to allocate memory in two places 1)
> a module_parm_cb callback for when the pm_save_enable parameter changes at
> run-time and 2) at probe time to find out the initial value of the
> pm_save_enable which may be set by kernel command line.
> 
> As the module_parm_cb callback is file-static we wouldn't know which drvdata
> to allocate the memory for. We could allocate it for any etmdrvdata member
> that isn't NULL - but this seems to add a lot of complexity - is this worth
> it to avoid a GFP_ATOMIC allocation? (If we fail the allocation we can return
> NOTIFY_BAD and stop the PM event.)

Ok, fair enough. If we are going for the GFP_ATOMIC allocation, please could we
release it once we have restored ? We don't want to be holding onto the ATOMIC
allocated memory forever.

Cheers
Suzuki

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel