[PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Geert Uytterhoeven]

	Hi all,

During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
clock register state is lost.  Note that as the boot loader skips most
initialization after resume, clock register state differs from the state
encountered during normal system boot, too.

Hence after s2ram, some operations may fail because module clocks are
disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":

    ravb e6800000.ethernet eth0: failed to switch device to config mode
    ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
    ravb e6800000.ethernet eth0: failed to switch device to config
    PM: Device e6800000.ethernet failed to resume: error -110

In addition, some clocks that were disabled by clk_disable_unused() may
have been re-enabled, wasting power.

This RFC is a second attempt to fix this issue by restoring clock registers
during system resume.

Note that while this fixes EtherAVB operation after resume from s2ram,
EtherAVB cannot be used as an actual wake-up source from s2ram, only
from s2idle, due to PSCI limitations.

Changes compared to v1 (more details in the individual patches):
  - Save module clock registers in suspend_noirq instead of constantly
    updating shadow registers,
  - Restore all module clocks under our control, not just the ones we ever
    changed,
  - Also restore DIV6, SDHI, and R clocks, thus covering all supported
    programmable core clocks on R-Car Gen3.

As clock register restore is only needed on R-Car Gen3 with PSCI, although
harmless on other systems, perhaps the save/restore code should be
protected by #ifdef CONFIG_ARM_PSCI_FW?

This series is against clk-next, with "clk: renesas: div6: Document fields
used for parent selection" applied on top.

This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resume,
but NFS fails with "server not responding", probably not due to a clock
issue.

Thanks for your comments!

Geert Uytterhoeven (5):
  [RFC] clk: renesas: cpg-mssr: Restore module clocks during resume
  [RFC] clk: renesas: cpg-mssr: Add support to restore core clocks
    during resume
  [RFC] clk: renesas: div6: Restore clock state during resume
  [RFC] clk: renesas: rcar-gen3: Restore SDHI clocks during resume
  [RFC] clk: renesas: rcar-gen3: Restore R clock during resume

 drivers/clk/renesas/clk-div6.c         | 38 ++++++++++++++-
 drivers/clk/renesas/clk-div6.h         |  3 +-
 drivers/clk/renesas/rcar-gen2-cpg.c    |  7 ++-
 drivers/clk/renesas/rcar-gen2-cpg.h    |  6 +--
 drivers/clk/renesas/rcar-gen3-cpg.c    | 79 +++++++++++++++++++++++++-------
 drivers/clk/renesas/rcar-gen3-cpg.h    |  3 +-
 drivers/clk/renesas/renesas-cpg-mssr.c | 84 +++++++++++++++++++++++++++++++++-
 drivers/clk/renesas/renesas-cpg-mssr.h |  3 +-
 8 files changed, 193 insertions(+), 30 deletions(-)

-- 
2.7.4

Gr{oetje,eeting}s,

						Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds

[PATCH/RFC v2 1/5] clk: renesas: cpg-mssr: Restore module clocks during resume

[Geert Uytterhoeven]

During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
clock register state is lost.  Note that as the boot loader skips most
initialization after system resume, clock register state differs from
the state encountered during normal system boot, too.

Hence after s2ram, some operations may fail because module clocks are
disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":

    ravb e6800000.ethernet eth0: failed to switch device to config mode
    ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
    ravb e6800000.ethernet eth0: failed to switch device to config
    PM: Device e6800000.ethernet failed to resume: error -110

In addition, some module clocks that were disabled by
clk_disable_unused() may have been re-enabled, wasting power.

To fix this, restore all bits of the SMSTPCR registers that represent
clocks under control of Linux.

Note that while this fixes EtherAVB operation after resume from s2ram,
EtherAVB cannot be used as an actual wake-up source from s2ram, only
from s2idle, due to PSCI limitations.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - Add Tested-by, Reviewed-by,
  - Mark cpg_mssr_resume_noirq __maybe_unused to kill warning if PM=n,
  - Move shadow register updates inside the RMW spinlock to protect
    against concurrent updates of multiple MSTP clocks in the same bank,
    just like for the actual registers,
  - Add more comments,
  - Save registers in suspend_noirq instead of constantly updating
    shadow registers
  - Let smstpcr_saved[].mask represent all clocks under our control, not
    just the ones we ever changed.  As clk_disable_unused() doesn't
    touch clocks that are not enabled, such clocks were not marked for
    restore, and thus may stay enabled after resume (resume state is
    different from boot state, as the boot loader disables some clocks
    during normal boot),
  - Drop Tested-by, Reviewed-by, as the code has changed substantially.
---
 drivers/clk/renesas/renesas-cpg-mssr.c | 69 ++++++++++++++++++++++++++++++++++
 1 file changed, 69 insertions(+)

diff --git a/drivers/clk/renesas/renesas-cpg-mssr.c b/drivers/clk/renesas/renesas-cpg-mssr.c
index 1f607c806f9b9ec3..d27738521dc9a8bb 100644
--- a/drivers/clk/renesas/renesas-cpg-mssr.c
+++ b/drivers/clk/renesas/renesas-cpg-mssr.c
@@ -106,6 +106,8 @@ static const u16 srcr[] = {
  * @num_core_clks: Number of Core Clocks in clks[]
  * @num_mod_clks: Number of Module Clocks in clks[]
  * @last_dt_core_clk: ID of the last Core Clock exported to DT
+ * @smstpcr_saved[].mask: Mask of SMSTPCR[] bits under our control
+ * @smstpcr_saved[].val: Saved values of SMSTPCR[]
  */
 struct cpg_mssr_priv {
 #ifdef CONFIG_RESET_CONTROLLER
@@ -119,6 +121,11 @@ struct cpg_mssr_priv {
 	unsigned int num_core_clks;
 	unsigned int num_mod_clks;
 	unsigned int last_dt_core_clk;
+
+	struct {
+		u32 mask;
+		u32 val;
+	} smstpcr_saved[ARRAY_SIZE(smstpcr)];
 };
 
 
@@ -382,6 +389,7 @@ static void __init cpg_mssr_register_mod_clk(const struct mssr_mod_clk *mod,
 
 	dev_dbg(dev, "Module clock %pC at %pCr Hz\n", clk, clk);
 	priv->clks[id] = clk;
+	priv->smstpcr_saved[clock->index / 32].mask |= BIT(clock->index % 32);
 	return;
 
 fail:
@@ -723,6 +731,7 @@ static int __init cpg_mssr_probe(struct platform_device *pdev)
 	if (!clks)
 		return -ENOMEM;
 
+	dev_set_drvdata(dev, priv);
 	priv->clks = clks;
 	priv->num_core_clks = info->num_total_core_clks;
 	priv->num_mod_clks = info->num_hw_mod_clks;
@@ -759,10 +768,70 @@ static int __init cpg_mssr_probe(struct platform_device *pdev)
 	return 0;
 }
 
+static int cpg_mssr_suspend_noirq(struct device *dev)
+{
+	struct cpg_mssr_priv *priv = dev_get_drvdata(dev);
+	unsigned int reg;
+
+	/* Save module registers with bits under our control */
+	for (reg = 0; reg < ARRAY_SIZE(priv->smstpcr_saved); reg++) {
+		if (priv->smstpcr_saved[reg].mask)
+			priv->smstpcr_saved[reg].val =
+				readl(priv->base + SMSTPCR(reg));
+	}
+	return 0;
+}
+
+static __maybe_unused int cpg_mssr_resume_noirq(struct device *dev)
+{
+	struct cpg_mssr_priv *priv = dev_get_drvdata(dev);
+	unsigned int reg, i;
+	u32 mask, oldval, newval;
+
+	/* Restore module clocks */
+	for (reg = 0; reg < ARRAY_SIZE(priv->smstpcr_saved); reg++) {
+		mask = priv->smstpcr_saved[reg].mask;
+		if (!mask)
+			continue;
+
+		oldval = readl(priv->base + SMSTPCR(reg));
+		newval = oldval & ~mask;
+		newval |= priv->smstpcr_saved[reg].val & mask;
+		if (newval == oldval)
+			continue;
+
+		writel(newval, priv->base + SMSTPCR(reg));
+
+		/* Wait until enabled clocks are really enabled */
+		mask &= ~priv->smstpcr_saved[reg].val;
+		if (!mask)
+			continue;
+
+		for (i = 1000; i > 0; --i) {
+			oldval = readl(priv->base + MSTPSR(reg));
+			if (!(oldval & mask))
+				break;
+			cpu_relax();
+		}
+
+		if (!i)
+			dev_warn(dev, "Failed to enable SMSTP %p[0x%x]\n",
+				 priv->base + SMSTPCR(reg), oldval & mask);
+	}
+
+	return 0;
+}
+
+static const struct dev_pm_ops cpg_mssr_pm = {
+	SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(cpg_mssr_suspend_noirq,
+				      cpg_mssr_resume_noirq)
+};
+
 static struct platform_driver cpg_mssr_driver = {
 	.driver		= {
 		.name	= "renesas-cpg-mssr",
 		.of_match_table = cpg_mssr_match,
+		.pm = &cpg_mssr_pm,
 	},
 };
 
-- 
2.7.4

[PATCH/RFC v2 2/5] clk: renesas: cpg-mssr: Add support to restore core clocks during resume

[Geert Uytterhoeven]

On R-Car Gen3 systems, PSCI system suspend powers down the SoC, possibly
losing clock configuration.  Hence add a notifier chain that can be used
by core clocks to save/restore clock state during system suspend/resume.

The implementation of the actual clock state save/restore operations is
clock-specific, and to be registered with the notifier chain in the SoC
or family-specific cpg_mssr_info.cpg_clk_register() callback.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - New.
---
 drivers/clk/renesas/rcar-gen2-cpg.c    |  7 +++----
 drivers/clk/renesas/rcar-gen2-cpg.h    |  6 +++---
 drivers/clk/renesas/rcar-gen3-cpg.c    |  3 ++-
 drivers/clk/renesas/rcar-gen3-cpg.h    |  3 ++-
 drivers/clk/renesas/renesas-cpg-mssr.c | 12 +++++++++++-
 drivers/clk/renesas/renesas-cpg-mssr.h |  3 ++-
 6 files changed, 23 insertions(+), 11 deletions(-)

diff --git a/drivers/clk/renesas/rcar-gen2-cpg.c b/drivers/clk/renesas/rcar-gen2-cpg.c
index 123b1e622179308e..feb14579a71b3bf4 100644
--- a/drivers/clk/renesas/rcar-gen2-cpg.c
+++ b/drivers/clk/renesas/rcar-gen2-cpg.c
@@ -262,10 +262,9 @@ static unsigned int cpg_pll0_div __initdata;
 static u32 cpg_mode __initdata;
 
 struct clk * __init rcar_gen2_cpg_clk_register(struct device *dev,
-					       const struct cpg_core_clk *core,
-					       const struct cpg_mssr_info *info,
-					       struct clk **clks,
-					       void __iomem *base)
+	const struct cpg_core_clk *core, const struct cpg_mssr_info *info,
+	struct clk **clks, void __iomem *base,
+	struct raw_notifier_head *notifiers)
 {
 	const struct clk_div_table *table = NULL;
 	const struct clk *parent;
diff --git a/drivers/clk/renesas/rcar-gen2-cpg.h b/drivers/clk/renesas/rcar-gen2-cpg.h
index 9eba07ff8b11e20e..020a3baad0154231 100644
--- a/drivers/clk/renesas/rcar-gen2-cpg.h
+++ b/drivers/clk/renesas/rcar-gen2-cpg.h
@@ -34,9 +34,9 @@ struct rcar_gen2_cpg_pll_config {
 };
 
 struct clk *rcar_gen2_cpg_clk_register(struct device *dev,
-				       const struct cpg_core_clk *core,
-				       const struct cpg_mssr_info *info,
-				       struct clk **clks, void __iomem *base);
+	const struct cpg_core_clk *core, const struct cpg_mssr_info *info,
+	struct clk **clks, void __iomem *base,
+	struct raw_notifier_head *notifiers);
 int rcar_gen2_cpg_init(const struct rcar_gen2_cpg_pll_config *config,
 		       unsigned int pll0_div, u32 mode);
 
diff --git a/drivers/clk/renesas/rcar-gen3-cpg.c b/drivers/clk/renesas/rcar-gen3-cpg.c
index 3dee900522b703bd..5367944c4ab8491d 100644
--- a/drivers/clk/renesas/rcar-gen3-cpg.c
+++ b/drivers/clk/renesas/rcar-gen3-cpg.c
@@ -272,7 +272,8 @@ static const struct soc_device_attribute cpg_quirks_match[] __initconst = {
 
 struct clk * __init rcar_gen3_cpg_clk_register(struct device *dev,
 	const struct cpg_core_clk *core, const struct cpg_mssr_info *info,
-	struct clk **clks, void __iomem *base)
+	struct clk **clks, void __iomem *base,
+	struct raw_notifier_head *notifiers)
 {
 	const struct clk *parent;
 	unsigned int mult = 1;
diff --git a/drivers/clk/renesas/rcar-gen3-cpg.h b/drivers/clk/renesas/rcar-gen3-cpg.h
index 073be54b5d038ae3..f4ecadaf41f226b4 100644
--- a/drivers/clk/renesas/rcar-gen3-cpg.h
+++ b/drivers/clk/renesas/rcar-gen3-cpg.h
@@ -35,7 +35,8 @@ struct rcar_gen3_cpg_pll_config {
 
 struct clk *rcar_gen3_cpg_clk_register(struct device *dev,
 	const struct cpg_core_clk *core, const struct cpg_mssr_info *info,
-	struct clk **clks, void __iomem *base);
+	struct clk **clks, void __iomem *base,
+	struct raw_notifier_head *notifiers);
 int rcar_gen3_cpg_init(const struct rcar_gen3_cpg_pll_config *config,
 		       unsigned int clk_extalr, u32 mode);
 
diff --git a/drivers/clk/renesas/renesas-cpg-mssr.c b/drivers/clk/renesas/renesas-cpg-mssr.c
index d27738521dc9a8bb..ab12b2ec61a3c041 100644
--- a/drivers/clk/renesas/renesas-cpg-mssr.c
+++ b/drivers/clk/renesas/renesas-cpg-mssr.c
@@ -106,6 +106,7 @@ static const u16 srcr[] = {
  * @num_core_clks: Number of Core Clocks in clks[]
  * @num_mod_clks: Number of Module Clocks in clks[]
  * @last_dt_core_clk: ID of the last Core Clock exported to DT
+ * @notifiers: Notifier chain to save/restore clock state for system resume
  * @smstpcr_saved[].mask: Mask of SMSTPCR[] bits under our control
  * @smstpcr_saved[].val: Saved values of SMSTPCR[]
  */
@@ -122,6 +123,7 @@ struct cpg_mssr_priv {
 	unsigned int num_mod_clks;
 	unsigned int last_dt_core_clk;
 
+	struct raw_notifier_head notifiers;
 	struct {
 		u32 mask;
 		u32 val;
@@ -311,7 +313,8 @@ static void __init cpg_mssr_register_core_clk(const struct cpg_core_clk *core,
 	default:
 		if (info->cpg_clk_register)
 			clk = info->cpg_clk_register(dev, core, info,
-						     priv->clks, priv->base);
+						     priv->clks, priv->base,
+						     &priv->notifiers);
 		else
 			dev_err(dev, "%s has unsupported core clock type %u\n",
 				core->name, core->type);
@@ -736,6 +739,7 @@ static int __init cpg_mssr_probe(struct platform_device *pdev)
 	priv->num_core_clks = info->num_total_core_clks;
 	priv->num_mod_clks = info->num_hw_mod_clks;
 	priv->last_dt_core_clk = info->last_dt_core_clk;
+	RAW_INIT_NOTIFIER_HEAD(&priv->notifiers);
 
 	for (i = 0; i < nclks; i++)
 		clks[i] = ERR_PTR(-ENOENT);
@@ -779,6 +783,9 @@ static int cpg_mssr_suspend_noirq(struct device *dev)
 			priv->smstpcr_saved[reg].val =
 				readl(priv->base + SMSTPCR(reg));
 	}
+
+	/* Save core clocks */
+	raw_notifier_call_chain(&priv->notifiers, PM_EVENT_SUSPEND, NULL);
 	return 0;
 }
 
@@ -788,6 +795,9 @@ static __maybe_unused int cpg_mssr_resume_noirq(struct device *dev)
 	unsigned int reg, i;
 	u32 mask, oldval, newval;
 
+	/* Restore core clocks */
+	raw_notifier_call_chain(&priv->notifiers, PM_EVENT_RESUME, NULL);
+
 	/* Restore module clocks */
 	for (reg = 0; reg < ARRAY_SIZE(priv->smstpcr_saved); reg++) {
 		mask = priv->smstpcr_saved[reg].mask;
diff --git a/drivers/clk/renesas/renesas-cpg-mssr.h b/drivers/clk/renesas/renesas-cpg-mssr.h
index 43d7c7f6832df0b2..c86dc91b0a0facde 100644
--- a/drivers/clk/renesas/renesas-cpg-mssr.h
+++ b/drivers/clk/renesas/renesas-cpg-mssr.h
@@ -127,7 +127,8 @@ struct cpg_mssr_info {
 	struct clk *(*cpg_clk_register)(struct device *dev,
 					const struct cpg_core_clk *core,
 					const struct cpg_mssr_info *info,
-					struct clk **clks, void __iomem *base);
+					struct clk **clks, void __iomem *base,
+					struct raw_notifier_head *notifiers);
 };
 
 extern const struct cpg_mssr_info r8a7743_cpg_mssr_info;
-- 
2.7.4

[PATCH/RFC v2 3/5] clk: renesas: div6: Restore clock state during resume

[Geert Uytterhoeven]

On R-Car Gen3 systems, PSCI system suspend powers down the SoC, losing
clock configuration.  Register an (optional) notifier to restore the
DIV6 clock state during system resume.

As DIV6 clocks can be picky w.r.t. modifying multiple register fields at
once, restore is not implemented by blindly restoring the register
value, but by using the existing cpg_div6_clock_{en,dis}able() helpers.

Note that this does not yet support DIV6 clocks with multiple parents,
which do not exist on R-Car Gen3 SoCs.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - New.
---
 drivers/clk/renesas/clk-div6.c         | 38 ++++++++++++++++++++++++++++++++--
 drivers/clk/renesas/clk-div6.h         |  3 ++-
 drivers/clk/renesas/renesas-cpg-mssr.c |  3 ++-
 3 files changed, 40 insertions(+), 4 deletions(-)

diff --git a/drivers/clk/renesas/clk-div6.c b/drivers/clk/renesas/clk-div6.c
index 3e0040c0ac87a14b..6bc9469e769d035c 100644
--- a/drivers/clk/renesas/clk-div6.c
+++ b/drivers/clk/renesas/clk-div6.c
@@ -14,8 +14,10 @@
 #include <linux/init.h>
 #include <linux/io.h>
 #include <linux/kernel.h>
+#include <linux/notifier.h>
 #include <linux/of.h>
 #include <linux/of_address.h>
+#include <linux/pm.h>
 #include <linux/slab.h>
 
 #include "clk-div6.h"
@@ -32,6 +34,7 @@
  * @src_shift: Shift to access the register bits to select the parent clock
  * @src_width: Number of register bits to select the parent clock (may be 0)
  * @parents: Array to map from valid parent clocks indices to hardware indices
+ * @nb: Notifier block to save/restore clock state for system resume
  */
 struct div6_clock {
 	struct clk_hw hw;
@@ -40,6 +43,7 @@ struct div6_clock {
 	u32 src_shift;
 	u32 src_width;
 	u8 *parents;
+	struct notifier_block nb;
 };
 
 #define to_div6_clock(_hw) container_of(_hw, struct div6_clock, hw)
@@ -176,6 +180,29 @@ static const struct clk_ops cpg_div6_clock_ops = {
 	.set_rate = cpg_div6_clock_set_rate,
 };
 
+static int cpg_div6_clock_resume(struct notifier_block *nb,
+				 unsigned long action, void *data)
+{
+	struct div6_clock *clock = container_of(nb, struct div6_clock, nb);
+
+	switch (action) {
+	case PM_EVENT_RESUME:
+		/*
+		 * TODO: This does not yet support DIV6 clocks with multiple
+		 * parents, as the parent selection bits are not restored.
+		 * Fortunately so far such DIV6 clocks are found only on
+		 * R/SH-Mobile SoCs, while the resume functionality is only
+		 * needed on R-Car Gen3.
+		 */
+		if (__clk_get_enable_count(clock->hw.clk))
+			cpg_div6_clock_enable(&clock->hw);
+		else
+			cpg_div6_clock_disable(&clock->hw);
+		return NOTIFY_OK;
+	}
+
+	return NOTIFY_DONE;
+}
 
 /**
  * cpg_div6_register - Register a DIV6 clock
@@ -183,11 +210,13 @@ static const struct clk_ops cpg_div6_clock_ops = {
  * @num_parents: Number of parent clocks of the DIV6 clock (1, 4, or 8)
  * @parent_names: Array containing the names of the parent clocks
  * @reg: Mapped register used to control the DIV6 clock
+ * @notifiers: Optional notifier chain to save/restore state for system resume
  */
 struct clk * __init cpg_div6_register(const char *name,
 				      unsigned int num_parents,
 				      const char **parent_names,
-				      void __iomem *reg)
+				      void __iomem *reg,
+				      struct raw_notifier_head *notifiers)
 {
 	unsigned int valid_parents;
 	struct clk_init_data init;
@@ -258,6 +287,11 @@ struct clk * __init cpg_div6_register(const char *name,
 	if (IS_ERR(clk))
 		goto free_parents;
 
+	if (notifiers) {
+		clock->nb.notifier_call = cpg_div6_clock_resume;
+		raw_notifier_chain_register(notifiers, &clock->nb);
+	}
+
 	return clk;
 
 free_parents:
@@ -301,7 +335,7 @@ static void __init cpg_div6_clock_init(struct device_node *np)
 	for (i = 0; i < num_parents; i++)
 		parent_names[i] = of_clk_get_parent_name(np, i);
 
-	clk = cpg_div6_register(clk_name, num_parents, parent_names, reg);
+	clk = cpg_div6_register(clk_name, num_parents, parent_names, reg, NULL);
 	if (IS_ERR(clk)) {
 		pr_err("%s: failed to register %s DIV6 clock (%ld)\n",
 		       __func__, np->name, PTR_ERR(clk));
diff --git a/drivers/clk/renesas/clk-div6.h b/drivers/clk/renesas/clk-div6.h
index 567b31d2bfa5269c..da4807299dda735b 100644
--- a/drivers/clk/renesas/clk-div6.h
+++ b/drivers/clk/renesas/clk-div6.h
@@ -2,6 +2,7 @@
 #define __RENESAS_CLK_DIV6_H__
 
 struct clk *cpg_div6_register(const char *name, unsigned int num_parents,
-			      const char **parent_names, void __iomem *reg);
+			      const char **parent_names, void __iomem *reg,
+			      struct raw_notifier_head *notifiers);
 
 #endif
diff --git a/drivers/clk/renesas/renesas-cpg-mssr.c b/drivers/clk/renesas/renesas-cpg-mssr.c
index ab12b2ec61a3c041..a766ac079d86a901 100644
--- a/drivers/clk/renesas/renesas-cpg-mssr.c
+++ b/drivers/clk/renesas/renesas-cpg-mssr.c
@@ -302,7 +302,8 @@ static void __init cpg_mssr_register_core_clk(const struct cpg_core_clk *core,
 
 		if (core->type == CLK_TYPE_DIV6P1) {
 			clk = cpg_div6_register(core->name, 1, &parent_name,
-						priv->base + core->offset);
+						priv->base + core->offset,
+						&priv->notifiers);
 		} else {
 			clk = clk_register_fixed_factor(NULL, core->name,
 							parent_name, 0,
-- 
2.7.4

[PATCH/RFC v2 4/5] clk: renesas: rcar-gen3: Restore SDHI clocks during resume

[Geert Uytterhoeven]

On R-Car Gen3 systems, PSCI system suspend powers down the SoC, losing
clock configuration.  Register a notifier to save/restore SDHI clock
registers during system suspend/resume.

This is implemented using the cpg_simple_notifier abstraction, which can
be reused for others clocks that just need to save/restore a single
register.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - New.
---
 drivers/clk/renesas/rcar-gen3-cpg.c | 63 +++++++++++++++++++++++++++++--------
 1 file changed, 50 insertions(+), 13 deletions(-)

diff --git a/drivers/clk/renesas/rcar-gen3-cpg.c b/drivers/clk/renesas/rcar-gen3-cpg.c
index 5367944c4ab8491d..1abbf1d6d0a0c889 100644
--- a/drivers/clk/renesas/rcar-gen3-cpg.c
+++ b/drivers/clk/renesas/rcar-gen3-cpg.c
@@ -19,6 +19,7 @@
 #include <linux/err.h>
 #include <linux/init.h>
 #include <linux/io.h>
+#include <linux/pm.h>
 #include <linux/slab.h>
 #include <linux/sys_soc.h>
 
@@ -29,6 +30,36 @@
 #define CPG_PLL2CR		0x002c
 #define CPG_PLL4CR		0x01f4
 
+struct cpg_simple_notifier {
+	struct notifier_block nb;
+	void __iomem *reg;
+	u32 saved;
+};
+
+static int cpg_simple_notifier_call(struct notifier_block *nb,
+				    unsigned long action, void *data)
+{
+	struct cpg_simple_notifier *csn =
+		container_of(nb, struct cpg_simple_notifier, nb);
+
+	switch (action) {
+	case PM_EVENT_SUSPEND:
+		csn->saved = readl(csn->reg);
+		return NOTIFY_OK;
+
+	case PM_EVENT_RESUME:
+		writel(csn->saved, csn->reg);
+		return NOTIFY_OK;
+	}
+	return NOTIFY_DONE;
+}
+
+static void cpg_simple_notifier_register(struct raw_notifier_head *notifiers,
+					 struct cpg_simple_notifier *csn)
+{
+	csn->nb.notifier_call = cpg_simple_notifier_call;
+	raw_notifier_chain_register(notifiers, &csn->nb);
+}
 
 /*
  * SDn Clock
@@ -55,8 +86,8 @@ struct sd_div_table {
 
 struct sd_clock {
 	struct clk_hw hw;
-	void __iomem *reg;
 	const struct sd_div_table *div_table;
+	struct cpg_simple_notifier csn;
 	unsigned int div_num;
 	unsigned int div_min;
 	unsigned int div_max;
@@ -99,7 +130,7 @@ static int cpg_sd_clock_enable(struct clk_hw *hw)
 	u32 val, sd_fc;
 	unsigned int i;
 
-	val = readl(clock->reg);
+	val = readl(clock->csn.reg);
 
 	sd_fc = val & CPG_SD_FC_MASK;
 	for (i = 0; i < clock->div_num; i++)
@@ -112,7 +143,7 @@ static int cpg_sd_clock_enable(struct clk_hw *hw)
 	val &= ~(CPG_SD_STP_MASK);
 	val |= clock->div_table[i].val & CPG_SD_STP_MASK;
 
-	writel(val, clock->reg);
+	writel(val, clock->csn.reg);
 
 	return 0;
 }
@@ -121,14 +152,14 @@ static void cpg_sd_clock_disable(struct clk_hw *hw)
 {
 	struct sd_clock *clock = to_sd_clock(hw);
 
-	writel(readl(clock->reg) | CPG_SD_STP_MASK, clock->reg);
+	writel(readl(clock->csn.reg) | CPG_SD_STP_MASK, clock->csn.reg);
 }
 
 static int cpg_sd_clock_is_enabled(struct clk_hw *hw)
 {
 	struct sd_clock *clock = to_sd_clock(hw);
 
-	return !(readl(clock->reg) & CPG_SD_STP_MASK);
+	return !(readl(clock->csn.reg) & CPG_SD_STP_MASK);
 }
 
 static unsigned long cpg_sd_clock_recalc_rate(struct clk_hw *hw,
@@ -139,7 +170,7 @@ static unsigned long cpg_sd_clock_recalc_rate(struct clk_hw *hw,
 	u32 val, sd_fc;
 	unsigned int i;
 
-	val = readl(clock->reg);
+	val = readl(clock->csn.reg);
 
 	sd_fc = val & CPG_SD_FC_MASK;
 	for (i = 0; i < clock->div_num; i++)
@@ -190,10 +221,10 @@ static int cpg_sd_clock_set_rate(struct clk_hw *hw, unsigned long rate,
 	if (i >= clock->div_num)
 		return -EINVAL;
 
-	val = readl(clock->reg);
+	val = readl(clock->csn.reg);
 	val &= ~(CPG_SD_STP_MASK | CPG_SD_FC_MASK);
 	val |= clock->div_table[i].val & (CPG_SD_STP_MASK | CPG_SD_FC_MASK);
-	writel(val, clock->reg);
+	writel(val, clock->csn.reg);
 
 	return 0;
 }
@@ -208,8 +239,8 @@ static const struct clk_ops cpg_sd_clock_ops = {
 };
 
 static struct clk * __init cpg_sd_clk_register(const struct cpg_core_clk *core,
-					       void __iomem *base,
-					       const char *parent_name)
+	void __iomem *base, const char *parent_name,
+	struct raw_notifier_head *notifiers)
 {
 	struct clk_init_data init;
 	struct sd_clock *clock;
@@ -226,7 +257,7 @@ static struct clk * __init cpg_sd_clk_register(const struct cpg_core_clk *core,
 	init.parent_names = &parent_name;
 	init.num_parents = 1;
 
-	clock->reg = base + core->offset;
+	clock->csn.reg = base + core->offset;
 	clock->hw.init = &init;
 	clock->div_table = cpg_sd_div_table;
 	clock->div_num = ARRAY_SIZE(cpg_sd_div_table);
@@ -240,8 +271,13 @@ static struct clk * __init cpg_sd_clk_register(const struct cpg_core_clk *core,
 
 	clk = clk_register(NULL, &clock->hw);
 	if (IS_ERR(clk))
-		kfree(clock);
+		goto free_clock;
+
+	cpg_simple_notifier_register(notifiers, &clock->csn);
+	return clk;
 
+free_clock:
+	kfree(clock);
 	return clk;
 }
 
@@ -337,7 +373,8 @@ struct clk * __init rcar_gen3_cpg_clk_register(struct device *dev,
 		break;
 
 	case CLK_TYPE_GEN3_SD:
-		return cpg_sd_clk_register(core, base, __clk_get_name(parent));
+		return cpg_sd_clk_register(core, base, __clk_get_name(parent),
+					   notifiers);
 
 	case CLK_TYPE_GEN3_R:
 		if (cpg_quirks & RCKCR_CKSEL) {
-- 
2.7.4

[PATCH/RFC v2 5/5] clk: renesas: rcar-gen3: Restore R clock during resume

[Geert Uytterhoeven]

On R-Car Gen3 systems, PSCI system suspend powers down the SoC, losing
clock configuration.  Register a notifier to save/restore the RCKCR
register during system suspend/resume.

Signed-off-by: Geert Uytterhoeven <geert+renesas@glider.be>
---
v2:
  - New.
---
 drivers/clk/renesas/clk-div6.c      |  6 +++---
 drivers/clk/renesas/rcar-gen3-cpg.c | 13 +++++++++++--
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/drivers/clk/renesas/clk-div6.c b/drivers/clk/renesas/clk-div6.c
index 6bc9469e769d035c..151336d2ba59e689 100644
--- a/drivers/clk/renesas/clk-div6.c
+++ b/drivers/clk/renesas/clk-div6.c
@@ -180,8 +180,8 @@ static const struct clk_ops cpg_div6_clock_ops = {
 	.set_rate = cpg_div6_clock_set_rate,
 };
 
-static int cpg_div6_clock_resume(struct notifier_block *nb,
-				 unsigned long action, void *data)
+static int cpg_div6_clock_notifier_call(struct notifier_block *nb,
+					unsigned long action, void *data)
 {
 	struct div6_clock *clock = container_of(nb, struct div6_clock, nb);
 
@@ -288,7 +288,7 @@ struct clk * __init cpg_div6_register(const char *name,
 		goto free_parents;
 
 	if (notifiers) {
-		clock->nb.notifier_call = cpg_div6_clock_resume;
+		clock->nb.notifier_call = cpg_div6_clock_notifier_call;
 		raw_notifier_chain_register(notifiers, &clock->nb);
 	}
 
diff --git a/drivers/clk/renesas/rcar-gen3-cpg.c b/drivers/clk/renesas/rcar-gen3-cpg.c
index 1abbf1d6d0a0c889..949116f4222226f5 100644
--- a/drivers/clk/renesas/rcar-gen3-cpg.c
+++ b/drivers/clk/renesas/rcar-gen3-cpg.c
@@ -378,18 +378,27 @@ struct clk * __init rcar_gen3_cpg_clk_register(struct device *dev,
 
 	case CLK_TYPE_GEN3_R:
 		if (cpg_quirks & RCKCR_CKSEL) {
+			struct cpg_simple_notifier *csn;
+
+			csn = kzalloc(sizeof(*csn), GFP_KERNEL);
+			if (!csn)
+				return ERR_PTR(-ENOMEM);
+
+			csn->reg = base + CPG_RCKCR;
+
 			/*
 			 * RINT is default.
 			 * Only if EXTALR is populated, we switch to it.
 			 */
-			value = readl(base + CPG_RCKCR) & 0x3f;
+			value = readl(csn->reg) & 0x3f;
 
 			if (clk_get_rate(clks[cpg_clk_extalr])) {
 				parent = clks[cpg_clk_extalr];
 				value |= BIT(15);
 			}
 
-			writel(value, base + CPG_RCKCR);
+			writel(value, csn->reg);
+			cpg_simple_notifier_register(notifiers, csn);
 			break;
 		}
 
-- 
2.7.4

Re: [PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Niklas Söderlund]

Hi Geert,

Thanks for your hard work!

On 2017-06-29 15:25:15 +0200, Geert Uytterhoeven wrote:
> 	Hi all,
> 
> During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
> clock register state is lost.  Note that as the boot loader skips most
> initialization after resume, clock register state differs from the state
> encountered during normal system boot, too.
> 
> Hence after s2ram, some operations may fail because module clocks are
> disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
> fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":
> 
>     ravb e6800000.ethernet eth0: failed to switch device to config mode
>     ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
>     ravb e6800000.ethernet eth0: failed to switch device to config
>     PM: Device e6800000.ethernet failed to resume: error -110
> 
> In addition, some clocks that were disabled by clk_disable_unused() may
> have been re-enabled, wasting power.
> 
> This RFC is a second attempt to fix this issue by restoring clock registers
> during system resume.
> 
> Note that while this fixes EtherAVB operation after resume from s2ram,
> EtherAVB cannot be used as an actual wake-up source from s2ram, only
> from s2idle, due to PSCI limitations.
> 
> Changes compared to v1 (more details in the individual patches):
>   - Save module clock registers in suspend_noirq instead of constantly
>     updating shadow registers,
>   - Restore all module clocks under our control, not just the ones we ever
>     changed,
>   - Also restore DIV6, SDHI, and R clocks, thus covering all supported
>     programmable core clocks on R-Car Gen3.
> 
> As clock register restore is only needed on R-Car Gen3 with PSCI, although
> harmless on other systems, perhaps the save/restore code should be
> protected by #ifdef CONFIG_ARM_PSCI_FW?
> 
> This series is against clk-next, with "clk: renesas: div6: Document fields
> used for parent selection" applied on top.
> 
> This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
> On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resume,
> but NFS fails with "server not responding", probably not due to a clock
> issue.

I tested this series using this setup:

- Base latest renesas-drivers, 710def1a48c7bc9d ("of_mdio: Fix broken 
  PHY IRQ in case of probe deferral"). This branch also includes the 
  RAVB WoL patches.

- Salvator-X H3 ES1.0

- The arm64 defconfig.

The test procedure I used:

    arm64 ~/shared/deep-sleep # cat sleep.sh
    #!/bin/bash

    ethtool -s eth0 wol g

    echo disabled > /sys/devices/platform/soc/e6800000.ethernet/power/wakeup

    echo 0 > /sys/module/printk/parameters/console_suspend

    i2cset -f -y 7 0x30 0x20 0x0F

    echo "Flip Switch"
    read -n 1

    echo mem > /sys/power/state

And for me the NFS root came up OK after flipping switch back. I wonder 
what is different in our test procedures. I would like to provide my 
tested-by tag but I first want to figure out why NFS (or maybe the whole 
net interface?) don't work for you after resume.

arm64 ~/shared/deep-sleep # ./sleep.sh
Flip Switch

[  720.992971] PM: Syncing filesystems ... done.
[  720.998841] Freezing user space processes ... (elapsed 0.001 seconds) done.
[  721.007530] OOM killer disabled.
[  721.010763] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[  721.600789] vsp1 fea38000.vsp: pipeline 0 stop timeout
[  722.112773] vsp1 fea30000.vsp: pipeline 0 stop timeout
[  722.624769] vsp1 fea28000.vsp: pipeline 0 stop timeout
[  723.136772] vsp1 fea20000.vsp: pipeline 0 stop timeout
[  723.147535] ohci-platform ee080000.usb: runtime PM trying to suspend device but active child
[  723.156051] phy_rcar_gen3_usb2 ee080200.usb-phy: runtime PM trying to suspend device but active child
[  723.167851] ohci-platform ee0c0000.usb: runtime PM trying to suspend device but active child
[  723.176313] ohci-platform ee0a0000.usb: runtime PM trying to suspend device but active child
[  723.184769] phy_rcar_gen3_usb2 ee0c0200.usb-phy: runtime PM trying to suspend device but active child
[  723.194001] phy_rcar_gen3_usb2 ee0a0200.usb-phy: runtime PM trying to suspend device but active child
[  723.203413] Disabling non-boot CPUs ...
[  723.229225] IRQ15 no longer affine to CPU1
[  723.229561] CPU1: shutdown
[  723.236412] psci: CPU1 killed.
[  723.277182] IRQ16 no longer affine to CPU2
[  723.277535] CPU2: shutdown
[  723.284367] psci: CPU2 killed.
[  723.329121] IRQ17 no longer affine to CPU3
[  723.329404] CPU3: shutdown
[  723.336229] psci: CPU3 killed.

!! Flipping switch back here !!

NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57) Rev.1.0.12
NOTICE:  BL2: PRR is R-Car H3 ES1.0
NOTICE:  BL2: Boot device is HyperFlash(160MHz)
NOTICE:  BL2: LCM state is CM
NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x52
NOTICE:  BL2: DDR1600(rev.0.20)[WARM_BOOT]..0
NOTICE:  BL2: DRAM Split is 4ch
NOTICE:  BL2: QoS is default setting(rev.0.33)
NOTICE:  BL2: v1.3(release):c040be5
NOTICE:  BL2: Built : 16:03:29, Jan 30 2017
NOTICE:  BL2: Normal boot
�[  723.351187] Enabling non-boot CPUs ...dToRAM)
[  723.367103] Detected PIPT I-cache on CPU1
[  723.367163] CPU1: Booted secondary processor [411fd073]
[  723.367448]  cache: parent cpu1 should not be sleeping
[  723.381957] CPU1 is up
[  723.394114] Detected PIPT I-cache on CPU2
[  723.394134] CPU2: Booted secondary processor [411fd073]
[  723.394306]  cache: parent cpu2 should not be sleeping
[  723.408755] CPU2 is up
[  723.426652] Detected PIPT I-cache on CPU3
[  723.426673] CPU3: Booted secondary processor [411fd073]
[  723.426845]  cache: parent cpu3 should not be sleeping
[  723.441328] CPU3 is up
[  723.565306] usb usb2: root hub lost power or was reset
[  723.565325] usb usb1: root hub lost power or was reset
[  723.569644] usb usb5: root hub lost power or was reset
[  723.649254] ravb e6800000.ethernet eth0: limited PHY to 100Mbit/s
[  723.655345] Micrel KSZ9031 Gigabit PHY e6800000.ethernet-ffffffff:00: attached PHY driver [Micrel KSZ9031 Gigabit PHY] (mii_bus:phy_addr=e6800000.ethernet-ffffffff:00, irq=236)
[  723.762737] usb usb3: root hub lost power or was reset
[  723.854727] usb usb4: root hub lost power or was reset
[  723.950730] usb usb6: root hub lost power or was reset
[  724.092718] ata1: link resume succeeded after 1 retries
[  724.153545] OOM killer enabled.
[  724.158915] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[  724.158918] [drm] No driver support for vblank timestamp query.
[  724.172372] Restarting tasks ... done.
[  724.202297] ata1: SATA link down (SStatus 0 SControl 300)
[  725.265366] ravb e6800000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
arm64 ~/shared/deep-sleep #

I also know you reported an IRQ storm when resuming using WoL which I in 
over 1000 suspend/resume cycles never have been able to reproduce. Maybe 
my test environment or procedure is to kind and/or something is falling 
thru the cracks :-( Do you notice any difference in test procedure or 
console printouts?

> 
> Thanks for your comments!
> 
> Geert Uytterhoeven (5):
>   [RFC] clk: renesas: cpg-mssr: Restore module clocks during resume
>   [RFC] clk: renesas: cpg-mssr: Add support to restore core clocks
>     during resume
>   [RFC] clk: renesas: div6: Restore clock state during resume
>   [RFC] clk: renesas: rcar-gen3: Restore SDHI clocks during resume
>   [RFC] clk: renesas: rcar-gen3: Restore R clock during resume
> 
>  drivers/clk/renesas/clk-div6.c         | 38 ++++++++++++++-
>  drivers/clk/renesas/clk-div6.h         |  3 +-
>  drivers/clk/renesas/rcar-gen2-cpg.c    |  7 ++-
>  drivers/clk/renesas/rcar-gen2-cpg.h    |  6 +--
>  drivers/clk/renesas/rcar-gen3-cpg.c    | 79 +++++++++++++++++++++++++-------
>  drivers/clk/renesas/rcar-gen3-cpg.h    |  3 +-
>  drivers/clk/renesas/renesas-cpg-mssr.c | 84 +++++++++++++++++++++++++++++++++-
>  drivers/clk/renesas/renesas-cpg-mssr.h |  3 +-
>  8 files changed, 193 insertions(+), 30 deletions(-)
> 
> -- 
> 2.7.4
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

-- 
Regards,
Niklas S�derlund

Re: [PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Niklas Söderlund]

Hi Geert,

Thanks for your hard work!

On 2017-06-29 15:25:15 +0200, Geert Uytterhoeven wrote:
> 	Hi all,
> 
> During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
> clock register state is lost.  Note that as the boot loader skips most
> initialization after resume, clock register state differs from the state
> encountered during normal system boot, too.
> 
> Hence after s2ram, some operations may fail because module clocks are
> disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
> fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":
> 
>     ravb e6800000.ethernet eth0: failed to switch device to config mode
>     ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
>     ravb e6800000.ethernet eth0: failed to switch device to config
>     PM: Device e6800000.ethernet failed to resume: error -110
> 
> In addition, some clocks that were disabled by clk_disable_unused() may
> have been re-enabled, wasting power.
> 
> This RFC is a second attempt to fix this issue by restoring clock registers
> during system resume.
> 
> Note that while this fixes EtherAVB operation after resume from s2ram,
> EtherAVB cannot be used as an actual wake-up source from s2ram, only
> from s2idle, due to PSCI limitations.
> 
> Changes compared to v1 (more details in the individual patches):
>   - Save module clock registers in suspend_noirq instead of constantly
>     updating shadow registers,
>   - Restore all module clocks under our control, not just the ones we ever
>     changed,
>   - Also restore DIV6, SDHI, and R clocks, thus covering all supported
>     programmable core clocks on R-Car Gen3.
> 
> As clock register restore is only needed on R-Car Gen3 with PSCI, although
> harmless on other systems, perhaps the save/restore code should be
> protected by #ifdef CONFIG_ARM_PSCI_FW?
> 
> This series is against clk-next, with "clk: renesas: div6: Document fields
> used for parent selection" applied on top.
> 
> This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
> On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resume,
> but NFS fails with "server not responding", probably not due to a clock
> issue.

I tested this series using this setup:

- Base latest renesas-drivers, 710def1a48c7bc9d ("of_mdio: Fix broken 
  PHY IRQ in case of probe deferral"). This branch also includes the 
  RAVB WoL patches.

- Salvator-X H3 ES1.0

- The arm64 defconfig.

The test procedure I used:

    arm64 ~/shared/deep-sleep # cat sleep.sh
    #!/bin/bash

    ethtool -s eth0 wol g

    echo disabled > /sys/devices/platform/soc/e6800000.ethernet/power/wakeup

    echo 0 > /sys/module/printk/parameters/console_suspend

    i2cset -f -y 7 0x30 0x20 0x0F

    echo "Flip Switch"
    read -n 1

    echo mem > /sys/power/state

And for me the NFS root came up OK after flipping switch back. I wonder 
what is different in our test procedures. I would like to provide my 
tested-by tag but I first want to figure out why NFS (or maybe the whole 
net interface?) don't work for you after resume.

arm64 ~/shared/deep-sleep # ./sleep.sh
Flip Switch

[  720.992971] PM: Syncing filesystems ... done.
[  720.998841] Freezing user space processes ... (elapsed 0.001 seconds) done.
[  721.007530] OOM killer disabled.
[  721.010763] Freezing remaining freezable tasks ... (elapsed 0.001 seconds) done.
[  721.600789] vsp1 fea38000.vsp: pipeline 0 stop timeout
[  722.112773] vsp1 fea30000.vsp: pipeline 0 stop timeout
[  722.624769] vsp1 fea28000.vsp: pipeline 0 stop timeout
[  723.136772] vsp1 fea20000.vsp: pipeline 0 stop timeout
[  723.147535] ohci-platform ee080000.usb: runtime PM trying to suspend device but active child
[  723.156051] phy_rcar_gen3_usb2 ee080200.usb-phy: runtime PM trying to suspend device but active child
[  723.167851] ohci-platform ee0c0000.usb: runtime PM trying to suspend device but active child
[  723.176313] ohci-platform ee0a0000.usb: runtime PM trying to suspend device but active child
[  723.184769] phy_rcar_gen3_usb2 ee0c0200.usb-phy: runtime PM trying to suspend device but active child
[  723.194001] phy_rcar_gen3_usb2 ee0a0200.usb-phy: runtime PM trying to suspend device but active child
[  723.203413] Disabling non-boot CPUs ...
[  723.229225] IRQ15 no longer affine to CPU1
[  723.229561] CPU1: shutdown
[  723.236412] psci: CPU1 killed.
[  723.277182] IRQ16 no longer affine to CPU2
[  723.277535] CPU2: shutdown
[  723.284367] psci: CPU2 killed.
[  723.329121] IRQ17 no longer affine to CPU3
[  723.329404] CPU3: shutdown
[  723.336229] psci: CPU3 killed.

!! Flipping switch back here !!

NOTICE:  BL2: R-Car Gen3 Initial Program Loader(CA57) Rev.1.0.12
NOTICE:  BL2: PRR is R-Car H3 ES1.0
NOTICE:  BL2: Boot device is HyperFlash(160MHz)
NOTICE:  BL2: LCM state is CM
NOTICE:  BL2: AVS setting succeeded. DVFS_SetVID=0x52
NOTICE:  BL2: DDR1600(rev.0.20)[WARM_BOOT]..0
NOTICE:  BL2: DRAM Split is 4ch
NOTICE:  BL2: QoS is default setting(rev.0.33)
NOTICE:  BL2: v1.3(release):c040be5
NOTICE:  BL2: Built : 16:03:29, Jan 30 2017
NOTICE:  BL2: Normal boot
ÿ[  723.351187] Enabling non-boot CPUs ...dToRAM)
[  723.367103] Detected PIPT I-cache on CPU1
[  723.367163] CPU1: Booted secondary processor [411fd073]
[  723.367448]  cache: parent cpu1 should not be sleeping
[  723.381957] CPU1 is up
[  723.394114] Detected PIPT I-cache on CPU2
[  723.394134] CPU2: Booted secondary processor [411fd073]
[  723.394306]  cache: parent cpu2 should not be sleeping
[  723.408755] CPU2 is up
[  723.426652] Detected PIPT I-cache on CPU3
[  723.426673] CPU3: Booted secondary processor [411fd073]
[  723.426845]  cache: parent cpu3 should not be sleeping
[  723.441328] CPU3 is up
[  723.565306] usb usb2: root hub lost power or was reset
[  723.565325] usb usb1: root hub lost power or was reset
[  723.569644] usb usb5: root hub lost power or was reset
[  723.649254] ravb e6800000.ethernet eth0: limited PHY to 100Mbit/s
[  723.655345] Micrel KSZ9031 Gigabit PHY e6800000.ethernet-ffffffff:00: attached PHY driver [Micrel KSZ9031 Gigabit PHY] (mii_bus:phy_addr=e6800000.ethernet-ffffffff:00, irq=236)
[  723.762737] usb usb3: root hub lost power or was reset
[  723.854727] usb usb4: root hub lost power or was reset
[  723.950730] usb usb6: root hub lost power or was reset
[  724.092718] ata1: link resume succeeded after 1 retries
[  724.153545] OOM killer enabled.
[  724.158915] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[  724.158918] [drm] No driver support for vblank timestamp query.
[  724.172372] Restarting tasks ... done.
[  724.202297] ata1: SATA link down (SStatus 0 SControl 300)
[  725.265366] ravb e6800000.ethernet eth0: Link is Up - 100Mbps/Full - flow control rx/tx
arm64 ~/shared/deep-sleep #

I also know you reported an IRQ storm when resuming using WoL which I in 
over 1000 suspend/resume cycles never have been able to reproduce. Maybe 
my test environment or procedure is to kind and/or something is falling 
thru the cracks :-( Do you notice any difference in test procedure or 
console printouts?

> 
> Thanks for your comments!
> 
> Geert Uytterhoeven (5):
>   [RFC] clk: renesas: cpg-mssr: Restore module clocks during resume
>   [RFC] clk: renesas: cpg-mssr: Add support to restore core clocks
>     during resume
>   [RFC] clk: renesas: div6: Restore clock state during resume
>   [RFC] clk: renesas: rcar-gen3: Restore SDHI clocks during resume
>   [RFC] clk: renesas: rcar-gen3: Restore R clock during resume
> 
>  drivers/clk/renesas/clk-div6.c         | 38 ++++++++++++++-
>  drivers/clk/renesas/clk-div6.h         |  3 +-
>  drivers/clk/renesas/rcar-gen2-cpg.c    |  7 ++-
>  drivers/clk/renesas/rcar-gen2-cpg.h    |  6 +--
>  drivers/clk/renesas/rcar-gen3-cpg.c    | 79 +++++++++++++++++++++++++-------
>  drivers/clk/renesas/rcar-gen3-cpg.h    |  3 +-
>  drivers/clk/renesas/renesas-cpg-mssr.c | 84 +++++++++++++++++++++++++++++++++-
>  drivers/clk/renesas/renesas-cpg-mssr.h |  3 +-
>  8 files changed, 193 insertions(+), 30 deletions(-)
> 
> -- 
> 2.7.4
> 
> Gr{oetje,eeting}s,
> 
> 						Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
> 							    -- Linus Torvalds

-- 
Regards,
Niklas Söderlund

Re: [PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Geert Uytterhoeven]

Hi Niklas,

On Thu, Jun 29, 2017 at 5:18 PM, Niklas Söderlund
<niklas.soderlund@ragnatech.se> wrote:
> On 2017-06-29 15:25:15 +0200, Geert Uytterhoeven wrote:
>> During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
>> clock register state is lost.  Note that as the boot loader skips most
>> initialization after resume, clock register state differs from the state
>> encountered during normal system boot, too.
>>
>> Hence after s2ram, some operations may fail because module clocks are
>> disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
>> fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":
>>
>>     ravb e6800000.ethernet eth0: failed to switch device to config mode
>>     ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
>>     ravb e6800000.ethernet eth0: failed to switch device to config
>>     PM: Device e6800000.ethernet failed to resume: error -110
>>
>> In addition, some clocks that were disabled by clk_disable_unused() may
>> have been re-enabled, wasting power.
>>
>> This RFC is a second attempt to fix this issue by restoring clock registers
>> during system resume.

>> This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
>> On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resume,
>> but NFS fails with "server not responding", probably not due to a clock
>> issue.
>
> I tested this series using this setup:
>
> - Base latest renesas-drivers, 710def1a48c7bc9d ("of_mdio: Fix broken
>   PHY IRQ in case of probe deferral"). This branch also includes the
>   RAVB WoL patches.
>
> - Salvator-X H3 ES1.0
>
> - The arm64 defconfig.
>
> The test procedure I used:
>
>     arm64 ~/shared/deep-sleep # cat sleep.sh
>     #!/bin/bash
>
>     ethtool -s eth0 wol g
>
>     echo disabled > /sys/devices/platform/soc/e6800000.ethernet/power/wakeup
>
>     echo 0 > /sys/module/printk/parameters/console_suspend
>
>     i2cset -f -y 7 0x30 0x20 0x0F
>
>     echo "Flip Switch"
>     read -n 1
>
>     echo mem > /sys/power/state

Looks good.

> And for me the NFS root came up OK after flipping switch back. I wonder
> what is different in our test procedures. I would like to provide my
> tested-by tag but I first want to figure out why NFS (or maybe the whole
> net interface?) don't work for you after resume.

Sorry for the confusion: NFS fails only on Salvator-XS H3 ES2.0.
It works fine on Salvator-X H3 ES1.0 and M3-W ES1.0.

> I also know you reported an IRQ storm when resuming using WoL which I in
> over 1000 suspend/resume cycles never have been able to reproduce. Maybe
> my test environment or procedure is to kind and/or something is falling
> thru the cracks :-( Do you notice any difference in test procedure or
> console printouts?

I saw the IRQ storm only once.

Thanks for testing!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Geert Uytterhoeven]

Hi Niklas,

On Thu, Jun 29, 2017 at 5:18 PM, Niklas S=C3=B6derlund
<niklas.soderlund@ragnatech.se> wrote:
> On 2017-06-29 15:25:15 +0200, Geert Uytterhoeven wrote:
>> During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
>> clock register state is lost.  Note that as the boot loader skips most
>> initialization after resume, clock register state differs from the state
>> encountered during normal system boot, too.
>>
>> Hence after s2ram, some operations may fail because module clocks are
>> disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
>> fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":
>>
>>     ravb e6800000.ethernet eth0: failed to switch device to config mode
>>     ravb e6800000.ethernet eth0: device will be stopped after h/w proces=
ses are done.
>>     ravb e6800000.ethernet eth0: failed to switch device to config
>>     PM: Device e6800000.ethernet failed to resume: error -110
>>
>> In addition, some clocks that were disabled by clk_disable_unused() may
>> have been re-enabled, wasting power.
>>
>> This RFC is a second attempt to fix this issue by restoring clock regist=
ers
>> during system resume.

>> This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
>> On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resum=
e,
>> but NFS fails with "server not responding", probably not due to a clock
>> issue.
>
> I tested this series using this setup:
>
> - Base latest renesas-drivers, 710def1a48c7bc9d ("of_mdio: Fix broken
>   PHY IRQ in case of probe deferral"). This branch also includes the
>   RAVB WoL patches.
>
> - Salvator-X H3 ES1.0
>
> - The arm64 defconfig.
>
> The test procedure I used:
>
>     arm64 ~/shared/deep-sleep # cat sleep.sh
>     #!/bin/bash
>
>     ethtool -s eth0 wol g
>
>     echo disabled > /sys/devices/platform/soc/e6800000.ethernet/power/wak=
eup
>
>     echo 0 > /sys/module/printk/parameters/console_suspend
>
>     i2cset -f -y 7 0x30 0x20 0x0F
>
>     echo "Flip Switch"
>     read -n 1
>
>     echo mem > /sys/power/state

Looks good.

> And for me the NFS root came up OK after flipping switch back. I wonder
> what is different in our test procedures. I would like to provide my
> tested-by tag but I first want to figure out why NFS (or maybe the whole
> net interface?) don't work for you after resume.

Sorry for the confusion: NFS fails only on Salvator-XS H3 ES2.0.
It works fine on Salvator-X H3 ES1.0 and M3-W ES1.0.

> I also know you reported an IRQ storm when resuming using WoL which I in
> over 1000 suspend/resume cycles never have been able to reproduce. Maybe
> my test environment or procedure is to kind and/or something is falling
> thru the cracks :-( Do you notice any difference in test procedure or
> console printouts?

I saw the IRQ storm only once.

Thanks for testing!

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k=
.org

In personal conversations with technical people, I call myself a hacker. Bu=
t
when I'm talking to journalists I just say "programmer" or something like t=
hat.
                                -- Linus Torvalds

Re: [PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Niklas Söderlund]

Hi Geert,

On 2017-06-29 18:26:38 +0200, Geert Uytterhoeven wrote:
> Hi Niklas,
> 
> On Thu, Jun 29, 2017 at 5:18 PM, Niklas S�derlund
> <niklas.soderlund@ragnatech.se> wrote:
> > On 2017-06-29 15:25:15 +0200, Geert Uytterhoeven wrote:
> >> During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
> >> clock register state is lost.  Note that as the boot loader skips most
> >> initialization after resume, clock register state differs from the state
> >> encountered during normal system boot, too.
> >>
> >> Hence after s2ram, some operations may fail because module clocks are
> >> disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
> >> fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":
> >>
> >>     ravb e6800000.ethernet eth0: failed to switch device to config mode
> >>     ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
> >>     ravb e6800000.ethernet eth0: failed to switch device to config
> >>     PM: Device e6800000.ethernet failed to resume: error -110
> >>
> >> In addition, some clocks that were disabled by clk_disable_unused() may
> >> have been re-enabled, wasting power.
> >>
> >> This RFC is a second attempt to fix this issue by restoring clock registers
> >> during system resume.
> 
> >> This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
> >> On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resume,
> >> but NFS fails with "server not responding", probably not due to a clock
> >> issue.
> >
> > I tested this series using this setup:
> >
> > - Base latest renesas-drivers, 710def1a48c7bc9d ("of_mdio: Fix broken
> >   PHY IRQ in case of probe deferral"). This branch also includes the
> >   RAVB WoL patches.
> >
> > - Salvator-X H3 ES1.0
> >
> > - The arm64 defconfig.
> >
> > The test procedure I used:
> >
> >     arm64 ~/shared/deep-sleep # cat sleep.sh
> >     #!/bin/bash
> >
> >     ethtool -s eth0 wol g
> >
> >     echo disabled > /sys/devices/platform/soc/e6800000.ethernet/power/wakeup
> >
> >     echo 0 > /sys/module/printk/parameters/console_suspend
> >
> >     i2cset -f -y 7 0x30 0x20 0x0F
> >
> >     echo "Flip Switch"
> >     read -n 1
> >
> >     echo mem > /sys/power/state
> 
> Looks good.
> 
> > And for me the NFS root came up OK after flipping switch back. I wonder
> > what is different in our test procedures. I would like to provide my
> > tested-by tag but I first want to figure out why NFS (or maybe the whole
> > net interface?) don't work for you after resume.
> 
> Sorry for the confusion: NFS fails only on Salvator-XS H3 ES2.0.
> It works fine on Salvator-X H3 ES1.0 and M3-W ES1.0.

Dooh, I should read the error descriptions more carefully before directly 
jumping to the conclusion that WoL is broken :-) For the entire series:

Tested-by: Niklas S�derlund <niklas.soderlund+renesas@ragnatech.se>

Lets hope I soon can get my hands on a Salvator-XS to try and figure out 
the RAVB issue!

> 
> > I also know you reported an IRQ storm when resuming using WoL which I in
> > over 1000 suspend/resume cycles never have been able to reproduce. Maybe
> > my test environment or procedure is to kind and/or something is falling
> > thru the cracks :-( Do you notice any difference in test procedure or
> > console printouts?
> 
> I saw the IRQ storm only once.
> 
> Thanks for testing!
> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds

-- 
Regards,
Niklas S�derlund

Re: [PATCH/RFC v2 0/5] clk: renesas: rcar-gen3: Restore clocks during resume

[Niklas Söderlund]

Hi Geert,

On 2017-06-29 18:26:38 +0200, Geert Uytterhoeven wrote:
> Hi Niklas,
> 
> On Thu, Jun 29, 2017 at 5:18 PM, Niklas Söderlund
> <niklas.soderlund@ragnatech.se> wrote:
> > On 2017-06-29 15:25:15 +0200, Geert Uytterhoeven wrote:
> >> During PSCI system suspend, R-Car Gen3 SoCs are powered down, and their
> >> clock register state is lost.  Note that as the boot loader skips most
> >> initialization after resume, clock register state differs from the state
> >> encountered during normal system boot, too.
> >>
> >> Hence after s2ram, some operations may fail because module clocks are
> >> disabled, while drivers expect them to be still enabled.  E.g. EtherAVB
> >> fails when Wake-on-LAN has been enabled using "ethtool -s eth0 wol g":
> >>
> >>     ravb e6800000.ethernet eth0: failed to switch device to config mode
> >>     ravb e6800000.ethernet eth0: device will be stopped after h/w processes are done.
> >>     ravb e6800000.ethernet eth0: failed to switch device to config
> >>     PM: Device e6800000.ethernet failed to resume: error -110
> >>
> >> In addition, some clocks that were disabled by clk_disable_unused() may
> >> have been re-enabled, wasting power.
> >>
> >> This RFC is a second attempt to fix this issue by restoring clock registers
> >> during system resume.
> 
> >> This has been tested on Salvator-X with R-Car H3 ES1.0 and M3-W ES1.0.
> >> On Salvator-XS with R-Car H3 ES2.0, EtherAVB restarts after system resume,
> >> but NFS fails with "server not responding", probably not due to a clock
> >> issue.
> >
> > I tested this series using this setup:
> >
> > - Base latest renesas-drivers, 710def1a48c7bc9d ("of_mdio: Fix broken
> >   PHY IRQ in case of probe deferral"). This branch also includes the
> >   RAVB WoL patches.
> >
> > - Salvator-X H3 ES1.0
> >
> > - The arm64 defconfig.
> >
> > The test procedure I used:
> >
> >     arm64 ~/shared/deep-sleep # cat sleep.sh
> >     #!/bin/bash
> >
> >     ethtool -s eth0 wol g
> >
> >     echo disabled > /sys/devices/platform/soc/e6800000.ethernet/power/wakeup
> >
> >     echo 0 > /sys/module/printk/parameters/console_suspend
> >
> >     i2cset -f -y 7 0x30 0x20 0x0F
> >
> >     echo "Flip Switch"
> >     read -n 1
> >
> >     echo mem > /sys/power/state
> 
> Looks good.
> 
> > And for me the NFS root came up OK after flipping switch back. I wonder
> > what is different in our test procedures. I would like to provide my
> > tested-by tag but I first want to figure out why NFS (or maybe the whole
> > net interface?) don't work for you after resume.
> 
> Sorry for the confusion: NFS fails only on Salvator-XS H3 ES2.0.
> It works fine on Salvator-X H3 ES1.0 and M3-W ES1.0.

Dooh, I should read the error descriptions more carefully before directly 
jumping to the conclusion that WoL is broken :-) For the entire series:

Tested-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

Lets hope I soon can get my hands on a Salvator-XS to try and figure out 
the RAVB issue!

> 
> > I also know you reported an IRQ storm when resuming using WoL which I in
> > over 1000 suspend/resume cycles never have been able to reproduce. Maybe
> > my test environment or procedure is to kind and/or something is falling
> > thru the cracks :-( Do you notice any difference in test procedure or
> > console printouts?
> 
> I saw the IRQ storm only once.
> 
> Thanks for testing!
> 
> Gr{oetje,eeting}s,
> 
>                         Geert
> 
> --
> Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org
> 
> In personal conversations with technical people, I call myself a hacker. But
> when I'm talking to journalists I just say "programmer" or something like that.
>                                 -- Linus Torvalds

-- 
Regards,
Niklas Söderlund